Program, information processing method, and information processing device

Abstract

To provide a program etc. capable of accurately evaluating the load on a worker.SOLUTION: According to a program, a computer acquires a captured image of a worker and weight information relating to the weight of an object held by the worker. The computer then identifies a posture of the worker based on the acquired captured images, and determines load information relating to the load on the body of the worker based on the identified posture of the worker and the acquired weight information.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a program, an information processing method, and an information processing device. [Background technology]

[0002] Patent Document 1 discloses a technology for evaluating the posture of a moving object by extracting joint positions of the moving object from a captured image of the moving object and reconstructing the skeleton of the moving object based on the spatial distance to each extracted joint position. The technology disclosed in Patent Document 1 measures the distance to the moving object from images captured simultaneously with multiple optical cameras, or acquires images and distance information of the moving object by simultaneously capturing images with an optical camera and a distance camera. The technology disclosed in Patent Document 1 also uses the Ovako Working Posture Analyzing System (OWAS) method to evaluate the load applied to the moving object depending on the posture of the moving object. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-59753 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology disclosed in Patent Document 1 uses multiple optical cameras, or optical cameras and distance cameras, to capture moving objects, so all cameras must be positioned so that they can capture moving objects. Furthermore, the OWAS method evaluates (classifies) the load on a worker based on four factors: the posture of the worker's back, upper limbs (arms), and lower limbs (legs), and the weight (weight, load) of the object being held (carried) by the worker. However, the technology disclosed in Patent Document 1 does not measure the weight of the object being worked on, so accurate evaluation that takes the weight of the object into consideration is not possible, and there is a problem in that the tasks to be evaluated are limited to tasks that involve holding an object of a specified weight.

[0005] The present disclosure aims to provide a program or the like that can accurately evaluate the load on a worker. [Means for solving the problem]

[0006] A program according to one aspect of the present disclosure causes a computer to execute a process of acquiring an image of a worker, acquiring weight information relating to the weight of an object held by the worker, identifying the posture of the worker based on the acquired image, and identifying load information relating to the load applied to the worker's body based on the identified posture of the worker and the acquired weight information. [Effects of the Invention]

[0007] In one aspect of the present disclosure, the load applied to a worker can be accurately evaluated. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram illustrating an example of the configuration of an information processing system. [Figure 2] FIG. 2 is a block diagram showing an example of the configuration of an information processing device and a server. [Figure 3] FIG. 10 is an explanatory diagram of a process for extracting the joint positions of a worker. [Figure 4] FIG. 10 is an explanatory diagram showing an example of the configuration of a back determination table. [Figure 5] FIG. 10 is an explanatory diagram showing an example of the configuration of an upper limb determination table. [Figure 6] FIG. 10 is an explanatory diagram showing an example of the configuration of a lower limb determination table. [Figure 7] FIG. 10 is an explanatory diagram showing an example of the configuration of a posture evaluation table. [Figure 8] 10A and 10B are explanatory diagrams showing examples of record layouts of an evaluation result DB and a photographed image DB. [Figure 9] 10 is a flowchart illustrating an example of a load estimation process procedure. [Figure 10]10 is a flowchart illustrating an example of a load estimation process procedure. [Figure 11] 10 is a flowchart illustrating an example of a load estimation process procedure. [Figure 12] 10 is a flowchart illustrating an example of a load estimation process procedure. [Figure 13] FIG. 10 is an explanatory diagram showing an example of a screen. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, a program, an information processing method, and an information processing device according to the present disclosure will be described in detail with reference to the drawings illustrating embodiments thereof.

[0010] (Embodiment 1) In this embodiment, an information processing system is described that estimates the physical load of a worker performing a task based on an image of the worker. FIG. 1 is an explanatory diagram showing an example of the configuration of the information processing system. The information processing system of this embodiment includes an information processing device 10, a camera 20, a server 30, a weighing scale 40, etc., and these devices are communicatively connected via a network N. The network N may be the Internet or a public telephone network, or a local area network (LAN) established within the facility where the information processing system is installed. In this embodiment, images captured by the camera 20 and weights measured by the weighing scale 40 are transmitted to and stored on the server 30. The camera 20 and the weighing scale 40 may be configured to directly transmit and receive information to and from the server 30 via wired or wireless communication via a cable. Alternatively, the server 30 may not be provided, and the information processing device 10 may have the functions of the server 30. In this case, the camera 20 and the weighing scale 40 may be configured to directly transmit and receive information to and from the information processing device 10 via wired or wireless communication via a cable.

[0011] Camera 20 is provided at a site where workers are working, such as a manufacturing site or a construction site, and captures images of the workers while they are working. Camera 20 is an imaging device equipped with an imaging unit having a lens and an image sensor, a communication unit for connecting to network N, and the like. Camera 20 performs a process of capturing images using the imaging unit in response to, for example, an operation on a capture button, acquiring image data (hereinafter referred to as a captured image), and a process of transmitting the captured image to server 30 via the communication unit. Camera 20 of this embodiment performs a process of capturing images (video) at, for example, 15 or 30 frames per second, but may also be configured to capture one image (still image) in response to a single operation of the capture button. Camera 20 may also be configured to sequentially transmit captured images captured by capturing to server 30, or may be configured to include a memory unit, store captured images in the memory, and transmit the stored captured images collectively to server 30 in response to an operation on the operation unit or an instruction from server 30. The storage unit may be a storage unit built into the camera 20, such as a flash memory, a hard disk, or a solid state drive (SSD), or may be a storage unit detachable from the camera 20, such as a universal serial bus (USB) memory, an SD card, or a micro SD card. The camera 20 may be any camera capable of capturing an image of the entire body of the worker and capturing the state of the worker's work. For example, the camera 20 may be a camera whose shooting position is fixed using a tripod or a fixture, a camera held by the photographer, or a camera with a function for tracking a set target and capturing images while tracking the movement of the target. The camera 20 may be configured to capture an image of the worker from the front, rear, or side of the worker, as long as it can capture an image of the worker without any obstructions. The camera 20 may also be configured to capture an image of the worker's entire body using multiple cameras.

[0012] Weighing scale 40 is provided at a work site and measures the weight of an object held (carried) by a worker during work. Weighing scale 40 includes a weight measurement unit, a communication unit for connecting to network N, and performs the following processes: measuring the weight of an object placed on a platform using the measurement unit to obtain weight data; and transmitting the obtained weight data from the communication unit to server 30. For example, when an object is placed on the platform, weighing scale 40 measures the weight and transmits weight data including the measured weight and measurement date and time to server 30. Weighing scale 40 may be configured to sequentially transmit the obtained weight data to server 30, or may be configured to include a memory unit, accumulate weight data in the memory, and transmit the accumulated weight data collectively to server 30 in response to an operation on the operation unit or an instruction from server 30.

[0013] The information processing device 10 is a device capable of various information processing and transmitting and receiving information, such as a personal computer or a server computer. The information processing device 10 performs processing to estimate the load on the body of the worker being photographed, based on the images captured by the camera 20 and the weight data measured by the weighing scale 40. The server 30 is an image processing device capable of various information processing and transmitting and receiving information, such as a server computer or a personal computer. The server 30 acquires the images captured by the camera 20 and the weight data measured by the weighing scale 40, stores the acquired images and weight data, and transmits the stored images and weight data to the information processing device 10 in response to a request from the information processing device 10.

[0014] 2 is a block diagram showing an example of the configuration of the information processing device 10 and the server 30. The information processing device 10 includes a control unit 11, a storage unit 12, a communication unit 13, an input unit 14, a display unit 15, a reading unit 16, etc., and these units are connected to each other via a bus. The control unit 11 includes one or more processors such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), or a GPU (Graphics Processing Unit). The control unit 11 executes a program 12P stored in the storage unit 12 as needed to perform information processing and control processing to be performed by the information processing device 10. Note that if the control unit 11 includes multiple processors, the control unit 11 may execute each process using a different processor.

[0015] The storage unit 12 includes a RAM (Random Access Memory), a flash memory, a hard disk, an SSD (Solid State Drive), etc. The storage unit 12 stores a program 12P (program product) executed by the control unit 11 and various data required for executing the program 12P. The storage unit 12 also temporarily stores data and the like generated when the control unit 11 executes the program 12P. The storage unit 12 also stores a back determination table 12a, an upper limb determination table 12b, a lower limb determination table 12c, a posture evaluation table 12d, and an evaluation result DB 12e, which will be described later.

[0016] The communication unit 13 is a communication module for performing processes related to wired or wireless communication, and transmits and receives information to other devices via the network N. The input unit 14 accepts operation input by the user and sends a control signal corresponding to the operation content to the control unit 11. The display unit 15 is a liquid crystal display, an organic EL (electroluminescence) display, or the like, and displays various information according to instructions from the control unit 11. A part of the input unit 14 and the display unit 15 may be a touch panel configured as an integrated unit.

[0017] The reading unit 16 reads information stored in a portable storage medium 10a, which may include a CD (Compact Disc), a DVD (Digital Versatile Disc), a USB (Universal Serial Bus) memory, an SD (Secure Digital) card, etc. The program 12P and various data stored in the storage unit 12 may be read by the control unit 11 from the portable storage medium 10a via the reading unit 16 and stored in the storage unit 12. The program 12P and various data may be written to the storage unit 12 during the manufacturing stage of the information processing device 10, or may be downloaded by the control unit 11 from another device via the communication unit 13 and stored in the storage unit 12.

[0018] The information processing device 10 may be a multi-computer including multiple computers, or may be a virtual machine virtually constructed by software within a single device. When the information processing device 10 is configured as a server computer, the information processing device 10 may be a local server installed in the facility where the information processing device 10 is installed, or a cloud server connected to the information processing device 10 via a network such as the Internet. The following description assumes that the information processing device 10 is a single computer. The program 12P may be deployed on a single computer or a single site, or may be distributed across multiple sites and executed on multiple computers interconnected via a network. Furthermore, the information processing device 10 does not necessarily require the input unit 14 and the display unit 15. The information processing device 10 may be configured to accept operations through a connected computer or to output information to be displayed to an external display device. The memory unit 12 of the information processing device 10 may be configured with multiple storage devices or may include an external storage device connected to the information processing device 10.

[0019] Server 30 includes a control unit 31, a storage unit 32, a communication unit 33, etc., which are interconnected via a bus. Control unit 31, storage unit 32, and communication unit 33 each have the same configuration as control unit 11, storage unit 12, and communication unit 13 of information processing device 10, respectively, and therefore detailed description of the configurations will be omitted. Note that storage unit 32 of server 30 stores, in addition to a program (not shown), a captured image DB 32a that stores captured images received from camera 20 together with weight data received from weighing scale 40. Furthermore, server 30 may have, in addition to the above-described configuration, an input unit that accepts operation inputs from a user, a display unit such as a liquid crystal display, a reading unit that reads information stored in a portable storage medium, etc.

[0020] The information processing device 10 of this embodiment extracts the joint positions of a worker for each image frame captured by the camera 20, and estimates (evaluates) the worker's posture based on the extracted joint positions. FIG. 3 is an explanatory diagram of the process for extracting the worker's joint positions. FIG. 3A shows an example of a captured image of one frame of a worker captured from behind, and FIG. 3B shows the result of performing the process for extracting joint positions on the captured image of FIG. 3A. In FIG. 3B, the extracted joints are indicated by white circles, and the skeletal lines are indicated by lines connecting the joints. The information processing device 10 of this embodiment estimates the joint positions of the worker (person) in the captured image and obtains the three-dimensional coordinates of the joint positions using a technology for estimating the skeleton (joint positions) of a person in the captured image, such as openpifpaf or tf-pose-estimation. The information processing device 10 defines the floor surface in the captured image as the XY coordinate plane, and an arbitrary position on the XY coordinate plane as the origin 0, and identifies the coordinate values ​​of each joint position in a Cartesian coordinate space where the X, Y, and Z axes intersect. In the example of Figure 3B, the information processing device 10 extracts 14 joint positions of the worker: neck P1, left shoulder P2, left elbow P3, left wrist P4, right shoulder P5, right elbow P6, right wrist P7, center of pelvis P8 (tailbone), left end of pelvis P9 (left buttocks, left groin), left knee P10, left toe P11, right end of pelvis P12 (right buttocks, right groin), right knee P13, and right toe P14.

[0021] FIG. 4 is an explanatory diagram showing an example of the configuration of back determination table 12a. Back determination table 12a includes an index table 12a1 that stores indexes used to determine whether or not a worker's back (spine, back) is in each of multiple states, and a condition table 12a2 that stores conditions for determining whether or not a worker is in each state based on each index. Index table 12a1 includes a back state column, a state number column, an index number column, an index column, a level column, etc., and stores, for each back state, a state number assigned to each state and information on the index set for each state. The state numbers are assigned as 1, 2, 3, and 4 in order of decreasing load on the worker. The index information includes the index number assigned to each index, the content of each index, and the level set for each index. The indices used to determine the state of the back include the intersection angle (index number 1) between a line connecting the coordinate point P1 of the neck and the coordinate point P8 of the pelvis center and a line extended from the coordinate point P8 of the pelvis center toward the zenith (Z-axis direction), the intersection angle (index number 2) between a line connecting the coordinate point P1 of the neck and the coordinate point P8 of the pelvis center and a line connecting the coordinate point P8 of the pelvis center and a coordinate point P9 of the left end of the pelvis, the intersection angle (index number 3) between a line connecting the coordinate point P1 of the neck and the coordinate point P8 of the pelvis center and a line connecting the coordinate point P8 of the pelvis center and a coordinate point P12 of the right end of the pelvis, and the smaller intersection angle (index number 4) between a line connecting the coordinate point P9 of the left end of the pelvis and a coordinate point P12 of the right end of the pelvis and a line connecting the coordinate point P5 of the right shoulder and the coordinate point P2 of the left shoulder. Condition table 12a2 stores conditions for determining each state of the back in association with each state. For example, for the states of "straight" and "bending forward or backward", a condition (logical formula) that all of the indexes of index numbers 1 to 4 are satisfied is stored. For the state of "twisting or bending sideways", a condition that the index of index number 1 is satisfied, and also the index of index number 2 or 3 is satisfied, and also the index of index number 4 is satisfied is stored. For the state of "twisting and bending sideways", a condition that the index of index number 1 is satisfied is stored. The contents stored in the back determination table 12a are not limited to the example shown in FIG. 4.

[0022] FIG. 5 is an explanatory diagram showing an example of the configuration of the upper limb determination table 12b. The upper limb determination table 12b includes an index table 12b1 that stores indexes used to determine whether or not a worker's upper limb (arm) is in each of multiple states, and a condition table 12b2 that stores conditions for determining whether or not a worker is in each state based on each index. The index table 12b1 includes an upper limb state column, a state number column, an index number column, an index column, a level column, etc., and stores, for each upper limb state, a state number assigned to each state and information about the index set for each state. The state numbers are assigned as 1, 2, and 3 in order of increasing load on the worker. The index information includes an index number assigned to each index, the content of each index, and the level set for each index. The indexes used to determine the state of the upper limbs include the height component (Z coordinate value) of the coordinate point P6 of the right elbow or the coordinate point P7 of the right wrist (index number 1), and the height component (Z coordinate value) of the coordinate point P3 of the left elbow or the coordinate point P4 of the left wrist (index number 2). The condition table 12b2 stores conditions for determining each state in association with each state of the upper limbs. For example, for the states of "both arms below shoulder height" and "both arms above shoulder height", a condition (logical formula) that both indexes of index numbers 1 and 2 are satisfied is stored. For the state of "one arm above shoulder height", a condition that either index number 1 or 2 is satisfied is stored. The contents stored in the upper limb determination table 12b are not limited to the example shown in FIG. 5.

[0023] FIG. 6 is an explanatory diagram showing an example of the configuration of the lower limb determination table 12c. The lower limb determination table 12c includes an index table 12c1 that stores indexes used to determine whether or not a worker's lower limbs (legs) are in each state, and a condition table 12c2 that stores conditions for determining whether or not a worker is in each state based on each index. The index table 12c1 includes a lower limb state column, a state number column, an index number column, an index column, a level column, etc., and stores, for each lower limb state, a state number assigned to each state and information on the index set for each state. The state numbers are assigned from 1 to 6 in ascending order of the load applied to the worker. The index information includes an index number assigned to each index, the content of each index, and the level set for each index. The indices used to determine the condition of the lower limbs include the height component (Z coordinate value) of the coordinate point P13 of the right knee, the height component of the coordinate point P10 of the left knee, the distance between the coordinate point P14 of the right toe and the coordinate point of the pelvis on the XY coordinate plane, the distance between the coordinate point P11 of the left toe and the coordinate point of the pelvis on the XY coordinate plane, the distance between the coordinate point of the midpoint between both toes and the coordinate point of the pelvis on the XY coordinate plane, the height component of the coordinate point P14 of the right toe, the intersection angle of the line connecting the coordinate point P14 of the right toe and the coordinate point P13 of the right knee with the line connecting the coordinate point P13 of the right knee and the coordinate point P12 of the right groin (right end of the pelvis), the height component of the coordinate point P11 of the left toe, and the intersection angle of the line connecting the coordinate point P11 of the left toe and the coordinate point P10 of the left knee with the coordinate point P9 of the left groin (left end of the pelvis). The distance between coordinate point P14 of the right toe and the coordinate point of the pelvis on the XY coordinate plane is, for example, the distance between coordinate point P14 of the right toe on the XY coordinate plane and the line connecting coordinate point P9 of the left end of the pelvis and coordinate point P12 of the right end of the pelvis; the distance between coordinate point P11 of the left toe and the coordinate point of the pelvis on the XY coordinate plane is the distance between coordinate point P11 of the left toe and the line connecting coordinate point P9 of the left end of the pelvis and coordinate point P12 of the right end of the pelvis on the XY coordinate plane; and the distance between the coordinate point of the midpoint between the toes and the coordinate point of the pelvis on the XY coordinate plane is the distance between the coordinate point of the midpoint between coordinate point P11 of the left toe and coordinate point P14 of the right toe on the XY coordinate plane and the line connecting coordinate point P9 of the left end of the pelvis and coordinate point P12 of the right end of the pelvis.Additionally, the level for the indicators using the height component of each coordinate point and the distance between two coordinate points is calculated by multiplying a predetermined coefficient by the distance between the coordinates of a portion of the worker's work whose length can be reliably detected, such as the distance between the center of the worker's pelvis and the neck, or the distance between the right end of the pelvis and the right knee. The predetermined coefficient can be a value that most accurately calculates the load (load score) on the worker's body based on experimental results of posture detection processing for several videos. The condition table 12c2 stores conditions for determining each state of the lower limbs in association with each state. For example, for the states "with one or both knees on the floor" and "standing with the leg on which the center of gravity is placed, either bent or crouching," a condition (logical formula) that satisfies index number 1 or 2 is stored. For the states "standing with both feet straight" and "standing with both knees bent or crouching," a condition that both index numbers 1 and 2 are satisfied is stored. For the state of "sitting", a condition that none of the indicators of indicator numbers 1 to 3 is satisfied is stored, and for the state of "standing with one leg on which the center of gravity is placed straight", a condition that either the indicators of indicator numbers 1 and 2 are satisfied or the indicators of indicator numbers 3 and 4 are satisfied is stored. The contents stored in the lower limb determination table 12c are not limited to the example shown in FIG. 6.

[0024] FIG. 7 is an explanatory diagram showing an example of the configuration of the posture evaluation table 12d. The posture evaluation table 12d stores load scores (load information) that indicate the load on the worker's body, which are evaluated (classified) by the OWAS method based on the worker's posture and the weight of the object being held by the worker. Specifically, the posture evaluation table 12d stores load scores associated with each combination of four types of back posture, three types of upper limb states, six types of lower limb states, and three types of object weight. In the example of FIG. 7, three weight levels are used for the weight of the object: 10 kg or less, over 10 kg to 20 kg or less, and over 20 kg, and weight numbers 1, 2, and 3 are assigned to each weight in ascending order. In addition, although the example of FIG. 7 uses load scores of 1 to 4, the values ​​of the load scores are not limited to these.

[0025] 8A is an explanatory diagram showing an example of the record layout of the evaluation result DB 12e and the captured image DB 32a. The evaluation result DB 12e is a database that stores evaluation results related to the posture of the worker who is the subject of image capture, which are identified based on the captured images captured by the camera 20. The evaluation result DB 12e shown in FIG. 8A includes a camera ID column, an image ID column, a start date and time column, an end date and time column, a worker ID column, a back information column, an upper limb information column, a lower limb information column, a weight information column, a score information column, etc., and stores the evaluation results identified based on the captured images captured by the camera 20 in association with the camera ID assigned to the camera 20. The image ID column stores the image ID assigned to the captured image, and the start date and time column and the end date and time column store the capture start date and time and the capture end date and time of the image frame to be evaluated in the captured images. The worker ID column stores the worker ID assigned to the worker captured in the image frame to be evaluated. The back information column, upper limb information column, lower limb information column, and weight information column store the worker's posture (state of the back, upper limbs, and lower limbs) and the weight of the object being held by the worker, determined based on each image frame to be evaluated. The score information column stores a load score indicating the load on the worker's body, estimated based on the worker's posture and the weight of the object determined for each image frame. The back information column, upper limb information column, lower limb information column, weight information column, and score information column store each piece of information in association with information assigned to each image frame (e.g., frame ID, time from the start of shooting, shooting date and time, etc.), thereby storing time-series data of each piece of information based on the captured images (video) from the start date and time to the end date and time. The stored contents of the evaluation result DB 12e are not limited to the example shown in FIG. 8A.

[0026] The captured image DB 32a is a database that stores images captured by the camera 20 and weight data measured by the weighing scale 40. The captured image DB 32a shown in FIG. 8B is provided in association with a camera ID and includes an image ID column, a start date and time column, an end date and time column, a worker ID column, an image data column, and a weight data column. The captured image DB 32a stores information about the captured image and weight data with the same measurement date and time as the captured image, in association with the image ID assigned to the captured image. The start date and time column and the end date and time column store the start date and time and the end date and time of the capture of the captured image. The worker ID column stores the worker ID of the worker photographed in the captured image. The image data column stores image data of the captured image. The weight data column stores weight data measured when the captured image was captured. The image data and weight data of the captured image may be stored in a predetermined area of ​​the storage unit 32 or in a storage device connected to the server 30. In this case, the image data column and the weight data column store information for reading the data (e.g., a folder name and file name indicating the storage location of the data). The contents stored in the photographed image DB 32a are not limited to the example shown in FIG. 8B.

[0027] The following describes the process in which the information processing device 10 of this embodiment estimates the load on the worker's body based on an image of the worker captured by the camera 20 and the weight of an object measured by the weighing scale 40. FIGS. 9 to 12 are flowcharts showing an example of the load estimation process, and FIG. 13 is an explanatory diagram showing an example screen. The control unit 11 of the information processing device 10 executes the following process in accordance with a program 12P stored in the storage unit 12. Note that the back condition determination process shown in FIG. 10 is the process of step S14 in FIG. 9, the upper limb condition determination process shown in FIG. 11 is the process of step S15 in FIG. 9, and the lower limb condition determination process shown in FIG. 12 is the process of step S16 in FIG. 9.

[0028] The control unit 11 of the information processing device 10 acquires captured images to be processed in the load estimation process and weight data corresponding to the captured images (S11). For example, the control unit 11 requests the server 30 to acquire captured images and weight data for which the load estimation process has not been performed, among the captured images and weight data stored in the captured image DB 32a of the server 30. If the server 30 deletes the captured images and weight data stored in the captured image DB 32a from the captured image DB 32a after transmitting them to the information processing device 10, the control unit 11 may acquire all captured images and weight data stored in the captured image DB 32a. Alternatively, the control unit 11 may request and acquire the captured images and weight data designated as the processing target via the input unit 14 from the server 30. The control unit 11 may acquire the captured images directly from the camera 20 or may acquire the weight data directly from the weighing scale 40. Furthermore, if a photographed image has been previously acquired from the server 30 or the camera 20 and stored in the storage unit 12, the control unit 11 may read out the photographed image to be processed from the storage unit 12. If weight data has been previously acquired from the server 30 or the weighing scale 40 and stored in the storage unit 12, the control unit 11 may read out the weight data to be processed from the storage unit 12. The control unit 11 may extract a portion of the acquired photographed image and weight data as the photographed image and weight data to be processed. Note that the weight data corresponding to the photographed image to be processed can be identified based on the shooting date and time of each image frame in the photographed image and the measurement date and time of each weight in the weight data. The control unit 11 stores the acquired photographed image and weight data to be processed in the storage unit 12.

[0029] The control unit 11 extracts one frame of a captured image (image frame) from the captured image to be processed (S12), and based on the extracted image frame, performs skeletal estimation of the worker appearing in the image frame and extracts the worker's joint positions (S13). Specifically, the control unit 11 estimates the worker's joint positions in the image frame using openpifpaf, tf-pose-estimation, or the like, and acquires three-dimensional coordinates of joint positions P1 to P14 as shown in FIG. 3B. The control unit 11 may also acquire two-dimensional coordinates of the joint positions using openpose, or the like, and convert the acquired two-dimensional coordinates into coordinates in a three-dimensional coordinate space using 3d-pose-estimation, or the like, to acquire three-dimensional coordinates of the joint positions. Note that the control unit 11 may acquire three-dimensional coordinates of the joint positions by any process as long as it can acquire coordinate values ​​in a Cartesian coordinate space in which the floor surface appearing in the image frame is defined as an XY coordinate plane and an arbitrary position on the XY coordinate plane is defined as origin 0, and the X, Y, and Z axes intersect at right angles.

[0030] After acquiring the three-dimensional coordinates of the joint positions P1 to P14, the control unit 11 performs a process of determining the state of the worker's back based on the three-dimensional coordinates of the joint positions P1 to P14 (S14). In the process shown in FIG. 10, the control unit 11 determines whether or not each index corresponding to each state registered in the index table 12a1 of the back determination table 12a satisfies a standard. The control unit 11 first determines whether or not each index corresponding to "straight" satisfies the standard (the appropriateness of each index) (S31). Specifically, the control unit 11 calculates an equation representing, as the index with index number 1, a line connecting the coordinate point P1 of the neck and the coordinate point P8 of the pelvis center, and a line extending from the coordinate point P8 of the pelvis center toward the zenith (Z-axis direction), calculates the intersection angle between the two lines, and determines whether or not the calculated angle is less than 20 degrees. If the calculated angle is less than 20 degrees, the control unit 11 determines that the standard of the index is satisfied, and if the calculated angle is 20 degrees or greater, determines that the standard of the index is not satisfied. Furthermore, as an indicator of index number 2, the control unit 11 calculates an equation representing a line connecting the coordinate point P1 of the neck and the coordinate point P8 of the pelvis center, and a line connecting the coordinate point P8 of the pelvis center and the coordinate point P9 of the left end of the pelvis, calculates the intersection angle of the two lines, and determines whether the calculated angle is 70 degrees or greater. If the calculated angle is 70 degrees or greater, the control unit 11 determines that the level of the indicator is met, and if the calculated angle is less than 70 degrees, it determines that the level of the indicator is not met. Furthermore, as an indicator of index number 3, the control unit 11 calculates an equation representing a line connecting the coordinate point P1 of the neck and the coordinate point P8 of the pelvis center, and a line connecting the coordinate point P8 of the pelvis center and the coordinate point P12 of the right end of the pelvis, calculates the intersection angle of the two lines, and determines whether the calculated angle is 70 degrees or greater. If the calculated angle is 70 degrees or more, the control unit 11 determines that the level of the indicator is met, and if it is less than 70 degrees, the control unit 11 determines that the level of the indicator is not met. Furthermore, as the indicator of indicator number 4, the control unit 11 calculates an equation representing the line connecting the coordinate point P9 of the left end of the pelvis and the coordinate point P12 of the right end of the pelvis, and the line connecting the coordinate point P5 of the right shoulder and the coordinate point P2 of the left shoulder, calculates the smaller intersection angle of the two lines, and determines whether the calculated angle is less than 20 degrees.If the calculated angle is less than 20 degrees, the control unit 11 determines that the level of the index is met, and if the calculated angle is 20 degrees or more, it determines that the level of the index is not met. Depending on the determination result of whether or not the level is met for each of the indexes of index numbers 1 to 4, the control unit 11 determines whether or not the condition corresponding to "straight" registered in the condition table 12a2 is met (S32). Here, the control unit 11 determines whether or not the condition of meeting the level is met for all of the indexes of index numbers 1 to 4, and if it determines that the condition is met (S32: YES), it specifies the back state of the worker as state number 1 (straight) (S33), and returns to the processing of FIG. 9.

[0031] When the control unit 11 determines that the condition corresponding to "straight" is not satisfied (S32: NO), that is, when the standard is not met for any of the indicators of indicator numbers 1 to 4, it determines whether the standard is met for each indicator corresponding to "bend forward or backward" (the appropriateness of each indicator) (S34). Specifically, the control unit 11 determines, for the indicator of indicator number 1, whether the intersection angle between the line connecting the coordinate point P1 of the neck and the coordinate point P8 of the pelvis center and the line extending from the coordinate point P8 of the pelvis center toward the zenith is 20 degrees or more. Furthermore, for the indicator of index number 2, the control unit 11 determines whether the intersection angle between the line connecting the coordinate point P1 of the neck and the coordinate point P8 of the pelvis center and the line connecting the coordinate point P8 of the pelvis center and the coordinate point P9 of the left end of the pelvis is 70 degrees or more. Furthermore, the control unit 11 determines whether the angle of intersection between the line connecting the coordinate point P1 of the neck and the coordinate point P8 of the pelvis center and the line connecting the coordinate point P8 of the pelvis center and the coordinate point P12 of the right end of the pelvis is 70 degrees or more, as the index of index number 3. Furthermore, the control unit 11 determines whether the smaller angle of intersection between the line connecting the coordinate point P9 of the left end of the pelvis and the coordinate point P12 of the right end of the pelvis and the line connecting the coordinate point P5 of the right shoulder and the coordinate point P2 of the left shoulder is less than 20 degrees, as the index of index number 4. Then, the control unit 11 determines whether the condition corresponding to "bending forward or backward" is satisfied according to the determination result of whether or not each of the indexes of index numbers 1 to 4 satisfies the standard (S35). Here, the control unit 11 determines whether the condition that the level is met for all indicators of indicator numbers 1 to 4 is met, and if it determines that the condition is met (S35: YES), it identifies the worker's back condition as condition number 2 (bending forward or backward) (S36) and returns to the processing of Figure 9.

[0032] When the control unit 11 determines that the condition corresponding to "bending forward or backward" is not satisfied (S35: NO), that is, when the standard is not met for any of the indicators of indicator numbers 1 to 4, it determines whether or not the standard is met for each indicator corresponding to "twisting or bending sideways" (the appropriateness of each indicator) (S37). Specifically, the control unit 11 determines, for the indicator of indicator number 1, whether the angle of intersection between the line connecting the coordinate point P1 of the neck and the coordinate point P8 of the pelvis center and the line extending from the coordinate point P8 of the pelvis center toward the zenith is 20 degrees or more. Furthermore, for the indicator of index number 2, the control unit 11 determines whether the angle of intersection between the line connecting the coordinate point P1 of the neck and the coordinate point P8 of the pelvis center and the line connecting the coordinate point P8 of the pelvis center and the coordinate point P9 of the left end of the pelvis is less than 70 degrees. Furthermore, the control unit 11 determines whether the angle of intersection between the line connecting the coordinate point P1 of the neck and the coordinate point P8 of the pelvis center and the line connecting the coordinate point P8 of the pelvis center and the coordinate point P12 of the right end of the pelvis is less than 70 degrees, as the index of index number 3. Furthermore, the control unit 11 determines whether the smaller angle of intersection between the line connecting the coordinate point P9 of the left end of the pelvis and the coordinate point P12 of the right end of the pelvis and the line connecting the coordinate point P5 of the right shoulder and the coordinate point P2 of the left shoulder is 20 degrees or more, as the index of index number 4. Then, the control unit 11 determines whether the condition corresponding to "twisting or bending sideways" is satisfied according to the determination result of whether or not each of the indexes of index numbers 1 to 4 satisfies the standard (S38). Here, the control unit 11 determines whether the conditions that the index of index number 1 is satisfied, the index of index number 2 or 3 is satisfied, and the index of index number 4 is satisfied are satisfied. If it is determined that the conditions are satisfied (S38: YES), the control unit 11 specifies the back state of the worker as state number 3 (twisting or bending sideways) (S39), and returns to the processing of FIG. 9. If the control unit 11 determines that the condition corresponding to "twisting or bending sideways" is not satisfied (S38: NO), the control unit 11 specifies the back state of the worker as state number 4 (twisting and bending sideways) (S40), and returns to the processing of FIG. 9. Note that the control unit 11 stores, in the evaluation result DB 12e, information of the image frame to be processed (e.g., frame ID) and the state number specified in step S33, S36, S39, or S40 in association with the back information sequence corresponding to the image ID of the captured image to be processed.

[0033] Next, the control unit 11 performs a process of determining the state of the worker's upper limbs based on the three-dimensional coordinates of the joint positions P1 to P14 (S15). In the process shown in FIG. 11, the control unit 11 determines whether or not each index corresponding to each state registered in the index table 12b1 of the upper limb determination table 12b satisfies a standard. The control unit 11 first determines whether or not each index corresponding to "both arms below the shoulders" satisfies a standard (the appropriateness of each index) (S51). Specifically, the control unit 11 extracts the height components (Z coordinate values) of the coordinate point P6 of the right elbow, the coordinate point P7 of the right wrist, and the coordinate point P5 of the right shoulder as the index with index number 1, and determines whether or not the height component of the coordinate point P6 of the right elbow or the coordinate point P7 of the right wrist is less than the height component of the coordinate point P5 of the right shoulder. The control unit 11 determines that the level of the index is met if the height component of the right elbow coordinate point P6 or the right wrist coordinate point P7 is less than the height component of the right shoulder coordinate point P5, and determines that the level of the index is not met if the height component is equal to or greater than the height component of the right shoulder coordinate point P5. The control unit 11 also extracts the height components (Z coordinate values) of the left elbow coordinate point P3, the left wrist coordinate point P4, and the left shoulder coordinate point P2 as the index of index number 2, and determines whether the height component of the left elbow coordinate point P3 or the left wrist coordinate point P4 is less than the height component of the left shoulder coordinate point P2. The control unit 11 determines that the level of the index is met if the height component of the left elbow coordinate point P3 or the left wrist coordinate point P4 is less than the height component of the left shoulder coordinate point P2, and determines that the level of the index is not met if the height component is equal to or greater than the height component of the left shoulder coordinate point P2. The control unit 11 determines whether the condition corresponding to "both arms below the shoulders" registered in the condition table 12b2 is satisfied, depending on the determination result of whether the level is satisfied for each of the indices of index numbers 1 and 2 (S52). Here, the control unit 11 determines whether the condition of satisfying the level for both indices of index numbers 1 and 2 is satisfied, and if it determines that the condition is satisfied (S52: YES), it specifies the state of the upper limbs of the worker as state number 1 (both arms below the shoulders) (S53), and returns to the processing of FIG.

[0034] If the control unit 11 determines that the condition corresponding to "both arms below shoulder height" is not satisfied (S52: NO), that is, if the standard is not satisfied for any of the indicators of indicator numbers 1 and 2, the control unit 11 determines whether or not the standard is satisfied for each indicator corresponding to "one arm above shoulder height" (the appropriateness of each indicator) (S54). Specifically, the control unit 11 determines, as the indicator of index number 1, whether the height component of coordinate point P6 of the right elbow or coordinate point P7 of the right wrist is equal to or greater than the height component of coordinate point P5 of the right shoulder. The control unit 11 also determines, as the indicator of index number 2, whether the height component of coordinate point P3 of the left elbow or coordinate point P4 of the left wrist is equal to or greater than the height component of coordinate point P2 of the left shoulder. Then, the control unit 11 determines, depending on the result of the determination of whether or not the standard is satisfied for each of the indicators of index numbers 1 and 2, whether or not the condition corresponding to "one arm above shoulder height" is satisfied (S55). Here, the control unit 11 determines whether or not a condition that a level is met for any of the indexes of index numbers 1 to 2 is met, and if it determines that the condition is met (S55: YES), it specifies the worker's upper limb state as state number 2 (one arm is at or above shoulder height) (S56), and returns to the processing of FIG. 9. If the control unit 11 determines that the condition corresponding to "one arm is at or above shoulder height" is not met (S55: NO), it specifies the worker's upper limb state as state number 3 (both arms are at or above shoulder height) (S57), and returns to the processing of FIG. 9. Note that the control unit 11 stores, in the evaluation result DB 12e, information on the image frame to be processed (e.g., frame ID) and the state number specified in step S53, S56, or S57 in an upper limb information sequence corresponding to the image ID of the captured image to be processed, in association with each other.

[0035] Next, the control unit 11 performs a process of determining the state of the lower limbs of the worker based on the three-dimensional coordinates of the joint positions P1 to P14 (S16). In the process shown in FIG. 12, the control unit 11 determines whether or not each index corresponding to each state registered in the index table 12c1 of the lower limb determination table 12c satisfies a standard. The control unit 11 first determines whether or not each index corresponding to "placing one or both knees on the floor" satisfies a standard (the appropriateness of each index) (S61). Specifically, the control unit 11 extracts the height component (Z coordinate value) of the coordinate point P13 of the right knee as the index with index number 1 corresponding to "placing one or both knees on the floor" and determines whether or not it is less than 0.1. If the height component of the coordinate point P13 of the right knee is less than 0.1, the control unit 11 determines that the standard of the index is satisfied, and if it is 0.1 or greater, the control unit 11 determines that the standard of the index is not satisfied. The control unit 11 also extracts the height component (Z coordinate value) of the coordinate point P10 of the left knee as the index of index number 2, and determines whether it is less than 0.1. If it is less than 0.1, it determines that the level of the index is met, and if it is 0.1 or more, it determines that the level of the index is not met. The control unit 11 determines whether the condition corresponding to "one or both knees on the floor" registered in the condition table 12c2 is met, depending on the determination result of whether the level is met for each of the indexes of index numbers 1 to 2 (S62). Here, the control unit 11 determines whether the condition of meeting the level is met for any of the indexes of index numbers 1 to 2, and if it determines that the condition is met (S62: YES), it specifies the state of the worker's lower limbs as state number 6 (one or both knees on the floor) (S63), and returns to the processing of FIG. 9.

[0036] When the control unit 11 determines that the condition corresponding to "placing one or both knees on the floor" is not satisfied (S62: NO), it determines whether or not each indicator corresponding to "sitting" satisfies the standard (the appropriateness of each indicator) (S64). Specifically, the control unit 11 extracts the coordinate values ​​(X and Y coordinate values) of the coordinate point P14 of the right foot on the XY coordinate plane and the coordinate points P9, P8, and P12 of the left end, center, and right end of the pelvis as the indicator of indicator number 1 corresponding to "sitting," calculates an equation representing a line connecting the coordinate point P9 of the left end of the pelvis and the coordinate point P12 of the right end of the pelvis, calculates the distance between the calculated line and the coordinate point P14 of the right foot, and determines whether or not the calculated distance is less than 0.2. Furthermore, the control unit 11 calculates the distance between the coordinate point P11 of the left foot tip and the line connecting the coordinate point P9 of the left end of the pelvis and the coordinate point P12 of the right end of the pelvis on the XY coordinate plane as the index of index number 2, and determines whether the calculated distance is less than 0.2. Furthermore, the control unit 11 calculates the coordinate value of the midpoint between the coordinate points P14 and P11 of the right and left foot tips on the XY coordinate plane as the index of index number 3, and calculates the distance between the coordinate value of the midpoint and the line connecting the coordinate point P9 of the left end of the pelvis and the coordinate point P12 of the right end of the pelvis, and determines whether the calculated distance is less than 0.2. Then, the control unit 11 determines whether the condition corresponding to "sitting" is satisfied according to the determination result of whether or not each of the indexes of index numbers 1 to 3 satisfies the level (S65). Here, the control unit 11 determines whether the condition that the level is not met for all of the indicators of indicator numbers 1 to 3 is met, and if it determines that the condition is met (S65: YES), it identifies the worker's lower limb condition as condition number 1 (sitting) (S66) and returns to the processing of Figure 9.

[0037] When the control unit 11 determines that the condition corresponding to "sitting" is not satisfied (S65: NO), it determines whether or not the level of each indicator corresponding to "standing with the leg on which the center of gravity is placed straight" is satisfied (the appropriateness of each indicator) (S67). Specifically, the control unit 11 extracts the height component (Z coordinate value) of the coordinate point P14 of the right foot as the indicator of indicator number 1 corresponding to "standing with the leg on which the center of gravity is placed straight" and determines whether or not it is less than 0.1. When the height component of the coordinate point P14 of the right foot is less than 0.1, the control unit 11 determines that the level of the indicator is satisfied, and when it is 0.1 or more, it determines that the level of the indicator is not satisfied. The control unit 11 also calculates an equation representing a line connecting the coordinate point P14 of the right toe and the coordinate point P13 of the right knee, and a line connecting the coordinate point P13 of the right knee and the coordinate point P12 of the right groin (the right end of the pelvis), as an index of index number 2, calculates the intersection angle between the two lines, and determines whether the calculated angle is 150 degrees or greater. The control unit 11 also extracts the height component of the coordinate point P11 of the left toe, as an index of index number 3, and determines whether it is less than 0.1. The control unit 11 also calculates the intersection angle between the line connecting the coordinate point P11 of the left toe and the coordinate point P10 of the left knee, and a line connecting the coordinate point P10 of the left knee and the coordinate point P9 of the left groin (the left end of the pelvis), as an index of index number 4, and determines whether the calculated angle is 150 degrees or greater. Then, the control unit 11 determines whether the condition corresponding to "standing with the leg on which the center of gravity is placed straight" is met, depending on the determination result of whether the level is met for each of the indexes of index numbers 1 to 4 (S68). Here, the control unit 11 determines whether the condition of meeting both of the indexes of index numbers 1 and 2, or both of the indexes of index numbers 3 and 4, is met, and if it determines that the condition is met (S68: YES), it specifies the state of the lower limbs of the worker as state number 3 (standing with the leg on which the center of gravity is placed straight) (S69), and returns to the processing of FIG. 9.

[0038] When the control unit 11 determines that the condition corresponding to "standing with one leg on which the center of gravity is placed straight" is not satisfied (S68: NO), it determines whether or not the standard is satisfied for each indicator corresponding to "standing with both legs straight" (the appropriateness of each indicator) (S70). Specifically, the control unit 11 determines, as the indicator of indicator number 1 corresponding to "standing with both legs straight," whether the angle of intersection between the line connecting the coordinate point P14 of the right toe and the coordinate point P13 of the right knee and the line connecting the coordinate point P13 of the right knee and the coordinate point P12 of the right groin (the right end of the pelvis) is 150 degrees or more. Also, as the indicator of indicator number 2, the control unit 11 determines whether the angle of intersection between the line connecting the coordinate point P11 of the left toe and the coordinate point P10 of the left knee and the coordinate point P9 of the left groin (the left end of the pelvis) is 150 degrees or more. Then, the control unit 11 determines whether the condition corresponding to "standing with both legs straight" is satisfied (S71) depending on the determination result of whether the level is satisfied for each of the indicators of indicator numbers 1 and 2. Here, the control unit 11 determines whether the condition of satisfying both indicators of indicator numbers 1 and 2 is satisfied, and if it determines that the condition is satisfied (S71: YES), it specifies the lower limb condition of the worker as condition number 2 (standing with both legs straight) (S72), and returns to the processing of FIG.

[0039] When the control unit 11 determines that the condition corresponding to "standing with both feet straight" is not satisfied (S71: NO), it determines whether or not the standard is satisfied for each indicator corresponding to "standing with the leg on which the center of gravity is placed bent, or in a half-squat position" (S73). Specifically, the control unit 11 determines, as the indicator of indicator number 1 corresponding to "standing with the leg on which the center of gravity is placed bent, or in a half-squat position," whether or not the angle of intersection between the line connecting the coordinate point P14 of the right toe and the coordinate point P13 of the right knee and the line connecting the coordinate point P13 of the right knee and the coordinate point P12 of the right hip is 150 degrees or more. Also, as the indicator of indicator number 2, the control unit 11 determines whether or not the angle of intersection between the line connecting the coordinate point P11 of the left toe and the coordinate point P10 of the left knee and the coordinate point P9 of the left hip is 150 degrees or more. Then, the control unit 11 determines whether the condition corresponding to "standing with one leg on which the center of gravity is placed, or squatting" is met, depending on the determination result of whether the level is met for each of the indicators with indicator numbers 1 to 2 (S74). Here, the control unit 11 determines whether the condition of meeting any of the indicators with indicator numbers 1 to 2 is met, and if it determines that the condition is met (S74: YES), it identifies the state of the lower limbs of the worker as state number 5 (standing with one leg on which the center of gravity is placed, or squatting) (S75), and returns to the processing of Fig. 9. If the control unit 11 determines that the condition corresponding to "standing with one leg on which the center of gravity is placed, or squatting" is not met (S74: NO), it identifies the state of the lower limbs of the worker as state number 4 (standing with both knees bent, or squatting) (S76), and returns to the processing of Fig. 9. In addition, in the evaluation result DB12e, the control unit 11 stores the lower limb information sequence corresponding to the image ID of the captured image to be processed in association with the information of the image frame to be processed (e.g., frame ID) and the condition number identified in step S63, S66, S69, S72, S75, or S76.

[0040] Returning to the process of FIG. 9, the control unit 11 determines the weight of the object being held by the worker in the image frame extracted in step S12 (S17). Here, based on the photographing date and time of the image frame extracted in step S12, the control unit 11 extracts the weight having the photographing date and time as the measurement date and time from the weight data of the processing target acquired in step S11, thereby identifying the weight of the object. Based on the identified weight of the object, the control unit 11 identifies the weight information of the object from 1 to 3 (1: 10 kg or less, 2: more than 10 kg and less than 20 kg, 3: more than 20 kg). The control unit 11 associates information of the image frame to be processed (e.g., frame ID) with the determined weight information and stores it in the evaluation result DB 12e in a weight information sequence corresponding to the image ID of the captured image to be processed.

[0041] The control unit 11 evaluates the posture of the worker based on the states of the worker's back, upper limbs, and lower limbs determined in steps S14 to S16 and the weight of the object determined in step S17 (S18). Here, the control unit 11 identifies a load score (load information) corresponding to the combination of the back state determined in step S14, the upper limb state determined in step S15, the lower limb state determined in step S16, and the weight of the object determined in step S17, based on the posture evaluation table 12d. For example, if the back state is determined to be "bent forward or backward (state number 2)," the upper limb state is determined to be "both arms below the shoulders (state number 1)," the lower limb state is determined to be "standing with both legs straight (state number 2)," and the weight is determined to be "over 20 kg (weight number 3)," the control unit 11 identifies a load score of "3." Through this process, a load score is obtained that quantifies the load applied to the worker's body depending on the worker's posture and the weight of the object. The control unit 11 stores the identified load score in the score information sequence corresponding to the image ID of the captured image to be processed in the evaluation result DB 12e in association with information (for example, frame ID) of the image frame to be processed (S19).

[0042] The control unit 11 determines whether or not there are any unprocessed image frames that have not yet been subjected to the process for evaluating the worker's posture among the image frames included in the photographed images to be processed acquired in step S11 (S20). If it is determined that there are unprocessed image frames (S20: YES), the control unit 11 returns to the process of step S12 and executes the processes of steps S12 to S19 for the unprocessed image frames. The control unit 11 repeats steps S12 to S20 until it determines that there are no unprocessed image frames. As a result, for each image frame included in the photographed images to be processed, a load score is determined based on the posture of the worker captured in each image frame (the state of the back, upper limbs, and lower limbs) and the weight of the object held by the worker, and the load score is stored in the evaluation result DB 12e.

[0043] If it is determined that there are no unprocessed image frames (S20: NO), the control unit 11 generates an evaluation result screen based on the information stored in the evaluation result DB 12e (S21). FIG. 13 shows an example of the evaluation result screen, illustrating the evaluation results when processing images captured over a 3-second period at 15 frames per second. The screen shown in FIG. 13 displays the worker ID of the worker being evaluated, the image ID and capture date and time (start date and end date and time of capture) of the captured image being evaluated, and the number of image frames being evaluated (number of evaluations). The evaluation result screen also displays, as evaluation results, a frequency distribution table A1 of stress scores 1 to 4, a bar graph A2, a graph A3 showing time-series changes in stress scores, and a graph A4 showing time-series changes in the evaluation of the condition of each body part and the weight of the object (body part evaluation). The frequency distribution table A1 shows the number of image frames (number of cases) determined to have each stress score and the ratio (relative frequency) of the number of cases of each stress score to the number of image frames being evaluated. Bar graph A2 is a stacked graph showing the ratio of the number of occurrences of each stress score. Graph A3 is a graph showing time on the horizontal axis and stress scores on the vertical axis, plotting stress scores determined based on image frames captured at each time. Graph A4 is a graph showing time on the horizontal axis and evaluations of the worker's posture and object weight (condition numbers and weight numbers for each part of the back, upper limbs, and lower limbs) on the vertical axis, plotting the condition numbers and weight numbers for each part determined based on image frames captured at each time in a manner assigned to each part and weight. In graph A4 of FIG. 13, the condition of the back is shown with a dashed line, the condition of the upper limbs with a one-dot chain line, the condition of the lower limbs with a solid line, and weight with a two-dot chain line.

[0044] 13, the control unit 11 counts the number of image frames to be evaluated and the number of image frames judged to have each of the stress scores 1 to 4 based on the stress scores stored in the score information column of the evaluation result DB 12e, and generates a frequency distribution table A1 and a bar graph A2 from the counting results. The control unit 11 also generates graph A3 by plotting the stress scores stored in the score information column of the evaluation result DB 12e in association with the information of the image frames used to identify each stress score (e.g., frame ID, time from the start of shooting, shooting date and time). The control unit 11 also generates graph A4 by plotting the condition numbers and weight numbers of the back, upper limbs, and lower limbs stored in the back information column, upper limbs information column, lower limbs information column, and weight information column of the evaluation result DB 12e in association with the information of the image frames (e.g., frame ID, time from the start of shooting, shooting date and time). Furthermore, the control unit 11 identifies image frames with a load score of 3 or more, and adds a mark M indicating "high load" and a message (text data) to the identified image frames in graphs A3 and A4.

[0045] The control unit 11 generates a screen as shown in FIG. 13 through the above-described processing, and displays the generated evaluation result screen on the display unit 15 (S22). By displaying the screen shown in FIG. 13, it is possible to provide the time progression of the stress applied to the worker's body as the subject of image capture based on the captured images, and the ratio of each stress score in the captured images to be processed. Therefore, it is possible to understand the stress level applied to the worker's body for each task performed by the worker. In addition to displaying the generated evaluation result screen on the display unit 15, the control unit 11 may transmit the generated evaluation result screen to another device via the network N, or may transmit it to a printer connected via the network or directly and print it. In the processing shown in FIG. 9, the processes of steps S14 to S17 may be performed in any order, or may be performed in parallel.

[0046] Conventionally, worker posture has been measured by visual confirmation by the measurer, which can lead to variations in measurement results depending on the measurer's experience, fatigue level, and other factors. In contrast, in this embodiment, the worker's posture is measured based on a captured image of the worker, thereby obtaining objective measurement results. Furthermore, in this embodiment, the load level (load score) is evaluated according to the OWAS method, which takes into account the objectively measured worker's posture and the weight of the object the worker is holding during work. This allows the load level to be indicated using an objective index (load score). Therefore, the load level imposed on a worker can be evaluated using the same load score for various tasks at a workplace, making it applicable to a variety of tasks and convenient. Furthermore, in this embodiment, the worker is photographed with a single camera 20, making it possible to realize a system that can visualize the worker's load level based on the captured image with a simple system configuration. Furthermore, in this embodiment, based on the objective index (load score), it is possible to consider improving the worker's working posture in order to reduce the load level (load score) for tasks with a high load level, for example. This reduces the worker's load, thereby improving work safety.

[0047] In this embodiment, the load imposed on the worker captured in the image frame is evaluated (a load score is determined) for each image frame included in the captured image to be processed, but this configuration is not limited to this. For example, the evaluation process may be performed at predetermined time intervals, such as every 1 / 3 second, every 1 / 2 second, or every second. For example, if the camera 20 captures 30 frames per second, the evaluation process may be performed every 5 frames (every 1 / 6 second), every 10 frames (every 1 / 3 second), or every 15 frames (every 1 / 2 second).

[0048] In the present embodiment, the control unit 11 may be configured to display a message or advice corresponding to the evaluation result on the evaluation result screen. For example, a message or advice regarding the worker's working posture may be stored in advance in the storage unit 12 for each balance of the ratios of the load scores 1 to 4, and the control unit 11 may calculate the ratio of each load score, and then read out the message or advice corresponding to the balance of the ratios of each load score from the storage unit 12 and display it on the evaluation result screen. This allows the worker to improve their posture in accordance with the message or advice, thereby reducing the load during work.

[0049] (Embodiment 2) A modified example of the method for measuring the weight of an object held by a worker in the above-described first embodiment will be described. The information processing system of this embodiment does not include the weighing scale 40 in the information processing system of the first embodiment shown in Fig. 1, and the information processing device 10, camera 20, and server 30 have the same configurations as the devices 10, 20, and 30 in the first embodiment. Also, in this embodiment, the captured image DB 32a stored in the storage unit 32 of the server 30 does not have a weight data string in the configuration shown in Fig. 8B.

[0050] In the information processing system of this embodiment, the information processing device 10 can execute processes similar to those shown in FIGS. 9 to 12. In step S11, the control unit 11 of the information processing device 10 of this embodiment acquires only the captured image to be processed from the server 30. In step S17, the control unit 11 detects an object in the image frame by, for example, performing object detection processing on the image frame extracted in step S12, and identifies the weight of the detected object. The object detection processing can be performed using an object detection model trained on machine learning to output a result of determining whether an object included in an input image frame is one of previously learned objects. The object detection processing can also be performed by template matching using a template image that indicates image features of a captured image of an object handled by a worker. Furthermore, the object detection processing can be achieved by reading information for identifying the object or a code indicating the information (such as a barcode or QR code (registered trademark)) attached to the object. The weight of the detected object can be identified, for example, using a table in which the weights of objects handled by workers are registered in advance. In this case, the control unit 11 identifies the weight of the object detected by the object detection process from the table. Note that the control unit 11 may estimate the weight of the detected object based on the type and size of the object detected by the object detection process.

[0051] The control unit 11 may also identify the work content being performed by the worker in the image frame based on the image frame, thereby identifying the object held by the worker in the identified work content and determining the weight of the identified object. The process of identifying the work content based on the image frame can be performed using a learning model that has been machine-learned to output a result of determining the work content of the worker captured in the image frame when the image frame is input. The process of identifying the work content can also be performed by template matching using a template image that indicates image features of the captured image of the worker performing each work. Alternatively, the control unit 11 may be configured to identify the work content from the input information by the worker or the user of the information processing device 10 inputting information about the work content (e.g., a work process number) via the input unit 14 or another terminal capable of communicating with the information processing device 10. The object handled in the identified work content can be identified, for example, using a table that registers information about the object held by the worker in each work in association with the work content. When using a table that registers the weight of the object held by the worker in each work in association with the work content, the control unit 11 can identify the weight of the object corresponding to the work content based on the identified work content.

[0052] Furthermore, the control unit 11 may accept input of the weight of the object here via the input unit 14 or another terminal capable of communicating with the information processing device 10. For example, the control unit 11 may acquire the weight of the object by having a worker or a user of the information processing device 10 input the weight of the object measured using a weighing scale provided at the work site via the input unit 14 or another terminal capable of communicating with the information processing device 10. The control unit 11 determines weight information of the object from 1 to 3 based on the identified weight of the object. Note that, in addition to identifying the weight of the object, the control unit 11 may also be configured to determine whether the weight of the object is 10 kg or less, more than 10 kg and less than 20 kg, or more than 20 kg.

[0053] The information processing device 10 of this embodiment performs the same processing as in the first embodiment except for determining the weight of an object held by a worker based on a captured image of the worker, and therefore provides the same effects as in the first embodiment. Furthermore, this embodiment does not use a weighing scale 40, so the worker does not need to measure the weight of the object being held, improving operability. The modified examples described in the first embodiment above can also be applied to this embodiment as appropriate.

[0054] The following additional notes are disclosed regarding the above embodiments including the first and second embodiments.

[0055] (Appendix 1) Acquire a photographed image of the worker, acquiring weight information regarding the weight of an object held by the worker; Identifying the posture of the worker based on the acquired photographed image; Identifying load information relating to the load applied to the body of the worker based on the identified posture of the worker and the acquired weight information. A program that causes a computer to perform a process.

[0056] (Appendix 2) Using a table in which scores relating to the loads applied to the worker's body are associated with each combination of a plurality of types of postures and weight information of the object, the score corresponding to the identified posture of the worker and the acquired weight information is identified. 2. The program according to claim 1, which causes the computer to execute the process.

[0057] (Appendix 3) The posture includes a posture of the back of the worker, a posture of the upper limbs of the worker, and a posture of the lower limbs of the worker. 1. A program according to claim 1 or 2.

[0058] (Appendix 4) Using a table in which a plurality of types of postures are associated with each of the back, upper limbs, and lower limbs of the worker, postures of the back, upper limbs, and lower limbs of the worker in the captured image are identified. 4. The program according to claim 3, which causes the computer to execute the process.

[0059] (Appendix 5) Outputting information corresponding to the identified load information 5. The program according to any one of claims 1 to 4, which causes the computer to execute the process.

[0060] (Appendix 6) Identifying load information relating to a load applied to the body of the worker based on a posture of the worker identified based on each of a plurality of photographed images of the worker taken in time series and weight information of an object held by the worker captured in each of the plurality of photographed images; Outputs the time series changes in the load information 6. The program according to any one of claims 1 to 5, which causes the computer to execute the process.

[0061] (Appendix 7) the posture includes a posture of the back of the worker, a posture of the upper limbs of the worker, and a posture of the lower limbs of the worker, outputting time-series changes in posture of the back, upper limbs, and lower limbs of the worker identified based on each of the plurality of photographed images photographed in time series; 7. The program according to claim 6, which causes the computer to execute the process.

[0062] (Appendix 8) Acquire a photographed image of the worker, acquiring weight information regarding the weight of an object held by the worker; Identifying the posture of the worker based on the acquired photographed image; Identifying load information relating to the load applied to the body of the worker based on the identified posture of the worker and the acquired weight information. An information processing method in which processing is performed by a computer.

[0063] (Appendix 9) In an information processing device having a control unit, The control unit Acquire a photographed image of the worker, acquiring weight information regarding the weight of an object held by the worker; Identifying the posture of the worker based on the acquired photographed image; Identifying load information relating to the load applied to the body of the worker based on the identified posture of the worker and the acquired weight information. Information processing device.

[0064] The technical features (constituent elements) described in each embodiment can be combined with each other, and by combining them, new technical features can be formed. Furthermore, the independent and dependent claims described in the claims can be combined with each other in any combination, regardless of the reference format. Furthermore, while the claims use a format in which a claim references two or more other claims (multiple claim format), this is not limited to this format. Multiple claims (multi-multi claims) that reference at least one other multiple claim may also be used.

[0065] The embodiments disclosed herein are to be considered in all respects as illustrative and not restrictive. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0066] 10. Information processing equipment 11 Control section 12 Storage section 13 Communications Department 20 Camera 30 servers 31 Control Unit 32 Memory 33 Ministry of Communications 40 weight scale

Claims

1. Acquire a photographed image of the worker, acquiring weight information regarding the weight of an object held by the worker; Identifying the posture of the worker based on the acquired photographed image; Identifying load information relating to the load applied to the body of the worker based on the identified posture of the worker and the acquired weight information. A program that causes a computer to perform a process.

2. Using a table in which scores relating to the loads applied to the worker's body are associated with each combination of a plurality of types of postures and weight information of the object, the score corresponding to the identified posture of the worker and the acquired weight information is identified. The program according to claim 1 , which causes the computer to execute a process.

3. The posture includes a posture of the back of the worker, a posture of the upper limbs of the worker, and a posture of the lower limbs of the worker. The program according to claim 1 or 2.

4. Using a table in which a plurality of types of postures are associated with each of the back, upper limbs, and lower limbs of the worker, postures of the back, upper limbs, and lower limbs of the worker in the captured image are identified. The program according to claim 3, which causes the computer to execute processing.

5. Outputting information corresponding to the identified load information 3. The program according to claim 1, which causes the computer to execute processing.

6. Identifying load information relating to a load applied to the body of the worker based on a posture of the worker identified based on each of a plurality of photographed images of the worker in time series and weight information of an object held by the worker photographed in each of the plurality of photographed images; Outputs the time series changes in the load information 3. The program according to claim 1, which causes the computer to execute processing.

7. the posture includes a posture of the back of the worker, a posture of the upper limbs of the worker, and a posture of the lower limbs of the worker, outputting time-series changes in posture of the back, upper limbs, and lower limbs of the worker identified based on each of the plurality of photographed images photographed in time series; The program according to claim 6, which causes the computer to execute processing.

8. Acquire a photographed image of the worker, acquiring weight information regarding the weight of an object held by the worker; Identifying the posture of the worker based on the acquired photographed image; Identifying load information relating to the load applied to the body of the worker based on the identified posture of the worker and the acquired weight information. An information processing method in which processing is performed by a computer.

9. In an information processing device having a control unit, The control unit Acquire a photographed image of the worker, acquiring weight information regarding the weight of an object held by the worker; Identifying the posture of the worker based on the acquired photographed image; Identifying load information relating to the load applied to the body of the worker based on the identified posture of the worker and the acquired weight information. Information processing device.

Images