PROGRAM, INFORMATION PROCESSING METHOD, AND INFORMATION PROCESSING APPARATUS

Abstract

To provide a program etc. capable of accurately evaluating the load applied to a worker. According to a program, a computer acquires an image of a worker and weight information related to the weight of an object held by the worker. The computer then identifies the posture of the worker based on the acquired image, and identifies load information related to the load applied to the body of the worker based on the identified posture of the worker and the acquired weight information.

Description

[Technical field]

[0001] The present disclosure relates to a program, an information processing method, and an information processing device. [Background technology]

[0002] Patent Document 1 discloses a technique for evaluating the posture of a moving object by extracting joint positions of the moving object from an image of the moving object and reconstructing the skeleton of the moving object based on the spatial distance to each extracted joint position. In the technique disclosed in Patent Document 1, the distance to the moving object is measured from images simultaneously captured by multiple optical cameras, or the image and distance information of the moving object are obtained by simultaneously capturing images by an optical camera and a distance camera. In addition, the technique disclosed in Patent Document 1 uses the Ovako Working Posture Analyzing System (OWAS) method to evaluate the load applied to the moving object depending on the posture of the moving object. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2023-59753 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the technology disclosed in Patent Document 1, moving objects are photographed by multiple optical cameras, or an optical camera and a distance camera, so all cameras must be placed in positions where they can photograph moving objects. In addition, in the OWAS method, the load on a worker is evaluated (classified) based on four items: the posture of the worker's back, upper limbs (arms), and lower limbs (legs), and the weight (weight, load) of an object held (carried) by the worker. However, in the technology disclosed in Patent Document 1, the weight of the object on which the worker works is not measured, so accurate evaluation that takes the weight of the object into consideration is not possible, and there is a problem in that the work to be evaluated is limited to work of holding an object of a specified weight.

[0005] The present disclosure aims to provide a program etc. that can accurately evaluate the load applied to a worker. [Means for solving the problem]

[0006] A program according to one aspect of the present disclosure causes a computer to execute a process of acquiring an image of a worker, acquiring weight information regarding the weight of an object held by the worker, identifying the posture of the worker based on the acquired image, and identifying load information regarding the load applied to the body of the worker based on the identified posture of the worker and the acquired weight information. Effect of the Invention

[0007] In one aspect of the present disclosure, the load applied to a worker can be accurately evaluated. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram illustrating an example of the configuration of an information processing system. [Diagram 2] FIG. 2 is a block diagram showing a configuration example of an information processing device and a server. [Diagram 3] 11 is an explanatory diagram of a process for extracting a joint position of a worker. FIG. [Figure 4] FIG. 13 is an explanatory diagram showing an example of the configuration of a back determination table. [Diagram 5] FIG. 13 is an explanatory diagram showing an example of the configuration of an upper limb determination table. [Figure 6] FIG. 13 is an explanatory diagram showing an example of the configuration of a lower limb determination table. [Figure 7] FIG. 11 is an explanatory diagram showing an example of the configuration of a posture evaluation table; [Figure 8] 11 is an explanatory diagram showing an example of the record layout of an evaluation result DB and a captured image DB. FIG. [Figure 9] 13 is a flowchart illustrating an example of a load estimation process. [Figure 10]13 is a flowchart illustrating an example of a load estimation process. [Figure 11] 13 is a flowchart illustrating an example of a load estimation process. [Figure 12] 13 is a flowchart illustrating an example of a load estimation process. [Figure 13] FIG. 13 is an explanatory diagram showing an example of a screen. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, a program, an information processing method, and an information processing device according to the present disclosure will be described in detail with reference to the drawings showing embodiments thereof.

[0010] (Embodiment 1) In this embodiment, an information processing system that estimates the load on the body of a worker performing a task based on an image of the worker performing the task is described. FIG. 1 is an explanatory diagram showing an example of the configuration of the information processing system. The information processing system of this embodiment includes an information processing device 10, a camera 20, a server 30, a weighing scale 40, etc., and these devices are connected to each other via a network N. The network N may be the Internet or a public telephone line network, or a LAN (Local Area Network) constructed in a facility where the information processing system is installed. This embodiment is configured such that the image captured by the camera 20 and the weight measured by the weighing scale 40 are transmitted to and stored in the server 30, and the camera 20, the weighing scale 40, and the server 30 may be configured to directly transmit and receive information by wired communication or wireless communication via a cable. Alternatively, the server 30 may not be provided, and the information processing device 10 may have the function of the server 30. In this case, the camera 20, the weighing scale 40, and the information processing device 10 may be configured to directly transmit and receive information by wired communication or wireless communication via a cable.

[0011] The camera 20 is prepared at a site where a worker works, such as a manufacturing site or a construction site, and captures the worker while working. The camera 20 is an imaging device equipped with an imaging unit having a lens and an imaging element, a communication unit for connecting to the network N, and the like. The camera 20 performs a process of acquiring image data (hereinafter referred to as a captured image) by capturing an image using the imaging unit in accordance with an operation on a capture button, for example, and a process of transmitting the captured image from the communication unit to the server 30. The camera 20 of this embodiment performs a process of acquiring an image (video) of 15 frames or 30 frames per second, for example, but may also be configured to acquire one image (still image) in accordance with one operation on the capture button. The camera 20 may also be configured to sequentially transmit the captured images acquired by capturing images to the server 30, or may be configured to have a storage unit, accumulate the captured images in the storage unit, and transmit the accumulated captured images collectively to the server 30 in accordance with an operation on the operation unit or an instruction from the server 30. The storage unit may be a storage unit built into the camera 20, such as a flash memory, a hard disk, or a solid state drive (SSD), or may be a storage unit removable from the camera 20, such as a universal serial bus (USB) memory, an SD card, or a micro SD card. The camera 20 may be any camera that can capture the entire body of the worker and capture the state of the work performed by the worker. For example, the camera may be a camera in a state where the shooting position is fixed using a tripod or a fixing device, a camera in a state where the camera is held by the photographer, or a camera that has a function of tracking a set shooting target and captures the image while tracking the movement of the shooting target. The camera 20 may be configured to capture the image of the worker from the front side, the back side (behind), or the side of the worker, as long as it can capture the image of the worker without any obstruction. The camera 20 may be configured to capture the entire body of the worker using multiple cameras.

[0012] Weighing scale 40 is prepared at a work site and measures the weight of an object held (carried) by a worker during work. Weighing scale 40 includes a weight measuring unit, a communication unit for connecting to network N, and performs a process of measuring the weight of an object placed on a placement platform using the measuring unit to obtain weight data, and a process of transmitting the obtained weight data from the communication unit to server 30. Weighing scale 40 measures the weight when an object is placed on the placement platform, for example, and transmits weight data including the measured weight and measurement date and time to server 30. Weighing scale 40 may be configured to sequentially transmit the obtained weight data to server 30, or may be configured to include a storage unit, accumulate weight data in the storage unit, and transmit the accumulated weight data collectively to server 30 in accordance with an operation on an operation unit or an instruction from server 30.

[0013] The information processing device 10 is a device capable of various information processing and transmitting and receiving information, and is a personal computer, a server computer, or the like. The information processing device 10 performs processing to estimate the load applied to the body of the worker being photographed, based on the captured image taken by the camera 20 and the weight data measured by the weighing scale 40. The server 30 is an image processing device capable of various information processing and transmitting and receiving information, and is a server computer, a personal computer, or the like. The server 30 performs processing to acquire the captured image taken by the camera 20 and the weight data measured by the weighing scale 40, to store the acquired captured image and weight data, and to transmit the stored captured image and weight data to the information processing device 10 in response to a request from the information processing device 10.

[0014] 2 is a block diagram showing an example of the configuration of the information processing device 10 and the server 30. The information processing device 10 includes a control unit 11, a storage unit 12, a communication unit 13, an input unit 14, a display unit 15, a reading unit 16, etc., and these units are connected to each other via a bus. The control unit 11 includes one or more processors such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), or a GPU (Graphics Processing Unit). The control unit 11 executes information processing and control processing to be performed by the information processing device 10 by appropriately executing a program 12P stored in the storage unit 12. Note that when the control unit 11 includes multiple processors, the control unit 11 may execute each process by a different processor.

[0015] The storage unit 12 includes a RAM (Random Access Memory), a flash memory, a hard disk, an SSD (Solid State Drive), etc. The storage unit 12 stores a program 12P (program product) executed by the control unit 11 and various data required for executing the program 12P. The storage unit 12 also temporarily stores data and the like generated when the control unit 11 executes the program 12P. The storage unit 12 further stores a back determination table 12a, an upper limb determination table 12b, a lower limb determination table 12c, a posture evaluation table 12d, and an evaluation result DB 12e, which will be described later.

[0016] The communication unit 13 is a communication module for performing processes related to wired or wireless communication, and transmits and receives information to and from other devices via the network N. The input unit 14 accepts operation input by a user, and sends a control signal corresponding to the operation content to the control unit 11. The display unit 15 is a liquid crystal display or an organic EL (electroluminescence) display, etc., and displays various information according to instructions from the control unit 11. A part of the input unit 14 and the display unit 15 may be a touch panel configured as an integrated unit.

[0017] The reading unit 16 reads information stored in a portable storage medium 10a including a CD (Compact Disc), a DVD (Digital Versatile Disc), a USB (Universal Serial Bus) memory, an SD (Secure Digital) card, etc. The program 12P and various data stored in the storage unit 12 may be read by the control unit 11 from the portable storage medium 10a via the reading unit 16 and stored in the storage unit 12. In addition, the program 12P and various data may be written in the storage unit 12 during the manufacturing stage of the information processing device 10, or the control unit 11 may download them from another device via the communication unit 13 and store them in the storage unit 12.

[0018] The information processing device 10 may be a multi-computer including a plurality of computers, or may be a virtual machine virtually constructed by software in one device. When the information processing device 10 is configured as a server computer, the information processing device 10 may be a local server installed in a facility where the information processing device 10 is installed, or may be a cloud server connected to the information processing device 10 via a network such as the Internet. In the following, the information processing device 10 is described as being one computer. The program 12P may be arranged on a single computer or at one site, or may be distributed across multiple sites and executed on multiple computers connected to each other via a network. Furthermore, the input unit 14 and the display unit 15 are not essential for the information processing device 10, and the information processing device 10 may be configured to accept operations through a connected computer, or may be configured to output information to be displayed to an external display device. The storage unit 12 of the information processing device 10 may be configured by multiple storage devices, or may include an external storage device connected to the information processing device 10.

[0019] The server 30 includes a control unit 31, a storage unit 32, a communication unit 33, etc., and these units are connected to each other via a bus. The control unit 31, the storage unit 32, and the communication unit 33 each have the same configuration as the control unit 11, the storage unit 12, and the communication unit 13 of the information processing device 10, respectively, and therefore detailed description of the configuration is omitted. In addition to a program (not shown), the storage unit 32 of the server 30 stores a captured image DB 32a that stores captured images received from the camera 20 together with weight data received from the weighing scale 40. In addition to the above-mentioned configuration, the server 30 may also have an input unit that accepts operation input by a user, a display unit such as a liquid crystal display, a reading unit that reads information stored in a portable storage medium, etc.

[0020] The information processing device 10 of this embodiment performs a process of extracting the joint positions of the worker for each image frame of the captured image taken by the camera 20, and estimates (evaluates) the posture of the worker based on the extracted joint positions. FIG. 3 is an explanatory diagram of the extraction process of the joint positions of the worker. FIG. 3A shows an example of a captured image of one frame in which the worker is captured from the back side, and FIG. 3B shows the result of the process of extracting the joint positions for the captured image of FIG. 3A. In FIG. 3B, the extracted joints are shown with white circles, and the skeletal lines are shown with straight lines connecting the joints. The information processing device 10 of this embodiment estimates the joint positions of the worker (person) in the captured image using a technique for estimating the skeleton (joint positions) of a person in the captured image, such as openpifpaf, tf-pose-estimation, and the like, and obtains the three-dimensional coordinates of the joint positions. The information processing device 10 specifies the coordinate values ​​of each joint position in the orthogonal coordinate space in which the XYZ axes are orthogonal to each other, with the floor surface shown in the captured image as the XY coordinate plane and any position on the XY coordinate plane as the origin 0. In the example of Figure 3B, the information processing device 10 extracts 14 joint positions: the worker's neck P1, left shoulder P2, left elbow P3, left wrist P4, right shoulder P5, right elbow P6, right wrist P7, center of pelvis P8 (tailbone), left end of pelvis P9 (left buttock, base of left foot), left knee P10, left toe P11, right end of pelvis P12 (right buttock, base of right foot), right knee P13, and right toe P14.

[0021] FIG. 4 is an explanatory diagram showing an example of the configuration of the back judgment table 12a. The back judgment table 12a includes an index table 12a1 that stores indexes used to judge whether or not a plurality of states of the back (spine, back) of the worker are in each state, and a condition table 12a2 that stores conditions for judging whether or not each state is in each state based on each index. The index table 12a1 includes a back state column, a state number column, an index number column, an index column, a level column, etc., and stores, for each back state, a state number assigned to each state and information on an index set for each state. The state numbers are assigned as 1, 2, 3, and 4 in order of the smallest load applied to the worker. The information on the index includes an index number assigned to each index, the content of each index, and the level set for each index. The indices used to determine the state of the back include the intersection angle (index number 1) between the line connecting the coordinate point P1 of the neck and the coordinate point P8 of the pelvis center and the line extending from the coordinate point P8 of the pelvis center toward the zenith direction (Z-axis direction), the intersection angle (index number 2) between the line connecting the coordinate point P1 of the neck and the coordinate point P8 of the pelvis center and the line connecting the coordinate point P8 of the pelvis center and the coordinate point P9 of the left end of the pelvis, the intersection angle (index number 3) between the line connecting the coordinate point P9 of the left end of the pelvis and the coordinate point P12 of the right end of the pelvis and the smaller intersection angle (index number 4) between the line connecting the coordinate point P9 of the left end of the pelvis and the coordinate point P12 of the right end of the pelvis and the line connecting the coordinate point P5 of the right shoulder and the coordinate point P2 of the left shoulder. The condition table 12a2 stores conditions for determining each state in association with each state of the back. For example, for the states of "straight" and "bending forward or backward", a condition (logical formula) that all of the indexes of index numbers 1 to 4 are satisfied is stored. For the state of "twisting or bending sideways", a condition that the index of index number 1 is satisfied, and also the index of index numbers 2 or 3 is satisfied, and also the index of index number 4 is satisfied is stored, and for the state of "twisting and bending sideways", a condition that the index of index number 1 is satisfied is stored. The contents stored in the back determination table 12a are not limited to the example shown in FIG. 4.

[0022] FIG. 5 is an explanatory diagram showing an example of the configuration of the upper limb determination table 12b. The upper limb determination table 12b includes an index table 12b1 that stores indexes used for determining whether or not a plurality of states of the upper limbs (arms) of a worker are in each state, and a condition table 12b2 that stores conditions for determining whether or not a state is in each state based on each index. The index table 12b1 includes an upper limb state column, a state number column, an index number column, an index column, a level column, etc., and stores, for each state of the upper limbs, a state number assigned to each state and information on an index set for each state. The state numbers are assigned 1, 2, and 3 in order of increasing load on the worker. The information on the index includes an index number assigned to each index, the content of each index, and a level set for each index. The indexes used to determine the state of the upper limbs include the height component (Z coordinate value) of the coordinate point P6 of the right elbow or the coordinate point P7 of the right wrist (index number 1), and the height component (Z coordinate value) of the coordinate point P3 of the left elbow or the coordinate point P4 of the left wrist (index number 2). The condition table 12b2 stores conditions for determining each state in association with each state of the upper limbs. For example, for the states of "both arms below shoulder height" and "both arms above shoulder height", a condition (logical formula) that both indexes of index numbers 1 to 2 are satisfied is stored. For the state of "one arm above shoulder height", a condition that index number 1 or 2 is satisfied is stored. The stored contents of the upper limb determination table 12b are not limited to the example shown in FIG. 5.

[0023] FIG. 6 is an explanatory diagram showing an example of the configuration of the lower limbs determination table 12c. The lower limbs determination table 12c includes an index table 12c1 that stores indexes used for determining whether or not a worker is in each state for a plurality of states of the worker's lower limbs (legs), and a condition table 12c2 that stores conditions for determining whether or not a worker is in each state based on each index. The index table 12c1 includes a lower limbs state column, a state number column, an index number column, an index column, a level column, etc., and stores, for each state of the lower limbs, a state number assigned to each state and information on an index set for each state. The state numbers are assigned from 1 to 6 in ascending order of the load applied to the worker. The information on the index includes an index number assigned to each index, the content of each index, and a level set for each index. The indices used to determine the condition of the lower limbs include the height component (Z coordinate value) of the coordinate point P13 of the right knee, the height component of the coordinate point P10 of the left knee, the distance between the coordinate point P14 of the right toe and the coordinate point of the pelvis on the XY coordinate plane, the distance between the coordinate point P11 of the left toe and the coordinate point of the pelvis on the XY coordinate plane, the distance between the coordinate point of the midpoint between both toes and the coordinate point of the pelvis on the XY coordinate plane, the height component of the coordinate point P14 of the right toe, the intersection angle of the line connecting the coordinate point P14 of the right toe and the coordinate point P13 of the right knee with the line connecting the coordinate point P13 of the right knee and the coordinate point P12 of the right base of the right foot (right end of the pelvis), the height component of the coordinate point P11 of the left toe, and the intersection angle of the line connecting the coordinate point P11 of the left toe and the coordinate point P10 of the left knee with the coordinate point P10 of the left base of the left foot (left end of the pelvis). The distance between the coordinate point P14 of the right foot tip and the coordinate point of the pelvis on the XY coordinate plane is, for example, the distance between the coordinate point P14 of the right foot tip and the coordinate point P9 of the left end of the pelvis and the coordinate point P12 of the right end of the pelvis on the XY coordinate plane; the distance between the coordinate point P11 of the left foot tip and the coordinate point of the pelvis on the XY coordinate plane is the distance between the coordinate point P11 of the left foot tip and the coordinate point P9 of the left end of the pelvis and the coordinate point P12 of the right end of the pelvis on the XY coordinate plane; and the distance between the coordinate point of the midpoint between both feet on the XY coordinate plane and the coordinate point of the pelvis on the XY coordinate plane is the distance between the coordinate point of the midpoint between the coordinate point P11 of the left foot tip and the coordinate point P14 of the right foot tip on the XY coordinate plane and the coordinate point P9 of the left end of the pelvis and the coordinate point P12 of the right end of the pelvis on the XY coordinate plane.In addition, for the level of the index using the height component of each coordinate point and the distance between two coordinate points, a threshold value calculated by multiplying the coordinate distance of a part whose length can be stably detected in a series of operations performed by the worker, such as the distance between the center of the worker's pelvis and the neck, or the distance between the right end of the pelvis and the right knee, by a predetermined coefficient is used. Note that the predetermined coefficient can be a value that can most accurately calculate the load (load score) applied to the body of the worker from the experimental results of the posture detection process of the worker for several videos. The condition table 12c2 stores conditions for determining each state in association with each state of the lower limbs. For example, for the states of "putting one or both knees on the floor" and "standing with one leg bent or crouching on which the center of gravity is placed", a condition (logical formula) that the index of index number 1 or 2 is satisfied is stored, and for the states of "standing with both legs straight" and "standing with both knees bent or crouching", a condition that both indexes of index numbers 1 to 2 are satisfied is stored. For the state of "sitting", a condition is stored that none of the indicators of indicator numbers 1 to 3 is satisfied, and for the state of "standing with one leg on which the center of gravity is placed straight" a condition is stored that the indicators of indicator numbers 1 and 2 are satisfied, or the indicators of indicator numbers 3 and 4 are satisfied. The contents stored in the lower limb determination table 12c are not limited to the example shown in FIG.

[0024] FIG. 7 is an explanatory diagram showing an example of the configuration of the posture evaluation table 12d. The posture evaluation table 12d stores load scores (load information) indicating the load applied to the body of the worker, which are evaluated (classified) by the OWAS method based on the posture of the worker and the weight of the object held by the worker. Specifically, the posture evaluation table 12d stores load scores in association with each combination of four types of back states, three types of upper limb states, six types of lower limb states, and three types of weight of the object as the posture of the worker. In the example of FIG. 7, three weight levels are used as the weight of the object: 10 kg or less, more than 10 kg and less than 20 kg, and more than 20 kg, and weight numbers 1, 2, and 3 are assigned to each weight in ascending order of lightness. In the example of FIG. 7, load scores 1 to 4 are used, but the values ​​of the load scores are not limited to these.

[0025] Fig. 8 is an explanatory diagram showing an example of the record layout of the evaluation result DB 12e and the photographed image DB 32a. The evaluation result DB 12e is a database that stores evaluation results related to the posture of the worker to be photographed, which are identified based on the photographed image photographed by the camera 20. The evaluation result DB 12e shown in Fig. 8A includes a camera ID column, an image ID column, a start date and time column, an end date and time column, a worker ID column, a back information column, an upper limb information column, a lower limb information column, a weight information column, a score information column, and the like, and stores the evaluation results identified based on the photographed image photographed by the camera 20 in association with the camera ID assigned to the camera 20. The image ID column stores the image ID assigned to the photographed image, and the start date and time column and the end date and time column store the photographing start date and time and the photographing end date and time of the image frame to be evaluated in the photographed image. The worker ID column stores the worker ID assigned to the worker photographed in the image frame to be evaluated, and the back information column, upper limb information column, lower limb information column, and weight information column store the posture of the worker (state of the back, upper limbs, and lower limbs) identified based on each image frame to be evaluated and the weight of the object held by the worker. The score information column stores a load score indicating the load applied to the body of the worker estimated based on the posture of the worker identified for each image frame and the weight of the object. The back information column, upper limb information column, lower limb information column, weight information column, and score information column store each piece of information in association with information assigned to each image frame (e.g., frame ID, time from start of shooting, shooting date and time, etc.), so that time-series data of each piece of information based on the photographed images (video) from the start date and time to the end date and time is stored. The stored contents of the evaluation result DB 12e are not limited to the example shown in FIG. 8A.

[0026] The captured image DB 32a is a database that stores the captured images captured by the camera 20 and the weight data measured by the weighing scale 40. The captured image DB 32a shown in FIG. 8B is provided in association with the camera ID, and includes an image ID column, a start date and time column, an end date and time column, a worker ID column, an image data column, a weight data column, and the like, and stores information about the captured images and weight data of the measurement date and time that is the same as the capture date and time of the captured images in association with the image IDs assigned to the captured images. The start date and time column and the end date and time column store the capture start date and time and the capture end date and time of the captured images, the worker ID column stores the worker ID of the worker photographed in the captured images, and the image data column stores image data of the captured images. The weight data column stores weight data measured at the time of capturing the captured images. The image data and weight data of the captured images may be stored in a predetermined area of ​​the storage unit 32 or in a storage device connected to the server 30, and in this case, the image data column and the weight data column store information for reading out the data (for example, a folder name and a file name indicating the storage location of the data). The stored contents of the captured image DB 32a are not limited to the example shown in FIG. 8B.

[0027] The following describes the process in which the information processing device 10 of this embodiment estimates the load on the body of the worker based on the image of the worker captured by the camera 20 and the weight of an object measured by the weighing scale 40. FIGS. 9 to 12 are flowcharts showing an example of the load estimation process procedure, and FIG. 13 is an explanatory diagram showing an example screen. The following process is executed by the control unit 11 of the information processing device 10 in accordance with the program 12P stored in the storage unit 12. Note that the back state determination process shown in FIG. 10 is the process of step S14 in FIG. 9, the upper limb state determination process shown in FIG. 11 is the process of step S15 in FIG. 9, and the lower limb state determination process shown in FIG. 12 is the process of step S16 in FIG. 9.

[0028] The control unit 11 of the information processing device 10 acquires the captured image to be processed in the load estimation process and the weight data corresponding to the captured image (S11). For example, the control unit 11 requests the server 30 to acquire the captured image and weight data for which the load estimation process has not been executed among the captured image and weight data stored in the captured image DB 32a of the server 30. When the server 30 deletes the captured image and weight data stored in the captured image DB 32a from the captured image DB 32a after transmitting them to the information processing device 10, the control unit 11 may acquire all the captured images and weight data stored in the captured image DB 32a. The control unit 11 may also request the server 30 to acquire the captured image and weight data designated as the processing target via the input unit 14. The control unit 11 may acquire the captured image directly from the camera 20, or may acquire the weight data directly from the weighing scale 40. Furthermore, when a photographed image is acquired in advance from the server 30 or the camera 20 and stored in the storage unit 12, the control unit 11 may read out the photographed image to be processed from the storage unit 12, and when weight data is acquired in advance from the server 30 or the weighing scale 40 and stored in the storage unit 12, the control unit 11 may read out the weight data to be processed from the storage unit 12. The control unit 11 may extract a part of the acquired photographed image and weight data as the photographed image and weight data to be processed. Note that the weight data corresponding to the photographed image to be processed can be identified based on the photographing date and time of each image frame in the photographed image and the measurement date and time of each weight in the weight data. The control unit 11 stores the acquired photographed image and weight data to be processed in the storage unit 12.

[0029] The control unit 11 extracts one frame of a photographed image (image frame) from the photographed image to be processed (S12), and performs skeleton estimation of the worker appearing in the image frame based on the extracted image frame, and extracts the joint positions of the worker (S13). Specifically, the control unit 11 estimates the joint positions of the worker in the image frame using openpifpaf, tf-pose-estimation, or the like, and obtains three-dimensional coordinates of the joint positions P1 to P14 as shown in FIG. 3B. The control unit 11 may also obtain two-dimensional coordinates of the joint positions using openpose, or the like, and convert the obtained two-dimensional coordinates into coordinates in a three-dimensional coordinate space using 3d-pose-estimation, or the like, to obtain three-dimensional coordinates of the joint positions. Note that the control unit 11 may obtain the three-dimensional coordinates of the joint positions by any process, as long as it can obtain coordinate values ​​in an orthogonal coordinate space in which the floor surface appearing in the image frame is the XY coordinate plane, and an arbitrary position on the XY coordinate plane is the origin 0 and the XYZ axes are orthogonal to each other.

[0030] After acquiring the three-dimensional coordinates of the joint positions P1 to P14, the control unit 11 performs a process of judging the state of the back of the worker based on the three-dimensional coordinates of the joint positions P1 to P14 (S14). In the process shown in FIG. 10, the control unit 11 judges whether or not the standard is satisfied for each index corresponding to each state registered in the index table 12a1 of the back judgment table 12a. The control unit 11 first judges whether or not the standard is satisfied for each index corresponding to "straight" (the suitability of each index) (S31). Specifically, the control unit 11 calculates an equation representing a straight line connecting the coordinate point P1 of the neck and the coordinate point P8 of the center of the pelvis as the index of index number 1, and a straight line extending from the coordinate point P8 of the center of the pelvis toward the zenith direction (Z-axis direction), calculates the intersection angle of the two straight lines, and judges whether or not the calculated angle is less than 20 degrees. If the calculated angle is less than 20 degrees, the control unit 11 judges that the standard of the index is satisfied, and if it is 20 degrees or more, judges that the standard of the index is not satisfied. The control unit 11 also calculates an equation representing a line connecting the coordinate point P1 of the neck and the coordinate point P8 of the pelvis center, and a line connecting the coordinate point P8 of the pelvis center and the coordinate point P9 of the left end of the pelvis, as an index of index number 2, calculates an intersection angle between the two lines, and judges whether the calculated angle is 70 degrees or more. If the calculated angle is 70 degrees or more, the control unit 11 judges that the level of the index is met, and if it is less than 70 degrees, it judges that the level of the index is not met. The control unit 11 also calculates an equation representing a line connecting the coordinate point P1 of the neck and the coordinate point P8 of the pelvis center, and a line connecting the coordinate point P8 of the pelvis center and the coordinate point P12 of the right end of the pelvis, as an index of index number 3, calculates an intersection angle between the two lines, and judges whether the calculated angle is 70 degrees or more. If the calculated angle is 70 degrees or more, the control unit 11 determines that the level of the index is met, and if it is less than 70 degrees, it determines that the level of the index is not met. Furthermore, as the index of index number 4, the control unit 11 calculates an equation expressing a line connecting coordinate point P9 of the left end of the pelvis and coordinate point P12 of the right end of the pelvis, and a line connecting coordinate point P5 of the right shoulder and coordinate point P2 of the left shoulder, calculates the smaller intersection angle of the two lines, and determines whether the calculated angle is less than 20 degrees.If the calculated angle is less than 20 degrees, the control unit 11 determines that the level of the index is met, and if the calculated angle is 20 degrees or more, it determines that the level of the index is not met. Depending on the result of the determination of whether or not the level is met for each index of index numbers 1 to 4, the control unit 11 determines whether or not the condition corresponding to "straight" registered in the condition table 12a2 is met (S32). Here, the control unit 11 determines whether or not the condition of meeting the level is met for all indexes of index numbers 1 to 4, and if it determines that the condition is met (S32: YES), it specifies the back state of the worker as state number 1 (straight) (S33), and returns to the processing of FIG. 9.

[0031] When the control unit 11 determines that the condition corresponding to "straight" is not satisfied (S32: NO), that is, when the standard is not satisfied for any of the indicators of indicator numbers 1 to 4, it judges whether or not the standard is satisfied for each indicator corresponding to "bend forward or backward" (the appropriateness of each indicator) (S34). Specifically, the control unit 11 judges, as the indicator of indicator number 1, whether the intersection angle between the line connecting the coordinate point P1 of the neck and the coordinate point P8 of the pelvis center and the line extending from the coordinate point P8 of the pelvis center toward the zenith is 20 degrees or more. Also, as the indicator of index number 2, the control unit 11 judges, as the indicator of index number 2, whether the intersection angle between the line connecting the coordinate point P1 of the neck and the coordinate point P8 of the pelvis center and the line connecting the coordinate point P8 of the pelvis center and the coordinate point P9 of the left end of the pelvis is 70 degrees or more. Furthermore, the control unit 11 determines whether the angle between the line connecting the coordinate point P1 of the neck and the coordinate point P8 of the pelvis center and the line connecting the coordinate point P8 of the pelvis center and the coordinate point P12 of the right end of the pelvis is 70 degrees or more, as the index of index number 3. Furthermore, the control unit 11 determines whether the smaller angle between the line connecting the coordinate point P9 of the left end of the pelvis and the coordinate point P12 of the right end of the pelvis and the line connecting the coordinate point P5 of the right shoulder and the coordinate point P2 of the left shoulder is less than 20 degrees, as the index of index number 4. Then, the control unit 11 determines whether the condition corresponding to "bend forward or backward" is satisfied according to the determination result of whether the level is satisfied for each of the index numbers 1 to 4 (S35). Here, the control unit 11 determines whether or not the condition that the levels are met for all of the indicators of indicator numbers 1 to 4 is met, and if it determines that the condition is met (S35: YES), it identifies the worker's back condition as condition number 2 (bending forward or backward) (S36), and returns to the processing of Figure 9.

[0032] When the control unit 11 determines that the condition corresponding to "bending forward or backward" is not satisfied (S35: NO), that is, when the standard is not satisfied for any of the indicators of the indicator numbers 1 to 4, it judges whether or not the standard is satisfied for each indicator corresponding to "twisting or bending sideways" (the appropriateness of each indicator) (S37). Specifically, the control unit 11 judges, as the indicator of the indicator number 1, whether the intersection angle between the line connecting the coordinate point P1 of the neck and the coordinate point P8 of the pelvis center and the line extending from the coordinate point P8 of the pelvis center toward the zenith is 20 degrees or more. Also, as the indicator of the indicator number 2, the control unit 11 judges, as the indicator of the indicator number 2, whether the intersection angle between the line connecting the coordinate point P1 of the neck and the coordinate point P8 of the pelvis center and the line connecting the coordinate point P8 of the pelvis center and the coordinate point P9 of the left end of the pelvis is less than 70 degrees. Furthermore, the control unit 11 determines whether the angle between the line connecting the coordinate point P1 of the neck and the coordinate point P8 of the pelvis center and the line connecting the coordinate point P8 of the pelvis center and the coordinate point P12 of the right end of the pelvis is less than 70 degrees as the index of index number 3. Furthermore, the control unit 11 determines whether the smaller angle between the line connecting the coordinate point P9 of the left end of the pelvis and the coordinate point P12 of the right end of the pelvis and the line connecting the coordinate point P5 of the right shoulder and the coordinate point P2 of the left shoulder is 20 degrees or more as the index of index number 4. Then, the control unit 11 determines whether the condition corresponding to "twisting or bending sideways" is satisfied according to the determination result of whether the level is satisfied for each of the index numbers 1 to 4 (S38). Here, the control unit 11 judges whether the conditions of satisfying the index of index number 1, satisfying the index of index number 2 or 3, and satisfying the index of index number 4 are satisfied. If it is judged that the conditions are satisfied (S38: YES), it specifies the back state of the worker as state number 3 (twisting or bending sideways) (S39) and returns to the processing of FIG. 9. If it is judged that the condition corresponding to "twisting or bending sideways" is not satisfied (S38: NO), it specifies the back state of the worker as state number 4 (twisting and bending sideways) (S40) and returns to the processing of FIG. 9. The control unit 11 stores, in the evaluation result DB 12e, the information of the image frame to be processed (e.g., frame ID) and the state number specified in step S33, S36, S39, or S40 in the back information sequence corresponding to the image ID of the photographed image to be processed in association with each other.

[0033] Next, the control unit 11 performs a process of determining the state of the upper limbs of the worker based on the three-dimensional coordinates of the joint positions P1 to P14 (S15). In the process shown in FIG. 11, the control unit 11 determines whether or not each index corresponding to each state registered in the index table 12b1 of the upper limb determination table 12b satisfies a standard. The control unit 11 first determines whether or not each index corresponding to "both arms below the shoulders" satisfies a standard (suitability of each index) (S51). Specifically, the control unit 11 extracts height components (Z coordinate values) of the coordinate point P6 of the right elbow, the coordinate point P7 of the right wrist, and the coordinate point P5 of the right shoulder as the index of index number 1, and determines whether or not the height component of the coordinate point P6 of the right elbow or the coordinate point P7 of the right wrist is less than the height component of the coordinate point P5 of the right shoulder. The control unit 11 determines that the level of the index is met when the height component of the coordinate point P6 of the right elbow or the coordinate point P7 of the right wrist is less than the height component of the coordinate point P5 of the right shoulder, and determines that the level of the index is not met when the height component of the coordinate point P5 of the right shoulder or more is equal to or greater than that. The control unit 11 also extracts the height components (Z coordinate values) of the coordinate point P3 of the left elbow, the coordinate point P4 of the left wrist, and the coordinate point P2 of the left shoulder as the index of index number 2, and determines whether the height component of the coordinate point P3 of the left elbow or the coordinate point P4 of the left wrist is less than the height component of the coordinate point P2 of the left shoulder. The control unit 11 determines that the level of the index is met when the height component of the coordinate point P3 of the left elbow or the coordinate point P4 of the left wrist is less than the height component of the coordinate point P2 of the left shoulder, and determines that the level of the index is not met when the height component of the coordinate point P2 of the left shoulder or more is equal to or greater than that. The control unit 11 judges whether or not the condition corresponding to "both arms below the shoulders" registered in the condition table 12b2 is satisfied according to the judgment result of whether or not the level is satisfied for each of the indexes of index numbers 1 to 2 (S52). Here, the control unit 11 judges whether or not the condition of satisfying the level is satisfied for both indexes of index numbers 1 to 2, and when it judges that the condition is satisfied (S52: YES), it specifies the upper limb state of the worker as state number 1 (both arms below the shoulders) (S53), and returns to the processing of FIG.

[0034] When the control unit 11 determines that the condition corresponding to "both arms are below the shoulders" is not satisfied (S52: NO), that is, when the standard is not satisfied for any of the indicators of the indicator numbers 1 to 2, the control unit 11 determines whether or not the standard is satisfied for each indicator corresponding to "one arm is above the shoulder height" (the suitability of each indicator) (S54). Specifically, the control unit 11 determines whether or not the height component of the coordinate point P6 of the right elbow or the coordinate point P7 of the right wrist is equal to or higher than the height component of the coordinate point P5 of the right shoulder, as the indicator of the indicator number 1. The control unit 11 also determines whether or not the height component of the coordinate point P3 of the left elbow or the coordinate point P4 of the left wrist is equal to or higher than the height component of the coordinate point P2 of the left shoulder, as the indicator of the indicator number 2. Then, the control unit 11 determines whether or not the condition corresponding to "one arm is above the shoulder height" is satisfied according to the determination result of whether or not the standard is satisfied for each of the indicators of the indicator numbers 1 to 2 (S55). Here, the control unit 11 judges whether or not a condition that a level is met for any of the indexes of index numbers 1 to 2 is met, and when it judges that the condition is met (S55: YES), it specifies the upper limb state of the worker as state number 2 (one arm is at or above shoulder height) (S56), and returns to the processing of FIG. 9. When it judges that the condition corresponding to "one arm is at or above shoulder height" is not met (S55: NO), it specifies the upper limb state of the worker as state number 3 (both arms are at or above shoulder height) (S57), and returns to the processing of FIG. 9. Note that the control unit 11 stores, in the evaluation result DB 12e, information of the image frame to be processed (e.g., frame ID) and the state number specified in step S53, S56, or S57 in the upper limb information sequence corresponding to the image ID of the captured image to be processed in association with each other.

[0035] Next, the control unit 11 performs a process of determining the state of the lower limbs of the worker based on the three-dimensional coordinates of the joint positions P1 to P14 (S16). In the process shown in FIG. 12, the control unit 11 determines whether or not each index corresponding to each state registered in the index table 12c1 of the lower limb determination table 12c satisfies a standard. The control unit 11 first determines whether or not each index corresponding to "put one or both knees on the floor" satisfies a standard (suitability of each index) (S61). Specifically, the control unit 11 extracts the height component (Z coordinate value) of the coordinate point P13 of the right knee as the index of index number 1 corresponding to "put one or both knees on the floor" and determines whether or not it is less than 0.1. If the height component of the coordinate point P13 of the right knee is less than 0.1, the control unit 11 determines that the standard of the index is satisfied, and if it is 0.1 or more, the control unit 11 determines that the standard of the index is not satisfied. The control unit 11 also extracts the height component (Z coordinate value) of the coordinate point P10 of the left knee as the index of index number 2, and judges whether it is less than 0.1. If it is less than 0.1, it judges that the level of the index is met, and if it is 0.1 or more, it judges that the level of the index is not met. The control unit 11 judges whether the condition corresponding to "one or both knees on the floor" registered in the condition table 12c2 is met according to the judgment result of whether the level is met for each of the indexes of index numbers 1 to 2 (S62). Here, the control unit 11 judges whether the condition of meeting the level is met for any of the indexes of index numbers 1 to 2, and if it judges that the condition is met (S62: YES), it specifies the lower limb state of the worker as state number 6 (one or both knees on the floor) (S63), and returns to the processing of FIG. 9.

[0036] When the control unit 11 determines that the condition corresponding to "put one or both knees on the floor" is not satisfied (S62: NO), it determines whether or not each index corresponding to "sitting" satisfies the standard (suitability of each index) (S64). Specifically, the control unit 11 extracts the coordinate values ​​(X and Y coordinate values) of the coordinate point P14 of the right foot on the XY coordinate plane and the coordinate points P9, P8, and P12 of the left end, center, and right end of the pelvis as the index of index number 1 corresponding to "sitting", calculates an equation representing a straight line connecting the coordinate point P9 of the left end of the pelvis and the coordinate point P12 of the right end of the pelvis, calculates the distance between the calculated straight line and the coordinate point P14 of the right foot, and determines whether or not the calculated distance is less than 0.2. Furthermore, the control unit 11 calculates the distance between the line connecting the coordinate point P9 of the left end of the pelvis and the coordinate point P12 of the right end of the pelvis on the XY coordinate plane and the coordinate point P11 of the left foot as the index of index number 2, and judges whether the calculated distance is less than 0.2. Furthermore, the control unit 11 calculates the coordinate value of the midpoint between the coordinate points P14, P11 of the right foot and the coordinate point P11 of the left foot on the XY coordinate plane as the index of index number 3, calculates the distance between the coordinate value of the midpoint and the line connecting the coordinate point P9 of the left end of the pelvis and the coordinate point P12 of the right end of the pelvis, and judges whether the calculated distance is less than 0.2. Then, the control unit 11 judges whether the condition corresponding to "sitting" is satisfied according to the judgment result of whether the level is satisfied for each of the indexes of index numbers 1 to 3 (S65). Here, the control unit 11 determines whether or not the condition that the level is not met for all of the indicators of indicator numbers 1 to 3 is met, and if it determines that the condition is met (S65: YES), it identifies the lower limb condition of the worker as condition number 1 (sitting) (S66), and returns to the processing of Figure 9.

[0037] When the control unit 11 determines that the condition corresponding to "sitting" is not satisfied (S65: NO), it determines whether or not the level is satisfied for each index corresponding to "standing with one leg on which the center of gravity is placed straight" (the suitability of each index) (S67). Specifically, the control unit 11 extracts the height component (Z coordinate value) of the coordinate point P14 of the right foot as the index of index number 1 corresponding to "standing with one leg on which the center of gravity is placed straight" and determines whether or not it is less than 0.1. When the height component of the coordinate point P14 of the right foot is less than 0.1, the control unit 11 determines that the level of the index is satisfied, and when it is 0.1 or more, it determines that the level of the index is not satisfied. The control unit 11 also calculates an equation representing a line connecting the coordinate point P14 of the right toe and the coordinate point P13 of the right knee, and a line connecting the coordinate point P13 of the right knee and the coordinate point P12 of the right groin (right end of the pelvis) as an index of index number 2, calculates the intersection angle of the two lines, and judges whether the calculated angle is 150 degrees or more. The control unit 11 also extracts a height component of the coordinate point P11 of the left toe as an index of index number 3, and judges whether it is less than 0.1. The control unit 11 also calculates an intersection angle of a line connecting the coordinate point P11 of the left toe and the coordinate point P10 of the left knee, and a line connecting the coordinate point P10 of the left knee and the coordinate point P9 of the left groin (left end of the pelvis) as an index of index number 4, and judges whether the calculated angle is 150 degrees or more. Then, the control unit 11 judges whether or not the condition corresponding to "standing with one leg on which the center of gravity is placed straight" is satisfied according to the judgment result of whether or not the level is satisfied for each of the indexes of index numbers 1 to 4 (S68). Here, the control unit 11 judges whether or not the condition of satisfying both of the indexes of index numbers 1 to 2 or both of the indexes of index numbers 3 to 4 is satisfied, and if it judges that the condition is satisfied (S68: YES), it specifies the lower limb condition of the worker as condition number 3 (standing with one leg on which the center of gravity is placed straight) (S69), and returns to the processing of FIG. 9.

[0038] When the control unit 11 determines that the condition corresponding to "standing with one leg on which the center of gravity is placed straight" is not satisfied (S68: NO), it judges whether or not the standard is satisfied for each index corresponding to "standing with both legs straight" (S70). Specifically, the control unit 11 judges, as the index of index number 1 corresponding to "standing with both legs straight", whether or not the intersection angle between the line connecting the coordinate point P14 of the right toe and the coordinate point P13 of the right knee and the line connecting the coordinate point P13 of the right knee and the coordinate point P12 of the right groin (right end of the pelvis) is 150 degrees or more. In addition, the control unit 11 judges, as the index of index number 2 corresponding to "standing with both legs straight", whether or not the intersection angle between the line connecting the coordinate point P11 of the left toe and the coordinate point P10 of the left knee and the coordinate point P9 of the left groin (left end of the pelvis) is 150 degrees or more. Then, the control unit 11 judges whether or not the condition corresponding to "standing with both legs straight" is satisfied according to the judgment result of whether or not the level is satisfied for each of the indexes of index numbers 1 to 2 (S71). Here, the control unit 11 judges whether or not the condition of satisfying both indexes of index numbers 1 to 2 is satisfied, and if it judges that the condition is satisfied (S71: YES), it specifies the lower limb condition of the worker as condition number 2 (standing with both legs straight) (S72), and returns to the processing of FIG. 9.

[0039] When the control unit 11 determines that the condition corresponding to "standing with both feet straight" is not satisfied (S71: NO), it judges whether or not the standard is satisfied for each index corresponding to "standing with one leg bent on which the center of gravity is placed, or in a half-squat position" (S73). Specifically, the control unit 11 judges, as the index of index number 1 corresponding to "standing with one leg bent on which the center of gravity is placed, or in a half-squat position", whether or not the intersection angle between the line connecting the coordinate point P14 of the right toe and the coordinate point P13 of the right knee and the line connecting the coordinate point P13 of the right knee and the coordinate point P12 of the right hip is 150 degrees or more. In addition, the control unit 11 judges, as the index of index number 2 corresponding to "standing with one leg bent on which the center of gravity is placed, or in a half-squat position", whether or not the intersection angle between the line connecting the coordinate point P11 of the left toe and the coordinate point P10 of the left knee and the coordinate point P9 of the left hip is 150 degrees or more. Then, the control unit 11 judges whether the condition corresponding to "standing with one leg on which the center of gravity is placed, or squatting" is satisfied according to the judgment result of whether the level is satisfied for each of the indexes of the index numbers 1 to 2 (S74). Here, the control unit 11 judges whether the condition of satisfying any of the indexes of the index numbers 1 to 2 is satisfied, and if it judges that the condition is satisfied (S74: YES), it specifies the lower limb state of the worker as state number 5 (standing with one leg on which the center of gravity is placed, or squatting) (S75), and returns to the processing of FIG. 9. If the control unit 11 judges that the condition corresponding to "standing with one leg on which the center of gravity is placed, or squatting" is not satisfied (S74: NO), it specifies the lower limb state of the worker as state number 4 (standing with both knees bent, or squatting) (S76), and returns to the processing of FIG. 9. In addition, in the evaluation result DB12e, the control unit 11 stores the lower limb information sequence corresponding to the image ID of the captured image to be processed in association with the information of the image frame to be processed (e.g., frame ID) and the condition number identified in step S63, S66, S69, S72, S75, or S76.

[0040] Returning to the process of FIG. 9, the control unit 11 determines the weight of the object held by the worker in the image frame extracted in step S12 (S17). Here, based on the photographing date and time of the image frame extracted in step S12, the control unit 11 extracts the weight having the photographing date and time as the measurement date and time from the weight data of the processing target acquired in step S11, thereby specifying the weight of the object here. Based on the specified weight of the object, the control unit 11 determines the weight information of the object from 1 to 3 (1: 10 kg or less, 2: more than 10 kg and less than 20 kg, 3: more than 20 kg). The control unit 11 stores information of the image frame to be processed (e.g., frame ID) and the determined weight information in a weight information sequence corresponding to the image ID of the photographed image to be processed in the evaluation result DB 12e in association with each other.

[0041] The control unit 11 evaluates the posture of the worker based on the states of the back, upper limbs, and lower limbs of the worker determined in steps S14 to S16 and the weight of the object determined in step S17 (S18). Here, the control unit 11 identifies a load score (load information) corresponding to a combination of the state of the back determined in step S14, the state of the upper limbs determined in step S15, the state of the lower limbs determined in step S16, and the weight of the object determined in step S17 based on the posture evaluation table 12d. For example, if the state of the back is determined to be "bent forward or backward (state number 2)", the state of the upper limbs is determined to be "both arms below the shoulders (state number 1)", the state of the lower limbs is determined to be "standing with both legs straight (state number 2)", and the weight is determined to be "over 20 kg (weight number 3)", the control unit 11 identifies the load score "3". Through such processing, a load score that quantifies the load applied to the body of the worker according to the posture of the worker and the weight of the object is obtained. The control unit 11 stores the identified load score in the score information string corresponding to the image ID of the captured image to be processed in the evaluation result DB 12e in association with information (eg, frame ID) of the image frame to be processed (S19).

[0042] The control unit 11 determines whether or not there is an unprocessed image frame that has not been subjected to the process of evaluating the posture of the worker among the image frames included in the photographed image to be processed acquired in step S11 (S20). If it is determined that there is an unprocessed image frame (S20: YES), the control unit 11 returns to the process of step S12 and executes the processes of steps S12 to S19 for the unprocessed image frame. The control unit 11 repeats steps S12 to S20 until it determines that there is no unprocessed image frame. As a result, for each image frame included in the photographed image to be processed, a load score based on the posture of the worker (state of the back, upper limbs, and lower limbs) photographed in each image frame and the weight of the object held by the worker is specified and stored in the evaluation result DB 12e.

[0043] When it is determined that there is no unprocessed image frame (S20: NO), the control unit 11 generates an evaluation result screen based on the information stored in the evaluation result DB 12e (S21). FIG. 13 shows an example of the evaluation result screen, and shows the evaluation result when the captured images for 3 seconds in which 15 frames are captured per second are processed. The screen shown in FIG. 13 displays the worker ID of the worker to be evaluated, the image ID and capture date and time (capture start date and time and capture end date and time) of the captured image to be evaluated, and the number of image frames to be evaluated (number of evaluations). The evaluation result screen also displays, as the evaluation results, a frequency distribution table A1 of the load scores 1 to 4, a bar graph A2, a graph A3 showing the time series change in the load score, and a graph A4 showing the time series change in the evaluation for the state of each part and the weight of the object (part-by-part evaluation). The frequency distribution table A1 shows the number of image frames determined to each load score (number of cases) and the ratio (relative frequency) of the number of each load score to the number of image frames to be evaluated. The bar graph A2 is a stacked graph showing the ratio of the number of each load score. The graph A3 shows time on the horizontal axis, load scores on the vertical axis, and is a graph plotting load scores determined based on image frames with the date and time of each image taken at each time. The graph A4 shows time on the horizontal axis, evaluation of the worker's posture and the weight of the object (condition numbers and weight numbers of each part of the back, upper limbs, and lower limbs) on the vertical axis, and is a graph plotting the condition numbers and weight numbers of each part determined based on image frames with the date and time of each image taken at each time in a manner assigned to each part and weight. In the graph A4 in FIG. 13, the state of the back is shown by a dashed line, the state of the upper limbs by a dashed line, the state of the lower limbs by a solid line, and the weight by a dashed double-dot line.

[0044] When generating the screen of Fig. 13, the control unit 11 counts the number of image frames of the evaluation target and the number of image frames judged to have each of the stress scores 1 to 4 based on the stress scores stored in the score information column of the evaluation result DB 12e, and generates a frequency distribution table A1 and a bar graph A2 from the counting results. The control unit 11 also generates a graph A3 by plotting the stress scores stored in the score information column of the evaluation result DB 12e in association with the information of the image frames used to identify each stress score (e.g., frame ID, time from the start of shooting, shooting date and time). The control unit 11 also generates a graph A4 by plotting the condition numbers and weight numbers of the back, upper limbs, and lower limbs stored in the back information column, upper limbs information column, lower limbs information column, and weight information column of the evaluation result DB 12e in association with the information of the image frames (e.g., frame ID, time from the start of shooting, shooting date and time). Furthermore, the control unit 11 specifies image frames with a load score of 3 or more, and adds a mark M indicating "high load" and a message (text data) to the specified image frames in the graphs A3 and A4.

[0045] The control unit 11 generates a screen as shown in FIG. 13 by the above-mentioned process, and displays the generated evaluation result screen on the display unit 15 (S22). By displaying the screen shown in FIG. 13, it is possible to provide the time transition of the load applied to the worker's body based on the photographed image, and the ratio of each load score in the photographed image to be processed. Therefore, it is possible to grasp the load level applied to the worker's body for each task performed by the worker. In addition to displaying the generated evaluation result screen on the display unit 15, the control unit 11 may transmit it to another device via the network N, or may transmit it to a printer connected via the network or directly and print it. In the process shown in FIG. 9, the processes of steps S14 to S17 may be executed in any order, or may be executed in parallel.

[0046] Conventionally, the posture of a worker is measured by visual confirmation by a measurer, and there is a possibility that the measurement result may vary depending on the experience or fatigue level of the measurer. On the other hand, in this embodiment, the posture of a worker is measured based on a photographed image of the worker, so that an objective measurement result can be obtained. In addition, in this embodiment, the load level (load score) is evaluated according to the OWAS method that takes into account the objectively measured posture of the worker and the weight of the object held by the worker during work, so that the load level can be indicated by an objective index (load score). Therefore, the load level applied to the worker can be evaluated by the load score based on the same standard for various tasks at the work site, so that it can be applied to various tasks and is convenient. In addition, in this embodiment, the worker is photographed by one camera 20, so that a system that can visualize the load level of the worker based on the photographed image can be realized with a simple system configuration. Furthermore, in this embodiment, it is possible to consider improving the working posture of the worker to reduce the load level (load score) for tasks with a high load level, for example, based on the objective index (load score), and the safety of the work can be improved by reducing the load on the worker.

[0047] In this embodiment, the load applied to the worker captured in the image frame is evaluated (load score is determined) for each image frame included in the captured image to be processed, but the present invention is not limited to this configuration. For example, the evaluation process may be performed at a predetermined time interval, such as every 1 / 3 second, every 1 / 2 second, or every second. For example, when the camera 20 captures 30 frames per second, the evaluation process may be performed every 5 frames (every 1 / 6 second), every 10 frames (every 1 / 3 second), or every 15 frames (every 1 / 2 second).

[0048] In this embodiment, the control unit 11 may be configured to display a message or advice according to the evaluation result on the evaluation result screen. For example, a message or advice regarding the worker's working posture for each balance of the ratios of the load scores 1 to 4 may be stored in advance in the storage unit 12, and the control unit 11 may calculate the ratio of each load score, and then read out the message or advice according to the balance of the ratios of each load score from the storage unit 12 and display it on the evaluation result screen. This allows the worker to improve his or her posture in accordance with the message or advice, thereby realizing a reduction in the load during work.

[0049] (Embodiment 2) A modified example of the method for measuring the weight of an object held by a worker in the above-mentioned embodiment 1 will be described. The information processing system of this embodiment does not include the weighing scale 40 in the information processing system of embodiment 1 shown in Fig. 1, and the information processing device 10, camera 20, and server 30 have the same configurations as the devices 10, 20, and 30 in embodiment 1. Also, in this embodiment, the captured image DB 32a stored in the storage unit 32 of the server 30 does not have a weight data string in the configuration shown in Fig. 8B.

[0050] In the information processing system of this embodiment, the information processing device 10 can execute the same processes as those shown in Figs. 9 to 12. In step S11, the control unit 11 of the information processing device 10 of this embodiment acquires only the captured image to be processed from the server 30. In step S17, the control unit 11 detects an object in the image frame by performing an object detection process on the image frame extracted in step S12, for example, and identifies the weight of the detected object. The object detection process can be performed using an object detection model that has been machine-learned to output a result of determining whether an object included in an image frame is one of the objects previously learned when an image frame is input. In addition, the object detection process can be performed by template matching using a template image that indicates the image feature amount of the captured image of the object handled by the worker, for example. Furthermore, the object detection process may be realized by reading information for identifying the object or a code indicating the information (barcode, QR code (registered trademark), etc.) attached to the object. The weight of the detected object can be identified, for example, by using a table in which the weights of objects handled by the worker are registered in advance. In this case, the control unit 11 specifies the weight of the object detected by the object detection process from the table. Note that the control unit 11 may estimate the weight of the object detected by the object detection process based on the type and size of the object.

[0051] The control unit 11 may also specify the work content being performed by the worker in the image frame based on the image frame, thereby specifying the object held by the worker in the specified work content, and specifying the weight of the specified object. The process of specifying the work content based on the image frame can be performed using a learning model that has been machine-learned to output a result of determining the work content by the worker photographed in the image frame when the image frame is input. The process of specifying the work content can also be performed by template matching using a template image that indicates the image feature amount of the photographed image of the worker performing each work. Note that the configuration may be such that the worker or the user of the information processing device 10 inputs information on the work content (e.g., a work process number) via the input unit 14 or another terminal that can communicate with the information processing device 10, and the control unit 11 specifies the work content from the input information. The object handled in the specified work content can be specified, for example, using a table in which information on the object held by the worker in each work is registered in association with the work content. Note that when a table in which the weight of the object held by the worker in each work is registered in association with the work content is used, the control unit 11 can specify the weight of the object corresponding to the work content based on the specified work content.

[0052] Furthermore, the control unit 11 may accept input of the weight of the object here via the input unit 14 or another terminal capable of communicating with the information processing device 10. For example, the control unit 11 may acquire the weight of the object by having a worker or a user of the information processing device 10 input the weight of the object measured using a weighing scale prepared at the work site via the input unit 14 or another terminal capable of communicating with the information processing device 10. The control unit 11 determines the weight information of the object from 1 to 3 based on the identified weight of the object. Note that the control unit 11 may be configured to determine whether the weight of the object is 10 kg or less, more than 10 kg and less than 20 kg, or more than 20 kg, in addition to identifying the weight of the object.

[0053] The information processing device 10 of this embodiment performs the same processes as those of the first embodiment except for the process of determining the weight of an object held by a worker based on a captured image of the worker, and therefore provides the same effects as those of the first embodiment. Furthermore, since the weighing scale 40 is not used in this embodiment, the worker does not need to measure the weight of the object being held, improving operability. The modified examples described in the first embodiment above can also be applied to this embodiment as appropriate.

[0054] Regarding the above embodiments including the first and second embodiments, the following supplementary notes are disclosed.

[0055] (Appendix 1) Acquire a photographed image of the worker, Obtaining weight information regarding a weight of an object held by the worker; Identifying the posture of the worker based on the acquired photographed image; Identifying load information related to a load applied to the body of the worker based on the identified posture of the worker and the acquired weight information. A program that causes a computer to carry out processing.

[0056] (Appendix 2) Using a table in which a score related to a load applied to the body of the worker is associated with each combination of a plurality of types of postures and weight information of an object, the score corresponding to the identified posture of the worker and the acquired weight information is identified. 2. The program according to claim 1, which causes the computer to execute the process.

[0057] (Appendix 3) The posture includes a posture of the back of the worker, a posture of the upper limbs of the worker, and a posture of the lower limbs of the worker. 3. The program according to claim 1 or 2.

[0058] (Appendix 4) Using a table in which a plurality of types of postures are associated with each of the back, upper limbs, and lower limbs of the worker, postures of the back, upper limbs, and lower limbs of the worker in the captured image are identified. 4. The program according to claim 3, which causes the computer to execute the process.

[0059] (Appendix 5) Output information corresponding to the identified load information. 5. The program according to claim 1, which causes the computer to execute the process.

[0060] (Appendix 6) Identifying load information relating to a load applied to a body of the worker based on a posture of the worker identified based on each of a plurality of photographed images of the worker in time series and weight information of an object held by the worker captured in each of the plurality of photographed images; Outputting the time series changes of the load information 6. The program according to any one of claims 1 to 5, for causing the computer to execute the process.

[0061] (Appendix 7) The posture includes a posture of the back of the worker, a posture of the upper limbs of the worker, and a posture of the lower limbs of the worker, Outputting time-series changes in posture of the back, upper limbs, and lower limbs of the worker identified based on each of the plurality of photographed images photographed in the time series. 7. The program according to claim 6, which causes the computer to execute the process.

[0062] (Appendix 8) Acquire a photographed image of the worker, Obtaining weight information regarding a weight of an object held by the worker; Identifying the posture of the worker based on the acquired photographed image; Identifying load information related to a load applied to the body of the worker based on the identified posture of the worker and the acquired weight information. An information processing method in which processing is performed by a computer.

[0063] (Appendix 9) In an information processing device having a control unit, The control unit is Acquire a photographed image of the worker, Obtaining weight information regarding a weight of an object held by the worker; Identifying the posture of the worker based on the acquired photographed image; Identifying load information related to a load applied to the body of the worker based on the identified posture of the worker and the acquired weight information. Information processing device.

[0064] The technical features (constituent elements) described in each embodiment can be combined with each other, and by combining them, new technical features can be formed. In addition, the independent claims and dependent claims described in the claims can be combined with each other in all combinations regardless of the citation format. Furthermore, the claims use a format in which a claim cites two or more other claims (multiple claim format), but this is not limited to this. They may also be written using a format in which a multiple claim cites at least one other multiple claim (multi-multi claim).

[0065] The embodiments disclosed herein are illustrative in all respects and should not be considered as limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the scope and meaning equivalent to the claims. [Explanation of symbols]

[0066] 10. Information processing device 11 Control section 12 Storage section 13. Communications Department 20 Camera 30 Servers 31 Control Unit 32 Memory unit 33 Communication unit 40 Weighing scale

Claims

1. a storage unit stores an image of the worker in association with weight information on the weight of an object held by the worker, the weight information being measured by a weighing scale on the same date and time as the image was taken; The photographed image and weight information measured at the same date and time as the photographed image are acquired from the storage unit; Identifying the posture of the worker based on the acquired photographed image; Identifying load information related to a load applied to the body of the worker based on the identified posture of the worker and the acquired weight information. A program that causes a computer to carry out processing.

2. Using a table in which a score related to a load applied to the body of the worker is associated with each combination of a plurality of types of postures and weight information of an object, the score corresponding to the identified posture of the worker and the acquired weight information is identified. The program according to claim 1 , which causes the computer to execute a process.

3. The posture includes a posture of the back of the worker, a posture of the upper limbs of the worker, and a posture of the lower limbs of the worker.

3. The program according to claim 1 or 2.

4. Using a table in which a plurality of types of postures are associated with each of the back, upper limbs, and lower limbs of the worker, postures of the back, upper limbs, and lower limbs of the worker in the captured image are identified. The program according to claim 3, which causes the computer to execute a process.

5. Output information corresponding to the identified load information.

3. The program according to claim 1, which causes the computer to execute a process.

6. Identifying load information relating to a load applied to a body of the worker based on a posture of the worker identified based on each of a plurality of photographed images of the worker in time series and weight information of an object held by the worker captured in each of the plurality of photographed images; Outputting the time series changes of the load information 3. The program according to claim 1, which causes the computer to execute a process.

7. The posture includes a posture of the back of the worker, a posture of the upper limbs of the worker, and a posture of the lower limbs of the worker, Outputting time-series changes in posture of the back, upper limbs, and lower limbs of the worker identified based on each of the plurality of photographed images photographed in the time series. The program according to claim 6, which causes the computer to execute a process.

8. a storage unit stores an image of the worker in association with weight information on the weight of an object held by the worker, the weight information being measured by a weighing scale on the same date and time as the image was taken; The photographed image and weight information measured at the same date and time as the photographed image are acquired from the storage unit; Identifying the posture of the worker based on the acquired photographed image; Identifying load information related to a load applied to the body of the worker based on the identified posture of the worker and the acquired weight information. An information processing method in which processing is performed by a computer.

9. In an information processing device having a control unit, The control unit is a storage unit stores an image of the worker in association with weight information on the weight of an object held by the worker, the weight information being measured by a weighing scale on the same date and time as the image was taken; The photographed image and weight information measured at the same date and time as the photographed image are acquired from the storage unit; Identifying the posture of the worker based on the acquired photographed image; Identifying load information related to a load applied to the body of the worker based on the identified posture of the worker and the acquired weight information. Information processing device.

Images