Information processing system and information processing method

Abstract

To provide an information processing system and an information processing method capable of presenting a result of analyzing posture of a worker and reliability of the analysis result to a user.SOLUTION: An information processing system 1 is equipped with an information terminal device 10, and a server device 20 which communicates with the information terminal device 10 over a communication network 30. The server device 20 includes: a communication unit 23 which receives a work video from the information terminal device 10, and transmits information indicating an analysis result and information indicating reliability to the information terminal device 10; an analysis unit which identifies a direction of a worker 40 presented in a work image and analyzes the posture of the worker 40 for each work image constituting the work video; and a reliability addition unit 202 which adds reliability to the result analyzed by the analysis unit, based on the direction of the worker 40 indicated in the work image. The information terminal device 10 includes a communication unit 13 which receives the information indicating the analysis result and the information indicating the reliability from the server device 20.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing system and an information processing method. [Background technology]

[0002] Conventionally, there is known a technique for identifying the load on a worker while analyzing the posture of the worker based on a work video generated by capturing images of the worker's work using a wearable camera or the like (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-48017 Summary of the Invention [Problem to be solved by the invention]

[0004] Here, if a part of the worker is hidden by an obstruction or if the worker's posture is inappropriate for analysis, it may be difficult to properly analyze the worker's posture based on the work video. If the worker's posture is analyzed based on a work video in which the worker's posture is inappropriate for analysis, an erroneous analysis result may be obtained. With the technology disclosed in Patent Document 1, even if the worker's posture can be analyzed, the accuracy of the analysis result, that is, the reliability, cannot be grasped. Therefore, it is necessary to check whether there are any defects in the analysis by checking the analysis result against the entire video. [Means for solving the problem]

[0005] An information processing system that achieves the above-mentioned object is an information processing system comprising an information terminal device and a server device that communicates with the information terminal device via a communication network, wherein the server device comprises a first receiving unit that receives from the information terminal device a work video in which a plurality of work images generated by imaging a worker are successively displayed in chronological order, an analysis unit that identifies the orientation of the worker shown in the work image and analyzes the posture of the worker for each of the work images that constitute the work video, a reliability adding unit that adds reliability to the analysis result of the analysis unit based on the orientation of the worker shown in the work image, and a first transmitting unit that transmits the analysis result and the reliability to the information terminal device, and the information terminal device comprises a second transmitting unit that transmits the work video to the server device, and a second receiving unit that receives the analysis result and the reliability from the server device.

[0006] According to this configuration, it is possible to present to the user the analysis results of the analysis of the worker's posture and the reliability of the analysis results. In an information processing system that achieves the above-mentioned objective, the information terminal device may include an operation unit that accepts user operations, a display unit that displays the analysis result and the reliability, and a correction unit that corrects the analysis result based on the user's operation to correct the analysis result.

[0007] According to this configuration, the analysis results can be easily corrected. In the information system for achieving the above object, the reliability adding section may add a higher reliability to the work image as the worker's orientation in the work image becomes more landscape.

[0008] According to this configuration, the more appropriate the orientation of the worker's posture for analysis, the higher the reliability of the analysis result can be. In an information system that achieves the above-mentioned objective, when the analysis result of the analysis unit indicates that the worker's posture cannot be analyzed based on the work image, the reliability addition unit may assign a lower reliability compared to when the analysis unit can analyze the worker's posture.

[0009] According to this configuration, by presenting an analysis result with a low reliability to the user, it is possible to prompt the user to correct the analysis result. In an information system that achieves the above-mentioned objective, the worker's postures analyzed by the analysis unit include a fist-raising posture in which the worker raises his / her fist, a leaning forward posture in which the worker bends his / her waist, and a knee-bending posture in which the worker bends his / her knees.

[0010] According to this configuration, it is possible to analyze postures of the worker that impose a particularly high load on the worker. In an information system that achieves the above-mentioned objective, the analysis unit may analyze, based on the position of the worker's shoulder and the position of the worker's wrist or elbow shown in the work image, that the worker is in a fist-raised posture when the position of the wrist or elbow is higher than the position of the shoulder.

[0011] According to this configuration, the posture of the worker can be analyzed based on the worker's feature points that are easy to identify. In an information system that achieves the above-mentioned objective, the analysis unit may analyze that the worker is in a bending forward posture with the waist bent, when the angle between a line segment connecting the center of the worker's waist and the base of the worker's neck shown in the work image and a line segment connecting the worker's knee and the center of the waist on the back side of the worker is a predetermined angle or greater.

[0012] According to this configuration, the posture of the worker can be analyzed based on the worker's feature points that are easy to identify. In an information system that achieves the above-mentioned objective, the analysis unit may analyze that the worker is in a knee bending posture with his / her knees bent, if the angle is less than a predetermined angle based on the inner angle of the worker's left knee or the inner angle of the worker's right knee shown in the work image.

[0013] According to this configuration, the posture of the worker can be analyzed based on the worker's feature points that are easy to identify. In an information system for achieving the above-mentioned object, the server device may further include a subtitle file generation unit that generates a subtitle file showing at least one of the analysis result of the analysis unit and the reliability added by the reliability addition unit as a subtitle for the work video, and the first transmission unit may transmit the subtitle file generated by the subtitle file generation unit to the information terminal device.

[0014] According to this configuration, the analysis results and the reliability can be easily presented to the user as subtitles for the video. In an information system that achieves the above-mentioned objective, the information terminal device includes an operation unit that accepts user operations, a display unit that displays the analysis result and the reliability, and a correction unit that, when the operation unit accepts an operation by the user to correct the analysis result, corrects the analysis result based on the operation, and the correction unit may correct the subtitle file based on the operation by the user.

[0015] According to this configuration, the subtitle file can be easily corrected while referring to the correction file. In an information system that achieves the above-mentioned objective, the server device may further include a load calculation unit that calculates a load for each posture of the worker based on a predetermined weighting for each posture of the worker and the analysis result, and the first transmission unit may transmit the load for each posture calculated by the load calculation unit to the information terminal.

[0016] According to this configuration, it is possible to present to the user information indicating the load on the worker based on the analysis result of the worker's posture. An information processing method for achieving the above-mentioned object is an information processing method for controlling an apparatus equipped with a display unit, and is characterized in that, based on a work video in which a plurality of work images generated by imaging a worker are successively displayed in chronological order, the orientation of the worker shown in each of the work images constituting the work video is identified, and the posture of the worker is analyzed, and based on the orientation of the worker shown in the identified work image, a reliability is added to the analysis result of the posture of the worker, and the analysis result and the reliability are displayed on the display unit.

[0017] According to this configuration, the same effects as those of the above-mentioned information processing system can be obtained. Effect of the Invention

[0018] According to the present invention, it is possible to present to a user the analysis result of the posture of a worker and the reliability of the analysis result. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is a diagram illustrating an example of an overall configuration of an information processing system. [Diagram 2] FIG. 2 is a diagram used to explain the configuration of the information processing system. [Diagram 3] FIG. 3 is a diagram showing an example of a display of the analysis results. [Figure 4] FIG. 4 is a diagram showing an example of a display of the correction screen. [Diagram 5] FIG. 5 is a sequence diagram used to explain the operation of the information processing system. [Figure 6] FIG. 6 is a diagram showing another example of displaying the analysis results. [Figure 7] FIG. 7 is a diagram showing another example of the display of subtitles. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] <Embodiment> Hereinafter, an information processing system and an information processing method according to specific embodiments will be described with reference to the drawings.

[0021] [Overall configuration] 1, an information processing system 1 includes, for example, one or more information terminal devices 10 and one or more server devices 20. The information terminal devices 10 and the server devices 20 are connected to each other so as to be able to transmit and receive information, for example, via a communication network 30. The communication network 30 includes, for example, the Internet, a Wide Area Network (WAN), a Local Area Network (LAN), a provider terminal, a wireless communication network, a wireless base station, a dedicated line, and the like.

[0022] The information terminal device 10 is connected to an imaging unit 50 that captures an image of the worker 40 to be imaged. The imaging unit 50 is, for example, a digital camera using a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). An example of the imaging unit 50 is a digital camera using a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), or a smartphone equipped with a digital camera. The imaging unit 50 captures an image of the worker 40 working to generate a work video in which a plurality of work images are generated in a chronological order. The information terminal device 10 transmits the generated work video to the server device 20. The work video is, for example, a video generated by capturing an image of the worker 40 performing a certain series of work from start to finish.

[0023] The server device 20 analyzes the posture of the worker 40 for each work image constituting the work video based on the work video received from the information terminal device 10. Here, some or all of the body parts of the worker 40 required for analyzing the posture of the worker 40 may be covered by an obstruction or the worker 40 himself, or the posture of the worker 40 itself may be difficult to analyze. In this case, it is difficult for the server device 20 to appropriately analyze the posture of the worker 40 based on the work video. Accordingly, the server device 20 adds a reliability to the analysis result. The reliability is higher when the orientation of the worker 40 shown in the work image, which is the original data of the analysis result, is more suitable for analyzing the posture of the worker 40. In addition, the reliability is higher when the worker 40 is clearly shown in the work image, which is the original data of the analysis result. The worker 40 is clearly shown in the work image when the worker 40 is not covered by an obstruction. The server device 20 transmits information indicating the analysis result and information indicating the reliability to the information terminal device 10.

[0024] Furthermore, the server device 20 calculates the load value, the load, and the integrated load value for each posture of the worker 40 based on a weighting that is predetermined for each posture of the worker 40 and the posture of the worker 40. The server device 20 transmits information indicating the calculated load to the information terminal device 10.

[0025] The information terminal device 10 presents to the user the analysis result of the posture of the worker 40 shown in the work video, the reliability of the analysis result, and the integrated value of the load for each posture of the worker 40. The information terminal device 10 accepts an operation by the user to correct an analysis result with a low reliability. The information terminal device 10 corrects the analysis result based on the accepted operation. Specific configurations and processes of the information terminal device 10 and the server device 20 will be described below.

[0026] [Configuration of server device 20] 2, the server device 20 includes, for example, a communication unit 23, a control unit 200, and a storage unit 250. The communication unit 23 includes, for example, a communication circuit or a communication module, and transmits and receives various information to and from the information terminal device 10 and other devices based on the control of the control unit 200. The communication unit 23 is an example of a first transmitting unit in the process of transmitting various information to the information terminal device 10, and is an example of a first receiving unit in the process of receiving various information from the information terminal device 10.

[0027] The control unit 200 is realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device (not shown) equipped with a non-transitory storage medium such as an HDD (Hard Disk Drive) or a flash memory equipped in the storage unit 250.

[0028] The storage unit 250 may be realized by the above-mentioned various storage devices, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a ROM (Read Only Memory), a RAM (Random Access Memory), etc. In the storage unit 250, a trained model 251 is stored in advance.

[0029] The trained model 251 is a model that has been trained by machine learning to identify key points of the worker 40 shown in the work image. The key points are, for example, some or all of the head, base of the neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, right toe, left hip, left knee, left ankle, and left toe of the worker 40.

[0030] An example of the trained model 251 is a neural network. When a work image is input as an input variable to the trained model 251, the trained model 251 identifies two-dimensional key points of the worker 40 shown in the work image. Then, the trained model 251 calculates or estimates the coordinates of three-dimensional key points of the worker 40 in the virtual space based on the identified two-dimensional key points of the worker 40.

[0031] The control unit 200 includes, for example, an analysis unit 201, a reliability adding unit 202, a load calculation unit 203, a subtitle file generation unit 204, and a correction file generation unit 205. The analysis unit 201 analyzes the posture of the worker 40 for each work image constituting the work video based on the work video received by the communication unit 23. First, the analysis unit 201 inputs the work video to the trained model 251. The analysis unit 201 acquires, as an output of the trained model 251, the coordinates of three-dimensional key points in the virtual space for each work image constituting the work video. Hereinafter, the coordinate system of the virtual space is assumed to have the width direction as the X axis, the height direction as the Y axis, and the depth direction as the Z axis.

[0032] The analysis unit 201 analyzes the posture of the worker 40 for each work image based on the coordinates of the three-dimensional key points. Hereinafter, the analysis unit 201 analyzes a fist-raising posture in which the worker 40 raises his / her fist, a forward-bending posture in which the worker 40 bends his / her waist, and a knee-bending posture in which the worker 40 bends his / her knees. Specifically, the analysis unit 201 analyzes that the worker 40 has a fist-raising posture when the condition (A1) or (A2) is satisfied. Furthermore, the analysis unit 201 analyzes that the worker 40 has a forward-bending posture when the condition (A3) is satisfied. Furthermore, the analysis unit 201 analyzes that the worker 40 has a knee-bending posture when the condition (A4) or (A5) is satisfied. In the following description, the center of the line segment connecting the right hip key point and the left hip key point is also referred to as the hip center.

[0033] (A1) The position of the key point of the right wrist of the worker 40 is higher than the position of the key point of the right shoulder. (A2) The position of the key point of the left wrist of the worker 40 is higher than the position of the key point of the left shoulder.

[0034] (A3) The angle between the line segment connecting the center of the waist of the worker 40 to the key point at the base of the neck and the straight line connecting the knee of the worker 40 to the center of the waist, on the back side of the worker 40, is 30 degrees or more.

[0035] (A4) The angle between the line segment connecting the right hip key point and the right knee key point of worker 40 and the line segment connecting the right knee key point and the right ankle key point of worker 40, the angle on the inside of the right knee of worker 40, is less than 140 degrees.

[0036] (A5) The angle between the line segment connecting the left hip key point and the left knee key point of worker 40 and the line segment connecting the left knee key point and the left ankle key point of worker 40, the angle on the inside of the left knee of worker 40, is less than 140 degrees.

[0037] The reliability adding unit 202 adds reliability to the analysis result analyzed by the analysis unit 201. Specifically, the reliability adding unit 202 adds a higher reliability as the orientation of the worker 40 shown in the work image, which is the original data of the analysis result, becomes more horizontal. This is because the more the worker 40 appears horizontally in the work image, the easier it is to analyze that the worker is in a bending posture or a knee-bending posture. Specifically, when the angle formed by the line segment connecting the key point of the right hip and the key point of the left hip of the worker 40 shown in the work image and the X-axis is in the range of 45 degrees to 135 degrees, it can be said that the worker 40 is horizontal.

[0038] Hereinafter, a case where the reliability adding unit 202 adds three reliability levels, "high", "medium" and "low", to the analysis result will be described. The reliability adding unit 202 adds a reliability level of "high" to the analysis result when the angle between the line segment connecting the key point of the right hip and the key point of the left hip of the worker 40 and the X-axis is in the range of 45 degrees to 135 degrees. Furthermore, the reliability adding unit 202 adds a reliability level of "medium" to the analysis result when the angle between the line segment connecting the key point of the right hip and the key point of the left hip of the worker 40 and the X-axis is outside the range of 45 degrees to 135 degrees. Furthermore, the reliability adding unit 202 adds a reliability level of "low" to the analysis result when some or all of the key points of the worker 40 used in the analysis cannot be acquired due to being blocked or the like.

[0039] The load calculation unit 203 calculates the load value, the load, and the integrated value of the load for the posture of the worker 40 based on the analysis result of the analysis unit 201 and a weighting determined in advance for each posture of the worker 40.

[0040] The load calculation unit 203 calculates a load value of the posture of the worker 40 based on the posture of the worker 40 analyzed by the analysis unit 201. The load value is a value that represents a coefficient of the severity of the posture of the worker 40 associated with the angle of twisting or bending, etc. Specifically, the load value increases as the angle of bending forward or bending the knees increases. More specifically, the load value of bending forward is "1" when the bending angle is 30 degrees, and is "1.1" when the bending angle is 40 degrees. The load calculation unit 203, for example, specifies the angle of twisting or bending based on the coordinates of three-dimensional key points acquired by the analysis unit 201, and calculates the load value based on the specified angle. Note that the numerical value of the load value is merely an example and is not limited thereto.

[0041] The load calculation unit 203 specifies the weighting based on the posture of the worker 40 analyzed by the analysis unit 201. The weighting is such that the higher the load of a posture among the postures that the worker 40 can take, the higher the weighting is. Hereinafter, the order of load of postures among the postures that the worker 40 can take is the bending forward posture, the bending knee posture, and the raising fist posture, and the weighting is "3" for the bending forward posture, "2" for the bending knee posture, and "1" for the raising fist posture. Note that the weighting values ​​are merely examples and are not limited to these.

[0042] The load calculation unit 203 calculates the load due to the posture of the worker 40 based on the calculated load value and a weighting corresponding to the posture of the worker 40, using the formula (1). Load of posture of worker 40 = (load value x weighting)... (1) The load calculation unit 203 calculates an integrated value of the load of the worker 40 for each posture. Specifically, the load calculation unit 203 calculates a new load for the posture of the analysis result using the above-mentioned formula (1), and calculates a new integrated value for the posture by adding the calculated value to the previous integrated value for the posture. In this example, the load calculation unit 203 calculates an integrated value of the load for the fist-raised posture, an integrated value of the load for the forward bending posture, and an integrated value of the load for the knee-bending posture. Then, the load calculation unit 203 calculates the sum of the integrated values ​​of the load of the worker 40 for each posture as a total integrated value.

[0043] Furthermore, each time the analysis unit 201 analyzes a posture of the worker 40, the load calculation unit 203 calculates the load of the worker 40 related to the posture and calculates an integrated value of the load of the posture. That is, the load calculation unit 203 calculates a total integrated value for each work image constituting the work video.

[0044] The subtitle file generating unit 204 generates a subtitle file based on the analysis result of the analyzing unit 201, the reliability added by the reliability adding unit 202, and the total integrated value calculated by the load calculating unit 203. The subtitle file is a file used for displaying subtitles indicating the posture analysis result, the reliability, and the total integrated value by superimposing them on the work video using text overlay.

[0045] The subtitle file generating unit 204 generates a subtitle file so that, at the timing when a work image analyzed as a raised fist posture is displayed during the work video, the text of "raised fist posture" in the subtitles is colored red, and at that timing, the text of "bending forward posture" and "bending knee posture" in the subtitles is colored gray, to indicate that the analysis has been performed as a "bending forward posture" and not a "bending forward posture" or a "bending knee posture." The subtitle file generating unit 204 also generates a subtitle file so that, at the timing when a work image analyzed as a raised fist posture is displayed during the work video, the text of "bending forward posture" in the subtitles is colored red, and at that timing, the text of "raised fist posture" and "bending knee posture" in the subtitles is colored gray, to indicate the analysis result. In addition, the subtitle file generating unit 204 generates a subtitle file so that, at the timing during the work video when a work image analyzed as a bent knee posture is displayed, the text of "bent knee posture" in the subtitles is colored red, while the text of "raised fist posture" and "bent forward posture" in the subtitles is colored gray at that timing to show the analysis result.

[0046] Here, the analysis unit 201 may analyze, based on a certain work image, that the worker 40 is in multiple postures at the same time, i.e., that there is a load in multiple analyzed postures. Specifically, the analysis unit 201 may analyze, based on the work image, that the worker 40 is in a "raised fist posture" and a "bent knee posture." Accordingly, when the analysis result in the work image shows multiple postures, the subtitle file generation unit 204 generates a subtitle file to show the analysis result by applying red to all of the text showing the multiple postures and applying gray to the text showing other postures.

[0047] Further, the subtitle file generating unit 204 generates a subtitle file based on the reliability added by the reliability adding unit 202 so that the text "data measurement section" in blue is expressed as the reliability at the timing when the work image with the reliability "high" added is displayed in the work video. The "data measurement section" is intended to mean that the posture of the worker 40 is appropriate when the analysis unit 201 performs the analysis, the analysis result is highly likely to be accurate, and the reliability of the analysis result of the posture of the worker 40 is "high". Further, the analysis unit 201 generates a subtitle file based on the reliability added by the reliability adding unit 202 so that the text "data estimation section" in yellow is expressed as the reliability at the timing when the work image with the reliability "medium" added is displayed in the work video. The "data estimation section" is intended to mean that the posture of the worker 40 is not appropriate when the analysis unit 201 performs the analysis, so that the analysis result may be inaccurate, and the reliability of the analysis result of the posture of the worker 40 is "medium". Furthermore, the analysis unit 201 generates a subtitle file so that, based on the reliability added by the reliability adding unit 202, at the timing when a work image with a reliability of "low" is displayed during the work video, the red text "data missing section" is displayed as the reliability. The "data missing section" indicates that the posture of the worker 40 was not properly measured and estimated, data on at least a portion of the posture is missing, and the reliability of the analysis result of the posture of the worker 40 is "low".

[0048] Furthermore, the subtitle file generator 204 generates a subtitle file based on the integrated value calculated by the load calculator 203 so as to represent an integrated value corresponding to the timing of the work image being displayed during the work video.

[0049] The correction file generating unit 205 generates a correction file used when correcting the analysis result in the information terminal device 10. The correction file is, for example, a file executed in a spreadsheet software. The correction file generated by the correction file generating unit 205 will be described in detail later.

[0050] The control unit 200 causes the communication unit 23 to transmit the subtitle file generated by the subtitle file generation unit 204, the correction file generated by the correction file generation unit 205, and the analyzed video adjusted so that the subtitles in the subtitle file are displayed in the work video, to the information terminal device 10. The analyzed video is a file name of the work video linked to a subtitle file name. Therefore, the multiple images that make up the analyzed video and are consecutive in time series match the work images that make up the work video.

[0051] [Configuration of information terminal device 10] As shown in FIG. 2, the information terminal device 10 includes, for example, an operation unit 11, a display unit 12, a communication unit 13, a control unit 100, and a storage unit 150. The control unit 100 is realized by, for example, a hardware processor such as a CPU executing a program (software). In addition, some or all of these components may be realized by hardware (including a circuit unit) such as an LSI, ASIC, FPGA, or GPU, or may be realized by cooperation between software and hardware. The program may be stored in advance in a storage device (not shown) including a non-transitory storage medium such as an HDD or a flash memory included in the storage unit 150. The storage unit 150 may be realized by the various storage devices described above, or an EEPROM, a ROM, a RAM, or the like.

[0052] The operation unit 11 accepts operations by a user of the information terminal device 10. The operation unit 11 is realized, for example, by some or all of a keyboard, a mouse, a touch panel, etc. Hereinafter, the operation unit 11 is assumed to be realized by a keyboard and a mouse. The operation unit 11 outputs information indicating the accepted user operation to the control unit 100.

[0053] The display unit 12 displays various types of information as images, thereby presenting the various types of information to the user of the information terminal device 10. The display unit 12 is realized by, for example, a display. The display unit 12 displays various images based on the control of the control unit 100.

[0054] The communication unit 13 includes, for example, a communication circuit or a communication module, and transmits and receives various information to and from the server device 20 and other devices based on the control of the control unit 100. The communication unit 13 is an example of a second transmitting unit in the process of transmitting various information to the server device 20, and is an example of a second receiving unit in the process of receiving various information from the server device 20.

[0055] The control unit 100 includes, for example, a display control unit 101 and a correction unit 102. The display control unit 101 executes the post-analysis video and the subtitle file received from the server device 20 using video playback software, thereby causing the display unit 12 to display subtitles indicating the analysis result of the posture of the worker 40, the reliability of the posture, and the total integrated value of the worker 40, superimposed on the post-analysis video by text overlay. The display control unit 101 also executes the correction file received from the server device 20 using spreadsheet software, thereby prompting the user to correct the analysis result.

[0056] The correcting unit 102 corrects the analysis result based on an operation by the user to correct the analysis result, which is an operation accepted by the operation unit 11. The correction process by the correcting unit 102 will be described in detail later.

[0057] [Example of analysis results] 3 shows a scene of the post-analysis video displayed on the display unit 12 by the display control unit 101 executing the post-analysis video and the subtitle file with video playback software. By playing the post-analysis video with the video playback software, for example, a working image P12 constituting the post-analysis video and a subtitle P14 superimposed on the working image P12 by overlay are displayed on the display unit 12. The working image P12 is a working image P12 corresponding to the playback time of the post-analysis video, among the multiple working images P12 constituting the post-analysis video. The working image P12 realizes the post-analysis video by successively changing into working images P12 that are successive in time series.

[0058] The subtitles P14 include text indicating the analysis result corresponding to the displayed work image P12, the reliability of the analysis result, and the total integrated value of worker 40 from the start of the post-analysis video to the current work image P12. The display control unit 101 plays back the post-analysis video using the acquired subtitle file, thereby presenting the analysis result, the reliability, and the total integrated value of worker 40 to the user of the information terminal device 10 as subtitles P14 for the post-analysis video.

[0059] In the example shown in FIG. 3, the worker 40 in the work image P12 is bending forward. Therefore, the subtitle P14 shows the text "Bending posture" in red to indicate that the analysis has been performed to determine the posture as "Bending posture". The subtitle P14 also shows the text "Raised fist posture" and "Bent knee posture", which represent postures other than bending forward, in gray. In other words, the subtitle P14 indicates that the analysis has been performed to determine the posture as not "Raised fist posture" and "Bent knee posture". Furthermore, since the worker 40 in the work image P12 is facing sideways, the reliability added to the analysis result is "High". Therefore, the subtitle P14 shows the text "Data measurement section" corresponding to the reliability of "High" in blue. Furthermore, the total accumulated value of the worker 40 from the start of the video to the work image P12 is "150". Therefore, the subtitle P14 shows the text "Total accumulated value of the worker: 150".

[0060] [Example of the correction file execution screen] As shown in FIG. 4, when the operation unit 11 accepts an operation to open a revision file in the spreadsheet software, the display control unit 101 causes the display unit 12 to display a revision file execution screen P2.

[0061] The correction file execution screen P2 includes, for example, a data division table P21, a correction button P22, a reliability column P23, a fist-raised posture selection column P24, a bending posture selection column P25, and a knee-bending posture selection column P26. The correction file execution screen P2 also shows a time axis corresponding to the playback time of the analyzed video as the horizontal axis of the reliability column P23, the fist-raised posture selection column P24, and the bending posture selection column P25. This time axis is provided for the same length as the length of the analyzed video. The correction file execution screen P2 is operated by a pointer P20. The pointer P20 is linked to the operation of, for example, a mouse serving as the operation unit 11, and is used to perform various selection operations in the correction file execution screen P2.

[0062] The correction button P22 includes five button images of correction buttons P22a, P22b, P22c, P22d, and P22e. The correction button P22a is an image representing a button used when updating / saving data according to correction contents based on the operation of the user of the information terminal device 10 accepted by the operation unit 11. The correction button P22b is an image representing a button used when the user himself starts a "forced ON" operation to forcibly select the posture of the worker 40, contrary to the analysis result. The correction button P22c is an image representing a button used when the user himself starts a "forced OFF" operation to forcibly release the posture of the worker 40, contrary to the analysis result. The correction button P22d is an image representing a button used when starting an operation to specify a range of the post-analysis video that is not to be included in the analysis of the posture of the worker 40. The correction button P22e is an image representing a button used when canceling the previous operation.

[0063] The correction file generating unit 205 generates a correction file by providing a data division table P21, a correction button P22, a reliability column P23, a fist-raised posture selection column P24, a bending posture selection column P25, and a bending knee posture selection column P26 in appropriate cells, and providing a time axis corresponding to the playback time of the analyzed video as the horizontal axis. As a result, the number of cells in the data division table P21, the reliability column P23, the fist-raised posture selection column P24, and the bending forward posture selection column P25 are all the same, and are provided from the beginning to the end of the time axis corresponding to the playback time of the analyzed video. In addition, the correction file generating unit 205 generates a correction file by placing a correction button P22 in an appropriate cell.

[0064] The fist-raising posture selection field P24 is configured with input cells corresponding to the playback time of the analyzed video. For example, when the user of the information terminal device 10 forcibly changes the analysis result analyzed as a posture other than the fist-raising posture to the fist-raising posture, the user performs a "forced OFF" operation to delete the analysis result of the posture other than the fist-raising posture and a "forced ON" operation to add the analysis result of the fist-raising posture. Specifically, the "forced OFF" operation is performed by selecting a cell of the range of playback time to be deleted in the corresponding posture and operating the correction button P22c. Furthermore, the "forced ON" operation is performed by specifying the corresponding playback time range in the fist-raising posture selection field P24 and operating the correction button P22b. For the specification, the mouse as the operation unit 11 and the pointer P20 displayed on the display unit 12 in response to the mouse operation are used. The forward bending posture selection field P25 is configured with input cells corresponding to the playback time of the analyzed video. The knee bending posture selection field P26 is configured with input cells corresponding to the playback time of the analyzed video.

[0065] The “Force OFF” operation for deleting the analysis results of the forward bending posture using the forward bending posture selection field P25, the “Force ON” operation for adding the analysis results of the forward bending posture, the “Force OFF” operation for deleting the analysis results of the knee bending posture using the knee bending posture selection field P26, and the “Force ON” operation for adding the analysis results of the knee bending posture are similar to those in the case of the fist-raising posture described above, and therefore will not be described here.

[0066] Furthermore, the user of the information terminal device 10 may specify a range in which the posture of the worker 40 is not analyzed in an input cell of any of the fist-raised posture selection field P24, the bending posture selection field P25, and the bending knee posture selection field P26. Here, the beginning or end of the analyzed video may not include the state of the worker 40 working. In this case, it is preferable to exclude the beginning or end of the analyzed video as a range in which the posture of the worker 40 is not analyzed. When excluding a specific range from the analysis result, the user of the information terminal device 10 operates the correction button P22d, and then specifies the range of the playback time of the analyzed video to be excluded in an input cell of any of the fist-raised posture selection field P24, the bending posture selection field P25, and the bending knee posture selection field P26.

[0067] The reliability column P23 indicates the correspondence between the playback time of the analyzed video and the classification of the section. As described above, there are three types of reliability: "high", "medium", and "low". Therefore, the analyzed video includes a "data measurement section" with a "high" reliability, a "data estimation section" with a "medium" reliability, and a "data missing section" with a "low" reliability. In addition to this, the analyzed video includes a "data forced section" that is attached to a section that is "forced selected" by the above-mentioned "forced ON" or "forced OFF" operation, and a "data excluded section" that is attached to a section that is "excluded".

[0068] In the reliability column P23, a "data measurement section" of the analyzed moving image is indicated by a measurement cell P27a. In addition, in the reliability column P23, a section of the analyzed moving image that has been "forcefully selected" by the above-mentioned "forced ON" or "forced OFF" operation is indicated by a forced selection cell P27b. In addition, in the reliability column P23, a section of the analyzed moving image that has been "excluded" is indicated by an exclusion cell P27c. In addition, in the reliability column P23, a "data estimation section" of the analyzed moving image is indicated by an estimation cell P27d. In addition, in the reliability column P23, a "data missing section" of the analyzed moving image is indicated by a missing cell P27e.

[0069] The measured cell P27a, the forced choice cell P27b, the excluded cell P27c, the estimated cell P27d, and the missing cell P27e are realized by, for example, filling in the cells of the corresponding playback times in the reliability column P23 in different ways. Specifically, the measured cell P27a is shown in light blue, the forced choice cell P27b is shown in blue, the excluded cell P27c is shown in gray, the estimated cell P27d is shown in yellow, and the missing cell P27e is shown in red. Note that the colors of the cells are merely examples and are not limited to these.

[0070] In addition, in the fist-raised posture selection field P24, among the measurement cells P27a included in the reliability field P23, cells at positions corresponding to the measurement cells P27a whose analysis results indicate the "fist-raised posture" are displayed in the same manner as the measurement cells P27a. In the forward bending posture selection field P25, among the measurement cells P27a included in the reliability field P23, cells at positions corresponding to the measurement cells P27a whose analysis results indicate the "forward bending posture" are displayed in the same manner as the measurement cells P27a. In the knee bending posture selection field P26, among the measurement cells P27a included in the reliability field P23, cells at positions corresponding to the measurement cells P27a whose analysis results indicate the "knee bending posture" are displayed in the same manner as the measurement cells P27a.

[0071] In the fist-raised posture selection field P24, the bending posture selection field P25, and the bending knee posture selection field P26, cells corresponding to positions that have been "forcibly selected" by the "forced ON" operation are displayed in the same manner as the forcibly selected cell P27b. In the fist-raised posture selection field P24, the bending posture selection field P25, and the bending knee posture selection field P26, cells that have been removed from the range of the corresponding posture by the "forced OFF" operation are displayed as blanks.

[0072] In the example shown in FIG. 4, initially, the analysis result of the analysis unit 201 was "fist-raised posture" from the video playback time "00:00:01" to the video playback time "00:00:06". Meanwhile, as the user performed a "forced OFF" operation on the cell in the fist-raised posture selection field P24, the fist-raised posture was deleted from the video playback time "00:00:04" to the video playback time "00:00:05". Therefore, the fist-raised posture selection field P24 shows a blank from the video playback time "00:00:04" to the video playback time "00:00:05". However, as the fist-raised posture selection field P24 shows a "forced selection" from the video playback time "00:00:04" to the video playback time "00:00:05", the corresponding cell in the reliability field P23 is shown in the manner of the forced selection cell P27b.

[0073] The correction file generating unit 205 generates a correction file based on the reliability added by the reliability adding unit 202 so that the cell in the reliability column P23 corresponding to the time when the text "Data measurement section" in blue is displayed in the analyzed video is indicated by the aspect of the measurement cell P27a. Also, based on the subtitle file generated by the subtitle file generating unit 204, the correction file generating unit 205 generates a correction file based on the subtitle file generated by the subtitle file generating unit 204 so that the cell in the reliability column P23 corresponding to the time when the text "Data estimation section" in yellow is displayed in the analyzed video is indicated by the aspect of the estimation cell P27d. Also, based on the reliability added by the reliability adding unit 202, the correction file generating unit 205 generates a correction file based on the reliability added by the reliability adding unit 202 so that the cell in the reliability column P23 corresponding to the time when the text "Data missing section" in red is displayed in the analyzed video is indicated by the aspect of the missing cell P27e.

[0074] Furthermore, the correction file generating unit 205 generates a correction file including a setting indicating the cell in the reliability column P23 corresponding to the cell forcibly selected by the fist-raised posture selection column P24, the bending posture selection column P25, and the bending knee posture selection column P26 in the manner of the forcibly selected cell P27b. Furthermore, when a cell to be excluded from the analyzed moving image is designated, the correction file generating unit 205 generates a correction file including a setting indicating the cell in the reliability column P23 corresponding to the excluded cell in the manner of the missing cell P27e.

[0075] The user of the information terminal device 10 checks whether the analysis result of the posture of the worker 40 is correct by rewatching the post-analysis video, paying particular attention to the playback time period of the post-analysis video indicated by the state of the estimated cell P27d and the missing cell P27e in the reliability column P23. If the confirmed analysis result is incorrect, the user performs a "force OFF" operation to delete the incorrect posture analysis result by specifying the corresponding playback time range and selecting the correction button P22c, and then performs a "force ON" operation to add the correct posture analysis result by specifying the corresponding playback time range and selecting the correction button P22b.

[0076] The data division table P21 is an image showing the percentage of the analyzed video that each reliability classification, "forced selection", and "exclusion" occupy. Specifically, the correction file generation unit 205 generates a correction file in which the percentage of the measurement cell P27a in the entire cell of the reliability column P23 is displayed in the "subtotal" column of "data measurement" in the data division table P21. In other words, the correction file generation unit 205 generates a correction file including a setting to display the percentage of the total length of the "data measurement section" of the analyzed video in the "subtotal" column of "data measurement" in the data division table P21. In addition, the correction file generation unit 205 generates a correction file in which the percentage of the forced selection cell P27b in the entire cell of the reliability column P23 is displayed in the "subtotal" column of "forced selection" in the data division table P21. In other words, the correction file generating unit 205 generates a correction file including a setting to display the percentage of the total length of the "forced selection" sections of the analyzed video to the total length of the analyzed video in the "subtotal" column of "forced selection" in the data category table P21. The correction file generating unit 205 also generates a correction file in which the percentage of the excluded cell P27c to the total cells in the reliability column P23 is displayed in the "subtotal" column of "exclusion" in the data category table P21. In other words, the correction file generating unit 205 generates a correction file including a setting to display the percentage of the total length of the "excluded" sections of the analyzed video to the total length of the analyzed video in the "subtotal" column of "exclusion" in the data category table P21.

[0077] The correction file generating unit 205 also generates a correction file in which the proportion of the estimated cell P27d to all the cells in the reliability column P23 is displayed in the "Subtotal" column of "Data Estimation" in the data classification table P21. In other words, the correction file generating unit 205 generates a correction file including a setting to display the proportion of the total length of the "Data Estimation Sections" of the analyzed moving image to the total length of the analyzed moving image in the "Subtotal" column of "Data Estimation" in the data classification table P21. The correction file generating unit 205 also generates a correction file in which the proportion of the missing cell P27e to all the cells in the reliability column P23 is displayed in the "Subtotal" column of "Data Missing" in the data classification table P21. In other words, the correction file generation unit 205 generates a correction file that includes a setting to display in the "Subtotal" column of "Missing Data" in the data division table P21 the percentage of the total length of the post-analysis video that is the sum of the "data missing sections" of the post-analysis video to the entire length of the post-analysis video.

[0078] The correction file generating unit 205 generates a correction file including a function for displaying the total percentage of "data measurement", "forced selection", and "exclusion" in the corresponding "total" column of the data classification table P21. The correction file generating unit 205 also generates a correction file including a function for displaying the total percentage of "data estimation" and "missing data" in the corresponding "total" column of the data classification table P21.

[0079] The data division table P21 shown in FIG. 4 indicates that the length of the "data measurement section" with a reliability of "high" accounts for 79[%] of the entire length of the analyzed video. The data division table P21 also indicates that the length of the "forced selection" section accounts for 8[%] of the entire length of the analyzed video. The data division table P21 also indicates that the length of the "excluded" section accounts for 2[%] of the entire length of the analyzed video. The data division table P21 also indicates that the length of the "data estimation section" with a reliability of "medium" accounts for 8[%] of the entire length of the analyzed video. The data division table P21 also indicates that the length of the "data missing section" with a reliability of "low" accounts for 3[%] of the entire length of the analyzed video.

[0080] The data division table P21 also shows that the total length of the video for the "data measurement section", "forced selection" section, and "excluded" section accounts for 89% of the total length of the analyzed video. The data division table P21 also shows that the total length of the "data estimation section" and "data missing section" accounts for 11% of the total length of the analyzed video.

[0081] By referring to the data division table P21, if the proportion of the "data estimated section" or "data missing section" is not 0[%], the user of the information terminal device 10 rewatches the post-analysis video, paying particular attention to the "data estimated section" or "data missing section". This allows the user to check whether the analysis result of the posture of the worker 40 is correct for the section where the reliability of the analysis result is low. The user then performs an operation to specify "forced selection" or "exclusion" so that the proportion of the "data estimated section" or "data missing section" is 0[%].

[0082] When the user selects the correction button P22a after performing a forced selection operation or an exclusion operation in the above-mentioned fist-raised posture selection field P24, bending posture selection field P25, and bending knee posture selection field P26, the correction unit 102 updates / saves the subtitle file. Specifically, the correction unit 102 updates the subtitle file by changing the display mode of the subtitles corresponding to the forcedly selected range of playback time and the wording of the reliability.

[0083] For example, when the "forced selection" process of adding a fist-raised posture is set to the fist-raised posture selection field P24, the correction unit 102 changes the subtitle representing the reliability to "data forced section" while coloring the text of "fist-raised posture" representing the posture in red within the forcibly selected range of playback time, and updates / saves the subtitle file. Similarly, when the "forced selection" process of adding a forward bending posture is set to the forward bending posture selection field P25, the correction unit 102 changes the subtitle representing the reliability to "data forced section" while coloring the text of "forward bending posture" representing the posture in red within the forcibly selected range of playback time, and updates / saves the subtitle file. Similarly, when the "forced selection" process of adding a knee-bending posture is set to the knee-bending posture selection field P26, the correction unit 102 changes the subtitle representing the reliability to "data forced section" while coloring the text of "knee-bending posture" representing the posture in red within the forcibly selected range of playback time, and updates / saves the subtitle file.

[0084] Furthermore, when the "forced selection" process for deleting the analysis result of the fist-raised posture is set to the fist-raised posture selection field P24, the correction unit 102 changes the subtitle representing the reliability to "data forced section" while graying the text of "fist-raised posture" representing the posture within the forcibly selected range of the playback time, and updates / saves the subtitle file. Similarly, when the "forced selection" process for deleting the analysis result of the forward bending posture is set to the forward bending posture selection field P25, the correction unit 102 changes the subtitle representing the reliability to "data forced section" while graying the text of "forward bending posture" representing the posture within the forcibly selected range of the playback time, and updates / saves the subtitle file. Similarly, when the "forced selection" process for deleting the analysis result of the knee bending posture is set to the knee bending posture selection field P26, the correction unit 102 changes the subtitle representing the reliability to "data forced section" while graying the text of "knee bending posture" representing the posture within the forcibly selected range of the playback time, and updates / saves the subtitle file.

[0085] [Operation of information processing system 1] 5, first, the information terminal device 10 acquires a work video from the imaging unit 50 (step S100). Next, the communication unit 13 transmits the work video to the server device 20 via the communication network 30 (step S102).

[0086] Next, the communication unit 23 of the server device 20 receives the work video from the information terminal device 10 (step S104). Next, the analysis unit 201 analyzes the received work video (step S106). The analysis unit 201 inputs the work video to the trained model 251. Then, the analysis unit 201 acquires, as an output of the trained model 251, the coordinates of two-dimensional key points and the coordinates of three-dimensional key points of the worker 40 in the virtual space for each work image constituting the work video. The analysis unit 201 determines whether or not the conditions (A1) to (A5) are satisfied based on the coordinates of the three-dimensional key points. If the condition (A1) or (A2) is satisfied, the analysis unit 201 analyzes that the worker 40 is in a fist-raising posture. Furthermore, if the condition (A3) is satisfied, the analysis unit 201 analyzes that the worker 40 is in a forward-leaning posture. Furthermore, when the condition (A4) or the condition (A5) is satisfied, the analysis unit 201 analyzes that the worker 40 is in a knee bending posture.

[0087] Next, the reliability adding unit 202 adds reliability to the analysis result analyzed by the analysis unit 201 (step S108). When the angle between the line segment connecting the key points of the right hip and the key points of the left hip of the worker 40 and the X-axis is in the range of 45 degrees to 135 degrees, the reliability adding unit 202 adds reliability of "high" to the analysis result. When the angle between the line segment connecting the key points of the right hip and the key points of the left hip of the worker 40 and the X-axis is outside the range of 45 degrees to 135 degrees, the reliability adding unit 202 adds reliability of "medium" to the analysis result. When some or all of the key points of the worker 40 used in the analysis could not be acquired due to being blocked or the like, the reliability adding unit 202 adds reliability of "low" to the analysis result. The reliability adding unit 202 adds a reliability to each of the analysis results for each work image analyzed by the analysis unit 201 .

[0088] Next, the load calculation unit 203 calculates the load value, load, and total integrated value of the posture of the worker 40 based on the analysis result of the analysis unit 201 and the weighting determined in advance for each posture of the worker 40 (step S110). The load calculation unit 203 calculates the load value of the posture of the worker 40 based on the posture of the worker 40 analyzed by the analysis unit 201. The load calculation unit 203 specifies the weighting based on the posture of the worker 40 analyzed by the analysis unit 201. The load calculation unit 203 calculates the load due to the posture using the formula (1) based on the calculated load value and the weighting corresponding to the posture of the worker 40. The load calculation unit 203 calculates the integrated value of the load of the worker 40 for each posture. Specifically, the load calculation unit 203 calculates a new load for the posture of the analysis result using the above-mentioned formula (1), and calculates a new integrated value for the posture by adding the calculated value to the previous integrated value for the posture. Then, the load calculation unit 203 calculates the sum of the integrated values ​​of the load of the worker 40 for each posture as a total integrated value.

[0089] Next, the subtitle file generating unit 204 generates a subtitle file based on the analysis result of the analysis unit 201, the reliability added by the reliability adding unit 202, and the integrated value calculated by the load calculating unit 203 (step S112). The subtitle file generating unit 204 generates a subtitle file so as to express the analysis result by adding red to the text of the posture analyzed to be loaded in the subtitles at the timing when the work image of the posture analyzed to be loaded is displayed in the post-analysis video, and adding gray to the text of the posture other than the posture analyzed to be loaded at that timing in the subtitles. In addition, the subtitle file generating unit 204 generates a subtitle file so as to express the reliability by the text corresponding to the added reliability and the color of the text at the timing when the work image to which any one of the reliability levels "high", "medium", and "low" is added is displayed in the work video, based on the reliability added by the reliability adding unit 202. Furthermore, the subtitle file generator 204 generates a subtitle file based on the integrated value calculated by the load calculator 203 so as to represent an integrated value corresponding to the timing of the work image being displayed during the work video.

[0090] Next, the correction file generating unit 205 generates a correction file to be used when correcting the analysis result in the information terminal device 10 (step S114). Next, the communication unit 23 transmits the subtitle file generated by the subtitle file generation unit 204, the correction file generated by the correction file generation unit 205, and the analyzed video adjusted so that the subtitles in the subtitle file are displayed in the work video to the information terminal device 10 via the communication network 30 (step S116). Next, the communication unit 13 receives the subtitle file, the correction file, and the analyzed video from the server device 20 (step S118).

[0091] Next, the display control unit 101 causes the display unit 12 to display a correction file execution screen P2 by executing the correction file using a spreadsheet software in conjunction with playing the post-analysis video using the video playback software (step S120). This allows the user of the information terminal device 10 to check the contents of the correction file while checking the post-analysis video. Next, the display control unit 101 causes the display unit 12 to display the subtitles of the subtitle file by text overlay on the post-analysis video while playing the post-analysis video using the video playback software (step S122). This allows the display control unit 101 to present the analysis result, the reliability, and the integrated value of the load of the worker 40 to the user by the subtitles P14.

[0092] With the correction file execution screen P2 displayed on the display unit 12, the operation unit 11 accepts an operation by the user to correct the analysis result (step S124). The correction unit 102 updates / saves the correction file while updating / saves the subtitle file based on the operation accepted in step S126 (step S126). Then, the correction unit 102 ends the series of processes.

[0093] [Effects of the embodiment] According to the above embodiment, the following effects can be obtained. (1) The communication unit 13 of the information terminal device 10 transmits to the server device 20 a work video in which a plurality of work videos, which are generated by the imaging unit 50 capturing images of the worker 40, are successive in time series.

[0094] The communication unit 23 of the server device 20 receives the work video from the information terminal device 10. The analysis unit 201 analyzes the posture of the worker 40 for each work image constituting the work video based on the work video. The reliability addition unit 202 adds reliability to the analysis result of the analysis unit 201 based on the orientation of the worker 40 shown in the work video. The communication unit 23 of the server device 20 transmits the analysis result and the reliability to the information terminal device 10.

[0095] The display control unit 101 of the information terminal device 10 causes the display unit 12 to display the information indicating the analysis result and the information indicating the reliability as subtitles of the analyzed moving image. With this configuration, the information processing system 1 can present the analysis result of analyzing the posture of the worker 40 and the reliability of the analysis result to the user.

[0096] In the information processing system 1, the process of analyzing the posture of the worker 40 from the work video is performed in the server device 20 via the communication network 30. On the other hand, the information terminal device 10 performs processes with a lower load than the server device 20, such as transmitting the work video and presenting the analysis results and reliability to the user. Therefore, in the information processing system 1, the information terminal device 10 does not need to have high functionality compared to the server device 20 that performs the analysis process. In this case, the information processing system 1 can reduce the cost required for the user to install the information terminal device 10 compared to a case in which all the information terminal devices 10 have the same functionality as the server devices 20.

[0097] (2) The information terminal device 10 includes an operation unit 11 and a display unit 12. The operation unit 11 accepts a user's operation. The display unit 12 displays a subtitle P14 indicating the analysis result and the reliability. The information terminal device 10 also includes a correction unit 102 that corrects a subtitle file indicating the analysis result based on the operation when the operation unit 11 accepts an operation by the user to correct the analysis result. Specifically, the display control unit 101 executes a correction file in a spreadsheet software while playing the post-analysis video, and displays a correction file execution screen P2 on the display unit 12. When the operation unit 11 accepts an operation by the user to correct the analysis result based on the correction file execution screen P2, the correction unit 102 updates / saves the subtitle file indicating the analysis result according to the correction content based on the operation of the user of the information terminal device 10 accepted by the operation unit 11. With this configuration, the information processing system 1 can easily correct the analysis result.

[0098] In the information processing system 1, the information terminal device 10 executes the correction file using a spreadsheet software, thereby displaying a correction file execution screen P2 on the display unit 12. Generally used computers often come with spreadsheet software pre-installed. Therefore, in the information processing system 1, the analysis results can be corrected by simple means compared to the case where dedicated software is used.

[0099] (3) The reliability adding unit 202 adds a higher reliability to the work image of the worker 40, the more sideways the worker 40 is shown in the work image. As described above, the more sideways the worker 40 is shown in the work image, the easier it is to analyze that the worker is in a bent-forward posture or a bent-knee posture. Therefore, the reliability of the analysis result based on the work image in which the worker 40 is shown sideways is high. On the other hand, the more forward or backward the worker 40 is shown, the more difficult it is to analyze that the worker is in a bent-forward posture or a bent-knee posture. Therefore, the reliability of the analysis result based on the work image in which the worker 40 is shown facing forward or backward is low. With this configuration, the information processing system 1 can add a higher reliability to the analysis result, the more appropriate the orientation is for analyzing the posture of the worker 40.

[0100] (4) When the analysis result of the analysis unit 201 indicates that the posture of the worker 40 cannot be analyzed, the reliability adding unit 202 adds a lower reliability than when the analysis can be performed. A case in which the analysis unit 201 cannot analyze the posture of the worker 40 is, for example, a case in which some or all of the key points of the worker 40 used in the analysis are blocked. With this configuration, the information processing system 1 can present an analysis result with a low reliability to the user, thereby urging the user to correct the analysis result.

[0101] (5) The analysis unit 201 analyzes, among the postures of the worker 40, a fist-raised posture in which the worker 40 raises his / her fist, a forward-bending posture in which the worker 40 bends his / her waist, and a knee-bending posture in which the worker 40 bends his / her knees. In general, the fist-raised posture, the forward-bending posture, and the knee-bending posture are particularly high-stress postures among the postures of the worker 40. Therefore, the information processing system 1 can analyze postures that are particularly high-stress for the worker 40, based on the work video.

[0102] (6) Based on the shoulder position of the worker 40 and the wrist position of the worker 40 shown in the work image, the analysis unit 201 analyzes that the worker 40 is in a fist-raising posture if the wrist position is higher than the shoulder position. In general, when analyzing a person's posture, it is preferable that the end points, joints, etc. of the person are specified as feature points such as key points. With this configuration, the information processing system 1 can analyze that the worker 40 is in a fist-raising posture based on the shoulder position of the worker 40 and the elbow position of the worker 40, which are feature points that are easy to use in analyzing the posture of the worker 40.

[0103] (7) The analysis unit 201 analyzes that the worker 40 is in a bending-forward posture when the angle between the line segment connecting the waist center of the worker 40 and the base of the neck of the worker 40 shown in the work image and the straight line connecting the knees of the worker 40 and the waist center of the worker 40 on the back side of the worker 40 is equal to or greater than a predetermined angle. With this configuration, the information processing system 1 can analyze that the worker 40 is in a bending-forward posture based on characteristic points of the worker 40, such as the knees, base of the neck, and waist center, which are easy to use in analyzing the posture of the worker 40.

[0104] (8) The analysis unit 201 analyzes that the worker 40 is in a bent knee posture, if the bending angle is less than a predetermined angle, based on the inner angle of the left knee of the worker 40 or the inner angle of the right knee of the worker 40 shown in the work image. With this configuration, the information processing system 1 can analyze that the worker 40 is in a bent knee posture, based on feature points of the worker 40, such as the knees, which are easy to use in analyzing the posture of the worker 40.

[0105] (9) The server device 20 further includes a subtitle file generator 204. The subtitle file generator 204 generates a subtitle file that indicates, as subtitles for the work video, the analysis result of the analysis unit 201, the reliability added by the reliability adding unit 202, and the integrated value of the load of the worker 40 calculated by the load calculation unit 203. With this configuration, the information processing system 1 can simply present the analysis result, the reliability, and the integrated value to the user as subtitles for the analyzed video.

[0106] (10) The information terminal device 10 further includes a correction unit 102. When the operation unit 11 receives an operation by the user to correct the analysis result, the correction unit 102 corrects the analysis result of the analysis unit 201 based on the operation. The server device 20 further includes a correction file generation unit 205. The correction file generation unit 205 generates a correction file indicating the analysis result of the analysis unit 201 and the reliability added by the reliability addition unit 202. The correction unit 102 corrects the analysis result and reliability indicated in the subtitle file by correcting the subtitle file based on the user's operation on the correction file. With this configuration, the information processing system 1 can enable the user to easily correct the subtitle file while referring to the correction file.

[0107] (11) The server device 20 further includes a load calculation unit 203. The load calculation unit 203 calculates a load value, a load, and an integrated load value for each posture of the worker 40 based on a weighting determined in advance for each posture of the worker 40 and an analysis result of the posture of the worker 40. The display control unit 101 displays text indicating the integrated load value included in the subtitle file by text overlay on the analyzed moving image. With this configuration, the information processing system 1 can present information indicating the integrated load value of the worker 40 to the user.

[0108] <Modification> The above-described embodiments may be modified as follows: The above-described embodiments and the following examples may be combined with each other as long as they are not technically inconsistent.

[0109] Although the analyzed video is described as being composed of chronologically consecutive work images, this is not limited to this. For example, a two-dimensional skeleton model may be superimposed on the analyzed video. The two-dimensional skeleton model is a model that connects the coordinates of two-dimensional key points output by the trained model 251 to the shape of the worker 40.

[0110] As shown in FIG. 6, in the composite work image P12a constituting the post-analysis video, a two-dimensional skeleton model is displayed superimposed on the worker 40 shown in the composite work image P12a. As described above, the analysis unit 201 acquires the coordinates of two-dimensional key points in the virtual space for each work image constituting the work video as the output of the learned model 251. The analysis unit 201 in this example generates a two-dimensional skeleton model image representing the two-dimensional skeleton model based on the coordinates of the acquired key points. The analysis unit 201 synthesizes the two-dimensional skeleton model image by general image processing so as to superimpose it on the worker 40 shown in the work image, thereby generating the composite work image P12a. The analysis unit 201 generates the composite work image P12a while generating a two-dimensional skeleton model image for each work image of the work video. The analysis unit 201 generates a video constituted by the composite work images P12a that are successive in time series as the post-analysis video.

[0111] According to this configuration, the information processing system 1 can present the coordinates of the detected two-dimensional key points to the user in an easy-to-understand manner. In addition, the information processing system 1 can present the correspondence between the coordinates of the detected two-dimensional key points and the posture of the worker 40 to the user in an easy-to-understand manner by displaying the two-dimensional skeletal model superimposed on the corresponding position of the worker 40 in the composite work image P12a.

[0112] 6, the post-analysis video may include a three-dimensional skeletal model image P16 in addition to the work image P12 and the composite work image P12a. The three-dimensional skeletal model image P16 is generated from a three-dimensional skeletal model calculated or estimated based on a two-dimensional skeletal model. With this configuration, the information processing system 1 can present the coordinates of the three-dimensional key points used in the posture analysis to the user in an easy-to-understand manner. Furthermore, the information processing system 1 can present the correspondence between the three-dimensional skeletal model used in the posture analysis and the posture of the worker 40 in an easy-to-understand manner by displaying the three-dimensional skeletal model alongside the worker 40 in the composite work image P12a.

[0113] The method by which the analysis unit 201 generates a video composed of chronologically consecutive composite work images P12a as the analyzed video is not limited to this, and may be realized by other methods. For example, the analysis unit 201 may obtain the analyzed video by using an application that inputs the coordinates of two-dimensional key points in a virtual space and a work image of the original data of the coordinates, and outputs an image in which a two-dimensional skeletal model is arranged so as to be superimposed on the worker 40.

[0114] In addition to the above information, the communication unit 23 may transmit information indicating the coordinates of three-dimensional key points in the virtual space acquired for each work image constituting the work video to the information terminal device 10. In this case, the display control unit 101 may generate a three-dimensional skeletal model image representing a three-dimensional skeletal model based on the received information. Also, instead of (or in addition to) the analysis unit 201, the display control unit 101 may execute a process of generating a composite work image P12a while generating a three-dimensional skeletal model image for each work image of the work video, and a process of generating a video composed of chronologically consecutive composite work images P12a as a post-analysis video.

[0115] The imaging unit 50 may not be connected to the information terminal device 10. The information terminal device 10 may acquire the work video by connecting a storage device (not shown) having a non-transient storage medium such as a flash memory or an SD (Safe Driver) card in which data of the work video generated by the imaging unit 50 is stored to the information terminal device 10. The information terminal device 10 and the imaging unit 50 may be connected to be able to transmit and receive information via a network, and the imaging unit 50 may transmit the work image via the network, so that the information terminal device 10 acquires the work image. In this case, the network may be the above-mentioned communication network 30, or may be a LAN, a wireless communication network, a dedicated line, or the like between the information terminal device 10 and the imaging unit 50.

[0116] The server device 20 does not need to execute the process of calculating the load of the worker 40. In this case, the control unit 200 does not need to include the load calculation unit 203, and the process of step S110 shown in Fig. 5 does not need to be performed. Also, it is possible to calculate only the load value and the load for each posture, without calculating the integrated value of the load.

[0117] The key point of the condition (A1) used by the analysis unit 201 when analyzing the fist-raising posture may be the right elbow instead of the right wrist. Also, the key point of the condition (A2) used by the analysis unit 201 when analyzing the fist-raising posture may be the left elbow instead of the left wrist.

[0118] In the condition (A3) used when the analysis unit 201 analyzes the forward bending posture, instead of the straight line connecting the knees and the waist center of the worker 40, a direction perpendicular to the ground surface on which the worker 40 is present may be used. In this case, the condition (A3) is that the angle between the line segment connecting the waist center of the worker 40 and the key point at the base of the neck and the vertical line passing through the waist center (or a straight line perpendicular to the ground surface on which the worker 40 is present) on the back side of the worker 40 is 30 degrees or more. Note that 30 degrees is an example of a predetermined angle and is not limited to this. Note that 30 degrees is an example of a predetermined angle and is not limited to this. In the condition (A3), instead of the angle on the back side of the worker 40, an angle on the abdominal side of the worker 40 may be used. In this case, the predetermined angle is, for example, 150 degrees, and the condition (A3) is that the angle between the above-mentioned two line segments on the abdominal side of the worker 40 is less than 150 degrees.

[0119] Although the analysis unit 201 analyzes whether the worker 40 is in a knee-bending posture based on the inner angle of the knee of the worker 40 in the condition (A4) and the condition (A5), the analysis unit 201 may analyze that the worker 40 is in a knee-bending posture in the condition (A4) when the angle between the straight line connecting the waist key point of the worker 40 and the right knee key point and the line segment connecting the right knee key point and the right ankle key point of the front side of the right foot of the worker 40 is 40 degrees or more. The analysis unit 201 analyzes that the worker 40 is in a knee-bending posture in the condition (A5) when the angle between the straight line connecting the waist key point of the worker 40 and the left knee key point and the line segment connecting the left knee key point and the left ankle key point of the front side of the left foot of the worker 40 is 40 degrees or more.

[0120] The key points used by the analysis unit 201 for analysis are merely examples and are not limited thereto. For example, instead of the wrist key points, key points on the hand, thumb, and fingertips may be used. Also, instead of the neck key points, key points on the center of the shoulder may be used. Also, instead of the ankle key points, key points on the feet may be used. Also, when the key points used for analysis are occluded, the analysis unit 201 may analyze the posture using the alternative key points described above.

[0121] In condition (A3), 30 degrees and 150 degrees are examples of the predetermined angle, and are not limited to these. In condition (A4) and condition (A5), 140 degrees and 40 degrees are examples of the predetermined angle, and are not limited to these.

[0122] Although the subtitle file generating unit 204 generates a subtitle file showing the analysis result by the subtitle of the analyzed video based on the analysis result, this is not limited to the above. The analysis unit 201 may generate a file including details of the analysis result information such as the load for each posture, the integrated value of the load for each posture, the analyzed angle information, the coordinates of two-dimensional key points, and the coordinates of two-dimensional key points in a table by the playback time of the analyzed video, and transmit the file to the information terminal device 10.

[0123] The analysis unit 201 may generate analysis result information including one or more records that associate the analysis result with the playback time of the work video. In this case, the record of the analysis result information is data such as "XX hours YY minutes 0.03 (playback time): knee bending posture (analysis result) ---> XX hours YY minutes 0.06 (playback time): knee bending posture (analysis result)".

[0124] Also, the case where the subtitle file generating unit 204 generates a subtitle file that indicates the reliability by the subtitles of the video after analyzing the reliability based on the reliability has been described, but this is not limited to this. The reliability adding unit 202 may generate reliability result information that includes one or more records that associate the reliability with the playback time of the work video. In this case, the record of the reliability result information is data such as "XX hours YY minutes 0.03 (playback time): high (reliability) ---> XX hours YY minutes 0.06 (playback time): high (reliability)".

[0125] Also, the case has been described where the subtitle file generating unit 204 generates a subtitle file that indicates the total integrated value by the subtitles of the video after analyzing the total integrated value based on the total integrated value of the worker 40, but this is not limited to the above. The load calculating unit 203 may generate integrated value result information that includes one or more records that associate the total integrated value with the playback time of the work video. In this case, the record of the integrated value result information is data such as "XX hours YY minutes 0.03 (playback time): 5 (total integrated value) --> XX hours YY minutes 0.06 (playback time): 8 (total integrated value)" and the like.

[0126] In this case, the correction unit 102 corrects the analysis result information based on the playback time corresponding to the range specified in the correction file and the posture that is forcibly selected. With this configuration, the information processing system 1 can present the analysis result, the reliability, and the integrated value to the user as more concisely summarized data.

[0127] In the above description, the analysis unit 201 analyzes three postures, namely, "fist-raised posture", "bending posture", and "bending knee posture", but this is not limited thereto. The analysis unit 201 may analyze three or more postures. In addition, the subtitles P14 include text of the posture analyzed by the analysis unit 201, text indicating the reliability, and text indicating the total integrated value of the worker 40, but this is not limited thereto. The subtitle file generation unit 204 may generate a subtitle file to produce another form of subtitle P14.

[0128] As shown in FIG. 7, the caption P14a of this example includes text indicating only the analysis result of the analysis unit 201 regarding the posture of the worker 40 shown in the composite work image P12a, and does not include text regarding other postures. Specifically, the caption P14a includes text of "bending posture" representing the posture of the worker 40 shown in the composite work image P12a. The caption P14a also includes text indicating a load value identified by the load calculation unit 203 based on the posture of the worker 40 shown in the composite work image P12a. Specifically, the caption P14a includes text of "1" as the load value of the "bending posture" of the worker 40 shown in the composite work image P12a. The caption P14a also includes text indicating the reliability added by the reliability addition unit 202 based on the orientation of the worker 40 shown in the composite work image P12a. Specifically, the caption P14a includes the text "Data measurement section," which indicates that the reliability is "high" because the orientation of the worker 40 shown in the composite work image P12a is landscape.

[0129] Here, when there are many postures to be analyzed by the analysis unit 201 and the analysis results are shown in the form of subtitles P14, the amount of text describing the postures increases, making the analysis results, reliability, and load information difficult to understand. With this configuration, the subtitle file generation unit 204 generates a subtitle file to produce simple subtitles P14a, so that the analysis results, reliability, and load information can be presented to the user in an easy-to-understand manner.

[0130] Furthermore, when it is analyzed that the worker 40 is performing multiple postures simultaneously, the subtitle file generating unit 204 may generate a subtitle file so as to display, as the subtitle P14a, text representing a posture among the multiple postures with a larger load value calculated by the load calculating unit 203 and the load value of the posture. For example, in a certain playback time, there is a case where the worker 40 is performing a fist-raising posture and a knee-bending posture, and the load value of the fist-raising posture is larger than the load value of the knee-bending posture. In this case, the subtitle file generating unit 204 generates a subtitle file that displays, as the subtitle P14a, the text of "fist-raising posture" as the posture with the highest load, and "2", which is the load value of the fist-raising posture. Furthermore, when it is analyzed that the worker 40 is performing multiple postures simultaneously, the subtitle file generating unit 204 may display, together with the text representing the multiple postures and the load value for each posture. In this case, the subtitle file generator 204 generates a subtitle file of subtitle P14a which includes the text "raised fist posture" and the load value of the raised fist posture of "2" and the text "bent knee posture" and the load value of the bent knee posture of "1." Note that subtitle P14a may include the load of the posture of worker 40 instead of the load value of the posture of worker 40.

[0131] The subtitle file may contain a load instead of (or in addition to) the total integrated value or the load value described above. In this case, the subtitle file generating unit 204 generates a subtitle file based on the load of the posture of the worker 40 calculated by the load calculating unit 203 so that text indicating the load of the posture of the worker 40 shown in each work image in the work video is displayed as a subtitle at the timing when the work image is displayed.

[0132] The load calculation unit 203 may specify a load value instead of calculating a load value. For example, the storage unit 250 may store information in which the torsion or bending angle and the load value are associated with each other. In this case, the analysis unit 201 specifies the torsion or bending angle based on the coordinates of the three-dimensional key points acquired, and acquires the load value based on the specified angle and the information.

[0133] In the above-mentioned subtitle P14, the total accumulated value of the worker 40 up to the currently displayed work image in the analyzed video and the analysis result of the posture of the worker 40 shown in the work image are shown by subtitles, but this is not limited to this. The subtitle file generation unit 204 may generate a subtitle file so that the maximum value of the load calculated by the load calculation unit 203 using formula (1) up to the currently displayed work image in the analyzed video is shown in the subtitle P14. The subtitle file generation unit 204 may also generate a subtitle file so that the text indicating the posture of the analysis result analyzed by the analysis unit 201 based on the work image that was the original data of the maximum value is shown in the subtitle P14 or subtitle P14a.

[0134] The subtitle file generating unit 204 may vary the color mode of the text indicating the analysis result according to the posture of the object of analysis of the analyzing unit 201. The subtitle file generating unit 204 may also vary the display mode of the text indicating the total integrated value according to a change in the total integrated value of the worker 40. Specifically, the subtitle file generating unit 204 may vary the display mode such that the color tone of the text indicating the integrated value changes from a cool color to a warm color as the total integrated value of the worker 40 increases.

[0135] The subtitle file generation unit 204 may generate a subtitle file so as to display the integrated value for each posture of the worker 40 in subtitles, instead of (or in addition to) the total integrated value of the worker 40 in subtitles.

[0136] The subtitle file generating unit 204 may generate a subtitle file based on the load value calculated by the load calculating unit 203 using the formula (1) based on the currently displayed work image in the analyzed video, so as to represent the load value corresponding to the timing of the work image being displayed during the work video. In this case, if the load value is equal to or less than the first threshold, the subtitle file generating unit 204 regards the load as light and represents the text representing the load value in blue. If the load value is equal to or more than the first threshold and less than the second threshold, the subtitle file generating unit 204 regards the load as medium and represents the text representing the load value in yellow. If the load value is equal to or more than the second threshold, the subtitle file generating unit 204 regards the load as high and represents the text representing the load value in red.

[0137] The subtitle file generator 204 may generate a subtitle file in which the posture analyzed by the analyzer 201 is displayed as the subtitle P14, and the display mode of the text representing the analyzed posture is changed according to the load of the analyzed posture. For example, the subtitle file generator 204 displays the text representing the analyzed posture in blue when the load value is equal to or less than a first threshold, displays the text in yellow when the load value is equal to or more than the first threshold and less than a second threshold, and displays the text in red when the load value is equal to or more than the second threshold. In this case, the subtitle P14 does not need to include text indicating the accumulated load value.

[0138] When displaying the posture analyzed by the analysis unit 201 as the subtitle P14, the subtitle file generation unit 204 may generate a subtitle file in which the display mode of the text representing the analyzed posture is changed according to the reliability. For example, the subtitle file generation unit 204 displays the text representing the analyzed posture in blue when the reliability is "high", displays the text in yellow when the reliability is "medium", and displays the text in red when the reliability is "low". In this case, the subtitle P14 does not need to include the text indicating the reliability.

[0139] The subtitle file generator 204 may generate multiple subtitle files that display subtitles with different display modes as described above. In the information terminal device 10, the user may select a subtitle file that displays the items the user wants to check about the earthquake as subtitles in an easily viewable manner, and display the selected subtitle file on the video after analysis by using a text overlay.

[0140] In the above description, the reliability adding unit 202 adds three reliability levels, "high", "medium", and "low", to the analysis result, but this is not limited to the above. The reliability adding unit 202 may add a value according to the orientation of the worker 40 shown in the work image as the reliability.

[0141] Specifically, when the angle between the line segment connecting the right hip key point and the left hip key point and the X-axis is in the range of 45 degrees to 135 degrees, it can be said that the worker 40 is facing sideways. In the following description, the line segment connecting the right hip key point and the left hip key point is also referred to as the hip orientation. Therefore, the reliability adding unit 202 adds a high reliability to the analysis result when the angle between the hip orientation and the X-axis is in the range of 45 degrees to 135 degrees. Hereinafter, the reliability adding unit 202 adds a value from 0 to 100[%] as the reliability. In this case, the reliability adding unit 202 adds a reliability of 100[%] when the angle between the hip orientation and the X-axis is 90 degrees, and adds a reliability of 50[%] when the angle is 45 degrees or 135 degrees.

[0142] Furthermore, if the angle between the waist direction and the X-axis is between 90 degrees and 45 degrees, the confidence adding unit 202 adds a confidence between 100[%] and 50[%]. In this case, the confidence adding unit 202 adds a confidence closer to 100[%] the closer to 90 degrees the angle is, and adds a confidence closer to 50[%] the closer to 45 degrees the angle is. Furthermore, if the angle between the waist direction and the X-axis is between 90 degrees and 135 degrees, the confidence adding unit 202 adds a confidence between 100[%] and 50[%]. In this case, the confidence adding unit 202 adds a confidence closer to 100[%] the closer to 90 degrees the angle is, and adds a confidence closer to 50[%] the closer to 135 degrees the angle is.

[0143] Furthermore, when the angle between the waist direction and the X-axis is in the range of 0 degrees to less than 45 degrees, or in the range of more than 135 degrees to 180 degrees, it can be said that the orientation of the worker 40 shown in the working image, which is the original data of the analysis result, is not sideways but facing forward or backward. Therefore, the reliability adding unit 202 adds a low reliability to the analysis result when the angle between the waist direction and the X-axis is in the range of 0 degrees to less than 45 degrees, or in the range of more than 135 degrees to 180 degrees.

[0144] Furthermore, the confidence adding unit 202 adds a confidence between 50[%] and 0[%] for angles less than 45 degrees to 0 degrees and angles greater than 135 degrees to 180 degrees. Therefore, the modifying unit 102 adds a confidence closer to 50[%] the closer to 45 degrees or 135 degrees the angle is, and adds a confidence closer to 0[%] the closer to 0 degrees or 180 degrees the angle is.

[0145] There are cases where appropriate three-dimensional keypoint coordinates cannot be obtained, such as when the keypoints of the worker 40 in the work image are occluded by an obstruction. In such cases, the reliability adding unit 202 may add a low reliability to the analysis results analyzed using the occluded keypoints.

[0146] Specifically, when the reliability adding unit 202 determines, by general image processing, that at least one of the right wrist and right shoulder used to determine condition (A1) is occluded and the analysis result is a raised fist, it adds a low reliability to the analysis result. Also, when the reliability adding unit 202 determines, by general image processing, for a working image that is the original data of the analysis result, that at least one of the left wrist and left shoulder used to determine condition (A2) is occluded and the analysis result is a raised fist posture, it adds a low reliability to the analysis result. Also, when the reliability adding unit 202 determines, by general image processing, for a working image that is the original data of the analysis result, that at least one of the right hip, left hip, and base of the neck used to determine condition (A3) is occluded and the analysis result is a leaning forward posture, it adds a low reliability to the analysis result. Furthermore, when the reliability adding unit 202 determines, by general image processing, that at least one of the right hip, right knee, and right ankle used to determine condition (A4) is occluded for the working image that is the original data of the analysis result, and the analysis result is a knee-bent posture, it adds a low reliability to the analysis result. Furthermore, when the reliability adding unit 202 determines, by general image processing, that at least one of the left hip, left knee, and left ankle used to determine condition (A5) is occluded for the working image that is the original data of the analysis result, and the analysis result is a knee-bent posture, it adds a low reliability to the analysis result. The low reliability added when the key points used to determine conditions (A1) to (A5) are occluded is, for example, a reliability of 0[%].

[0147] On the other hand, even if an occluded keypoint exists, if the keypoint is not used in the analysis by the analysis unit 201, the confidence adding unit 202 does not add a low confidence to the occluded keypoint.

[0148] The analysis unit 201 may analyze postures of the worker 40 other than the raised fist posture, the bent forward posture, and the bent knee posture. The analysis unit 201 may analyze whether some or all of the raised fist posture, the bent forward posture, and the bent knee posture occur simultaneously. The analysis unit 201 may not analyze some or all of the raised fist posture, the bent forward posture, and the bent knee posture.

[0149] In the above, the case where the information terminal device 10 includes the correction unit 102 has been described, but this is not limited thereto. The correction unit 102 may be included in the control unit 200 of the server device 20. In this case, the communication unit 13 of the information terminal device 10 transmits information indicating an operation to correct the analysis result, which is a user operation accepted by the operation unit 11, to the server device 20. The correction unit 102 of the server device 20 corrects the analysis result based on the received information indicating the operation to correct the analysis result. In addition, the information processing system 1 may not have a function to correct the analysis result as long as it can present information indicating the analysis result and information indicating the reliability to the user. In this case, the display control unit 101 may not display the correction file execution screen P2 on the display unit 12, and the control unit 100 may not have the correction unit 102.

[0150] In the information processing system 1, the information terminal device 10 does not need to have high functionality compared to the server device 20 that performs the analysis process, but this is not limited to the above. The information terminal device 10 and the server device 20 may have the same functionality.

[0151] Although the display control unit 101 has been described as implementing a spreadsheet software when displaying the correction file execution screen P2, this is not limiting. When presenting the analysis results and reliability to the user and accepting corrections to the analysis results, the display control unit 101 and the correction unit 102 may execute dedicated software having these functions in addition to general-purpose software such as a spreadsheet software.

[0152] In the information processing system 1, the analysis of the work video and the addition of the reliability are performed while various information is transmitted and received between the information terminal device 10 and the server device 20 via the communication network 30. However, this is not limited to the above. The information processing system 1 may be realized by a single device. In this case, the single device includes an operation unit 11, a display unit 12, a control unit 200, and a trained model 251. The control unit 200 also includes a display control unit 101, a correction unit 102, an analysis unit 201, a reliability addition unit 202, and a load calculation unit 203. With this configuration, the single device does not need to transmit and receive the work video, information indicating the analysis result, information indicating the reliability, and load information via the communication network 30.

[0153] Although the work video has been described as a video generated by capturing images of a series of work performed by the worker 40 from start to finish, the present invention is not limited to this. The work video may be a video generated by capturing images of a part of a series of work performed by the worker 40. The work image may be a video generated by capturing images of a part of a series of work performed by the worker 40 multiple times.

[0154] The subtitle file generator 204 may generate a subtitle file that indicates some of the analysis result, the reliability, and the integrated value through subtitles. The operation unit 11 may be realized by a microphone and a voice recognition function, instead of a configuration realized by some or all of a keyboard, a mouse, a touch panel, etc. In this case, the user performs various operations by voice, and the information terminal device 10 recognizes the voice indicating the various operations picked up by the microphone by the voice recognition function.

[0155] In the above, the weighting is determined in advance for each posture of the worker 40, but this is not limiting. The weighting may be determined by the user of the information terminal device 10. In this case, the information terminal device 10 acquires information indicating the weighting designated by the user for each posture of the worker 40 that is the analysis target of the analysis unit 201. The information terminal device 10 transmits the information to the server device 20. The load calculation unit 203 of the server device 20 calculates the load of the posture of the worker 40 based on the received information.

[0156] The technical ideas that can be understood from the above-described embodiment and modified examples will be described. [Mode 1] An information processing system comprising an information terminal device and a server device which communicates with the information terminal device via a communication network, the server device comprising a first receiving unit which receives from the information terminal device a work video in which a plurality of work images generated by imaging a worker are successively displayed in chronological order, an analysis unit which identifies the orientation of the worker shown in each work image constituting the work video and analyzes the posture of the worker, a reliability adding unit which adds reliability to the analysis result of the analysis unit based on the orientation of the worker shown in the work image, and a first transmitting unit which transmits the analysis result and the reliability to the information terminal device, and the information terminal device comprises a second transmitting unit which transmits the work video to the server device, and a second receiving unit which receives the analysis result and the reliability from the server device.

[0157] [Aspect 2] The information terminal device of the information processing system described in [Aspect 1] includes an operation unit that accepts user operations, a display unit that displays the analysis result and the reliability, and a correction unit that corrects the analysis result based on the user's operation to correct the analysis result.

[0158] [Aspect 3] An information processing system according to aspect 1 or aspect 2, wherein the reliability adding unit adds a higher reliability the more sideways the orientation of the worker shown in the work image.

[0159] [Aspect 4] An information processing system described in any one of [Aspect 1] to [Aspect 3], wherein when the analysis result of the analysis unit indicates that the worker's posture cannot be analyzed based on the work image, the reliability addition unit assigns a lower reliability compared to when the analysis unit can analyze the worker's posture.

[0160] [Aspect 5] An information processing system described in any one of [Aspect 1] to [Aspect 4], wherein the worker's posture analyzed by the analysis unit includes a fist-raised posture in which the worker raises his / her fist, a bending forward posture in which the worker bends his / her waist, and a bending knee posture in which the worker bends his / her knees.

[0161] [Aspect 6] An information processing system described in any one of [Aspect 1] to [Aspect 5], wherein the analysis unit analyzes, based on the position of the worker's shoulder and the position of the worker's wrist or elbow shown in the work image, that if the position of the wrist or elbow is higher than the position of the shoulder, the worker is in a fist-raised pose.

[0162] [Aspect 7] An information processing system described in any one of [Aspect 1] to [Aspect 6], wherein the analysis unit analyzes that the worker is in a bending forward posture with the waist bent if the angle between a line segment connecting the center of the worker's waist and the base of the neck shown in the work image and a line segment connecting the worker's knee and the center of the waist on the back side of the worker is a predetermined angle or greater.

[0163] [Aspect 8] An information processing system described in any one of [Aspect 1] to [Aspect 7], wherein the analysis unit analyzes that the worker is in a knee bending posture with his / her knees bent if the angle is less than a predetermined angle based on the inner angle of the worker's left knee or the inner angle of the worker's right knee shown in the work image.

[0164] [Aspect 9] An information processing system described in any one of [Aspect 1] to [Aspect 8], wherein the server device further includes a subtitle file generation unit that generates a subtitle file that shows at least one of the analysis result of the analysis unit and the reliability added by the reliability addition unit as a subtitle for the work video, and the first transmission unit transmits the subtitle file generated by the subtitle file generation unit to the information terminal device.

[0165] [Aspect 10] The information terminal device comprises an operation unit that accepts user operations, a display unit that displays the analysis result and the reliability, and a correction unit that, when the operation unit accepts an operation by the user to correct the analysis result, corrects the analysis result based on the operation, and the correction unit corrects the subtitle file based on the user's operation, the information processing system described in [Aspect 9].

[0166] [Aspect 11] The server device further includes a load calculation unit that calculates a load for each posture of the worker based on a weighting that is predetermined for each posture of the worker and the analysis result, The information processing system according to any one of [Aspect 1] to [Aspect 10], wherein the first transmission unit transmits the load for each posture calculated by the load calculation unit to the information terminal device.

[0167] [Aspect 12] An information processing method for controlling a device equipped with a display unit, which, based on a work video in which a plurality of work images are generated by imaging a worker and are successive in time series, identifies the orientation of the worker shown in each work image constituting the work video and analyzes the posture of the worker, adds a degree of reliability to the analysis result of the posture of the worker based on the identified orientation of the worker shown in the work image, and displays the analysis result and the degree of reliability on the display unit. [Explanation of symbols]

[0168] P12, P12a...working image, P14, P14a...subtitles, P16...3D skeletal model image, P2...edit file execution screen, P20...pointer image, P21...analysis result table, P22, P22a, P22b, P22c, P22d, P22e...edit buttons, P23...confidence column, P24...fist-raising posture selection column, P25...forward bending posture selection column, P26...knee bending posture selection column, P27a...measurement cell, P27b...forced selection cell, P27c...exclusion Cell, P27d...estimated cell, P27e...missing cell, 1...information processing system, 10...information terminal device, 11...operation unit, 12...display unit, 13...communication unit, 20...server device, 23...communication unit, 30...communication network, 40...operator, 50...imaging unit, 100, 200...control unit, 101...display control unit, 102...correction unit, 150, 250...memory unit, 201...analysis unit, 202...reliability addition unit, 203...load calculation unit, 251...trained model.

Claims

1. An information processing system including an information terminal device and a server device that communicates with the information terminal device via a communication network, The server device includes: a first receiving unit that receives from the information terminal device a work video in which a plurality of work images generated by capturing images of a worker are successive in time series; an analysis unit that identifies a direction of the worker shown in each of the work images constituting the work video and analyzes a posture of the worker; a reliability adding unit that adds reliability to an analysis result of the analysis unit based on a direction of the worker shown in the work image; a first transmission unit that transmits the analysis result and the reliability to the information terminal device; The information terminal device A second transmission unit that transmits the work video to the server device; a second receiving unit that receives the analysis result and the reliability from the server device; Information processing system.

2. The information terminal device an operation unit that accepts user operations; A display unit that displays the analysis result and the reliability; a correction unit that corrects the analysis result based on an operation of the user to correct the analysis result, The information processing system according to claim 1 .

3. The reliability adding unit adds a higher reliability as the orientation of the worker shown in the work image becomes more horizontal. The information processing system according to claim 1 .

4. When the analysis result of the analysis unit indicates that the posture of the worker cannot be analyzed based on the work image, the reliability adding unit adds a lower reliability compared to when the analysis unit can analyze the posture of the worker. The information processing system according to claim 1 .

5. The posture of the worker analyzed by the analysis unit includes a fist-raising posture in which the worker raises his / her fist, a forward-leaning posture in which the worker bends his / her waist, and a knee-bending posture in which the worker bends his / her knees. The information processing system according to claim 1 .

6. The analysis unit analyzes that the worker is in a fist-raising posture when the position of the wrist or elbow is higher than the position of the shoulder based on the position of the shoulder of the worker and the position of the wrist or elbow of the worker shown in the work image. The information processing system according to claim 1 .

7. the analysis unit analyzes that the worker is in a forward bending posture with the waist bent, when an angle formed by a line segment connecting the center of the worker's waist and the base of the neck of the worker and a line segment connecting the knee of the worker and the center of the waist shown in the work image on the back side of the worker is equal to or greater than a predetermined angle; The information processing system according to claim 1 .

8. The analysis unit analyzes that the worker is in a knee bending posture when an angle of an inner side of the left knee of the worker or an angle of an inner side of the right knee of the worker shown in the work image is less than a predetermined angle. The information processing system according to claim 1 .

9. The server device includes: Further comprising a subtitle file generating unit that generates a subtitle file showing at least one of the analysis result of the analysis unit and the reliability added by the reliability adding unit as a subtitle of the work video, The first transmission unit transmits the subtitle file generated by the subtitle file generation unit to the information terminal device. The information processing system according to claim 1 .

10. The information terminal device an operation unit that accepts user operations; A display unit that displays the analysis result and the reliability; a correction unit that corrects the analysis result based on an operation received by the operation unit from the user to correct the analysis result, The modification unit modifies the subtitle file based on the operation of the user. The information processing system according to claim 9.

11. The server device includes: a load calculation unit that calculates a load for each posture of the worker based on a weighting that is predetermined for each posture of the worker and the analysis result, The first transmission unit transmits the load for each posture calculated by the load calculation unit to the information terminal device. The information processing system according to claim 1 .

12. An information processing method for controlling a device having a display unit, comprising: Based on a work video in which a plurality of work images generated by imaging a worker are successively produced in a time series, a direction of the worker shown in each of the work images constituting the work video is identified and a posture of the worker is analyzed; adding a reliability to the analysis result of the posture of the worker based on the orientation of the worker shown in the identified work image; displaying the analysis result and the reliability on the display unit; 23. An information processing method comprising:

Images