Information processing device, information processing method, and program

Abstract

To achieve a technique for easily grasping the deviation from an action plan about the action performed by a person.SOLUTION: An information processing device (1) includes: a detection part (11) which detects a person and an object based on the sensor information; a recognition part (12) which recognizes the action of the person based on relevance between the person and the object; a measurement part (13) which measures the time during which the person has continued the action based on a recognition result of the action; and a generation part (14) which generates information indicating the degree of deviation from an action plan related to the recognized action of the person based on the time during which the action has continued and the time during which the action included in the action should be continued.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] There is disclosed a technique for measuring the time taken by an operator to perform work based on an image of the operator working.

[0003] Patent Document 1 discloses a work management apparatus that performs image analysis on an image of an operator at the start of work and an image of the operator at the end of work, and measures the work time required for work related to one work item.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] For example, at a construction site, a work plan is made in advance, and work is carried out according to the plan. However, in reality, the work may be delayed or advanced compared to the plan. Therefore, it is required to grasp the deviation from the plan for the work actually performed. However, even if the work time of the work performed by the operator is measured using the technique described in Patent Document 1, it is difficult to grasp the deviation from the plan of the work.

[0006] One aspect of the present invention has been made in view of the above problems, and an example of the object is to provide a technique that can easily grasp the deviation from the action plan for the actions performed by a person.

Means for Solving the Problems

[0007] An information processing device according to one aspect of the present invention comprises: detection means for detecting people and objects based on sensor information; recognition means for recognizing the actions of a person based on the relationship between the person and the object; measurement means for measuring the time the person continued the action based on the recognition result of the action; and generation means for generating information indicating the degree of deviation of the recognized person's actions from the action plan, based on the measured time the action continued and the time the action should be continued, which is included in the action plan that has been planned for the action.

[0008] An information processing method according to one aspect of the present invention includes an information processing device detecting a person and an object based on sensor information; recognizing the person's actions based on the relationship between the person and the object; measuring the time the person continued the actions based on the recognition result of the actions; and generating information indicating the degree of deviation of the recognized person's actions from the action plan based on the measured time the actions continued and the time the actions should be continued, which is included in the action plan that has been planned for the actions.

[0009] A program according to one aspect of the present invention is a program that causes a computer to function as an information processing device, the program causing the computer to function as: a detection means for detecting people and objects based on sensor information; a recognition means for recognizing the actions of a person based on the relationship between the person and the object; a measurement means for measuring the time the person continued the actions based on the recognition result of the actions; and a generation means for generating information indicating the degree of deviation of the recognized actions of the person from the action plan, based on the measured time the actions continued and the time the actions should be continued, which is included in the action plan that has been planned for the actions. [Effects of the Invention]

[0010] According to one aspect of the present invention, it is possible to easily identify deviations between a person's actions and an action plan. [Brief explanation of the drawing]

[0011] [Figure 1] This is a block diagram showing the configuration of the information processing device 1 according to Embodiment 1 of the present invention. [Figure 2] This is a flowchart showing the flow of the information processing method S1 according to Embodiment 1 of the present invention. [Figure 3] This is a schematic diagram of an information processing system according to Embodiment 2 of the present invention. [Figure 4] This is a block diagram showing the configuration of an information processing system according to Embodiment 2 of the present invention. [Figure 5] This figure shows an example of behavioral identification information in Embodiment 2 of the present invention. [Figure 6] This figure shows an example of a table illustrating the recognition results in Embodiment 2 of the present invention. [Figure 7] This figure shows an example of a method for measuring time when the measurement unit according to Embodiment 2 of the present invention recognizes an undecided action. [Figure 8] This figure shows an example of measurement results in Embodiment 2 of the present invention, and an example of the duration for which work should be continued, as included in the work plan. [Figure 9] This figure shows an example of an image showing measurement results in Embodiment 2 of the present invention, an example of an image showing a work plan, and an example of an image showing the degree of deviation. [Figure 10] This figure shows another example of an image illustrating the degree of deviation in Embodiment 2 of the present invention. [Figure 11] This figure shows another example of an image illustrating the degree of deviation in Embodiment 2 of the present invention. [Figure 12] This figure shows an example of an image output by a display unit according to a modified version of the present invention. [Figure 13] This figure shows an example of an image included in annotation information in a modified version of the present invention. [Figure 14] This figure shows an example of information indicating people and objects included in annotation information in a modified version of the present invention. [Figure 15] This figure shows an example of the configuration of the estimation model used by the recognition unit according to a modified version of the present invention. [Figure 16]This is a diagram showing an example of related information included in annotation information in a modified example of the present invention. [Figure 17] This is a diagram showing another example of related information included in annotation information in a modified example of the present invention. [Figure 18] This is a diagram showing an example of a table indicating recognition results in a modified example of the present invention. [Figure 19] This is a block diagram showing an example of the hardware configuration of an information processing apparatus according to each exemplary embodiment of the present invention.

Embodiments for Carrying Out the Invention

[0012] 〔Exemplary Embodiment 1〕 The first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is a basic form for the exemplary embodiments described later.

[0013] (Overview of Information Processing Apparatus 1) The information processing apparatus 1 according to this exemplary embodiment is an apparatus that detects a person and an object based on sensor information, recognizes a person's action based on the relevance of the detected person and object, and measures the time for which the person has continued the action based on the recognition result. Further, the information processing apparatus 1 is an apparatus that generates information indicating the degree of deviation from an action plan regarding the recognized person's action based on the measured time and the time included in the action plan.

[0014] "Sensor information" refers to information output from one or more sensors. Examples of "sensor information" include an image output from a camera, information indicating the distance to an object output from a Lidar (Light Detection And Ranging), a distance image based on the output from a depth sensor, a temperature image based on the output from an infrared sensor, position information output using a beacon, a first-person perspective image of the wearer output from a wearable camera, and audio data output from a microphone array composed of a plurality of microphones.

[0015] The method by which the information processing device 1 detects people and objects based on sensor information is not limited, and known methods can be used. Examples of methods by which the information processing device 1 detects people and objects based on sensor information include methods based on image features such as HOG (Histograms of Oriented Gradients), color histograms, and shape; methods based on local features around feature points such as SIFT (Scale-Invariant Feature Transform); and methods using machine learning models such as Faster R-CNN (Regions with Convolutional Neural Networks).

[0016] Furthermore, in order to measure the duration of a person's actions, the information processing device 1 detects the same person and object detected at a given point in time at multiple points in time or over a predetermined period. In other words, the information processing device 1 detects the same person and object detected based on certain sensor information, based on sensor information at a different timing than the aforementioned sensor information. The method for determining whether the person and object detected by the information processing device 1 based on certain sensor information is the same as the person and object detected based on sensor information output from the sensor at a different timing than the aforementioned sensor information is not limited, and known methods can be used.

[0017] For example, the information processing device 1 may use a method based on the degree of overlap between the bounding rectangle of a person (or object) detected based on certain sensor information and the bounding rectangle of a person (or object) detected based on sensor information at a different timing than the given sensor information; a method based on the similarity between the features within the bounding rectangle of a person (or object) detected based on certain sensor information and the features of the bounding rectangle of a person (or object) detected based on sensor information at a different timing than the given sensor information; and a method using a machine learning model (e.g., DeepSort).

[0018] Furthermore, "relationship between people and objects" refers to the nature of the relationship between a person and an object. Examples of "relationship between people and objects" include whether a person and an object have a relationship, or whether a person and an object do not have a relationship.

[0019] One example of how the information processing device 1 recognizes a person's actions based on the relationship between a person and an object is that, if the relationship between the person and the object is such that the person and the object have a relationship, the device recognizes that the person is performing an action using the object. Another example of how the information processing device 1 recognizes a person's actions based on the relationship between a person and an object is that, if the relationship between the person and the object is such that the person and the object do not have a relationship, the device recognizes that the person is performing an action not using the object. Thus, the actions recognized by the information processing device 1 may include actions using objects and actions not using objects.

[0020] Furthermore, an "action plan" is information that outlines a planned action, including the duration for which that action should be continued. An example of "information indicating the degree of deviation from the action plan regarding the perceived behavior of a person" is information that shows the extent to which the duration of the perceived behavior of the person deviates from the duration for which the action should be continued as included in the action plan.

[0021] (Configuration of Information Processing Device 1) The configuration of the information processing device 1 will be described with reference to Figure 1. Figure 1 is a block diagram showing the configuration of the information processing device 1 according to this exemplary embodiment.

[0022] As shown in Figure 1, the information processing device 1 comprises a detection unit 11, a recognition unit 12, a measurement unit 13, and a generation unit 14. In this exemplary embodiment, the detection unit 11, the recognition unit 12, the measurement unit 13, and the generation unit 14 are configured to implement a detection means, a recognition means, a measurement means, and a generation means, respectively.

[0023] The detection unit 11 detects people and objects based on sensor information. The method by which the detection unit 11 detects people and objects based on sensor information is as described above. The detection unit 11 supplies information indicating the detected people and objects to the recognition unit 12.

[0024] The recognition unit 12 recognizes human behavior based on the relationship between people and objects detected by the detection unit 11. The method by which the recognition unit 12 recognizes human behavior based on the relationship between people and objects is as described above. The recognition unit 12 supplies the recognition result to the measurement unit 13.

[0025] The measurement unit 13 measures the time a person continues an action based on the recognition result of the recognition unit 12. The measurement unit 13 supplies the measurement result indicating the time a person continues an action to the generation unit 14.

[0026] The generation unit 14 generates information indicating the degree of deviation from the action plan regarding the recognized person's behavior, based on the duration of the behavior measured by the measurement unit 13 and the duration of the behavior that should be continued as included in the action plan for that behavior.

[0027] As described above, the information processing device 1 according to this exemplary embodiment employs a configuration comprising: a detection unit 11 that detects people and objects based on sensor information; a recognition unit 12 that recognizes a person's actions based on the relationship between the person and the object; a measurement unit 13 that measures the time a person continues an action based on the recognition result of the action; and a generation means that generates information indicating the degree of deviation from the action plan regarding the recognized person's actions, based on the measured duration of the action and the duration of the action that should be continued, which is included in the action plan that has been planned for that action.

[0028] Therefore, according to the information processing device 1 of this exemplary embodiment, information is generated that shows the degree of discrepancy between the time the recognized action was continued and the time the action should be continued as included in the action plan. This provides the effect of easily understanding the deviation between the action taken by a person and the action plan.

[0029] (Information processing method S1 flow) The flow of the information processing method S1 according to this exemplary embodiment will be explained with reference to Figure 2. Figure 2 is a flowchart showing the flow of the information processing method S1 according to this exemplary embodiment.

[0030] (Step S11) In step S11, the detection unit 11 detects people and objects based on sensor information. The detection unit 11 supplies information indicating the detected people and objects to the recognition unit 12.

[0031] (Step S12) In step S12, the recognition unit 12 recognizes the person's actions based on the relationship between the person and the object detected by the detection unit 11. The recognition unit 12 supplies the recognition result to the measurement unit 13.

[0032] (Step S13) In step S13, the measurement unit 13 measures the time the person continued the action based on the recognition result of the recognition unit 12. The measurement unit 13 supplies the measurement result indicating the time the person continued the action to the generation unit 14.

[0033] (Step S14) In step S14, the generation unit 14 generates information indicating the degree of deviation from the action plan regarding the recognized person's behavior, based on the duration of the behavior measured by the measurement unit 13 and the duration of the behavior that should be continued as included in the action plan that was planned for that behavior.

[0034] As described above, the information processing method S1 according to this exemplary embodiment employs a configuration in which a detection unit 11 detects people and objects based on sensor information, a recognition unit 12 recognizes a person's actions based on the relationship between people and objects, a measurement unit 13 measures the time a person continues an action based on the recognition result of the action, and a generation unit 14 generates information indicating the degree of deviation from the action plan regarding the recognized person's actions, based on the measured duration of the action and the time the action should be continued, which is included in the action plan that was planned for that action. For this reason, the information processing method S1 according to this exemplary embodiment can be used to obtain the same effects as the information processing device 1 described above.

[0035] [Exemplary Embodiment 2] A second exemplary embodiment of the present invention will be described in detail with reference to the drawings. Components having the same function as those described in Exemplary Embodiment 1 will be denoted by the same reference numerals, and their descriptions will be omitted as appropriate.

[0036] (Overview of Information Processing System 100) An overview of the information processing system 100 according to this exemplary embodiment will be described with reference to Figure 3. Figure 3 is a schematic diagram of the information processing system 100 according to this exemplary embodiment.

[0037] The information processing system 100 is a system that detects people and objects based on sensor information, recognizes people's actions based on the relationships between the detected people and objects, and measures the time that people continued their actions based on the recognition results. Furthermore, the information processing system 100 is a system that generates information indicating the degree of deviation of the recognized people's actions from the action plan, based on the measured time and the time included in the action plan.

[0038] As an example, the information processing system 100 is configured to include an information processing device 2, a camera 6, and a display device 8, as shown in Figure 3. In the information processing system 100, the information processing device 2 acquires images output from the camera 6, which photographs a construction site where people are working with excavators and the like, as sensor information. Hereafter, actions performed by people at a construction site will also be referred to as "work."

[0039] Furthermore, the information processing device 2 detects people and objects at the construction site based on the acquired images. In this exemplary embodiment, we will describe the case where the person is a worker and the object is a work object. The information processing device 2 then recognizes the work being performed by the worker based on the relationship between the detected worker and the work object, and measures the time the worker continued the work based on the recognition result.

[0040] Furthermore, the information processing device 2 either displays the generated deviation degree information on its own device or outputs it to the display device 8. Here, the display device 8 is a device that provides information to the user, and an example of this is a device that displays images. In the information processing system 100, as an example, as shown in Figure 3, the information processing device 2 outputs the deviation degree information to at least one of the display device 8a and the tablet 8b.

[0041] (Configuration of Information Processing System 100) The configuration of the information processing system 100 according to this exemplary embodiment will be described with reference to Figure 4. Figure 4 is a block diagram showing the configuration of the information processing system 100 according to this exemplary embodiment.

[0042] As shown in Figure 4, the information processing system 100 comprises an information processing device 2, a camera 6, and a display device 8. The information processing device 2, camera 6, and display device 8 are each connected to each other via a network so that they can communicate with one another. The specific configuration of the network is not limited to this exemplary embodiment, but as an example, a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public telephone network, a mobile data communication network, or a combination of these networks can be used.

[0043] (Configuration of Information Processing Device 2) As shown in Figure 4, the information processing device 2 comprises a control unit 10, a display device 17, a communication unit 18, and a storage unit 19.

[0044] The display device 17 is a device that displays the image indicated by the image signal supplied from the control unit 10.

[0045] The communication unit 18 is a communication module that communicates with other devices connected via the network. For example, the communication unit 18 outputs data supplied from the control unit 10 to the display device 8, or supplies data output from the camera 6 to the control unit 10.

[0046] The memory unit 19 stores data that the control unit 10 references. For example, the memory unit 19 stores sensor information, action plans, and action identification information, which will be described later.

[0047] (Functions of the control unit 10) The control unit 10 controls each component of the information processing device 2. As shown in Figure 4, the control unit 10 also includes a detection unit 11, a recognition unit 12, a measurement unit 13, a generation unit 14, a display unit 15, and an acquisition unit 16. In this exemplary embodiment, the detection unit 11, the recognition unit 12, the measurement unit 13, the generation unit 14, and the display unit 15 are configured to realize a detection means, a recognition means, a measurement means, a generation means, and an output means, respectively.

[0048] The detection unit 11 detects workers and work objects based on sensor information. As an example, the detection unit 11 can detect multiple people. The method by which the detection unit 11 detects workers and work objects based on sensor information is as described above. The detection unit 11 supplies information indicating the detected workers and work objects to the recognition unit 12. An example of the process by which the detection unit 11 detects workers and work objects will be described later.

[0049] The recognition unit 12 recognizes the worker's actions based on the relationship between the worker and the work object detected by the detection unit 11. For example, the recognition unit 12 recognizes multiple actions based on the relationship between the worker and the work object detected by the detection unit 11. Also, if the detection unit 11 detects multiple people, the recognition unit 12 recognizes the actions of each person. Furthermore, the tasks recognized by the recognition unit 12 are tasks that are included in one of multiple processes. In other words, each of the multiple tasks recognized by the recognition unit 12 is a task that is included in one of multiple processes. An example of how the recognition unit 12 recognizes the worker's actions based on the relationship between the worker and the work object will be described later. The recognition unit 12 stores the recognition results in the storage unit 19.

[0050] The measurement unit 13 measures the duration of an action performed by the worker based on the recognition results of the recognition unit 12. For example, if multiple actions are recognized by the recognition unit 12, the measurement unit 13 measures the duration of each of the multiple actions. Also, if actions are recognized for each person by the recognition unit 12, the measurement unit 13 measures the duration of that action for each person. The measurement unit 13 supplies the measurement results to the display unit 15. An example of how the measurement unit 13 measures the duration of an action performed by a worker will be described later.

[0051] The generation unit 14 generates information indicating the degree of deviation from the action plan for the recognized person's behavior, based on the duration of the behavior measured by the measurement unit 13 and the duration of the behavior that should be continued as included in the action plan for that behavior. For example, if the measurement unit 13 measures the duration of each of several behaviors, the generation unit 14 generates information indicating the degree of deviation based on the duration of each of the measured behaviors and the duration of each behavior that should be continued as included in the action plan for all of the behaviors. Also, if the measurement unit 13 measures the duration of each behavior for each person, the generation unit 14 generates information indicating the degree of deviation for each person. The generation unit 14 supplies the generated information indicating the degree of deviation to the display unit 15. An example of how the generation unit 14 calculates the degree of deviation will be described later.

[0052] The display unit 15 displays information indicating the degree of deviation generated by the generation unit 14. As an example, the display unit 15 displays the information indicating the degree of deviation as an image via the display device 17. As another example, the display unit 15 outputs the information indicating the degree of deviation to the display device 8 via the communication unit 18, and displays it as an image via the display device 8. An example of an image displayed by the display unit 15 will be described later.

[0053] The acquisition unit 16 acquires data supplied from the communication unit 18. An example of data acquired by the acquisition unit 16 is an image output from the camera 6. The acquisition unit 16 stores the acquired data in the storage unit 19.

[0054] (Camera 6 configuration) As shown in Figure 4, camera 6 includes a camera control unit 60, a camera communication unit 68, and an imaging unit 69.

[0055] The camera communication unit 68 is a communication module that communicates with other devices connected via a network. For example, the camera communication unit 68 outputs data supplied from the camera control unit 60 to the information processing device 2.

[0056] The imaging unit 69 is a device that captures subjects included in the field of view. For example, the imaging unit 69 captures a construction site that includes workers and work objects in its field of view. The imaging unit 69 supplies the captured image to the camera control unit 60.

[0057] The camera control unit 60 controls each component of the camera 6. Furthermore, as shown in Figure 4, the camera control unit 60 includes an image acquisition unit 61 and an image output unit 62.

[0058] The image acquisition unit 61 acquires the image supplied from the imaging unit 69. The image acquisition unit 61 supplies the acquired image to the image output unit 62.

[0059] The image output unit 62 outputs data via the camera communication unit 68. For example, the image output unit 62 outputs the image supplied from the image acquisition unit 61 to the information processing device 2 via the camera communication unit 68.

[0060] (Configuration of display device 8) As shown in Figure 4, the display device 8 comprises a display device control unit 80, a display device communication unit 88, and a display device display unit 89.

[0061] The display device communication unit 88 is a communication module that communicates with other devices connected via a network. For example, the display device communication unit 88 supplies data output from the information processing device 2 to the display device control unit 80.

[0062] The display unit 89 is a device that displays the image indicated by the image signal. The display unit 89 displays the image indicated by the image signal supplied from the display control unit 80.

[0063] The display device control unit 80 controls each component of the display device 8. Furthermore, as shown in Figure 4, the display device control unit 80 includes an information acquisition unit 81 and a display control unit 82.

[0064] The information acquisition unit 81 acquires information indicating the degree of deviation supplied from the display device communication unit 88. The information acquisition unit 81 supplies the acquired information indicating the degree of deviation to the display control unit 82.

[0065] The display control unit 82 supplies the information indicating the degree of deviation, which is provided by the information acquisition unit 81, to the display device display unit 89 as an image signal.

[0066] (Example of a process in which the detection unit 11 tracks the worker) As described in Exemplary Embodiment 1, the detection unit 11 detects the same person and object that was detected based on certain sensor information, based on sensor information at a different timing than the aforementioned sensor information. Below, an example of the process of detecting the same person that was detected based on certain sensor information, based on sensor information at a different timing than the aforementioned sensor information, will be described.

[0067] First, the detection unit 11 detects a person based on the image acquired at time (t-1). Here, the detection unit 11 assigns a detection ID (for example, a worker ID, which will be described later) to the detected person to distinguish them from other people.

[0068] Next, the detection unit 11 detects a person based on the image acquired at time (t). Then, the detection unit 11 determines whether the person detected based on the image acquired at time (t) matches the person detected in the image acquired at time (t-1) and to whom a detection ID has been assigned.

[0069] As an example, the detection unit 11 calculates a degree of overlap indicating the extent to which the bounding rectangle of a person to whom a detection ID has been assigned overlaps with the bounding rectangle of a person detected based on the image acquired at time (t). Examples of the degree of overlap of bounding rectangles include the degree to which the positions of the two bounding rectangles overlap, the degree to which the sizes of the two bounding rectangles overlap, and the degree to which the features of the person within the two bounding rectangles overlap.

[0070] If the detection unit 11 determines that the person detected based on the image acquired at time (t) matches the person to whom a detection ID has been assigned, it assigns the detection ID that was assigned to the person detected based on the image acquired at time (t-1) to the person detected in the image acquired at time (t). With this configuration, the detection unit 11 can track the same person across images acquired at different times.

[0071] (Example 1 of how the recognition unit 12 recognizes the worker's actions) One example of how the recognition unit 12 recognizes the worker's actions is a method in which the recognition unit 12 recognizes the worker's actions based on the worker's position and the position of the work object.

[0072] For example, if the distance between the worker's position and the work object's position is less than or equal to a predetermined length, the recognition unit 12 recognizes that the worker is performing work using the work object. For instance, if the distance between the worker's position and the cart's position is less than or equal to a predetermined length (e.g., 30 cm), the recognition unit 12 recognizes that the worker is performing transportation work using the cart.

[0073] As another example, if the worker's position and the work object's position overlap, the recognition unit 12 recognizes that the worker is performing work using the work object. For example, if the worker's position and the excavator's position overlap, the recognition unit 12 recognizes that the worker is performing excavation, which is work using the excavator.

[0074] In this way, the recognition unit 12 recognizes the worker's actions based on the worker's position and the position of the work object, thereby enabling accurate recognition of the worker's actions using the work object.

[0075] (Example 2 of how the recognition unit 12 recognizes the worker's actions) Another example of how the recognition unit 12 recognizes the worker's actions is a method in which the recognition unit 12 recognizes the worker's actions detected by the detection unit 11 by referring to action identification information that shows the relationship between the worker's characteristics in a predetermined action and the characteristics of the work object involved in that predetermined action. The action identification information will be explained with reference to Figure 5. Figure 5 shows an example of action identification information in this exemplary embodiment.

[0076] As shown in Figure 5, the behavior identification information shows the relationship between human characteristics (human shape and posture in Figure 5) in a predetermined action ("carrying" and "pointing and confirming" in Figure 5) and the characteristics of an object involved in that predetermined action ("cart" in Figure 5). The recognition unit 12 determines whether the human characteristics of the worker and the characteristics of the object detected by the detection unit 11 match the human characteristics and the object in the behavior identification information.

[0077] Furthermore, in behavior-specific information, multiple "person characteristics" may be associated with a given behavior. For example, as shown in Figure 5, in behavior-specific information, the shape of a person, the posture of a person, and HGO may be associated with a given behavior, "transportation," as "person characteristics."

[0078] Furthermore, as shown in Figure 5, in the behavior-specific information, the shape of the person, the posture of the person, and the HGO in the "person characteristics" may also be multiple. For example, as shown in Figure 5, in the behavior-specific information, a predetermined behavior "pointing and confirming" may be associated with the shape of a person pointing downwards to the right, the shape of a person with a changed pointing angle (the shape of a person pointing horizontally to the right), and the shape of a person with a changed pointing direction (the shape of a person pointing downwards to the left) as "person characteristics."

[0079] In addition to human shape, posture, and HOG, other examples of human characteristics in behavioral identification information include color and local features.

[0080] If the human characteristics and object characteristics detected by the detection unit 11 match the human characteristics and object in the behavior identification information, the recognition unit 12 recognizes that the behavior associated with the human characteristics and object in the behavior identification information is the work being performed by the worker.

[0081] On the other hand, if the human characteristics and object characteristics of the worker detected by the detection unit 11 do not match the human characteristics and object characteristics in the behavior identification information, the recognition unit 12 recognizes the worker's work as an undetermined action, indicating that it could not be identified. In other words, the recognition unit 12 recognizes the worker's action as an undetermined action if it is not one of a predetermined set of actions.

[0082] Furthermore, as shown in Figure 5, the behavior identification information includes behaviors associated with the object "cart," such as the behavior "transportation," and behaviors not associated with an object, such as the behavior "pointing and confirming." In other words, the behaviors recognized by the recognition unit 12 include both behaviors using objects and behaviors not using objects.

[0083] In addition to human shape and posture, examples of human characteristics in behavioral identification information include HOG, color, and local features.

[0084] In this way, the recognition unit 12 can accurately recognize the worker's actions by referring to action identification information that shows the relationship between the characteristics of the worker in a predetermined action and the characteristics of the work object involved in that predetermined action, and by recognizing the worker's actions detected by the detection unit 11.

[0085] Furthermore, with this configuration, the recognition unit 12 can accurately recognize the worker's actions even when the worker performs actions that do not involve the use of an object.

[0086] (Example 3 of how the recognition unit 12 recognizes the worker's actions) Another example of how the recognition unit 12 recognizes the worker's actions is to recognize the worker's actions based on the environment, in addition to the worker and the work object.

[0087] For example, if the recognition unit 12 recognizes concrete in addition to the worker and the work object as part of the environment, the recognition unit 12 will recognize that the worker is performing the task "leveling concrete".

[0088] In this way, the recognition unit 12 can accurately recognize the worker's actions based on the environment, in addition to the worker and the work object.

[0089] (Example 4 of how the recognition unit 12 recognizes the worker's actions) As yet another example of how the recognition unit 12 recognizes the worker's actions, if sensor information is acquired from each of several sensors, and the objects recognized by the recognition unit 12 differ depending on the sensor information, the recognition unit 12 may determine the recognized object based on a majority vote.

[0090] For example, if the object recognized based on the sensor information output from sensor 1 is object 1, the object recognized based on the sensor information output from sensor 2 is object 2, and the object recognized based on the sensor information output from sensor 3 is object 1, the recognition unit 12 recognizes that the object is object 1.

[0091] In this way, when the recognition unit 12 acquires sensor information from each of the multiple sensors, it determines which object to recognize based on a majority vote, thereby reducing misrecognition.

[0092] (Example 1 of how the measurement unit 13 measures time) An example of how the measurement unit 13 measures the time the worker continues to perform the task will be explained with reference to Figure 6. Figure 6 is a diagram showing an example of a table indicating the recognition results in this exemplary embodiment.

[0093] First, the recognition unit 12 stores the recognition result in the storage unit 19, associating it with the time of recognition, the worker ID used to distinguish the recognized worker from other workers, and the work content. In this configuration, as shown in Figure 6, the storage unit 19 stores multiple recognition results associated with time, worker ID, and work content. The table shown in Figure 6 is a table when the recognition unit 12 recognizes work content every second.

[0094] The measurement unit 13 refers to the table shown in Figure 6 and measures the time the worker continued to perform the task. For example, when measuring the time a worker with worker ID "B" continued performing task "Task 1a", the measurement unit 13 extracts recognition results rr1 to rr4 for worker ID "B". Next, the measurement unit 13 extracts the time "8:00:00" when task "Task 1a" was first recognized from recognition results rr1 to rr4. Furthermore, the measurement unit 13 extracts the time "9:00:00" when a task other than task "Task 1a" was first recognized from recognition results rr1 to rr4. Then, the measurement unit 13 measures the difference "1 hour" between the extracted times "8:00:00" and "9:00:00" as the time the worker with worker ID "B" continued performing task "Task 1a".

[0095] Furthermore, if the same work content is recognized discretely on the time axis by the recognition unit 12, the measurement unit 13 may measure the sum of the times measured for a particular worker to have continued a certain work as the total time the work was continued. For example, suppose a worker with worker ID "A" continues work content "Work 1a" for 3 hours, then continues work content "Work 1b" for 2 hours, and then continues work content "Work 1a" again for 1 hour. In this case, the measurement unit 13 measures that the time the worker with worker ID "A" continued work content "Work 1a" is 3 hours + 1 hour = 4 hours.

[0096] Alternatively, the measurement unit 13 may measure the time a worker continued a certain task by multiplying the number of times a worker performed a certain task by the time interval at which the recognition unit 12 performs the recognition process. For example, when measuring the time a worker with worker ID "B" continued task "task 1a", the measurement unit 13 extracts the recognition results for worker ID "B" in the table shown in Figure 6. Here, since the recognition process is performed at 1-second intervals, the extracted recognition results are 3601, from rr1 ​​(time 8:00:00) to rr4 (9:00:00). Next, the measurement unit 13 calculates the number of recognition results associated with task "task 1a", which is "3600". Then, the measurement unit 13 multiplies the number "3600" by the time interval "1 second" at which the recognition unit 12 performs the recognition process, and measures that the time a worker with worker ID "B" continued task "task 1a" is 1 hour. Furthermore, by multiplying the number of times a worker has performed a particular task by the time interval between recognition processes, it is possible to measure the time taken to perform that task, regardless of whether the recognition results for the same task performed by a worker are discrete in terms of time.

[0097] (Example 2 of how the measurement unit 13 measures time) An example of a method for measuring the time a worker continues working when the worker's action recognized by the recognition unit 12 is an undecided action will be explained with reference to Figure 7. Figure 7 is a diagram showing an example of a method for measuring time when the measurement unit 13 recognizes an undecided action according to this exemplary embodiment.

[0098] An example of an undetermined action recognized by the recognition unit 12 is an action taken when a worker transitions from one task to another. In this case, the measurement unit 13 may consider the undetermined action to be either the task the worker performed immediately before the undetermined action, or the task the worker performed immediately after the undetermined action. Alternatively, the measurement unit 13 may determine which task the undetermined action is based on the positional relationship between the worker and the work object related to the task performed immediately before or immediately after the undetermined action. In other words, the measurement unit 13 may measure the time the worker continued the undetermined action by adding the time the worker continued other actions that are different from the undetermined action.

[0099] As an example, let's describe a configuration in which the measurement unit 13 measures the time the worker continues to work based on the distance between the worker and the work object. The image in Figure 7, which includes the worker and the work object, shows the transition from transporting goods using a cart to excavating with an excavator. In the image in Figure 7, the worker has no relationship with the cart and no relationship with the excavator, so the recognition unit 12 recognizes it as an undecided action. In this case, the measurement unit 13 first calculates the distance between the worker and the work object.

[0100] As an example of how the measurement unit 13 calculates the distance between the worker and the work object, as shown in Figure 7, it can calculate the distance between the center of the worker's circumscribed rectangle and the center of the work object's circumscribed rectangle. If the distance between the center of the worker's circumscribed rectangle and the center of the cart's circumscribed rectangle is distance 1, and the distance between the center of the worker's circumscribed rectangle and the center of the excavator's circumscribed rectangle is distance 2, the measurement unit 13 measures the time as a transport operation for the period during which distance 2 is longer than distance 1 within the period of undecided actions. On the other hand, the measurement unit 13 measures the time as an excavation operation for the period during which distance 2 is longer than or equal to distance 1 within the period of undecided actions.

[0101] In this way, the measurement unit 13 measures the time spent by adding the time spent on an undecided action to the time spent on another action that is different from the undecided action. This allows for measurement of any period of time spent on any action, even if there was a period during which the worker's actions could not be recognized, thus enabling a more accurate understanding of the time a person spent on an action.

[0102] (Example of how the generation unit 14 calculates the degree of deviation) An example of how the generation unit 14 calculates the degree of deviation will be explained using Figure 8. Figure 8 shows an example of the measurement results in this exemplary embodiment and an example of the time during which the work should be continued, which is included in the work plan.

[0103] The upper part of Figure 8 shows an example of measurement results measured by the measurement unit 13. As shown in the upper part of Figure 8, the measurement results are associated with a start time indicating the time when the work began, an end time indicating the time when the work ended, and the work content. For example, measurement result mr1 indicates that work content "Work 1a" was performed from time "8:00:00" to "9:30:00".

[0104] The lower part of Figure 8 shows an example of the duration of work included in a work plan. As shown in the lower part of Figure 8, a work plan is associated with a start time indicating when the work should begin, an end time indicating when the work should end, and the work content. For example, work plan wp1 indicates that work content "Task 1a" should be performed between the times "8:00:00" and "9:00:00".

[0105] The generation unit 14 calculates the degree of deviation by referring to the diagram shown in Figure 8. As an example, the generation unit 14 calculates the difference between the work time measured by the measurement unit 13 and the time during which the work included in the work plan should be continued as the degree of deviation.

[0106] For example, when the generation unit 14 calculates the degree of deviation for the work content "task 1a", it extracts the measurement result mr1, which includes the work content "task 1a", from the upper part of Figure 8, and the work plan wp1, which includes the work content "task 1a", from the lower part of Figure 8. The generation unit 14 then calculates the degree of deviation as the difference of "30 minutes" between the work time in the measurement result mr1 (1 hour and 30 minutes from "8:00:00" to "9:30:00") and the work time in the work plan wp1 (1 hour from "8:00:00" to "9:00:00").

[0107] As another example, the generation unit 14 calculates the deviation degree as the ratio of the work time measured by the measurement unit 13 to the time during which the work included in the work plan should be continued.

[0108] For example, when the generation unit 14 calculates the degree of deviation for the work content "task 2a", it extracts the measurement result mr2, which includes the work content "task 2a", from the upper part of Figure 8, and the work plan wp2, which includes the work content "task 2a", from the lower part of Figure 8. The generation unit 14 then calculates the degree of deviation as the ratio of the work time in the measurement result mr2 (4 hours from "9:30:00" to "13:30:00") to the work time in the work plan wp2 (3 hours from "9:00:00" to "12:00:00"), which is "133%".

[0109] (Example 1 of an image showing the degree of deviation) An example of an image showing the degree of deviation displayed by the display unit 15 will be explained with reference to Figure 9. Figure 9 is a diagram showing an example of an image showing the measurement results, an example of an image showing the work plan, and an example of an image showing the degree of deviation in this exemplary embodiment.

[0110] The image shown at the top of Figure 9 is an example of an image showing the measurement results measured by the measurement unit 13. The measurement results shown at the top of Figure 9 include the work content, the process including the work content, the worker who performed the work, the time the work started, and the time the work ended. As an example, the image shown at the top of Figure 9 shows that the work content "Work 1a" included in process "Process 1" was performed by worker "Worker A" from time "8:00:00" to "10:15:00".

[0111] The image shown in the center of Figure 9 is an example of an image showing a work plan. As shown in the center of Figure 9, a work plan includes the work content, the process that includes the work content, the time the work should start, and the time the work should end. As an example, the image shown in the center of Figure 9 shows that the work content "Work 1a" included in process "Process 1" should be performed from the time "8:00:00" to "10:00:00".

[0112] The image showing the measurement results measured by the measurement unit 13 is the image shown at the top of Figure 9, and if the work plan is the image shown in the center of Figure 9, then, as an example, the generation unit 14 calculates the deviation of "15 minutes" for "task 1a" included in "process 1". The generation unit 14 then supplies information indicating the deviation to the display unit 15.

[0113] The display unit 15 refers to the information indicating the degree of deviation generated by the generation unit 14 and displays the degree of deviation. As an example, it displays the image shown at the bottom of Figure 9. The image displayed by the display unit 15 at the bottom of Figure 9 includes the measurement results measured by the measurement unit 13 and the work plan. The image at the bottom of Figure 9 shows that the work content "Work 1a" included in process "Process 1" is delayed by "15 minutes" as the degree of deviation.

[0114] Furthermore, when the generation unit 14 generates information indicating the degree of deviation for each process, the display unit 15 displays an image showing the degree of deviation for each process, as shown in the lower part of Figure 9. In the image shown in the lower part of Figure 9, the display unit 15 displays the degree of deviation between the measurement results measured by the measurement unit 13 and the work plan for each of the processes "Process 1" and "Process 2". The image shown in the lower part of Figure 9 shows that process "Process 1" is delayed. In this way, the information processing device 2 can easily grasp the deviation between the work performed by the worker and the work plan for each process.

[0115] Furthermore, when the generation unit 14 generates information indicating the degree of deviation for each task, the display unit 15 displays an image indicating the degree of deviation for each task, as shown in the lower part of Figure 9. In the image shown in the lower part of Figure 9, the display unit 15 displays the degree of deviation between the measurement results measured by the measurement unit 13 and the work plan for each of the tasks "Task 1a", "Task 1b", "Task 1c", "Task 2a", and "Task 2b". The image shown in the lower part of Figure 9 shows that task "Task 1a" is delayed. In this way, the information processing device 2 can easily grasp the deviation between the work performed by the worker and the work plan for each task.

[0116] In this way, the display unit 15 displays information indicating the degree of deviation. Therefore, the display unit 15 can inform the user whether the person's work is proceeding according to the action plan. In addition, the display unit 15 displays the time that the action should be continued, which is included in the action plan. Therefore, the display unit 15 can effectively inform the user whether the person's work is proceeding according to the action plan.

[0117] (Example 2 of an image showing the degree of deviation) Other examples of images showing the degree of deviation displayed by the display unit 15 will be described with reference to Figure 10. Figure 10 is a diagram showing other examples of images showing the degree of deviation in this exemplary embodiment.

[0118] If the generation unit 14 generates information indicating the degree of deviation for each worker, the display unit 15 may display an image indicating the degree of deviation for each worker, as shown in Figure 10.

[0119] In the image shown in Figure 10, the display unit 15 shows the degree of deviation between the measurement results measured by the measurement unit 13 and the work plan for each of the workers "Worker A," "Worker B," and "Worker C." The image shown at the bottom of Figure 9 shows that worker "Worker A" is behind schedule. In this way, the information processing device 2 can easily grasp the deviation between the work performed by each worker and the work plan.

[0120] Furthermore, in the image shown in Figure 10, the display unit 15 indicates that the work of workers "Worker B" and "Worker C" is also delayed due to a delay in work "Task 1a" performed by worker "Worker A". Therefore, the display unit 15 can inform the user which worker (or which task) is causing the delay.

[0121] (Example 3 of an image showing the degree of deviation) Other examples of images showing the degree of deviation displayed by the display unit 15 will be described with reference to Figure 11. Figure 11 is a diagram showing other examples of images showing the degree of deviation in this exemplary embodiment.

[0122] The display unit 15 may also display the degree of deviation using text. In the image shown at the top of Figure 11, the display unit 15 displays the degree of deviation "30 minutes" for process "process 1a" using text. Furthermore, the display unit 15 displays an image that includes the text "30 minutes behind schedule" indicating that the degree of deviation "30 minutes" means that the work is 30 minutes behind schedule.

[0123] Similarly, in the image shown at the top of Figure 11, the display unit 15 displays an image that includes the deviation degree "-30 minutes" for process "process 1c", and the text "30 minutes ahead of schedule" indicating that the work is 30 minutes ahead of the work plan.

[0124] Furthermore, if the generation unit 14 calculates the deviation degree as the ratio of the work time measured by the measurement unit 13 to the time that the work included in the work plan should be continued, the display unit 15 may display this ratio as text as the deviation degree. In the image shown in the center of Figure 11, the display unit 15 displays the deviation degree of "125%" for process "process 1a" as text. The display unit 15 also displays an image that includes the text "25% behind schedule" indicating that the deviation degree of "125%" means that the work is 25% behind schedule.

[0125] Similarly, in the image shown in the center of Figure 11, the display unit 15 displays an image that includes the deviation of "75%" for process "process 1c," and the text "25% ahead of schedule" indicating that the deviation of "75" means the work is 25% ahead of schedule.

[0126] The display unit 15 may also display the degree of deviation using a graph. In the image shown at the bottom of Figure 11, the display unit 15 displays the measurement time and the time that the work included in the work plan should be continued for each process using a bar graph. In the image shown at the bottom of Figure 11, the display unit 15 shows, as an example, that for process "process 1a", the measurement time is "15 minutes" longer than the time that the work included in the work plan should be continued, and therefore the work is "15 minutes" behind schedule.

[0127] (Effects of Information Processing Device 2) The information processing device 2 employs a configuration comprising: a detection unit 11 that detects workers and work objects based on images; a recognition unit 12 that recognizes the worker's work based on the relationship between the worker and the work object; a measurement unit 13 that measures the time the worker continued the work based on the work recognition result; and a generation means that generates information indicating the degree of deviation from the work plan for the recognized person's work, based on the measured time the work continued and the time the work should continue as included in the work plan that has been planned for that work.

[0128] Therefore, according to the information processing device 2 of this exemplary embodiment, information is generated that shows the degree of discrepancy between the time the recognized work was continued and the time the work should be continued as included in the work plan. This provides the effect of easily understanding the deviation between the work performed by the worker and the work plan.

[0129] (Modified version of display unit 15) The display unit 15 may be configured to acquire information from the display device 8 indicating user operations on the display device 8 and to perform processing according to that information. This configuration will be explained with reference to Figure 12. Figure 12 shows an example of an image output by the display unit 15 according to this modified example.

[0130] As an example, let's assume that the display unit 15 outputs the image shown in the upper part of Figure 12, and the display device 8 is displaying that image. The image shown in the upper part of Figure 12 indicates that "task 1a" of "process 1" is delayed by "15 minutes". In this state, when the display device 8 receives confirmation that the user has selected a period ("10:00" to "10:15") during which "task 1a" of "process 1" is delayed by "15 minutes", it outputs information indicating this operation to the information processing device 2.

[0131] When information is received from the display device 8, the display unit 15 of the information processing device 2 refers to that information and outputs an image. For example, the display unit 15 outputs an image from the start time "8:00" to the end time "10:15" of the task "Task 1a" indicated by the received information.

[0132] As another example, the display unit 15 divides the image to be displayed into predetermined time periods and outputs an image that includes images from the period indicated by the acquired information, and in which the task "Task 1a" indicated by the acquired information is recognized. In this configuration, for example, let's assume that the display unit 15 acquires information indicating that the user has selected a period ("10:00" to "10:15") in which "Task 1a" of "Process 1" is delayed by "15 minutes". In this case, if the display unit 15 divides the image to be displayed into 30-minute intervals, it outputs an image that is recognized as "Task 1a" and covers a 30-minute period including "10:00" to "10:15" (for example, "9:45" to "10:15").

[0133] Here, the image displayed by the display unit 15 may be a moving image or a still image.

[0134] (Modified version of detection unit 11) The detection unit 11 may use a machine learning model to detect workers and work objects. When the detection unit 11 uses a machine learning model, the annotation information used in the machine learning of the machine learning model will be described below.

[0135] The machine learning model used by the detection unit 11 is trained using annotation information, which consists of sensor information and information indicating people and objects indicated by the sensor information. Below, we will explain using the case where an image is used as the sensor information as an example, referring to Figures 13 and 14. Figure 13 is a diagram showing an example of image AP1 included in the annotation information in this modified example. Figure 14 is a diagram showing an example of information indicating people and objects included in the annotation information in this modified example.

[0136] As shown in Figure 13, in the annotation information for image AP1, rectangle numbers are assigned to the bounding rectangles of people and objects included as subjects in image AP1. For example, the bounding rectangle of the person pushing the cart is assigned rectangle number "1", and the bounding rectangle of the cart is assigned rectangle number "4".

[0137] Next, as shown in the upper part of Figure 14, the information indicating people and objects included in the annotation information is associated with a rectangle number, an object label indicating whether the bounding rectangle of that rectangle number contains a person or an object, and position information indicating the location of the bounding rectangle. For example, rectangle number "1", which is the bounding rectangle of a person pushing a cart, is associated with the object label "person" and position information "x11, y11, x12, y12, x13, y13, x14, y14" indicating the positions of the four corners of that bounding rectangle.

[0138] Furthermore, positional information may be represented by information indicating the position of one of the four corners of the bounding rectangle, along with the width and height of the bounding rectangle. For example, as shown in Figure 13, rectangle number "4", which is the bounding rectangle of the trolley, is associated with the object label "object", and as positional information, "x41, y41" indicating the position of one of the four corners of the bounding rectangle, and the width "w2" and height "h2" of the bounding rectangle.

[0139] In this way, by training the machine learning model used by the detection unit 11 with annotation information that combines sensor information and information indicating people and objects indicated by the sensor information, the machine learning model can be trained with high accuracy.

[0140] (Modified version of the recognition unit 12) The recognition unit 12 may use an estimation model to recognize the worker's actions detected by the detection unit 11.

[0141] One example of an estimation model used by the recognition unit 12 is a model that takes information indicating a person's characteristics and information regarding an object as input and outputs information indicating the relationship between the person and the object in a predetermined action.

[0142] In this configuration, the recognition unit 12 inputs information indicating the worker's characteristics detected by the detection unit 11 and information regarding the object detected by the detection unit 11 into the estimation model. The recognition unit 12 then recognizes the worker's actions by referring to information output from the estimation model, which indicates the relationship between the person and the object in a predetermined action.

[0143] For example, if the information output from the estimation model indicating the relationship between a person and an object indicates that a person and an object have a relationship, the recognition unit 12 recognizes that the person is performing an action using the object. For instance, if the information output from the estimation model indicating the relationship between a person and an object indicates that a person and a cart have a relationship, the recognition unit 12 recognizes that the person's work is the work of "transporting" using the cart.

[0144] In this way, the recognition unit 12 takes information indicating the characteristics of a person and information regarding an object as input, and uses a model that outputs information indicating the relationship between the person and the object in a predetermined action to recognize the worker's actions detected by the detection unit 11, thus enabling accurate recognition of the worker's actions.

[0145] (Estimation model used by the recognition unit 12) An example of the configuration of the estimation model used by the recognition unit 12 will be explained using Figure 15. Figure 15 is a diagram showing an example of the configuration of the estimation model used by the recognition unit 12 in this modified example.

[0146] As shown in Figure 15, the recognition unit 12 includes a feature extractor 121, an object feature extractor 122, a weight calculator 123, and a classifier 124.

[0147] The feature extractor 121 takes an image of a person containing a person as input and outputs the features of the person contained in that image. The recognition unit 12 uses multiple feature extractors 1211 to 121, each outputting different features of a person, as shown in Figure 15. N The configuration may include the following: For example, the feature extractor 1211 may output the shape features of a person included as a subject in the image of a person, and the feature extractor 1212 may output the posture features of a person included as a subject in the image of a person.

[0148] The object feature extractor 122 takes an image of an object in which an object is included as a subject as input, and outputs information about the object included as a subject in the image of the object. The information about the object output by the object feature extractor 122 may be the characteristics of the object, or it may be the object name that identifies the object. Furthermore, the object feature extractor 122 may also include location information indicating the location of the object in the information about the object it outputs.

[0149] The weight calculator 123 is used by feature extractors 1211 to 121. N Each feature output from the system is assigned a weight. In other words, the recognition unit 12 refers to multiple features to which weights have been assigned.

[0150] The classifier 124 takes the features output from the feature extractor 121 and the information about the object output from the object feature extractor 122 as input and outputs information indicating the relationship between the person and the object in a predetermined action. In other words, the classifier 124 outputs information indicating the relationship between the person and the object in a predetermined action based on the features output from the feature extractor 121 and the information about the object output from the object feature extractor 122.

[0151] Furthermore, as described above, the classifier 124 uses multiple feature extractors 1211 to 121 N Multiple features output from the device may also be used as input. In other words, the recognition unit 12 may be configured to recognize a person's behavior based on the relationship between multiple features of a person and information about an object. With this configuration, the recognition unit 12 can recognize a person's behavior with high accuracy.

[0152] (Machine learning of estimation models) This section describes the annotation information used in the machine learning of the estimation model used by the recognition unit 12.

[0153] The estimation model used by the recognition unit 12 is trained using annotation information, which consists of sensor information and related information indicating the relationship between people and objects shown by the sensor information. In the following, we will use the image AP1 shown in Figure 13 as an example of sensor information and explain with reference to Figures 13, 16, and 17. Figure 16 is a diagram showing an example of related information included in the annotation information in this modified example. Figure 17 is a diagram showing another example of related information included in the annotation information in this modified example.

[0154] As shown in Figure 13, in the annotation information for image AP1, rectangle numbers are assigned to the bounding rectangles of people and objects included as subjects in image AP1. For example, the bounding rectangle of the person pushing the cart is assigned rectangle number "1", and the bounding rectangle of the cart is assigned rectangle number "4". Also, as shown in Figure 13, rectangle numbers are assigned to the bounding rectangles of people and objects that have a relationship with each other. For example, the bounding rectangles of the person pushing the cart and the cart are assigned rectangle number "7".

[0155] Next, in the related information included in the annotation information, as shown in the upper part of Figure 16, the rectangle number and the group number indicating the relationship are associated. For example, in the upper part of Figure 16, the rectangle number "1" representing the person pushing the cart and the rectangle number "4" representing the cart are related, so both are associated with the group number "1".

[0156] Furthermore, as shown in the lower part of Figure 16, the related information may also be in matrix format. For example, in the lower part of Figure 16, the value at the intersection of the column (or row) of rectangle number "1" representing the person pushing the cart and the row (or column) of rectangle number "4" representing the cart is "1," indicating that there is a relationship.

[0157] Furthermore, the related information indicating the relationship between people and objects may include an action label indicating a person's actions and location information. For example, as shown in Figure 17, the configuration may include location information "x71, y71, x72, y72, x73, y73, x74, y74" indicating the positions of the person pushing the cart and the four corners of the cart's circumscribing rectangle, and an action label "transport" indicating the work of the person pushing the cart.

[0158] In this way, by training the estimation model used by the recognition unit 12 with annotation information that combines sensor information and related information indicating the relationship between people and objects shown by the sensor information, the estimation model can be trained with high accuracy.

[0159] (Modified example of the measurement unit 13) The measurement unit 13 may be configured to include the duration of a task in the duration of the immediately preceding task or the immediately following task if the duration of the task is less than a predetermined time (for example, 15 seconds, 1 minute, etc.). This configuration will be explained with reference to Figure 18. Figure 18 is a diagram showing an example of a table indicating the recognition results in this modified example.

[0160] If the recognition result by the recognition unit 12 is as shown in the table in Figure 18, the measurement unit 13 refers to the table in Figure 18 and measures the time the worker continued the work. As an example, the measurement unit 13 extracts recognition results rr5 to rr7 for worker ID "B" and measures the time the worker with worker ID "B" continued the work. Here, we assume a configuration where, if the duration of the work is less than 15 seconds, the duration of that work is included in the duration of the immediately preceding work. In the example of the table shown in Figure 18, the time the worker with worker ID "B" performed work content "Work 1b" was "4 seconds", from "9:00:00" to "9:00:04". Therefore, the measurement unit 13 includes the period from "9:00:00" to "9:00:04" in the duration of the immediately preceding work, work content "Work 1a".

[0161] If the duration of the task is short, the recognition unit 12 is more likely to misrecognize it. However, as in this configuration, if the duration of the task is less than a predetermined time, the measurement unit 13 includes the duration of the task in the duration of the task performed immediately before or immediately after, thereby allowing for a more accurate determination of the time a person continued an action, even if the recognition unit 12 misrecognizes it.

[0162] (Other variations) In addition, the present exemplary embodiment may output information indicating the degree of deviation in other ways, instead of displaying it as an image, or in addition to other ways. For example, the present exemplary embodiment may have an audio output unit instead of, or in addition to, the display unit 15. In this case, the audio output unit may output audio indicating the degree of deviation to an audio output device.

[0163] [Examples of implementation using software] Some or all of the functions of the information processing devices 1 and 2 may be implemented by hardware such as integrated circuits (IC chips), or by software.

[0164] In the latter case, the information processing units 1 and 2 are implemented by a computer that executes instructions for a program, which is software that implements each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 19. Computer C comprises at least one processor C1 and at least one memory C2. The memory C2 stores a program P that causes computer C to operate as information processing units 1 and 2. In computer C, the processor C1 reads program P from memory C2 and executes it, thereby implementing each function of information processing units 1 and 2.

[0165] Processor C1 can include, for example, a CPU (Central Processing Unit), GPU (Graphic Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating Point Number Processing Unit), PPU (Physics Processing Unit), microcontroller, or a combination thereof. Memory C2 can include, for example, flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination thereof.

[0166] Computer C may also be equipped with RAM (Random Access Memory) for loading program P at runtime and for temporarily storing various data. Furthermore, computer C may be equipped with communication interfaces for sending and receiving data with other devices. Additionally, computer C may be equipped with input / output interfaces for connecting input / output devices such as keyboards, mice, displays, and printers.

[0167] Furthermore, program P can be recorded on a non-temporary, tangible recording medium M that is readable by computer C. Such a recording medium M could be, for example, tape, disk, card, semiconductor memory, or programmable logic circuitry. Computer C can acquire program P via such a recording medium M. Program P can also be transmitted via a transmission medium. Such a transmission medium could be, for example, a communication network or broadcast waves. Computer C can also acquire program P via such a transmission medium.

[0168] [Additional Note 1] The present invention is not limited to the embodiments described above, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the embodiments described above are also included in the technical scope of the present invention.

[0169] [Additional Note 2] Some or all of the embodiments described above may also be described as follows. However, the present invention is not limited to the embodiments described below.

[0170] (Note 1) An information processing device comprising: detection means for detecting people and objects based on sensor information; recognition means for recognizing the actions of a person based on the relationship between the person and the object; measurement means for measuring the time the person continued the action based on the recognition result of the action; and generation means for generating information indicating the degree of deviation of the recognized person's actions from the action plan, based on the measured time the action continued and the time the action should continue, which is included in the action plan that has been planned for the action.

[0171] (Note 2) The information processing apparatus according to Appendix 1, further comprising a display means for displaying information indicating the degree of deviation.

[0172] (Note 3) The display means is an information processing device according to Appendix 2 that displays the time for which the action to be continued, which is included in the action plan.

[0173] (Note 4) The information processing apparatus according to any one of the appendices 1 to 3, wherein the recognition means recognizes a plurality of actions, the measurement means measures the duration of each of the plurality of actions, and the generation means generates information indicating the degree of deviation based on the measured duration of each of the plurality of actions and the duration for which each action should be continued, which is included in the action plan planned for the plurality of actions as a whole.

[0174] (Note 5) The generation means is an information processing device as described in Appendix 4, which generates information indicating the degree of deviation for each action.

[0175] (Note 6) The information processing apparatus according to Appendix 4 or 5, wherein each of the multiple actions recognized by the recognition means is an operation included in any of the multiple processes, and the generation means generates information indicating the degree of deviation for each process.

[0176] (Note 7) The information processing apparatus according to any one of the appendices 1 to 6, wherein the detection means detects multiple people, the recognition means recognizes the behavior of each person, the measurement means measures the duration of the behavior for each person, and the generation means generates information indicating the degree of deviation for each person.

[0177] (Note 8) An information processing method comprising: an information processing device detecting a person and an object based on sensor information; recognizing the person's actions based on the relationship between the person and the object; measuring the time the person continued the actions based on the recognition result of the actions; and generating information indicating the degree of deviation of the recognized person's actions from the action plan based on the measured time the actions continued and the time the actions should be continued, which is included in the action plan that has been planned for the actions.

[0178] (Note 9) A program that causes a computer to function as an information processing device, the program causing the computer to function as: detection means for detecting people and objects based on sensor information; recognition means for recognizing the actions of a person based on the relationship between the person and the object; measurement means for measuring the time the person continued the actions based on the recognition result of the actions; and generation means for generating information indicating the degree of deviation of the recognized actions of the person from the action plan, based on the measured time the actions continued and the time the actions should have continued, which is included in the action plan that was planned for the actions.

[0179] (Note 10) An information processing device comprising at least one processor, the processor performing: a detection process for detecting people and objects based on sensor information; a recognition process for recognizing the actions of the person based on the relationship between the person and the object; a measurement process for measuring the time the person continued the action based on the recognition result of the action; and a generation process for generating information indicating the degree of deviation of the recognized person's actions from the action plan, based on the measured time the action continued and the time the action should have continued, which is included in the action plan that has been planned for the action.

[0180] Furthermore, this information processing device may also be equipped with memory, and this memory may store a program that causes the processor to execute the detection process, the recognition process, the measurement process, and the generation process. This program may also be recorded on a computer-readable, non-temporary, tangible recording medium. [Explanation of symbols]

[0181] 1, 2 Information Processing Devices 8 Display device 11 Detection Unit 12 Recognition part 13 Measurement Unit 14 Generation part 15 Display 16 Acquisition Department 100 Information Processing Systems

Claims

1. A detection means for detecting people and objects based on sensor information, A recognition means for recognizing the actions of the person based on the relationship between the person and the object, A measuring means for measuring the time the person continued the action, based on the result of recognizing the person's actions, A generation means that generates information indicating the degree of deviation from the action plan regarding the perceived behavior of the person, based on the measured duration of the behavior and the duration of the behavior that the person should continue, which is included in the action plan for the behavior of the person whose behavior was recognized. Equipped with, The aforementioned detection means detects multiple people, The recognition means recognizes multiple actions for each of the multiple people, The measurement means measures the duration of each of the multiple actions for each of the multiple people, The generation means generates information indicating the degree of deviation for each of the multiple actions, based on the duration for which each of the multiple actions was performed for each of the multiple people as measured, and the duration for which each of the multiple people should perform each action, which is included in the action plan that was planned for the entirety of the multiple actions for each of the multiple people in which the multiple actions were recognized. Information processing device.

2. The system further includes a display means for displaying information indicating the degree of deviation. The information processing apparatus according to claim 1.

3. The display means displays the time for which the multiple people should continue the action, as included in the action plan. The information processing apparatus according to claim 2.

4. The generation means generates information indicating the degree of deviation for each of the plurality of actions, The information processing apparatus according to claim 1.

5. Each of the plurality of actions recognized by the recognition means is an operation included in any of the plurality of processes, The generation means generates information indicating the degree of deviation for each step. The information processing apparatus according to claim 1.

6. A detection means for detecting user operations on the information indicating the degree of deviation displayed on the display means, Information output means that, upon detecting an operation by the user, outputs information indicating the actions of the multiple people detected based on the sensor information corresponding to the detected information indicating the degree of deviation, The information processing apparatus according to claim 2, further comprising:

7. The information processing apparatus according to claim 6, wherein the sensor information includes an image output by an image sensor, and the information output means outputs images of the actions of the plurality of people captured by the image sensor as information indicating the actions of the people.

8. The information processing device is Based on sensor information, it detects people and objects, Recognizing the person's actions based on the relationship between the person and the object, Based on the recognition result of the aforementioned behavior, the time during which the person continued the aforementioned behavior is measured, Based on the measured duration of the aforementioned behavior and the duration for which the person should continue the aforementioned behavior as included in the planned action plan for the aforementioned person's behavior as perceived, information is generated indicating the degree of deviation from the action plan regarding the perceived behavior of the person. Includes, The detection of persons and objects includes detecting multiple persons, Recognizing the actions of the aforementioned persons includes recognizing multiple actions for each of the aforementioned multiple persons, The measurement described above includes measuring the duration of each of the aforementioned actions for each of the aforementioned individuals. The generation includes generating information indicating the degree of deviation for each of the multiple people, based on the duration for which each of the multiple actions was performed for each of the multiple people as measured, and the duration for which each of the multiple people should perform each action as part of the action plan that was planned for the entirety of the multiple actions for each of the multiple people in which the multiple actions were recognized. Information processing methods.

9. A program that causes a computer to function as an information processing device, The aforementioned program, the computer, A detection means for detecting people and objects based on sensor information, A recognition means for recognizing the actions of the person based on the relationship between the person and the object, A measuring means for measuring the time the person continued the action based on the recognition result of the action, A generation means that generates information indicating the degree of deviation from the action plan regarding the perceived behavior of the person, based on the measured duration of the behavior and the duration of the behavior that the person should continue, which is included in the action plan for the behavior of the person whose behavior was recognized. To make it function as, The aforementioned detection means detects multiple people, The recognition means recognizes multiple actions for each of the multiple people, The measurement means measures the duration of each of the multiple actions for each of the multiple people, The generation means generates information indicating the degree of deviation for each of the multiple actions, based on the duration for which each of the multiple actions was performed for each of the multiple people as measured, and the duration for which each of the multiple people should perform each action, which is included in the action plan that was planned for the entirety of the multiple actions for each of the multiple people in which the multiple actions were recognized. program.

Images