Action analysis system and action analysis method

Abstract

To analyze an action of an operator.SOLUTION: An action analysis system comprises: an input part which receives measurement data on an operator and an object; a person operation estimation part which estimates an operation of the operator based upon the measurement data; an object position / attitude estimation part which estimates a position and an attitude of the object based upon the measurement data; an object model arrangement part which extracts an object model in the same shape with the object whose position and attitude are estimated from an object model database, and arranges the extracted object model in a virtual space at the position and in the attitude that the object position / attitude estimation part estimates; a body model arrangement part which extracts body model data of the operator whose operation is estimated from a body model database, and arranges the extracted boy model in the virtual space according to the operation that the person operation estimation part estimates; a mutual relationship analysis part which analyzes mutual relationship between the body model and object model based upon the position of the body model and the position and attitude of the object model arranged in the virtual space; and a mutual relationship output part which outputs the analyzed mutual relationship.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a behavior analysis system. [Background technology]

[0002] Behavioral analysis methods that measure the movements of objects, reconstruct tasks three-dimensionally, and evaluate task efficiency in detail are used in various fields such as equipment design.

[0003] The following prior art exists as background art in this technical field: Patent Document 1 (JP 2020-187778 A) is an information processing device that stores the position and orientation of a 3D model and is capable of communicating with a display device that displays a mixed reality image in which a real image and a 3D model are superimposed based on the position and orientation of the 3D model in space, the information processing device comprising: a storage means that stores a first model of the 3D models to be worked on and a second model that indicates a tool for working on the first model in association with each other; a real object detection means that detects a real object; a determination means that determines whether the position of the real object detected by the real object detection means and the position of the first model are within a predetermined distance; and a determination means that determines whether the position of the real object detected by the real object detection means and the position of the first model are within a predetermined distance, and The information processing device is characterized by comprising: an identification means for identifying the second model; a determination means for determining the spatial position and orientation of the second model identified by the identification means in accordance with the position and orientation of the physical object; a movement detection means for detecting that the physical object detected by the detection means has made a predetermined movement; a contact determination means for, when the movement detection means detects that the physical object has made a predetermined movement, determining whether the physical object making the predetermined movement or the second model whose position and orientation has been determined by the determination means in accordance with the position and orientation of the physical object has come into contact with another three-dimensional model; and a display control means for, when the contact determination means determines that contact has occurred, controlling the display device to display information indicating the contact on the display screen. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2020-187778 Summary of the Invention [Problem to be solved by the invention]

[0005] In the invention described in Patent Document 1, there is room for improvement in the three-dimensional model for analyzing the behavior of a worker relative to an object.

[0006] The present invention aims to perform a detailed analysis of the actions that a worker performs on a work object. [Means for solving the problem]

[0007] A representative example of the invention disclosed in the present application is as follows: That is, a behavior analysis system for analyzing a relationship between a worker and an object, the behavior analysis system being configured by a computer having an arithmetic unit that executes predetermined arithmetic processing and a storage device connected to the arithmetic unit, the arithmetic unit having an input unit that receives input of measurement data of the worker and the object, a human behavior estimation unit that estimates the behavior of the worker based on the measurement data, an object posture estimation unit that estimates the position and posture of the object based on the measurement data, and the arithmetic unit extracting an object model having the same shape as the object whose position and posture is to be estimated from an object model database, and storing the extracted object model as a previous object. the computing device comprises an object model placement unit that places the body model in a virtual space at the position and orientation estimated by the object position and orientation estimation unit; a body model placement unit that extracts a body model of a worker whose movement is estimated from a body model database and places the extracted body model in the virtual space according to the movement estimated by the human movement estimation unit; a correlation analysis unit that analyzes the correlation between the body model and the object model based on the position of the body model and the position and orientation of the object model placed in the virtual space; and a correlation output unit that outputs the correlation analyzed by the correlation analysis unit. [Effects of the Invention]

[0008] According to one aspect of the present invention, it is possible to provide correlation data that analyzes in detail the actions that a worker performs on a work object. Other objects, configurations, and advantages will become apparent from the following description of the preferred embodiment of the present invention. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 2 is a functional block diagram illustrating functions of a computing device according to the first embodiment. [Figure 2] FIG. 3 is a diagram illustrating an example of body posture data according to the first embodiment. [Figure 3] FIG. 3 is a diagram illustrating an example of object position and orientation data according to the first embodiment. [Figure 4] FIG. 2 is a hardware configuration diagram of a computing device according to the first embodiment. [Figure 5] 2A to 2C are diagrams illustrating an example of a three-dimensional body model and a three-dimensional object model configured in a virtual space according to the first embodiment. [Figure 6] 10 is a flowchart illustrating a process performed by a correlation analysis unit according to the first embodiment. [Figure 7] FIG. 3 illustrates an example of interrelation data according to the first embodiment; [Figure 8] FIG. 11 is a diagram illustrating an example of simultaneous recognition probability data between three-dimensional object models according to the second embodiment. [Figure 9] FIG. 10 is a functional block diagram showing functions of a computing device according to a fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] -First embodiment- A first embodiment of a behavior analysis system will be described below with reference to FIGS.

[0011] 1 is a functional block diagram showing the functions of a computing device 10 of a behavior analysis system according to a first embodiment. However, Fig. 1 also shows data stored in an auxiliary storage device 4 of the computing device 10.

[0012] The computing device 10 of this embodiment reads measurement data related to multiple objects, analyzes the relationships between them, and outputs the results. The computing device 10 has, as functional blocks, a measurement data input unit 11, a human motion estimation unit 12, an object position / posture estimation unit 13, a body model placement unit 14, an object model placement unit 15, a correlation analysis unit 16, and a correlation output unit 17.

[0013] The auxiliary storage device 4 stores a three-dimensional body model database 41 and a three-dimensional object model database 42. The three-dimensional body model database 41 stores three-dimensional mesh models whose postures can be manipulated by input values of multiple joint points. The three-dimensional object model database 42 stores three-dimensional mesh models of objects to be recognized. For example, the three-dimensional mesh models are configured using three-dimensional CAD (Computer Aided Design) data of the device components to be worked on, or mesh models created by performing three-dimensional measurements of the objects to be recognized in advance. The input image 43, body posture data 44, and object position and posture data 45 are stored by one of the measurement data input unit 11, the human motion estimation unit 12, and the object position and posture estimation unit 13, and the correlation data 46 is generated and stored by the correlation analysis unit 16.

[0014] The measurement data input unit 11 reads measurement data including information for identifying the posture of a person or an object, identifies the data by its data format, and inputs the identified data to an appropriate functional block. Specifically, when image data captured by a camera or the like is input, the measurement data input unit 11 inputs the data to the human movement estimation unit 12 and the object position / posture estimation unit 13. When position data measured by an inertial sensor or the like is input, the measurement data input unit 11 inputs the data to the body model placement unit 14 or the object model placement unit 15 according to the model corresponding to the position data. The image data input to the human movement estimation unit 12 and the object position / posture estimation unit 13 may be an RGB image including color information or a Depth image including distance information.

[0015] The human motion estimation unit 12 detects the body in the input image data, estimates motions represented by the time-series positions in virtual three-dimensional space of the detected feature points representing the body, such as joint points of the body, and stores the motions in body posture data 44. The three-dimensional coordinates of the feature points shown in FIG. 2 are then input as body posture data to the body model placement unit 14. The body posture data input to the body model placement unit 14 does not have to be data of the entire body, but may be data of a part of the body, such as a hand or a foot. Specifically, when using images from a camera mounted on an HMD (Head Mounted Display) worn by the worker, the posture data of the worker's arms is mainly estimated, so it is preferable to input posture data of the arms.

[0016] When image data is input to the human motion estimation unit 12, a machine learning model is used to infer body posture from the image. There are two main processing procedures: top-down and bottom-up. The top-down method extracts areas where people are present in the image and then infers the posture of each individual within each area. This allows for highly accurate posture inference even from images containing multiple people. On the other hand, the bottom-up method infers the positions of all body feature points from the entire image and then infers the posture connecting these feature points. While this method requires less computational effort than the top-down method, inference accuracy tends to decrease in images containing multiple people. The processing procedures of these methods are interchangeable depending on the purpose. Specifically, top-down processing is used for non-real-time applications, such as evaluating work quality after completion of a task, when multiple workers are photographed from a remote location with a camera. Meanwhile, bottom-up processing is used for real-time applications, such as visually presenting work analysis results to workers in real time, when only the bodies of the workers in question are visible on the camera.

[0017] When the input data includes a depth image, the human motion estimation unit 12 infers a two-dimensional posture from the RGB image, and then converts the two-dimensional posture into a three-dimensional posture by acquiring distance information of corresponding pixels from the depth image. On the other hand, when the input data is only an RGB image, the three-dimensional posture is inferred using a machine learning model that has learned to convert from two-dimensional posture to three-dimensional posture.

[0018] The object position and orientation estimation unit 13 extracts from the object 3D model database 42 three-dimensional object models whose shapes are similar to those of the objects in the input image data, and calculates the position and orientation data of each object using the extracted three-dimensional object models. As shown in Fig. 3, the object position and orientation data is represented by the X-axis coordinates, Y-axis coordinates, Z-axis coordinates, X-axis rotation, Y-axis rotation, and Z-axis rotation associated with each model ID. Furthermore, the model IDs and three-dimensional object models are associated with each other.

[0019] The object position and orientation estimation unit 13 infers object position and orientation data from RGB images and depth images generated using an object model stored in the object 3D model database 42, using a machine learning model trained using one or both of the images according to the actual input data format. Specifically, it extracts multiple feature points that are the greatest distance from each other on the surface of the object model, determines the 3D coordinates of each feature point from the image data, and calculates the object position and orientation so as to minimize the average distance error with each feature point in the object model. The accuracy of object position and orientation estimation can be improved by including processed images, in which image data of the object model photographed from various distances and directions is combined with images of an actual work site as the background, as training data for training the machine learning model.

[0020] When the image data input to the human movement estimation unit 12 and the object position / posture estimation unit 13 is RGB image data that does not include distance information, a general camera can be used as the device used for measurement, thereby providing an inexpensive and lightweight system. On the other hand, adding a depth image requires stereo vision using multiple cameras or a distance measurement function using a time-of-flight method, but using the depth image can improve the accuracy of estimating the posture of the body and object. Therefore, to be able to handle image input in either data format, the human movement estimation unit 12 and the object position / posture estimation unit 13 should preferably have a posture estimation model trained on RGB images and a posture estimation model trained on depth images.

[0021] The body model placement unit 14 extracts a three-dimensional body model from the three-dimensional body model database 41, and places each sub-model constituting the three-dimensional body model in the virtual three-dimensional space according to the coordinates of each feature point stored in the body posture data 44. Note that sub-models including feature points for which coordinate information is not stored because they are not shown in the input image 43 are hidden, and only the postures of the other parts are placed.

[0022] The object model placement unit 15 reads the object position and orientation data 45, extracts the 3D object model with the model ID corresponding to the position and orientation data from the 3D object model database 42, and places the 3D model of each object in the virtual 3D space according to the orientation corresponding to each model ID. If the input data is a continuous data sequence such as a video, an object model is placed for each frame image to avoid placing multiple identical objects at the same time and to express the object movement. However, if multiple objects of the same shape are captured in the same frame image, multiple object models corresponding to the same model ID will be placed simultaneously.

[0023] The interrelationship analysis unit 16 analyzes the relationship between the 3D body model and the 3D object model based on the position of the body model placed in the virtual 3D space and the positions and postures of the object models placed in the virtual 3D space. Specifically, it calculates whether or not there is a collision between the 3D body model and the 3D object model, and calculates whether or not there is contact with the object corresponding to each input data, and the contact location. Furthermore, it compares the input data with the frame number and time in the video, and analyzes the dynamic relationship of the object corresponding to the input data.

[0024] The interrelation output unit 17 outputs the body posture data 44, the object position and posture data 45, and the interrelation data to the input / output device 5 or the communication device 6 as the results of the analysis by the interrelation analysis unit 16.

[0025] FIG. 4 is a diagram showing the hardware configuration of the arithmetic device 10. As shown in FIG.

[0026] The computing device 10 has a CPU 1 which is a central processing unit, a ROM 2 which is a storage device from which data is read, a RAM 3 which is a storage device from which data can be read and written, an auxiliary storage device 4 which is a non-volatile storage device, an input / output device 5 which is a user interface, and a communication device 6 which controls communication with other devices. The CPU 1 deploys a program stored in the ROM 2 into the RAM 3 and executes it, thereby realizing a measurement data input unit 11, a human motion estimation unit 12, an object position / posture estimation unit 13, a body model placement unit 14, an object model placement unit 15, and a correlation analysis unit 16.

[0027] The ROM 2 stores unchanging programs (e.g., BIOS), etc. The RAM 3 is a high-speed, volatile storage element such as a DRAM (Dynamic Random Access Memory), and temporarily stores programs executed by the arithmetic device 10 and data used when the programs are executed.

[0028] The auxiliary storage device 4 is a large-capacity non-volatile storage device such as a magnetic storage device (HDD), a flash memory (SSD), etc. The auxiliary storage device 4 also stores data used by the CPU 1 when executing a program (for example, a three-dimensional body model database 41, a three-dimensional object model database 42, an input image 43, body posture data 44, object position and posture data 45, and correlation data 46), and the program executed by the CPU 1.

[0029] The input / output device 5 includes an interface such as a keyboard or a mouse that receives input from an operator, and an interface such as a display device or a wearable device (e.g., VR goggles) that outputs the results of program execution as images in a format that can be viewed by a user. Note that a terminal (not shown) connected to the arithmetic device 10 via a network may provide the input / output device 5. In this case, the arithmetic device 10 may have a web server function, and the terminal may access the arithmetic device 10 using a predetermined protocol (e.g., http).

[0030] The communication device 6 is a network interface device that controls communication with other devices in accordance with a predetermined protocol.

[0031] The arithmetic device 10 may be realized by a field programmable gate array (FPGA), which is a rewritable logic circuit, or an application specific integrated circuit (ASIC), which is an integrated circuit for a specific application, instead of the combination of the CPU 1, the ROM 2, and the RAM 3. Furthermore, the arithmetic device 10 may be realized by a combination of different configurations, for example, a combination of the CPU 1, the ROM 2, the RAM 3, and the FPGA, instead of the combination of the CPU 1, the ROM 2, and the RAM 3.

[0032] The program executed by the CPU 1 is provided to the arithmetic device 10 via a removable medium (such as a CD-ROM or flash memory) or a network, and is stored in a non-volatile auxiliary storage device 4, which is a non-transitory storage medium. For this reason, the arithmetic device 10 preferably has an interface for reading data from removable media.

[0033] The computing device 10 is a computer system configured on a single physical computer or on multiple logically or physically configured computers, and may operate on a virtual computer constructed on multiple physical computer resources. For example, each functional unit may operate on a separate physical or logical computer, or multiple functional units may be combined to operate on a single physical or logical computer.

[0034] FIG. 5 is a diagram showing an example of a three-dimensional body model and a three-dimensional object model constructed in a virtual space by the body model placement unit 14 and the object model placement unit 15, and shows the case where the object model OM is held by the right hand model RHM and touched by the left hand model LHM.

[0035] The interrelationship analysis unit 16 converts the coordinates of each feature point of the right hand model RHM and the left hand model LHM into the posture coordinate system of the object model OM. Next, if each body model is in contact with the object model OM and the coordinates of the feature points belonging to the body model do not move in the object model posture coordinate system, it is determined that the body model is grasping the object model OM. Next, if the coordinates of the feature points belonging to the body model move in the object model posture coordinate system, the body model trajectory information TR is recorded.

[0036] FIG. 6 is a flowchart showing the processing performed by the interrelationship analysis unit 16.

[0037] Before the interrelationship analysis unit 16 starts processing, the body model placement unit 14 and the object model placement unit 15 have completed placement of the three-dimensional body model and the three-dimensional object model in the virtual space. First, in step S101, the interrelationship analysis unit 16 performs collision detection between the body model and the object model placed in the virtual space, and calculates the presence or absence of contact and the contact location. In step S102, the interrelationship analysis unit 16 converts the feature point coordinates of the body model into object model posture coordinates. In step S103, the interrelationship analysis unit 16 determines whether the body model is gripping the object model. The grip detection method described with reference to FIG. 5 can be used. In step S104, the interrelationship analysis unit 16 stores the contact detection result, the contact location, and the grip detection result as interrelationship data 46, and ends the processing shown in FIG. 6.

[0038] The analysis method executed by the interrelationship analysis unit 16 in step S103 may be preset according to the work content. For example, in a carrying task, the interrelationship data 46 may be calculated based on the correlation coefficient between time-series data on the coordinates of feature points, such as joints, constituting a body model and time-series data on the coordinates of feature points of an object model. Alternatively, an analysis method may be employed in which, at a time when the correlation coefficient of the feature point coordinates is equal to or greater than a predetermined threshold, it is determined that the worker is grasping an object with a body model part, such as a hand, that includes the feature points. The analysis method may further calculate the carrying quality using the interrelationship data 46 obtained by comparing the acceleration of the body model with the acceleration of the object model being grasped by the body model, or by determining contact between the body model and the object model. Alternatively, an analysis method may be employed in which, in a screw tightening task, the interrelationship data 46 is calculated based on the amount of rotation or angular velocity of a feature point included in a hand part of the body model that has moved around the central axis of a screw or a screw hole in an object model that includes the screw hole, and the interrelationship data 46 is used to determine the work status, such as temporary tightening or retightening of the screw. Furthermore, in assembly work, when a hand part of a body model comes into contact with a certain object model, an analysis method can be employed in which the shape of the object model is analyzed to determine whether it is in a state that is a prerequisite for the work, for example, a state in which another object model is assembled, and correlation data 46 is calculated to determine whether the assembly order is correct. In wiping work, an analysis method can be employed in which correlation data 46 is calculated based on the surface of the object model that is wiped and that comes into contact with the hand part of the body model, to visualize the worked surface and quantify the progress of the work. In inspection work, an analysis method can be employed in which correlation data 46 is calculated based on contact between the hand part of the body model and a part of the object model registered as a palpation confirmation part, or based on the fact that when the hand part of the body model is in a pointing shape, the half line on which the feature points constituting the index finger lie is directed toward a part registered as a visual confirmation part.

[0039] The correlation data 46 stored in step S103 is output by the correlation output unit 17 to the input / output device 5 or the communication device 6 along with the body posture data 44 and the object position and posture data 45. FIG. 7 illustrates a continuous data sequence, such as a video, input by the measurement data input unit 11. It also illustrates an example of the correlation data 46 output when the preset analysis method is the contact determination shown in FIG. 5. The correlation data 46 includes the input data number determined to be in contact, a pair of the contacting object model ID (OM0) and the body model (LHM0, LHM1), and three-dimensional position coordinates indicating the contact position. The origin of the three-dimensional position coordinates is recorded uniformly in either the virtual space coordinate system or the object model posture coordinate system, but the three-dimensional position coordinates can be converted between the coordinate systems using the object position and posture data 45. By combining the output data output by the correlation output unit 17 with a three-dimensional mesh model stored in the object three-dimensional model database 42, the contact position can be visualized, for example, by rewriting the texture information in the three-dimensional mesh model with the contact position coordinate information.

[0040] According to the first embodiment described above, the following advantageous effects can be obtained.

[0041] (1) When image data is input to the measurement data input unit 11, even if the object to be worked on is partially hidden by the body, it can be reconstructed as a three-dimensional object model stored in the three-dimensional object model database 42, thereby providing a virtual space environment in which the shape of the object in the blind spot area can be complemented and the work content can be visually confirmed.

[0042] (2) It is possible to provide correlation data 46 that can be used to determine collisions between the three-dimensional object model and the three-dimensional body model, and to calculate the trajectory of the three-dimensional body model in the object model posture coordinate system, thereby analyzing in detail the actions that the worker performs on the object being worked on, such as contact and grasping.

[0043] (3) When analyzing worker behavior, it is possible to evaluate work on objects that is difficult to judge based on human movements alone. For example, it is possible to evaluate the safety of work by detecting events such as the acceleration of each part of an object during transportation and collisions between objects.

[0044] -Second embodiment- A second embodiment of a behavior analysis system will be described with reference to FIG. 7. In the following description, the same components as those in the first embodiment are assigned the same reference numerals, and differences will mainly be described. Points that are not particularly described are the same as those in the first embodiment. This embodiment differs from the first embodiment in that a simultaneous recognition probability between three-dimensional object models is used to extract a three-dimensional object model with the same shape as an object whose position and orientation are to be estimated.

[0045] FIG. 7 is a diagram illustrating an example of simultaneous recognition probability data between three-dimensional object models according to the second embodiment.

[0046] The simultaneous recognition probability data records the probability that objects will be simultaneously recognized in image data input to the object position and orientation estimation unit 13 for a combination of two model IDs registered in the object 3D model database 42.

[0047] The simultaneous recognition probability is calculated by counting the number of pairs of model IDs that are simultaneously recognized in one input image data using history information estimated by the object position and orientation estimation unit 13. For example, if, among 100 pieces of image data, three-dimensional object models with model IDs 0 and 1 are simultaneously recognized in 80 pieces of image data, the simultaneous recognition probability is calculated to be 0.8.

[0048] On the other hand, if it is difficult to obtain a sufficient amount of image data in advance, the three-dimensional object models registered in the three-dimensional object model database 42 are arranged in a virtual space in a typical positional relationship, the distance of the objects is calculated for each pair of model IDs, and the simultaneous recognition probability data is calculated by normalizing the relationship so that the simultaneous recognition probability for the pair of model IDs with the closest objects is 1 and the simultaneous recognition probability for the pair of model IDs with the farthest objects is 0. For example, a typical positional relationship is the arrangement of device components when they are assembled.

[0049] According to the second embodiment described above, the following advantageous effects can be obtained.

[0050] (1) Even if the number of 3D object models registered in the 3D object model database 42 increases, the system prioritizes recognition attempts using 3D object models with a high probability of simultaneous recognition with the 3D object model that initially recognized the object, thereby enabling efficient object recognition and speeding up the system. Furthermore, even when the system speeds up by interrupting processing after a predetermined number of object recognition attempts, a decrease in recognition accuracy can be suppressed.

[0051] (2) When there is a similar object model with a similar shape among the three-dimensional object models registered in the three-dimensional object model database 42, and when there is a similar object model with a higher simultaneous recognition probability than another three-dimensional object model recognized at the same time, the recognition accuracy can be improved by interchanging the recognition order with the similar object model. For example, when there are object models A and B with high erroneous recognition rates relative to each other in object position and orientation estimation, if object models A, C, and D are recognized simultaneously in an input image, the average simultaneous recognition probability PA between object models AC and object models AD is compared with the average simultaneous recognition probability PB between object models BC and object models BD, and if the average simultaneous recognition probability PB is greater, the recognized object model A is replaced with object model B and the object position and orientation estimation process is performed, thereby improving the recognition accuracy compared to recognition using a single object model.

[0052] -Third embodiment- A third embodiment of a behavior analysis system will be described. In the following description, the same components as those in the first embodiment will be assigned the same reference numerals, and differences will be mainly described. Points that are not particularly described are the same as those in the first embodiment. This embodiment differs from the first embodiment mainly in that the body model placement unit 14 and the object model placement unit 15 use body model and object model data placed in a virtual space to correct possible object positions and postures.

[0053] In the body model placement unit 14 and the object model placement unit 15, when the meshes constituting the body models and object models placed in the virtual space intersect with each other, a posture with a small average distance to the vertices constituting the object model placed in the initial posture is searched for among postures that reduce the amount of mesh intersection. Next, when a posture that avoids mesh intersection is obtained, it is replaced with the initial posture. This improves the accuracy of object position and posture estimation.

[0054] For example, when estimating the pose of a long rod-shaped object from an image captured with a hand in front of it, if the area obscured by the hand is the center of the rod-shaped object and both ends of the object are visible in the image, the object position and pose can be estimated from the feature points on both ends, with little impact on estimation accuracy. On the other hand, if the area obscured by the hand is on one side of the rod-shaped object, including its end, the pose must be estimated from the shape of the other end that is visible, and angular error has a significant impact on pose accuracy. In this case, if a situation that would normally not occur (for example, a situation where the object model slips into the body model) is created in the virtual space, the analysis of the correlation will be inaccurate.

[0055] Therefore, the posture of the object model is corrected to eliminate the intrusion of the object model into the body model. Specifically, a polygon through which a closed curve where the body model and object model intersect is extracted, and all polygons that are continuous with that polygon and exist inside the body model are extracted. Then, of the vertices that make up the extracted polygon, the vertex farthest from the surface on the other side is searched for. The coordinate point P at which the vertex V farthest from the surface reaches the outside of the body model in the shortest distance is calculated, and a new object position and posture is calculated in which the distances of the other vertex positions to the vertex positions before correction are shortest, under the condition that the positions of vertex V and coordinate point P match.

[0056] According to the third embodiment described above, even when most of an object in input image data is hidden by another object or the human body, the accuracy of estimating the object position and orientation can be improved.

[0057] -Fourth embodiment- A fourth embodiment of a behavior analysis system will be described with reference to FIG. 8. In the following description, the same components as those in the first embodiment are designated by the same reference numerals, and differences will be mainly described. Points that are not particularly described are the same as those in the first embodiment. This embodiment differs from the first embodiment mainly in that interrelation data 46 generated from multiple input data is stored in a behavior analysis database 47 by task content.

[0058] 8 is a functional block diagram showing the functions of the arithmetic device 10 of the behavior analysis system according to the fourth embodiment. Note that Fig. 8 also shows data stored in the auxiliary storage device 4 of the arithmetic device 10.

[0059] The computing device 10 of this embodiment has an activity classification unit 18, an activity evaluation unit 19, an analysis result presentation unit 20, a cause analysis unit 21, and a behavior analysis database 47 in addition to the configuration of the first embodiment.

[0060] The task classification unit 18 adds task classification information to the interrelation data 46 input from the interrelation output unit 17, using information input to the calculation device 10 by one of the following methods. A method of inputting the current task classification selected by the worker at the timing of task switching into the measurement data input section together with image data. A method in which task classifications corresponding to object model IDs registered in advance in the object 3D model database 42 are stored, and the task classification is searched for from the object model ID list recognized by the object position and orientation estimation unit 13. A classification method using a trained AI (artificial intelligence) model that has learned the relationship between input images and task classifications

[0061] The task evaluation unit 19 compares the correlation data 46 with data stored in the behavior analysis database 47 that is classified as the same task, and evaluates the task content. For example, it compares the body trajectory coordinates included in the task content with the trajectory information TR of the body model for the same task stored in the behavior analysis database 47 (e.g., statistical values of trajectory information, trajectory information selected as exemplary), and evaluates areas where the trajectory coordinates differ significantly as trajectory deviations. Furthermore, if the task content is wiping the surface of an object, it can extract areas determined to be contacts from the trajectory information, enabling more efficient evaluation.

[0062] The analysis result presentation unit 20 visually displays the analysis results by combining the body model arranged in the virtual space by the body model arrangement unit 14, the object model arranged in the virtual space by the object model arrangement unit 15, and data stored in the behavior analysis database 47. For example, if the work content is wiping the surface of an object, the color of the surface area of the object model where it is determined that the body model is in contact with the object model may be changed, and the image may be output to a display via the input / output device 5 or the communication device 6 as an image from a certain viewpoint in the virtual space, thereby informing the worker of the work completion area. Furthermore, if the work content is screwing an object, the work evaluation unit 19 may analyze the amount of rotation relative to the object model posture coordinates from the body model trajectory information TR, change the color of the screw model according to the analyzed amount of rotation, and output the image to a display via the input / output device 5 or the communication device 6 as an image from a certain viewpoint in the virtual space, thereby informing the worker of the work progress.

[0063] The cause analysis unit 21 adds the results of the judgment of whether the work is good or bad to the behavior analysis database 47, and extracts the correlation data 46 that has a large impact on the quality of the work. For example, if the work content is equipment maintenance or manufacturing work, the work quality results added by the cause analysis unit 21 are the results of equipment operation checks after the work and the equipment performance. The correlation data 46 related to the quality of the work is extracted based on the correlation between the correlation data 46 of each work result stored in the behavior analysis database 47 and the quality of the work results, and the magnitude of deviation from the average value of the correlation data in the quality judgment.

[0064] According to the above-described fourth embodiment, the following advantageous effects can be obtained.

[0065] (1) By storing correlation data 46 for different input data in a behavior analysis database 47 and classifying the correlation data by task, the variance of the correlation data can be determined by classification, and the importance of behavior in the task can be evaluated.

[0066] (2) The task evaluation unit 19 can evaluate the behavior of even an unfixed task object by comparing the behavior analysis database 47 and the correlation data 46 using the trajectory of the three-dimensional body model in the object model posture coordinate system. This allows for efficient training information to be provided to the trainee by providing information on task deviations.

[0067] (3) The analysis result presentation unit 20 combines the body model and object model placed in the virtual space by the body model placement unit 14 and the object model placement unit 15 with the data stored in the behavior analysis database 47 to visually present the analysis results of the interrelationships, thereby providing the worker with work progress information.

[0068] (4) The cause analysis unit 21 can extract correlation data 46 relating to the quality of the work and provide work content information useful for improving the quality of the work.

[0069] In each of the above-described embodiments, the configuration of the functional blocks is merely an example. Several functional configurations shown as separate functional blocks may be configured as a single unit, or a configuration shown as one functional block may be divided into two or more functional blocks. Furthermore, some of the functions of each functional block may be configured to be possessed by other functional blocks.

[0070] In each of the above-described embodiments and modifications, the program may be stored in the ROM 2 or the auxiliary storage device 4. Alternatively, the arithmetic device 10 may be provided with an input / output interface (not shown), and the program may be loaded from another device when necessary via the input / output interface and a medium usable by the arithmetic device 10. Here, the medium may be, for example, a storage medium detachable from the input / output interface or a communication medium. The communication medium may refer to a wired, wireless, optical, or other network, or a carrier wave or digital signal propagating through the network.

[0071] The present invention is not limited to the above-described embodiments, but includes various modifications and equivalent configurations within the spirit of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to configurations including all of the described configurations. Furthermore, part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Furthermore, the configuration of another embodiment may be added to the configuration of one embodiment. Furthermore, part of the configuration of each embodiment may be added, deleted, or replaced with other configurations.

[0072] Furthermore, the aforementioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole in hardware, for example by designing them as integrated circuits, or may be realized in software by a processor interpreting and executing a program that realizes each function.

[0073] Information such as programs, tables, and files that realize each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or in a recording medium such as an IC card, an SD card, or a DVD.

[0074] In addition, the control lines and information lines shown are those that are considered necessary for explanation, and do not necessarily represent all the control lines and information lines that are necessary for implementation. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0075] 1 CPU(Central Processing Unit) 2 ROM (Read Only Memory) 3. RAM (Random Access Memory) 4 Auxiliary storage 5 Input / Output Devices 6. Communications equipment 10 Arithmetic unit 11 Measurement data input section 12 Human motion estimation part 13 Object position and orientation estimation unit 14 Body model placement section 15 Object model placement section 16 Interrelationship Analysis Section 17 Correlation output section 18 Work classification department 19 Work Evaluation Department 20 Analysis result presentation section 21 Cause Analysis Department 41 3D Body Model Database 42 3D object model database 43 input images 44 Body Posture Data 45 Object position and orientation data 46 Correlation Data 47 Behavioral Analysis Database OM Object Model LHM Left-Handed Model RHM Right-Handed Model TR Track Information

Claims

1. A behavior analysis system that analyzes a mutual relationship between a worker and an object, The computer is configured by an arithmetic unit that executes predetermined arithmetic processing and a storage device connected to the arithmetic unit, an input unit that receives input of measurement data of the worker and the object; a human movement estimation unit that estimates a movement of a worker based on the measurement data; an object position and orientation estimation unit that estimates the position and orientation of the object based on the measurement data; an object model placement unit that extracts an object model having the same shape as the object whose position and orientation are to be estimated from an object model database, and places the extracted object model in a virtual space at the position and orientation estimated by the object position and orientation estimation unit; a body model placement unit that extracts a body model of a worker whose movement is to be estimated from a body model database, and places the extracted body model in the virtual space according to the movement estimated by the human movement estimation unit; a correlation analysis unit that analyzes a correlation between the body model and the object model based on a position of the body model and a position and posture of the object model arranged in the virtual space; A behavior analysis system comprising a correlation output unit that outputs the correlation analyzed by the correlation analysis unit.

2. The behavior analysis system according to claim 1, the object model database records joint recognition probabilities between stored object models; The behavior analysis system is characterized in that the object position and orientation estimation unit preferentially extracts object models recorded in the object model database that have a high probability of simultaneous recognition.

3. The behavior analysis system according to claim 1, a physical object model placement unit placing the extracted physical object model in the virtual space using the position of the physical body model placed by the physical body model placement unit;

4. The behavior analysis system according to claim 1, The behavior analysis system further comprises a behavior analysis database that stores the interrelationships analyzed by the interrelationship analysis unit by task content.

5. The behavior analysis system according to claim 4, A behavioral analysis system characterized by comprising a work evaluation unit that compares the interrelationships that serve as evaluation criteria stored in the behavioral analysis database with the interrelationships that are the evaluation targets analyzed by the interrelationship analysis unit, and evaluates the work content.

6. The behavior analysis system according to claim 4, The behavior analysis system further comprises an analysis result presentation unit that presents the interrelationships analyzed by the interrelationship analysis unit to the operator.

7. The behavior analysis system according to claim 4, A behavior analysis system characterized by having a cause analysis unit that records the results of judgments on the quality of work in the behavior analysis database and extracts correlations that have a large impact on the quality of work.

8. 8. The behavior analysis system according to claim 1, The interrelationship analysis unit Calculation of a correlation coefficient between the coordinates of the body model and the coordinates of the object model; comparing the acceleration of the body model with the acceleration of an object model being grasped by the body model; Determining contact between the body model and the object model; Calculation of the rotation amount or angular velocity of the body model and the rotation amount or angular velocity of the object model; Determining the shape of the object model when the object model and the body model are in contact; Calculating a contact surface area of ​​the body model using the object model; The positional relationship between the direction of the index finger of the body model and the object model, and A behavior analysis system characterized by analyzing at least one of the judgments of the shape of the hand part of the body model as the correlation.

9. A behavior analysis method in which a behavior analysis system analyzes a mutual relationship between a worker and an object, comprising: the behavior analysis system is configured by a computer having an arithmetic unit that executes predetermined arithmetic processing and a storage device connected to the arithmetic unit; The behavior analysis method includes the steps of: an input step for receiving input of measurement data of the worker and the object; a human motion estimation procedure for estimating a motion of a worker based on the measurement data; an object position and orientation estimation step of estimating the position and orientation of the object based on the measurement data; an object model placement procedure for extracting from an object model database an object model having the same shape as the object whose position and orientation are to be estimated, and placing the extracted object model in a virtual space at the position and orientation estimated in the object position and orientation estimation procedure; a body model placement step of extracting a body model of a worker whose motion is to be estimated from a body model database, and placing the extracted body model in the virtual space according to the motion estimated in the human motion estimation step; a correlation analysis step of analyzing a correlation between the body model and the object model based on the positions of the body model and the object model arranged in the virtual space; a correlation output step for outputting the correlation analyzed in the correlation analysis step.

Images