Task recognition device and task recognition method

Abstract

To provide a work recognition device and method which estimate the relevance between objects for general purposes with a simple algorithm when performing work recognition using connection between objects in a manufacturing site or the like.SOLUTION: A work recognition device comprises: an image acquisition unit which acquires image data captured by an imaging device; an analysis region detection unit which detects a region about an object by analyzing image data obtained from the image acquisition unit; a feature extraction unit which extracts a feature amount about an image of the region about the object detected by the analysis region detection unit to calculate a change amount of the feature amount about the image in a prescribed period; and a relevance estimation unit which calculates the object relation degree between the first object and the second object on the basis of the first change amount of the feature amount about the image detected as the region about the first object in the prescribed period and the second change amount of the feature amount about the image detected as the region about the second object in the same period as the prescribed period of the first change amount.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an activity recognition device and an activity recognition method, and more particularly to an activity recognition device and an activity recognition method suitable for performing activity recognition by associating relationships between workers or robots working at a manufacturing site, for example, and surrounding objects, for the purpose of activity monitoring. [Background technology]

[0002] In manufacturing sites, improvement activities are being carried out to ensure the safety of human work and to improve the accuracy of work content in order to maintain product quality. One of these improvement activities is the application of technological developments related to motion recognition in human work to the industrial field. For example, there are cases where it is necessary to detect whether actions are being performed according to standard operating procedures at manufacturing sites, or to calculate the amount of time required for a specific task. The manufacturing sites that this technology targets are assumed to be work sites where various machines and tools are installed, where machine movements differ depending on the process, and where different tools are used for each task.

[0003] Meanwhile, in computer vision technology related to task recognition, due to the rapid development of computer performance and the ease of collecting large amounts of general data, motion recognition technology using machine learning is being actively developed. In this motion recognition technology, video image data is input, and an information processing device recognizes the motions performed by people in the video. An example of a public database related to such motion recognition technology is Non-Patent Document 1, in which labels such as "Playing Piano", "Surfing", and "Table Tennis Shot" are assigned to each piece of image data.

[0004] Also, object algorithms for acquiring object regions from images are described in, for example, Non-Patent Documents 2 and 3.

[0005] In this regard, a method for linking objects present in a video using an information processing device is disclosed in, for example, Patent Document 1. Patent Document 1 proposes a method for finding a connection between a mirror image and a real image reflected in a mirror as an information processing device for counting people, and achieving accurate person counting even in an image in which a mirror image is reflected. To find the connection between a mirror image and a real image, two or more moving objects corresponding to the same real moving object among multiple moving objects are identified based on the similarity of the speed according to the positional changes of the multiple moving objects in the image, and it is determined whether the time change in the amount of movement is similar to each other, and the connection between the two objects (real image and mirror image) is estimated.

[0006] Moreover, a method for performing behavior recognition using sensors and cameras is described in, for example, Non-Patent Document 4.

[0007] In addition, Non-Patent Document 5 describes a technique for behavior recognition using a graph convolution network (GCN) as one aspect of machine learning. GCN is a field of so-called convolutional neural networks, and is a method for generating a graph consisting of nodes having information on the features and classes of a certain object and edges connecting the nodes, and inferring the target node class or the class of the entire graph using the features of adjacent nodes. In GCN, there is a method in which the connections between nodes are defined in advance or automatically determined according to changes in the video. Non-Patent Document 5 employs two methods. The first is a method in which the position of an object is detected in a video captured by a camera, an overlap rate is calculated from the degree of overlap between the objects, and then the relationship between the two objects is estimated according to the overlap rate. The second is a method in which similar objects are considered to be the same object when they are captured in frame images acquired in a time series, and the relationship between the objects in the frames in the time direction is calculated. With such a method, a graph can be generated by predefining or estimating the relationship between objects, making it possible to recognize behavior taking into account the relationship between objects. [Prior art documents] [Patent documents]

[0008]

Patent Document 1

Non-licensed literature

[0009]

Non-licensed literature 1

Non-licensed Document 2

Non-licensed Document 4

Non-licensed Document 5

[0010] In computer vision technology, when motion recognition technology is applied to workers in manufacturing sites, it is necessary to recognize unique motions that differ from one manufacturing site to another. In such cases, a common method is to capture the movements of the target worker only, but this method makes it difficult to distinguish between workers who perform multiple different tasks in the same posture. Furthermore, in manufacturing sites where machines are present, when performing a certain task, not only the worker but also the surrounding machines may move at the same time, or tools may be handled at the same time. In other words, there are cases where the object (person or object) being recognized as the task target and the surrounding objects change at the same time.

[0011] When applied to a manufacturing site, there are cases where the appearance of a machine changes but there is no significant movement. In Non-Patent Document 5, as described above, the overlap rate or similarity between objects is used to estimate the relevance between objects, but no matter which method is used, it is expected that the relationship between objects in the manufacturing site cannot be properly handled. For example, in a case where the door of a facility opens during processing work and a new workpiece is transported and inserted into the facility, even if they are operating at the same time, they may not overlap in the image space depending on the installation position of the camera. Therefore, it is necessary to estimate the relevance using a method that does not depend on the overlap or distance between objects.

[0012] Patent Document 1, which proposes a method for finding the connection between real images and mirror images, assumes that the target objects are the same. When performing a certain task, multiple objects (machines, workpieces, workers) present at the manufacturing site may move simultaneously, but the amount of movement over time between these machines, workpieces, workers, etc. differs for each object. Therefore, it is difficult to estimate the relevance of objects at the manufacturing site from the similarity of the change in the amount of movement over time between machines, workpieces, and workers.

[0013] Furthermore, in Patent Document 1 and Non-Patent Document 5, overlapping rate, similarity, etc. are adopted as indices for determining the connection between objects, and a threshold value is set for the judgment. Therefore, this method judges that there is a relation between the objects only when the indices have a dramatic value, and such a method cannot judge when the objects should be recognized as having a connection even if the overlapping rate or similarity of the indices does not exceed a certain threshold, and furthermore, a problem arises in that an algorithm for estimating the relation from the positional relationship of the objects, etc. is required.

[0014] An object of the present invention is to provide an activity recognition device that can generically estimate the associations between objects using a simple algorithm when performing activity recognition using connections between objects in task recognition in a manufacturing site or the like. [Means for solving the problem]

[0015] The configuration of the task recognition device of the present invention is preferably a task recognition device that analyzes image data and performs task recognition of objects related to the task by analyzing the image data, and has an image acquisition unit that acquires image data captured by a shooting device, an analysis area detection unit that analyzes the image data obtained from the image acquisition unit and detects an area related to the object, a feature extraction unit that extracts features of the image of the area related to the object detected by the analysis area detection unit and calculates an amount of change in the feature related to the image within a specified period of time, and a relevance estimation unit that calculates an object relevance between a first object and a second object based on a first amount of change in the feature related to the image detected as an area related to a first object within a specified period of time and a second amount of change in the feature related to the image detected as an area related to a second object within the same specified period as the first amount of change. Effect of the Invention

[0016] According to the present invention, when performing task recognition using connections between objects in a manufacturing site or the like, it is possible to provide an task recognition device that can generically estimate the relevance between objects using a simple algorithm. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 is a schematic configuration diagram of an activity recognition system. [Diagram 2] 1 is a functional configuration diagram of a task recognition device according to a first embodiment. FIG. [Diagram 3] FIG. 1 is a diagram showing the hardware and software configuration of the task recognition device. [Figure 4] 4 is a flowchart showing a series of processes from image acquisition to learning processing in the task recognition device according to the first embodiment. [Figure 5A] 13 is a flowchart showing details of a process for extracting an amount of change in a feature amount (in the case of a change in motion); [Figure 5B] 13 is a flowchart showing details of a feature amount change extraction process (in the case of texture change); [Figure 6] 13 is a flowchart showing details of a relevance estimation process. [Figure 7] 13 is a flowchart showing details of a graph generation process. [Figure 8] 1 is a flowchart showing a series of processes from image acquisition to inference processing by the task recognition device. [Figure 9] FIG. 11 is a diagram for explaining a specific image of calculating an object relevance from image data. [Figure 10] 11A and 11B are diagrams illustrating an example in which object relevance is applied to graph generation. [Figure 11] FIG. 13 is a diagram for explaining how a node work label is assigned to a node as an inference result. [Figure 12] FIG. 11 is a functional configuration diagram of a task recognition device according to a second embodiment. [Figure 13] 10 is a flowchart showing a series of processes from image acquisition to learning processing in the task recognition device according to the second embodiment. [Figure 14] FIG. 13 is a diagram showing an example of an object relevance editing screen. [Figure 15] FIG. 13 is a diagram for explaining a specific image of calculating an object relevance from image data according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, each embodiment of the present invention will be described with reference to FIGS.

[0019] [Embodiment 1] Hereinafter, a first embodiment of the present invention will be described with reference to FIGS.

[0020] In this embodiment, an example of a task recognition system will be described that detects a worker and surrounding objects from the way a user works, expresses the state in a graph structure, and recognizes the task actions by machine learning using a GCN technique based on the graph structure. Using a graph structure as a data structure for recognizing task actions is just one example, and a similar network structure or algorithm may be used for the application of this embodiment.

[0021] First, the configuration of the task recognition system will be described with reference to FIGS.

[0022] As shown in FIG. 1, the task recognition system 1 includes an image capture device 6, a sensor 3, and an task recognition device 100, which are connected via communication means 4.

[0023] The communication means 4 may be wired or wireless, may be a LAN (Local Area Network), or may be a WAN (Wide Area Network) such as the Internet, or may be a communication means conforming to a serial standard such as USB (Universal Serial Bus) or RS-232C.

[0024] The image acquisition device 6 (imaging device) is a device that acquires image data of the worker 2, surrounding objects 5, etc., and is, for example, a camera (digital camera (RGB camera), infrared camera, thermography camera, time-of-flight (TOF) camera, stereo camera, etc.) that acquires (takes) image data of videos or still images. Although the image acquisition device 6 is shown as one device in FIG. 1, multiple cameras may be used to use multiple images from different cameras when there are multiple subjects to be photographed, etc.

[0025] The sensor 3 is provided in the work environment where the worker 2 performs work, detects the state of the worker 2 and the work environment, and outputs physical information. The sensor 3 is, for example, a moving object detection sensor, a human sensor, a temperature sensor, a humidity sensor, an acceleration sensor, a speed sensor, an acoustic sensor (microphone), an ultrasonic sensor, a vibration sensor, a millimeter wave radar, a laser radar (LIDAR: Laser Imaging Detection and Ranging), or an infrared depth sensor.

[0026] The task recognition device 100 is a device that performs processing to recognize a task movement of a worker 2 or a task movement of a surrounding object 5 based on image data acquired by an image acquisition device 6.

[0027] Next, the functional configuration of the task recognition device will be described with reference to FIG. As shown in FIG. 2 , the task recognition device 100 has the following functional units: an image acquisition unit 101, an analysis area detection unit 102, a feature extraction unit 103, a relevance estimation unit 104, a graph generation unit 105, an task learning unit 106, an task reasoning unit 107, and a memory unit 110.

[0028] The image acquisition unit 101 is a functional unit that acquires image data 200 acquired from the image acquisition device 6. The image data 200 is, for example, frame data constituting still image data or video data sent from the image acquisition device 6.

[0029] The analysis region detection unit 102 is a functional unit that detects an object region from the image obtained by the image acquisition unit 101. The object region can be detected by a method in which a user manually sets a rectangle on the image by drawing it, or by a method in which the region is automatically acquired using an object detection algorithm as shown in, for example, Non-Patent Documents 2 and 3. Either the manual or automatic method may be used.

[0030] The feature extraction unit 103 is a functional unit that extracts feature amounts and the amount of change thereof in an image.

[0031] In the feature extraction function, the feature extraction unit 103 extracts features such as color features, motion features, and CNN (Convolutional Neural Network) features obtained from an image. Note that the features used in this embodiment are assumed to be scalar values. In the feature extraction function, the feature extraction unit 103 secures data in a time series such as previous and next frames, and extracts the amount of change in the feature using the difference in the features of the previous and next frames. Features may not only be extracted from a specified area, but also may be a change in which the area can be expressed by one point, such as position information.

[0032] The relevance estimation unit 104 is a functional unit that calculates an object relevance that estimates the strength of the connection between objects using the change in the feature amount output by the feature extraction unit 103. The relevance estimation unit 104 normalizes (adjusts the scale between 0 and 1) the change amount acquired from each object using the correction parameter 201. Then, the normalized change in the feature amount is calculated as the strength of the relationship between the objects, and the result of the calculation is output as the object relevance, for example, as the product thereof. The correction parameter 201 is a parameter for normalizing the change in the feature amount, and has the purpose of filling in the difference in the size of the rectangle between the objects and the average change amount. In the above description, an example was shown in which the strength of the relationship between the objects is calculated as the product of each change in the feature amount, but the calculation method may be a sum of each change in the feature amount, a weighted linear sum, or the like.

[0033] The graph generation unit 105 is a functional unit that generates a graph used in the GCN. The graph generation unit 105 generates graph nodes for the number of regions acquired by the analysis region detection unit 102, assigns features obtained by the feature extraction process of the feature extraction unit 103 to each node, and assigns a label of a node identification label 202. The label of the node identification label 202 is, for example, the name of an object in a manufacturing site, such as "robot", "worker", or "machine". It is assumed that there is a connection between each node, and an edge is provided between the nodes, and weights are assigned to the edges of the graph using the object relevance obtained from the relevance estimation unit 104.

[0034] The task learning unit 106 is a functional unit that performs machine learning using GCN based on the node task label 203, which is the correct label of the task, and the graph generated by the graph generation unit 105, to generate an inference model 210. In the machine learning process of the task learning unit 106, learning is performed using a GCN algorithm based on the generated graph, and as a result, the inference model 210 is output. The node task label 203 is a correct label that recognizes the task, for example, "welding" or "screw tightening" when the node is a "robot", or "control panel operation" or "floor cleaning" when the node is a "worker".

[0035] The task inference unit 107 is a functional unit that performs inference on image data 200 for which task recognition is desired based on an inference model 210 obtained as a result of learning by the task learning unit 106, and obtains an inference result 220.

[0036] The storage unit 110 is a functional unit that stores data. The storage unit 110 stores image data 200, correction parameters 201, node identification labels 202, node working labels 203, an inference model 210, and an inference result 220.

[0037] Next, the hardware and software configuration of the task recognition device will be described with reference to Fig. 3. The task recognition device 100 can be realized by a general information processing device (computer), and includes a processor 11, a main memory device 12, a communication device 13, an input device 14, an output device 15, and an auxiliary memory device 16, as shown in Fig. 3.

[0038] The processor 11 is a semiconductor device that executes programs on the main memory device 12 and controls each part of the task recognition device 100, and is, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), an AI (Artificial Intelligence) chip, etc.

[0039] The main memory device 12 is a device that temporarily stores programs and data, and is, for example, a random access memory (RAM) such as a static random access memory (SRAM) or a dynamic random access memory (DRAM).

[0040] The communication device 13 is an interface device that communicates with other devices via the communication means 4, and is, for example, a NIC (Network Interface Card), a wireless communication module, a USB (Universal Serial Interface) module, a serial communication module, or the like. The communication device 13 can also function as an input device that receives information from other devices that are communicatively connected. The communication device 13 can also function as an output device that transmits information to other devices that are communicatively connected. The task recognition device 100 communicates with the image acquisition device 6 and the sensor 3 via the communication means 4 by the communication device 13.

[0041] The input device 14 is a user interface device that accepts information from a user, and is, for example, a keyboard, a mouse, a card reader, a touch panel, or the like.

[0042] The output device 15 is a user interface device that outputs various types of information (display output, audio output, print output, etc.), and is, for example, a display device (LCD (Liquid Crystal Display), graphics card, etc.) that visualizes various types of information, an audio output device (speaker), a printing device, etc.

[0043] The auxiliary storage device 16 is a hard disk drive, a solid state drive (SSD), etc. The programs and data stored in the auxiliary storage device 16 are loaded into the main storage device 12 as needed, and the processor 11 refers to the loaded data and executes the programs.

[0044] In the auxiliary storage device 16 of this embodiment, an image acquisition program 161, an analysis area detection program 162, a feature extraction program 163, a relevance estimation program 164, a graph generation program 165, a work learning program 165, and a work inference program 166 are installed.

[0045] The image acquisition program 161, the analysis area detection program 162, the feature extraction program 163, the relevance estimation program 164, the graph generation program 165, the work learning program 165, and the work reasoning program 166 are programs that respectively realize the functions of the image acquisition unit 101, the analysis area detection unit 102, the feature extraction unit 103, the relevance estimation unit 104, the graph generation unit 105, the work learning unit 106, and the work reasoning unit 107.

[0046] In the above description, the various functions of the task recognition device 100 are implemented by the processor 11 reading and executing programs stored in the main memory device 12. However, the various functions may also be implemented by hardware (FPGA, ASIC, AI chip, etc.) that constitutes the task recognition device 100.

[0047] Next, the processing of the task recognition device will be described with reference to FIGS.

[0048] First, the image acquisition section 101 of the task recognition apparatus 100 acquires the image data 200 transmitted from the image acquisition device 6 (S100).

[0049] Next, the analysis region detection unit 102 detects an object region from the image of the image data 200 obtained by the image acquisition unit 101 .

[0050] Next, the feature extraction unit 103 extracts feature amounts between predetermined frames obtained from the images (S102).

[0051] Next, the feature extraction unit 103 secures data in time series, such as previous and next frames, and extracts the amount of change in the feature using the difference between the features of the previous and next frames (S103). Note that the process of extracting the amount of change in the feature will be described in detail later.

[0052] Next, the relevance estimation unit 104 calculates an object relevance degree (S104) that estimates the strength of the connection between the objects using the amount of change in the feature amount extracted by the process S103 of extracting the amount of change in the feature amount of the feature extraction unit 103. The process of calculating the object relevance degree between the objects will be described in detail later.

[0053] Next, the graph generation unit 105 generates a graph to be used in the GCN based on the object relevance between the objects in S104 (S105).

[0054] Next, the task learning unit 106 performs machine learning using the GCN based on the graph generated in S105, and generates an inference model 210 (S106).

[0055] Next, the process of extracting the amount of change in the feature amount will be described in detail with reference to FIGS. 5A and 5B.

[0056] This process corresponds to S103 in Fig. 4. In this embodiment, the case of a movement change and a texture change will be described in detail as examples of the feature amount, but the feature amount of the image data may be any data that contributes to task recognition.

[0057] First, a process of extracting a change in motion from a moving object in an image as a change in a feature amount will be described with reference to FIG. 5A.

[0058] First, a predetermined object region group is acquired (S200). Here, in order to acquire a region for which movement is to be acquired, it is not necessary for the region to be focused on the object itself, but it is sufficient for the region to be a predetermined region associated with the object in the image space.

[0059] Next, the amount of motion is calculated from within the region using the images of the previous and next frames by the optical flow method (S201). Optical flow is a method for analyzing the pattern of how the movement of an object between adjacent frames appears due to the movement of the object or the camera.

[0060] Next, the sum of the amount of movement within the area obtained for each object is calculated (S202).

[0061] Next, the sum of the calculated amounts of motion is output as the amount of change in the feature amount (S203).

[0062] Next, the process of extracting a change in image texture as a change in feature amount will be described with reference to FIG. 5B.

[0063] A change in the texture of an image refers to a change in the RGB values ​​or gray values ​​of the image.

[0064] First, a predetermined group of object regions is obtained (S210).

[0065] Next, a trained model capable of identifying an object or changes in an object is acquired as a feature extraction model (S211). For example, models for extracting local features of an image include a template feature model, a holistic feature model, a co-occurrence feature model, and a part feature model.

[0066] The image data of each object region is applied to a feature extraction model to obtain features for each object (S212).

[0067] Then, previous and next feature amounts are obtained from image data of previous and next frames, and the distance between the feature amounts of the previous and next frames is calculated (S212). The feature amount distance is, for example, an n-dimensional Euclidean distance when the feature amount is composed of n elements.

[0068] Next, the distance value calculated in S212 is output as the amount of change in the feature amount (S214).

[0069] Next, a process for estimating the relevance between objects will be described with reference to FIG. This process corresponds to S104 in FIG.

[0070] First, the amount of change in the feature amount of each object region extracted in S103 of FIG. 4 is obtained (S300).

[0071] Next, the amount of change in the feature amount is normalized (S301) using the correction parameters 201. Since the feature amount of each region may change significantly, it is meaningful to match the scale, and correction using the correction parameters 201 is not necessarily required.

[0072] Next, an object relevance is calculated based on the amount of change in the feature amounts of the two regions normalized in S301 (S302). This object relevance is an index showing the degree of object relation.

[0073] For example, the feature quantity of the region of object A is C A , the feature quantity of the region of object B is C B Then, the product of the two is taken and the object relevance is calculated using the following (Equation 1). Object relevance (object A, object B) = C A ×C B …(Formula 1)

[0074] Moreover, for example, the sum of the two is calculated, and the object relevance is calculated by the following (Equation 2). Object relevance (object A, object B) = C A +C B …(Formula 2)

[0075] Also, for example, a weighted linear sum of the two is taken, and the object relevance is calculated by the following (Equation 3). Object relevance (object A, object B) = K A C A +K B C B …(Formula 3)

[0076] Here, K A , K B are weighting coefficients determined by how much importance is attached to the changes in the feature quantities of object A and object B, respectively. Moreover, calculations other than these may be performed.

[0077] Next, the object relevance obtained in S313 is normalized (S303) using the correction parameter 201. This is to facilitate learning in the subsequent learning phase, and the correction using the correction parameter 201 is not necessarily required.

[0078] Next, the object relevance calculated in S303 is output (S304).

[0079] Next, the graph generation process will be described in detail with reference to FIG. The feature amount obtained by the feature amount extraction process in S102 of FIG. 4 is acquired for each object region (S400).

[0080] Next, a node-specific label assigned to each object region is obtained from the node-specific label 202 (S401).

[0081] Next, the object relevance between the objects obtained by the relevance estimation process in S104 of FIG. 4 is obtained (S402).

[0082] Next, the feature amount obtained in S411 and the node-specific label obtained in S412 are assigned to each node (S403).

[0083] Next, the object relevance is assigned as a weight for the edge between the nodes (S414). This weight is, for example, the edge weight used in the convolution layer of the GCN.

[0084] In this way, by using the generated graph, for example by learning using a GCN, it is possible to construct an inference model 210 that can infer the activity of an object corresponding to an edge that has been assigned a node-specific label.

[0085] Next, a series of processes from image acquisition to inference processing by the task recognition device will be described with reference to FIG. The processes of S100 to S105 are similar to those shown in Fig. 4. Here, the image data 200 input in S100 is image data 200 for making inference for object task recognition.

[0086] After the graph generation in S105, the task inference unit 107 performs inference for task recognition based on the inference model 210 obtained in S106 in Fig. 4, and outputs the inference result (S110). The inference result is a graph in which a node task label 203 is assigned to each node. A specific example of the inference result will be described in detail later.

[0087] Next, a specific image of calculating the object relevance from image data will be described with reference to FIG. Here, an example of calculating the object relevance of object A and object B for time series data such as video data will be described. Here, clock 1, clock 2, ... are the range of the time axis of the time series data. When object A and object B exist in an image, image features are obtained from image data obtained in the same frame. When focusing on clock 1 (C01) between a predetermined number of frames, a distance value is calculated using Euclidean distance or the like from the features obtained in feature extraction F1A in the first frame and feature extraction F2A in the last frame, and a change amount d1A in the feature amount of object A is obtained. Meanwhile, feature extraction is performed in the same way for object B, and a change amount d1B in the feature amount is obtained from the distance value of object B. From the change amount A11D and the change amount A21D, an object relevance P1 is obtained using the product thereof (in the case of (Equation 1)) or the like. P1 is the object relevance of clock 1 (C01). A similar process is performed for clock 2 (C02) to calculate object relevance P2. A similar process is performed for each clock every time video data is obtained. It is not necessary to use different frames in each clock, and some of the frames may be the same.

[0088] Next, an example of applying the object relevance to graph generation will be described with reference to FIG. Assume that for object A, a feature amount change F11A is obtained within a predetermined clock, and for object B, a feature amount change F11B is obtained within the same clock as for object A. At this time, the feature amount change is normalized by the correction parameter 201, and for object A, a normalized feature amount change F11AN is obtained, and for object B, a normalized feature amount change F11BN is obtained.

[0089] Next, object relevance P11 is calculated from the product of each change amount (in the case of (Equation 1)), and is applied as weight w11 to the edge connecting object A and object B shown in graph G01 from the change amount at a specified clock. In this way, similar processing is performed between each object, and weights are applied to the edges connecting the objects. Figure 10 shows an example in which the average object relevance of object A and object B in the time range of interest is 0.1, and this is applied as the weight of the edge connecting the robot node and the processing machine node.

[0090] Next, the task recognition result of the inference will be described with reference to FIG. FIG. 11 is a diagram for explaining how a node work label is assigned to a node as an inference result.

[0091] In this embodiment, task recognition is performed by task inference processing of S110 in FIG. 8 based on the inference model 210 generated by the task learning processing of S106 in FIG. 4, and node task labels 203 of objects represented as each node are obtained.

[0092] In the example of Fig. 11, as shown in Fig. 11(a), the node operation labels obtained at time T1 for node 1 "robot", node 2 "machine", and node 3 "workpiece" are "welding" for node 1, "stop" for node 2, and "welded" for node 3. Fig. 11(b) is a timing chart targeting node 1 "robot". This timing chart shows that the robot is performing welding work at time T1.

[0093] As described above, according to this embodiment, object relevance is calculated based on the amount of change in image feature amount, and GCN learning and inference are performed using the object relevance as the edge weight based on a graph in which objects are nodes and related objects are edges. As a result, when performing task recognition using connections between objects in task recognition at manufacturing sites, etc., task recognition can be performed by estimating the relevance between objects in a general purpose manner using a simple algorithm.

[0094] [Embodiment 2] Hereinafter, the second embodiment will be described with reference to FIGS.

[0095] In the first embodiment, an example of an activity recognition device has been described that calculates the object relevance of each object from the amount of change in the feature amount of the object and applies the object relevance as the edge weight of the GCN graph to perform activity recognition.

[0096] In this embodiment, an example will be described in which an activity recognition device performs activity recognition similar to that in the first embodiment, and further allows a user to edit the object relevance between objects corresponding to nodes.

[0097] The following description will focus on the differences from the first embodiment.

[0098] First, the functional configuration of the task recognition device according to the second embodiment will be described with reference to FIG. The task recognition apparatus 100 of this embodiment includes a relevance editing unit 108 in addition to the components of the task recognition apparatus 100 of the first embodiment shown in FIG.

[0099] The relevance editor 108 is a functional unit that edits the relevance value of an object recognized as a node.

[0100] Furthermore, the storage unit 110 stores association data 204. The association data 204 is data that holds values ​​of the association levels of objects recognized as nodes.

[0101] Next, a series of processes from image acquisition to learning processing by the task recognition device according to the second embodiment will be described with reference to FIG.

[0102] In the series of processes from image acquisition to learning in the task recognition device of this embodiment, in addition to the processes shown in FIG. 4 of the first embodiment, a relevance editing process S120 is added between S104 and S105.

[0103] The relevance editing process S120 is a process for editing the relevance values ​​of objects recognized as nodes of a graph. This process resets the object relevance defined by the user, and outputs the object relevance obtained in the relevance estimation process S104 and the object relevance defined by the user as relevance data 204, which is then passed to the graph generation process S105.

[0104] Next, the user interface provided by the task recognition device will be described with reference to FIG. The task recognition apparatus 100 of this embodiment can display and edit the object relevance assigned to the edges of the graph using an object relevance editing screen 400 shown in FIG. The task recognition device 100 reads the result of the relevance estimation process S04 or a setting file that allows a user to select a predetermined object from candidates. Next, the user selects two objects for linking the objects. In the example of FIG. 14, object A is selected in box 411 of object 1, and object B is selected in box 412 of object 2. The degree of relevance between the objects selected in each box can be entered in box 42, and the user can set any degree of relevance between the objects. In addition, the type of feature can be selected in box 43 of the feature, and the amount of change in the feature of the object can be set in box 441 of object 1 and box 442 of object 2, respectively, and the user has a function of setting any object relevance. In addition, the graph display 500 visualizes the relevance degree, and it is possible to visually recognize that nodes 511 and 512 are connected, that feature 53 and the like are displayed for each object, and that object A and object B are connected, and the numerical value of the object relevance between object A and object B can be edited in box 52.

[0105] As described above, the task recognition device of this embodiment allows the user to arbitrarily edit the amount of change in feature amount of each object and the object relevance.

[0106] [Embodiment 3] Hereinafter, the third embodiment of the present invention will be described with reference to FIG. In the first embodiment, a task recognition device has been described that obtains the amount of change in the feature amounts of object regions in an image, and in order to show the association between the objects, calculates the amount of change in the feature amounts in the region of one object and the amount of change in the feature amounts in the region of another object by calculating the object association between the two objects, and performs machine learning using this as the weight of the edges of a graph in a GCN.

[0107] In this embodiment, as in the first embodiment, the amount of change in the feature amount of an object region in an image is obtained, and then the object relevance between one object and another object is calculated. However, a method of calculating the object relevance is described that also focuses on the transition of image data over time.

[0108] The following description will focus on the differences from the first embodiment.

[0109] The task recognition device 100 of this embodiment has a functional configuration similar to that shown in FIG. 2 of the first embodiment, and calculates the object relevance between the two objects by calculating the amount of change in feature amount in the area of ​​an object and the amount of change in feature amount in the area of ​​another object, which is the relevance estimation process S104 in FIG. 4, in the same way; however, the method for calculating the object relevance between the two objects is different.

[0110] The relevance estimation process S104 in the third embodiment will be described below with reference to FIG. The relevance estimation unit 104 of the task recognition apparatus 100 acquires feature amount 1 (FA1) obtained by the optical flow or the like of the object A (60A) and feature amount 2 (FA2) obtained by 3DCNN or the like.

[0111] 3DCNN is a method for recognizing actions in videos that takes spatiotemporal information into account by combining spatial information (2D) and temporal information (1D) and performing 3D convolution.

[0112] Moreover, the relevance estimation unit 104 acquires a feature amount 1 (FB1) and a feature amount 2 (FB2) for the object B (60B) in the same manner as for the object A (60A).

[0113] Next, distance values ​​(d11, d12, d21, d22) are calculated between features obtained within the same clock between objects.

[0114] After that, the difference in distance values ​​between the previous and next clocks is calculated, and the different feature amount 1 and feature amount 2 are combined to calculate the object relevance. For example, the distance d11 between feature amount 1 of object A and feature amount 1 of object B obtained at clock 1 is calculated. Similarly, the distance d12 is calculated at clock 2. The change between clock 2 and clock 1 is obtained from the absolute value of the difference between distance d11 and distance d12, and the degree of change between the clocks (D1R) is obtained. For example, the reciprocal of the value obtained by adding 1 to the absolute value Δ1 of the difference between distance d11 and distance d12 is taken to obtain the degree of change between the clocks. That is, the degree of change between the clocks (D1R) is calculated using the following (Equation 4). D1R=1 / (Δ1+1) …(Formula 4)

[0115] Similarly, for feature amount 2 of object A and feature amount 2 of object B, the change between clocks 2 and 1 is obtained from the absolute value of the difference between distance d21 and distance d22, and the degree of change between the clocks (D2R) is obtained. For example, the reciprocal of the value obtained by adding 1 to the absolute value Δ2 of the difference between distance d21 and distance d22 is taken to obtain the degree of change between the clocks. In other words, the degree of change between the clocks (D2R) is calculated using the following (Equation 5). D2R=1 / (Δ2+1) …(Equation 5) Then, the arithmetic mean value of the change degrees D1R and D2R is calculated and set as the similarity S.

[0116] When generating the graph, when the edge weight of the graph of clock 1 is W1 and the edge weight of the graph of clock 2 is W2, the final edge weight W2' of clock 2 is calculated using the following (Equation 6). W2′=W1×S+W2 …(Equation 6)

[0117] In other words, when the degree of change between clocks is small, the weight of the edge in the graph of clock 2 is not emphasized, and when the degree of change between clocks is large, the weight of the edge in the graph of clock 2 is emphasized.

[0118] As described above, in this embodiment, by calculating the difference between objects and the difference between clocks, it is possible to calculate the object relevance taking into account time-series data and generate a GCN graph. [Explanation of symbols]

[0119] 1 ... task recognition system, 2 ... worker, 3 ... sensor, 4 ... communication means, 5 ... surrounding object, 6 ... image acquisition device, 100...work recognition device, 11: processor, 12: main memory device, 13: communication device, 14: input device, 15: output device, 16: auxiliary memory device, 101: image acquisition unit, 102: analysis area detection unit, 103: feature extraction unit, 104: relevance estimation unit, 105: graph generation unit, 106: work learning unit, 107: work reasoning unit, 108: relevance editing unit, 110: memory unit, 200: image data, 201: correction parameters, 202: node specific labels, 203: node working labels, 204: association data, 210: inference model, 220: inference results

Claims

1. A task recognition device that analyzes image data and performs task recognition of an object related to the task by analyzing the image data, an image acquisition unit that acquires image data captured by the imaging device; an analysis area detection unit that analyzes the image data obtained from the image acquisition unit and detects an area related to an object; a feature extraction unit that extracts a feature amount of an image of a region related to the object detected by the analysis region detection unit and calculates a change amount of the feature amount of the image within a predetermined period of time; and a relevance estimation unit that calculates an object relevance between a first object and a second object based on a first change in a feature amount for the image detected as an area related to a first object within a predetermined period of time and a second change in a feature amount for the image detected as an area related to a second object within the same predetermined period of time as the first change in the feature amount.

2. the first change amount of the feature quantity and the second change amount of the feature quantity are scalar quantities, 2. The task recognition device according to claim 1, wherein the object relevance between the first object and the second object is calculated by any one of a product, a sum, or a weighted linear sum of a first change amount in the feature amount and a second change amount in the feature amount.

3. The task recognition device according to claim 1, further comprising an task learning unit that performs machine learning for task recognition based on the object relevance calculated by the relevance estimation unit and feature amounts related to the image of the area related to the object extracted by the feature extraction unit, and outputs an inference model.

4. The task recognition system according to claim 3, further comprising a task inference unit that performs inference for task recognition based on image data captured by an imaging device and the inference model output by the task learning unit.

5. The data includes data expressing a graph in which regions related to the object are nodes and related objects are connected by edges; 4. The task recognition device according to claim 3, wherein an inference model is generated that generates a node task label for the node by using the object relevance as a weight between edges and a Graph Convolution Network (GCN).

6. The data includes data expressing a graph in which regions related to the object are nodes and related objects are connected by edges; 2. The task recognition apparatus according to claim 1, further comprising a display / editing unit for displaying and editing the feature amount of the area related to the object and the object relevance.

7. the relevance estimation unit calculates a first object relevance between the first object and the second object based on a first amount of change in a feature amount related to the image detected as a region related to a first object within a first period and a second amount of change in a feature amount related to the image detected as a region related to a second object within the first period; calculating a second object relevance between the first object and the second object based on a first change amount of a feature amount related to the image detected as a region related to a first object within a second period and a second change amount of a feature amount related to the image detected as a region related to a second object within the second period; calculating a degree of change in the object relevance based on an absolute value of a difference between the first object relevance and the second object relevance, and calculating a similarity based on a degree of change in the object relevance using a first amount of change within the first period; correcting a weight of a first edge corresponding to a first period in the graph by a similarity based on a degree of change in the object relevance; 6. The task recognition device according to claim 5, further comprising: a sum of a weight of a first edge corresponding to the corrected first period and a weight of a second edge in the second period, to calculate the corrected weight of the second edge in the second period.

8. A task recognition method using a task recognition device that analyzes image data and recognizes tasks of a worker and a task device by analyzing the image data, comprising: an image acquiring step in which an image acquiring unit acquires image data captured by the photographing device; an analysis region detection step in which an analysis region detection unit analyzes the image data obtained by the image acquisition step and detects a region related to the object; a feature extraction step of extracting a feature amount related to an image of the area related to the object detected by the analysis area detection step, and calculating a change amount of the feature amount related to the image within a predetermined period of time; a relevance estimation step of calculating an object relevance between the first object and the second object based on a first change amount of a feature amount of the image detected as an area related to a first object within a predetermined period and a second change amount of a feature amount of the image detected as an area related to a second object within the same predetermined period as the first change amount; a task learning step of performing machine learning for task recognition based on the object relevance calculated in the relevance estimation step and feature amounts related to the image of the area related to the object extracted in the feature extraction step, and outputting an inference model; A task recognition method comprising a task inference step of performing inference for task recognition based on image data captured by a photographing device and the inference model output by the task learning step.

9. the first change amount of the feature quantity and the second change amount of the feature quantity are scalar quantities, 9. The task recognition method according to claim 8, wherein the object relevance between the first object and the second object is calculated by any one of a product, a sum, or a weighted linear sum of a first change amount in the feature amount and a second change amount in the feature amount.

10. the task recognition device has data representing a graph in which regions related to the object are nodes and related objects are connected by edges; 9. The task recognition method according to claim 8, further comprising the steps of: generating an inference model that generates a node task label for the node by using the object relevance as a weight between edges and a graph convolution network (GCN).

Images