A camera array and data fusion method for human-machine collaborative assembly scene perception

Abstract

The present invention provides a camera array for human-machine collaborative assembly scene perception and its data fusion method, comprising a workbench, a robotic arm, a camera array, and a data processing system. The robotic arm is mounted in the left front area of ​​the workbench. The workbench is divided into an assembly area, a parts area, and a tool area. The camera array comprises five planar cameras arranged within the camera array space: cameras 1, 2, and 3 are positioned on the left, right, and right front sides of the workbench, with their placement heights similar to the assembler's height. Camera 4 is positioned on the left front side, higher than the workbench. Camera 5 is positioned directly above the workbench, higher than the workbench. By acquiring multi-source real-time data from cameras at multiple angles, stable perception of objects in the scene and the assembler's movements can be achieved even when a single camera is obscured, enhancing recognition reliability and stability and achieving stable perception of the current assembly environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of human-machine collaboration and intelligent manufacturing, and specifically relates to a camera array for human-machine collaborative assembly scene perception and a data fusion method thereof. Background Art

[0002] With the development of industry, robots assisting workers in assembly work in production workshops is becoming a reality. To achieve collaborative assembly, robots must accurately and quickly recognize and understand the assembly parts and human actions within the assembly scene. This recognition and understanding requires that robots be able to perceive the assembly parts and human actions within the scene.

[0003] Currently, the perception of assembly parts and human behavior in a scene mostly uses a single visual sensor to obtain scene images, and inputs object recognition models and human action recognition models to realize the perception of assembly objects and human behavior. However, the recognition rate of the recognition model for single visual data is unstable and cannot solve the problem of large-area occlusion. Summary of the Invention

[0004] The present invention provides a camera array for human-machine collaborative assembly scene perception and its data fusion method. By acquiring multi-source real-time data through cameras at multiple angles, it is possible to achieve stable perception of objects in the scene and the assembler's movements even when a single camera is blocked, thereby enhancing the reliability and stability of recognition and achieving stable perception of the current assembly environment.

[0005] In order to achieve the above technical features, the purpose of the present invention is achieved as follows: a camera array for human-machine collaborative assembly scene perception and a data fusion method thereof, comprising a workbench, a robotic arm, a camera array and a data processing system;

[0006] The robotic arm is installed in the left front area of ​​the workbench;

[0007] The workbench is divided into an assembly area, a parts area and a tool area;

[0008] The camera array consists of five planar cameras arranged in the camera array space: camera No. 1, camera No. 2 and camera No. 3 are arranged on the left, right and right front sides of the workbench, and their arrangement heights are similar to the height of the assembler; camera No. 4 is arranged on the left front side and is higher than the workbench; camera No. 5 is arranged directly above the workbench and is higher than the workbench.

[0009] Preferably, the fourth camera is at least 0.3m higher than the workbench; the fifth camera is at least 1m higher than the workbench.

[0010] Preferably, the fourth camera is 0.3m higher than the workbench; the fifth camera is 1m higher than the workbench.

[0011] Preferably, the camera array space is a quarter sphere space with a radius of one meter and a center of the workbench as the center of the sphere.

[0012] Preferably, the first camera, the second camera and the third camera are used for action recognition, and the third camera, the fourth camera and the fifth camera are used for object recognition.

[0013] Preferably, the camera No. 3 adopts high confidence for object recognition of the entire workbench when there is occlusion in the camera No. 4 or the camera No. 5, and adopts high confidence only for object recognition in the tool area and the parts area when there is no occlusion, and adopts low confidence for the assembly area; the camera No. 4 adopts high confidence for large-sized objects in the entire operating table when there is no occlusion itself; the camera No. 5 adopts high confidence for small-sized objects in the parts area when there is no occlusion itself, and adopts low confidence for the camera No. 4 and the camera No. 5 when there is occlusion, and the camera No. 5 adopts low confidence for small-sized objects.

[0014] Preferably, the data processing system includes a recognition module and a fusion module. The recognition module includes object recognition and action recognition. The fusion module uses DS theory to perform data fusion and finally outputs the category of the workbench object and its confidence and the category of the assembler's action and its confidence.

[0015] Preferably, using the data processing system comprises the following steps:

[0016] Step 1: Collect the data set required for human-machine collaborative assembly scenarios and train the recognition model;

[0017] Step 2: Arrange cameras according to the camera array to obtain multi-source image data;

[0018] Step 3: Multi-source image data is input into the data processing system;

[0019] Step 4: Obtain preliminary prediction results from the object recognition model and action recognition model;

[0020] Step 5: Convert the prediction results into recognition evidence and use DS evidence theory to fuse multiple pieces of evidence to obtain the final recognition result.

[0021] Preferably, the data processing system includes an object recognition model and an action recognition model. Object recognition uses YOLOv7, and the trained prior frame is obtained by K-means clustering of the assembly scene data set. Action recognition first uses the Alphapose model to detect and identify the posture of the assembler in the image, and estimates the human skeleton and skeleton node data based on the posture of the person, establishes a skeleton spatiotemporal topology map, and then uses the ST-GCN action recognition network to determine the assembler's action based on the posture of the person.

[0022] Preferably, multiple pieces of evidence are converted from the results predicted by the recognition model based on multi-source data input, the category is converted into a recognition framework, the confidence is converted into a trust distribution function, and the DS theory is used to fuse the multiple pieces of evidence. Finally, the object category of the workbench and its confidence and the assembler's action category and its confidence are output, which improves the stability and recognition rate of the model and realizes scene perception of multi-source data fusion.

[0023] Beneficial effects of the present invention:

[0024] 1. The present invention proposes a camera array solution that acquires multi-source real-time data through cameras at multiple angles. This allows for stable perception of objects in the scene and the assembler's movements even when a single camera is blocked, thereby enhancing the reliability and stability of recognition and achieving stable perception of the current assembly environment.

[0025] 2. The present invention adopts DS evidence theory to fuse and analyze multiple pieces of evidence, integrates data from multiple cameras to understand the current scene, eliminates data singularity, realizes scene perception by fusion of multi-source data, and increases the accuracy of scene perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The present invention will be further described below with reference to the accompanying drawings and examples.

[0027] Figure 1 Schematic diagram of a camera array according to an embodiment of the present invention.

[0028] Figure 2 Schematic diagram of a human-machine collaborative assembly scenario according to an embodiment of the present invention.

[0029] Figure 3 This is a flowchart of multi-source data fusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0031] Example 1:

[0032] In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the drawings are all in a very simplified form and use non-precise proportions, which are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention. The specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. In addition, the data involved in the embodiments of the present invention described below are only a preferred value proposed for the experimental scenario and are not used to limit the present invention. The technical features involved in the embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0033] YOLOv7 is used as the object recognition model. A dataset of images taken from multiple angles, including the front, side, and top, is used to ensure that the amount of data for each class is no less than 300. The dataset is then enhanced, and a priori boxes are selected based on the enhanced dataset. The K-means clustering algorithm is used to cluster the label data of our dataset to obtain a priori box anchors suitable for our own dataset.

[0034] Camera array such as Figure 1 The workbench area is determined according to the human-machine collaborative assembly scenario. A quarter sphere with a radius of one meter is made with the center point of the workbench area as the center of the sphere to divide the camera array space. Cameras 1, 2, and 3 are placed on the left, right, and right front sides of the workbench, with a height close to the assembler's height. Camera 4 is placed on the left front side, 0.3 meters above the workbench, and Camera 5 is placed directly above the workbench, 1 meter above the workbench.

[0035] like Figure 2 As shown, five cameras are arranged within the camera array. Data from cameras 1, 2, and 3 is used for human action recognition, while data from cameras 3, 4, and 5 is used for object recognition. Adjust the camera viewing angles to the appropriate positions: Cameras 1 and 2 should be adjusted so that the assembler is centered within the camera image. Camera 3 should be adjusted so that both the assembler and the workbench are centered within the image. Cameras 4 and 5 should be adjusted so that the workbench is centered within the image. The camera No. 3 adopts high confidence for object recognition of the entire workbench when there is occlusion in the camera No. 4 or the camera No. 5. When there is no occlusion, it adopts high confidence only for object recognition in the tool area and the parts area, and adopts low confidence for the farther assembly area. The camera No. 4 adopts high confidence for large objects in the entire operating table when there is no occlusion itself. The camera No. 5 adopts high confidence for small objects in the parts area when there is no occlusion itself. The camera No. 4 and the camera No. 5 adopt low confidence when there is occlusion, and the camera No. 5 adopts low confidence for small objects.

[0036] The process of multi-source data fusion method is as follows Figure 3 As shown in the figure, three cameras simultaneously acquire multi-source real-time data, which is input into the recognition model to obtain the recognition result. The recognition result is converted into multiple pieces of evidence and the evidence is fused according to the DS theory to obtain the final recognition result.

[0037] YOLOv7 for objects The prediction result is in is the index of the object recognition class, is the confidence of the object, is the spatial coordinate of the object center, The human action recognition uses the Alphapose model to detect and identify the posture of the assembler in the image. The human skeleton and skeleton node data are estimated based on the posture of the assembler, and the skeleton spatiotemporal topology map is established. The ST-GCN action recognition network is then used to determine the assembler's action based on the posture of the assembler. The output is in is the action category index, is the skeleton node coordinate matrix.

[0038] When people and robotic arms collaborate in the assembly area, three cameras simultaneously identify the assembly parts in the scene. The object recognition prediction results of the three cameras are:

[0039]

[0040] In the formula For the Camera No. The prediction results of the object class, For the The prediction result of the camera includes the recognized object category, confidence level and coordinate information.

[0041] For objects Establishing an identification framework , For objects A finite set of all possible values ​​of existence or non-existence, containing two elements :object Existing in the scene, :object Does not exist in the scene.

[0042] Constructing a basic trust allocation function for camera number one :

[0043]

[0044] When the No. 1 camera does not recognize the object In order to avoid a complete conflict between the evidence, we directly assign , .

[0045] Similarly, we can get the basic trust distribution function of camera 2 and camera 3: , :

[0046]

[0047]

[0048] In the recognition framework Next, for objects There are three sets of evidence. According to DS evidence theory, we can get the object Existence of trust :

[0049]

[0050]

[0051] When fusing object recognition results, the above basic trust function needs to be modified when different cameras have different confidence levels:

[0052]

[0053] Where a is the reliability value high reliability Low confidence , is the original basic trust allocation function, is the revised basic trust allocation function.

[0054] Substitute the modified trust allocation function into equations (5) and (6).

[0055] When the object Existence of trust If it is greater than 0.6, we consider that there is object i in the scene.

[0056] After the above steps, the multi-source data fusion results of object recognition can be obtained. The same method can also be used to obtain the multi-source data fusion results of human action recognition, realizing scene perception of multi-source data fusion.

Claims

1. A camera array data fusion method for human-machine collaborative assembly scene perception, characterized in that: Includes a workbench, robotic arm, camera array, and data processing system; The robotic arm is installed in the left front area of ​​the workbench; The workbench is divided into an assembly area, a parts area and a tool area; The camera array is composed of five planar cameras arranged in a camera array space: Camera 1, Camera 2, and Camera 3 are arranged on the left, right, and right front sides of the workbench, with a height close to that of the assembler; Camera 4 is arranged on the left front side and is higher than the workbench; Camera 5 is arranged directly above the workbench and is higher than the workbench; The camera array space is a quarter sphere space with a radius of one meter and a center of the workbench as the sphere center; The data processing system includes a recognition module and a fusion module. The recognition module includes object recognition and action recognition. The fusion module uses DS theory to perform data fusion and finally outputs the category and confidence of the workbench object and the category and confidence of the assembler's action. Using the data processing system involves the following steps: Step 1: Collect the data set required for human-machine collaborative assembly scenarios and train the recognition model; Step 2: Arrange cameras according to the camera array to obtain multi-source image data; Step 3: Multi-source image data is input into the data processing system; Step 4: Obtain preliminary prediction results from the object recognition model and action recognition model; Step 5: Convert the prediction results into recognition evidence and use DS evidence theory to fuse multiple pieces of evidence to obtain the final recognition result.

2. The camera array data fusion method for human-machine collaborative assembly scene perception according to claim 1, characterized in that: The fourth camera is at least 0.3m higher than the workbench; the fifth camera is at least 1m higher than the workbench.

3. The camera array data fusion method for human-machine collaborative assembly scene perception according to claim 2, characterized in that: The fourth camera is 0.3m higher than the workbench; the fifth camera is 1m higher than the workbench.

4. The camera array data fusion method for human-machine collaborative assembly scene perception according to claim 3, characterized in that: The first, second and third cameras are used for action recognition, and the third, fourth and fifth cameras are used for object recognition.

5. The camera array data fusion method for human-machine collaborative assembly scene perception according to claim 4, characterized in that: When there is occlusion on camera 4 or camera 5, camera 3 uses high confidence for object recognition on the entire workbench. When there is no occlusion, camera 3 uses high confidence only for object recognition in the tool area and parts area, and uses low confidence for object recognition in the assembly area. Camera 4 has high confidence for large objects in the entire operating table even when there is no occlusion. Camera 5 adopts high confidence for small objects in the parts area when there is no occlusion. Cameras 4 and 5 adopt low confidence when there is occlusion. Camera 5 adopts low confidence for small objects.

6. The camera array data fusion method for human-machine collaborative assembly scene perception according to claim 1, characterized in that: The data processing system includes an object recognition model and an action recognition model. Object recognition uses YOLOv7, and the trained prior frame is obtained by K-means clustering of the assembly scene data set. Action recognition first uses the Alphapose model to detect and identify the posture of the assembler in the image, and estimates the human skeleton and skeleton node data based on the posture, establishes the skeleton spatiotemporal topology map, and then uses the ST-GCN action recognition network to determine the assembler's action based on the posture.

7. The camera array data fusion method for human-machine collaborative assembly scene perception according to claim 1, characterized in that: Multiple pieces of evidence are converted from the results predicted by the recognition model based on multi-source data input. Categories are converted into recognition frameworks, and confidence levels are converted into trust distribution functions. DS theory is used to fuse multiple pieces of evidence, and finally the object category and confidence level of the workbench and the action category and confidence level of the assembler are output, which improves the stability and recognition rate of the model and realizes scene perception based on multi-source data fusion.

Images