Camera array for man-machine cooperation assembly scene perception and data fusion method thereof

Abstract

The invention provides a camera array for man-machine cooperation assembly scene perception and a data fusion method thereof. The camera array comprises a workbench, a mechanical arm, a camera array body and a data processing system. The mechanical arm is installed in the left front side area of the workbench. The workbench is divided into an assembling area, a part area and a tool area. The camera array comprises five plane cameras arranged in the camera array space, the first camera, the second camera and the third camera are arranged on the left side, the right side and the right front side of the workbench, the arrangement height is close to the height of an assembler, the fourth camera is arranged on the left front side and higher than the workbench, and the fifth camera is arranged over the workbench and higher than the assembler. Multi-source real-time data are acquired through the cameras at multiple angles, so that when a single camera is shielded, stable perception of objects in a scene and actions of assemblers can still be realized, the reliability and the stability of recognition are enhanced, and stable perception of a current assembly environment is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of human-machine collaboration and intelligent manufacturing, and specifically relates to a camera array for human-machine collaborative assembly scene perception and a data fusion method thereof. Background Art

[0002] With the development of industry, robots assisting workers in assembly in production workshops has gradually become a reality. To achieve human-machine collaborative assembly, robots are required to accurately and quickly recognize and understand the assembly parts and human behaviors in the assembly scene. The premise of recognition and understanding is that the robot can perceive the assembly parts and human behaviors in the scene.

[0003] Currently, the perception of assembly parts and human behavior in a scene mostly uses a single visual sensor to obtain scene images, and inputs object recognition models and human action recognition models to realize the perception of assembly objects and human behavior. However, the recognition rate of the recognition model for single visual data is unstable and cannot solve the problem of large-area occlusion. Summary of the invention

[0004] The present invention provides a camera array for human-machine collaborative assembly scene perception and a data fusion method thereof, which acquires multi-source real-time data through cameras at multiple angles, so that when a single camera is blocked, it is still possible to achieve stable perception of objects in the scene and the actions of the assembler, thereby enhancing the reliability and stability of recognition and achieving stable perception of the current assembly environment.

[0005] In order to realize the above technical features, the object of the present invention is achieved as follows: a camera array for human-machine collaborative assembly scene perception and a data fusion method thereof, comprising a workbench, a robotic arm, a camera array and a data processing system; The mechanical arm is installed in the left front area of ​​the workbench; The workbench is divided into an assembly area, a parts area and a tool area; The camera array consists of five planar cameras arranged in the camera array space: camera No. 1, camera No. 2 and camera No. 3 are arranged on the left side, right side and right front side of the workbench, and the arrangement height is close to the height of the assembler; camera No. 4 is arranged on the left front side and is higher than the workbench; camera No. 5 is arranged directly above the workbench and is higher than the workbench.

[0006] Preferably, the fourth camera is at least 0.3m higher than the workbench; the fifth camera is at least 1m higher than the workbench.

[0007] Preferably, the fourth camera is 0.3m higher than the workbench; the fifth camera is 1m higher than the workbench.

[0008] Preferably, the camera array space is a quarter sphere space with a radius of one meter and a center of the workbench as the sphere center.

[0009] Preferably, the first camera, the second camera and the third camera are used for action recognition, and the third camera, the fourth camera and the fifth camera are used for object recognition.

[0010] Preferably, the camera No. 3 adopts high confidence for object recognition of the entire workbench when there is occlusion in the camera No. 4 or the camera No. 5, and adopts high confidence only for object recognition in the tool area and the parts area when there is no occlusion, and adopts low confidence for the assembly area; the camera No. 4 adopts high confidence for large objects in the entire operating table when there is no occlusion itself; the camera No. 5 adopts high confidence for small objects in the parts area when there is no occlusion itself, the camera No. 4 and the camera No. 5 adopt low confidence when there is occlusion, and the camera No. 5 adopts low confidence for small objects.

[0011] Preferably, the data processing system includes a recognition module and a fusion module. The recognition module includes object recognition and action recognition. The fusion module uses DS theory to perform data fusion and finally outputs the category of the workbench object and its confidence and the category of the assembler's action and its confidence.

[0012] Preferably, using the data processing system comprises the following steps: Step 1: Collect the data set required for human-machine collaborative assembly scenarios and train the recognition model; Step 2: Arrange cameras according to the camera array to obtain multi-source image data; Step 3: Multi-source image data is input into the data processing system; Step 4: Obtain preliminary prediction results from the object recognition model and the action recognition model; Step 5: Convert the prediction results into recognition evidence, and use DS evidence theory to fuse multiple pieces of evidence to obtain the final recognition result.

[0013] Preferably, the data processing system includes an object recognition model and an action recognition model. Object recognition uses YOLOv7, and the trained prior frame is obtained by K-means clustering of the assembly scene data set. Action recognition first uses the Alphapose model to detect and identify the posture of the assembler in the image, and estimates the human skeleton and skeleton node data based on the posture of the person, establishes a skeleton spatiotemporal topology map, and then uses the ST-GCN action recognition network to determine the assembler's action based on the posture of the person.

[0014] Preferably, multiple pieces of evidence are converted based on the results predicted by the multi-source data input recognition model, the category is converted into a recognition framework, the confidence is converted into a trust allocation function, and the DS theory is used to fuse the multiple pieces of evidence, and finally the object category of the workbench and its confidence and the action category of the assembler and its confidence are output, so as to improve the stability and recognition rate of the model and realize scene perception of multi-source data fusion.

[0015] Beneficial effects of the present invention: 1. The present invention proposes a camera array solution, which acquires multi-source real-time data through cameras at multiple angles, so that when a single camera is blocked, it can still achieve stable perception of objects in the scene and the actions of the assembler, thereby enhancing the reliability and stability of recognition and achieving stable perception of the current assembly environment.

[0016] 2. The present invention adopts DS evidence theory to fuse and analyze multiple pieces of evidence, integrates data from multiple cameras to recognize the current scene, eliminates data singularity, realizes scene perception through multi-source data fusion, and increases the accuracy of scene perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0018] Figure 1 FIG. 4 is a schematic diagram of a camera array according to an embodiment of the present invention.

[0019] Figure 2 Schematic diagram of a human-machine collaborative assembly scenario according to an embodiment of the present invention.

[0020] Figure 3 The figure is a flowchart of multi-source data fusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The embodiments of the present invention are further described below in conjunction with the accompanying drawings.

[0022] Embodiment 1: In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be noted that the drawings are all in a very simplified form and use non-precise proportions, which are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention. The specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. In addition, the data involved in the embodiments of the present invention described below are only a preferred value proposed for the experimental scenario, and are not used to limit the present invention. The technical features involved in the embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0023] The object recognition model uses YOLOv7, and the image data set is taken from multiple angles such as the front, side and top. It ensures that the data volume of each class is no less than 300. The data set is enhanced, and the prior box is selected based on the enhanced data set. The K-means clustering algorithm is used to cluster the label data of our data set to obtain the prior box anchor suitable for our own data set.

[0024] Camera arrays such as Figure 1 , according to the human-machine collaborative assembly scenario, the workbench area is determined, and a quarter sphere with a radius of one meter is made with the center point of the workbench area as the center of the sphere to divide the camera array space. Cameras 1, 2, and 3 are arranged on the left, right, and right front sides of the workbench. The arrangement height is close to the height of the assembler. Camera 4 is arranged on the left front side, 0.3m higher than the workbench, and Camera 5 is arranged directly above the workbench, 1m higher than the workbench; like Figure 2 As shown, five cameras are arranged in the entire camera row space. The data of camera 1, camera 2 and camera 3 are used for human action recognition, and the data of camera 3, camera 4 and camera 5 are used for object recognition. Adjust the camera viewing angle to the appropriate position. The viewing angles of camera 1 and camera 2 need to be adjusted to the middle of the image where the assembler is located, the viewing angle of camera 3 needs to be adjusted to the middle of the image where the assembler and the workbench are both located, and the viewing angles of camera 4 and camera 5 need to be adjusted to the middle of the image where the workbench is located. The camera No. 3 adopts high confidence for object recognition of the entire workbench when the camera No. 4 or the camera No. 5 is obstructed. When there is no occlusion, it only adopts high confidence for object recognition in the tool area and the parts area, and adopts low confidence for the farther assembly area. The camera No. 4 adopts high confidence for large objects in the entire operating table when it is not obstructed. The camera No. 5 adopts high confidence for small objects in the parts area when it is not obstructed. The camera No. 4 and the camera No. 5 adopt low confidence when there is occlusion, and the camera No. 5 adopts low confidence for small objects.

[0025] The process of multi-source data fusion method is as follows Figure 3 As shown, three cameras simultaneously acquire multi-source real-time data, which are input into the recognition model to obtain the recognition result, and the recognition result is converted into multiple pieces of evidence, and the evidence is fused according to the DS theory to obtain the final recognition result.

[0026] YOLOv7 for objects The prediction result is in is the index of the object recognition class, is the confidence of the object, is the spatial coordinate of the center of the object, The Alphapose model is used to detect and identify the posture of the assembler in the image. The human skeleton and skeleton node data are estimated based on the posture of the assembler, and the skeleton spatiotemporal topology is established. The ST-GCN action recognition network is then used to determine the assembler's action based on the posture of the assembler. The output is in is the action category index, is the skeleton node coordinate matrix.

[0027] When people and robotic arms are working together in the assembly area, three cameras simultaneously identify the assembly parts in the scene: The object recognition prediction results of the three cameras are:

[0028] In the formula For the Camera No. The prediction results of the object class, For the The prediction result of the camera includes the recognized object category and confidence level and its coordinate information.

[0029] For objects Building an identification framework , For Objects A finite set of all possible values ​​of existence or non-existence, containing two elements :object Existing in the scene, :object Does not exist in the scene.

[0030] Constructing a basic trust allocation function for camera number one :

[0031] When Camera 1 does not recognize the object In order to avoid a complete conflict between the evidences, we directly assign , .

[0032] Similarly, we can get the basic trust distribution function of camera 2 and camera 3: , :

[0033]

[0034] In the identification framework Next, for the object There are three sets of evidence. According to DS evidence theory, we can get the object The degree of trust that exists :

[0035]

[0036] When fusing object recognition results, the above basic trust function needs to be modified when different cameras have different confidence levels:

[0037] Where a is the reliability value high reliability Low confidence , is the original basic trust allocation function, is the revised basic trust allocation function.

[0038] Substitute the modified trust allocation function into equation (5) and equation (6).

[0039] When the object The degree of trust that exists If it is greater than 0.6, we consider that there is object i in the scene.

[0040] After the above steps, the multi-source data fusion result of object recognition can be obtained. The same method can also be used to obtain the multi-source data fusion result of human action recognition, thus realizing scene perception of multi-source data fusion.

Claims

1. A camera array for human-machine collaborative assembly scene perception and a data fusion method thereof, characterized in that: Includes a workbench, robotic arm, camera array, and data processing system; The mechanical arm is installed in the left front area of ​​the workbench; The workbench is divided into an assembly area, a parts area and a tool area; The camera array consists of five planar cameras arranged in the camera array space: camera No. 1, camera No. 2 and camera No. 3 are arranged on the left side, right side and right front side of the workbench, and the arrangement height is close to the height of the assembler; camera No. 4 is arranged on the left front side and is higher than the workbench; camera No. 5 is arranged directly above the workbench and is higher than the workbench.

2. A camera array and data fusion method for human-machine collaborative assembly scene perception according to claim 1, characterized in that: The fourth camera is at least 0.3m higher than the workbench; the fifth camera is at least 1m higher than the workbench.

3. A camera array and data fusion method for human-machine collaborative assembly scene perception according to claim 2, characterized in that: The fourth camera is 0.3m higher than the workbench; the fifth camera is 1m higher than the workbench.

4. A camera array and data fusion method for human-machine collaborative assembly scene perception according to claim 3, characterized in that: The camera array space is a quarter spherical space with a radius of one meter and a center of the workbench as the sphere center.

5. A camera array and data fusion method for human-machine collaborative assembly scene perception according to claim 3, characterized in that: The first camera, the second camera and the third camera are used for action recognition, and the third camera, the fourth camera and the fifth camera are used for object recognition.

6. A camera array for human-machine collaborative assembly scene perception and a data fusion method thereof according to claim 4, characterized in that: When the fourth camera or the fifth camera is blocked, the third camera uses high confidence for object recognition of the entire workbench. When there is no blockage, the third camera uses high confidence only for object recognition of the tool area and the parts area, and uses low confidence for the assembly area. Camera 4 takes high confidence for large objects in the entire operating table when there is no occlusion. Camera 5 adopts high confidence for small objects in the parts area when there is no occlusion. Cameras 4 and 5 adopt low confidence when there is occlusion. Camera 5 adopts low confidence for small objects.

7. A camera array for human-machine collaborative assembly scene perception and a data fusion method thereof according to claim 1, characterized in that: The data processing system includes a recognition module and a fusion module. The recognition module includes object recognition and action recognition. The fusion module uses DS theory to perform data fusion and finally outputs the category and confidence of the workbench object and the category and confidence of the assembler's action.

8. A camera array and data fusion method for scene perception of human-machine collaborative assembly according to claim 7, characterized in that: Using the data processing system includes the following steps: Step 1: Collect the data set required for human-machine collaborative assembly scenarios and train the recognition model; Step 2: Arrange cameras according to the camera array to obtain multi-source image data; Step 3: Multi-source image data is input into the data processing system; Step 4: Obtain preliminary prediction results from the object recognition model and the action recognition model; Step 5: Convert the prediction results into recognition evidence, and use DS evidence theory to fuse multiple pieces of evidence to obtain the final recognition result.

9. A camera array for human-machine collaborative assembly scene perception and a data fusion method thereof according to claim 8, characterized in that: The data processing system includes an object recognition model and an action recognition model. Object recognition uses YOLOv7, and the trained prior frame is obtained by K-means clustering of the assembly scene data set. Action recognition first uses the Alphapose model to detect and recognize the posture of the assembler in the image, and estimates the human skeleton and skeleton node data based on the posture of the person, establishes a skeleton spatiotemporal topology map, and then uses the ST-GCN action recognition network to determine the assembler's action based on the posture of the person.

10. A camera array for human-machine collaborative assembly scene perception and a data fusion method thereof according to claim 8, characterized in that: Multiple pieces of evidence are converted from the results predicted by the recognition model based on multi-source data input. Categories are converted into recognition frameworks, and confidence is converted into trust allocation functions. DS theory is used to fuse multiple pieces of evidence, and finally the object category and confidence of the workbench and the action category and confidence of the assembler are output to improve the stability and recognition rate of the model and realize scene perception of multi-source data fusion.

Images