A part detection and positioning method and device based on multi-modal data

Abstract

This invention provides a method and apparatus for component detection and localization based on multimodal data. In an augmented reality-based manufacturing conformity inspection system, this method uses a multimodal neural network algorithm that fuses RGB and depth images. Features are extracted from and fused from both the RGB and depth images respectively. Finally, the two-dimensional detection results and depth information are used to perform three-dimensional spatial localization of the components. This method improves the robustness of the component detection algorithm in complex environments such as insufficient lighting, and enhances the model's accuracy, recall, and mean precision. Furthermore, this method can perform three-dimensional spatial localization of components based on the center points of the 2D bounding boxes output after detection and the aligned depth image.

Description

Technical Field

[0001] This invention relates to the field of target recognition, and more specifically to a method and apparatus for component detection and localization based on multimodal data. Background Technology

[0002] Object detection is a crucial task in computer vision, aiming to identify and locate objects of interest in images or videos. Currently, common object detection algorithms primarily use single-modal images relevant to the detection task as training data. These methods generally achieve good results, especially in well-lit environments where the target is clearly visible. However, the detection performance using only single-modal data remains limited in complex environments. For example, in low light, haze, or other visually disruptive conditions, the performance of these models often suffers significant degradation. For instance, RGB images are highly sensitive to changes in lighting. While they can provide rich color and texture information under good lighting conditions, they are prone to detail loss at night or in low light, resulting in unclear edges and features of the target object, making it difficult to provide sufficient information for object detection. In summary, while traditional single-modal data-based object detection algorithms can achieve good detection results in certain scenarios, they cannot fully meet the performance requirements of detection in complex environments. Summary of the Invention

[0003] The purpose of this invention is to provide a method and apparatus for component inspection and positioning based on multimodal data, which improves the accuracy and robustness of component inspection in complex scenarios of manufacturing conformity inspection, and at the same time performs precise three-dimensional spatial positioning of components.

[0004] The technical solution to achieve the purpose of this invention is as follows:

[0005] A method for component detection and localization based on multimodal data, comprising the following steps:

[0006] Obtain RGB and depth images of the components;

[0007] Image preprocessing includes filling holes in the depth image and aligning the depth image with the RGB image;

[0008] The depth image and RGB image are used as inputs to the YOLOv5 network to perform target detection on the parts.

[0009] By setting markers and using target detection data to locate the parts in three-dimensional space, the position of the parts can be obtained.

[0010] Furthermore, the RGB image and depth image are acquired by a camera.

[0011] Furthermore, the depth image and RGB image input to the YOLOv5 network are both 640 pixels in height and width.

[0012] Furthermore, the YOLOv5 network extends the CSPDarknet53 backbone network, including two parallel feature extraction layers, which extract features from depth images and RGB images respectively.

[0013] Furthermore, a feature fusion layer is added after the feature extraction layer of the YOLOv5 network to concatenate and fuse the feature maps of the RGB image and the depth image, and a 1×1 convolution operation is used to halve the number of channels in the concatenated feature map.

[0014] Furthermore, the feature map after splicing and fusion is as follows:

[0015]

[0016] Among them, F rgbd (c i ) represents the cth i The feature maps learned from each output channel, F rgb (k) represents the k-th RGB feature map, F d (j) represents the j-th depth feature map, n represents the number of channels of the RGB image features before fusion, and m represents the number of channels of the depth features before fusion.

[0017] Furthermore, the output of the YOLOv5 network includes the classified target, the detected bounding box position, and the confidence score.

[0018] Furthermore, the three-dimensional spatial positioning of the parts based on target detection data specifically includes:

[0019] Based on the detected bounding box position and the preprocessed depth image output by the YOLOv5 network, the 3D coordinates P of the component in the camera coordinate system are obtained. c for:

[0020]

[0021] Where X, Y, and Z represent the three-dimensional coordinates in the camera coordinate system; u and v represent the x and y coordinates of the component center point in the pixel coordinate system; c u and c v d represents the center pixel coordinates; f represents the pixel depth; c Indicates the camera's focal length;

[0022] Based on the current camera pose, the spatial coordinates P of the components in the SLAM world coordinate system are calculated. sw ,

[0023] P sw =R swc P c +t swc

[0024] Among them, R swc and t swc It is the rotation and translation matrix from the camera coordinate system to the SLAM world coordinate system;

[0025] Based on landmarks, the SLAM coordinate system is aligned with the geodetic coordinate system, and the spatial differences ΔR and Δt between the two are calculated.

[0026]

[0027] Among them, R aruco and t aruco It is the rotation and translation matrix labeled with ArUco in the real world, and is defined as the origin of the coordinate system in the real world. P aruco_in_SLAM and t aruco_in_SLAM It is the rotation and translation matrix of ArUco in the SLAM world coordinate system;

[0028] Based on the spatial differences ΔR and Δt, calculate the spatial position P of the component relative to the marker in the geodetic coordinate system. w P w =ΔRP sw +Δt.

[0029] Furthermore, the marker is marked using ArUco.

[0030] An apparatus for component detection and localization using the aforementioned multimodal data-based method includes an image acquisition module, an image preprocessing module, a target detection module, and a target localization module. The image acquisition module acquires RGB and depth images of the component. The image preprocessing module performs hole-filling processing on the depth image and aligns the depth image with the RGB image. The target detection module uses a YOLOv5 network, taking the depth and RGB images as inputs to the YOLOv5 network to detect the component. The target localization module performs three-dimensional spatial localization of the component based on the target detection data to obtain the component's position.

[0031] Compared with existing technologies, this invention has the following advantages: It uses a neural network algorithm that integrates RGB and depth images to more comprehensively describe the features of parts, improving the accuracy and robustness of part detection in complex manufacturing conformity inspection scenarios, while simultaneously enabling precise 3D spatial localization of parts. When the features of RGB images are affected by illumination intensity and direction, depth image information can help overcome the influence of illumination changes and complex backgrounds on part detection, improving the model's accuracy, recall, and mean precision. This method can perform 3D spatial localization of parts based on the center point of the 2D bounding box of the detected parts and the aligned depth image. Attached Figure Description

[0032] Figure 1 This is a flowchart of the component detection and positioning algorithm of the present invention.

[0033] Figure 2 This is a diagram of the YOLOv5 network structure that integrates RGB and depth images according to the present invention. Detailed Implementation

[0034] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0035] The component detection and localization method based on multimodal data in this embodiment relies on an augmented reality-based manufacturing conformity inspection system. The system's main control computer uses RGB and depth images acquired by an Intel RealSense DepthCamera D435i to run a real-time component detection and localization algorithm, and sends the processing results to a HoloLens 2 augmented reality head-mounted display for enhanced display. The flowchart of the component detection and localization algorithm is as described above. Figure 1 As shown, the method includes the following steps:

[0036] Step 1: The main control computer uses the Intel RealSense Depth Camera D435i to acquire RGB and depth images in real time;

[0037] Step 2: The main control computer first performs hole filling processing on the depth image, and then aligns the depth image with the RGB image;

[0038] Step 3: Use depth images and RGB images to perform target detection on the parts;

[0039] Step 4: Use depth images and component detection results to perform 3D spatial localization of the components;

[0040] Step 5: Send the component inspection and positioning results to the HoloLens2 augmented reality head-mounted display.

[0041] In one embodiment, the hole filling and alignment processing operation of the depth image in the second step of the above steps is mainly implemented by the hole_filling_filter and align_to_color methods provided in the Intel RealSense SDK.

[0042] In one embodiment, the component detection method is the YOLOv5 algorithm based on the fusion of RGB and depth images, and its network structure diagram is as described above. Figure 2 As shown, the specific method is as follows: An intermediate feature fusion strategy is used, based on an improved YOLOv5 network structure. Simultaneously, RGB images and depth images, each with a height and width of 640 pixels, are used as inputs to the network. The YOLOv5 CSPDarknet53 backbone network is extended, adding a feature extraction layer for extracting depth features and a feature fusion layer for fusing RGB image feature maps and depth image feature maps. The extended feature extraction layer is located at... Figure 2 Before the feature fusion layer in the image, the feature fusion layer concatenates and fuses the feature maps of the RGB image and the depth image, and uses a 1×1 convolution operation to halve the number of channels in the concatenated feature map, thus maintaining the original number of channels. After the 1×1 convolution operation, the fused feature map will have stronger expressive power and richer information than the arbitrary modal feature map before fusion. The fused feature map, obtained by concatenating the feature maps, can be represented by the following formula:

[0043]

[0044] Among them, F rgbd (c i ) represents the cth i The feature maps learned from each output channel, F rgb (k) represents the k-th RGB feature map, F d (j) represents the j-th depth feature map, n represents the number of channels of the RGB image features before fusion, and m represents the number of channels of the depth features before fusion.

[0045] In one embodiment, the component three-dimensional spatial positioning method uses the component detection results and the processed depth image for positioning. The specific method is as follows:

[0046] Step 1: Obtain the 3D coordinates P of the component in the camera coordinate system by comparing the center point of the 2D bounding box of the component after detection with the processed depth image. c As shown in the following formula:

[0047]

[0048] Where X, Y, and Z represent the three-dimensional coordinates in the camera coordinate system; u and v represent the x and y coordinates of the component center point in the pixel coordinate system; c u and c v d represents the center pixel coordinates; f represents the pixel depth; c This indicates the camera's focal length.

[0049] Step 2: Using ORB-SLAM2 as a visual odometry system, the camera pose is calculated in real time. Based on the camera pose, the spatial coordinates P of the components in the SLAM world coordinate system are calculated. sw As shown in the following formula:

[0050] P sw =R swc P c +t swc

[0051] Among them, R swc and t swc It is the rotation and translation matrix from the camera coordinate system to the SLAM world coordinate system.

[0052] Step 3: Use the detectMarkers method in the OpenCV library to identify ArUco markers, align the SLAM coordinate system with the real-world coordinate system (geocentric coordinate system), and calculate the spatial differences ΔR and Δt, as shown in the following formula:

[0053]

[0054] Among them, R aruco and t aruco These are the rotation and translation matrices labeled with ArUco in the real world, defined in the system as the origin of the real-world coordinate system; R aruco_in_SLAM and t aruco_in_SLAM It is the rotation and translation matrix of ArUco in the SLAM world coordinate system, obtained through geometric calculation.

[0055] Step 4: Calculate the spatial position P of the component in real space relative to the ArUco marker. w As shown in the following formula:

[0056] P w =ΔRP sw +Δt

[0057] In one embodiment, the results sent to the HoloLens2 augmented reality head-mounted display include component category information and its three-dimensional coordinates relative to the ArUco marker.

[0058] This invention addresses the problem of unstable detection results in current RGB image-based single-modal target detection methods, which are affected by factors such as illumination changes and occlusion. Depth images are more stable than RGB images and exhibit better robustness to illumination changes and occlusion. Therefore, fusing depth information with RGB images can compensate for the shortcomings of single-modal methods and improve the robustness and reliability of target detection.

[0059] The above embodiments are descriptions of specific implementations of the present invention, and not limitations thereof. Those skilled in the art can make various modifications and changes without departing from the spirit and scope of the present invention to obtain corresponding equivalent technical solutions. Therefore, all equivalent technical solutions should be included in the patent protection scope of the present invention.

Claims

1. A method for part detection and localization based on multi-modal data, the method comprising: Including the following steps: Obtain RGB and depth images of the components; Image preprocessing includes filling holes in the depth image and aligning the depth image with the RGB image; The depth image and RGB image are used as inputs to the YOLOv5 network to perform target detection on the parts. Set up markers, and perform three-dimensional spatial positioning of the parts based on the target detection results to obtain the position of the parts; The YOLOv5 network includes two parallel feature extraction layers, which extract features from depth images and RGB images respectively. The feature fusion layer is added behind the feature extraction layer of the YOLOv5 network, the feature map of the RGB image is spliced and fused with the feature map of the depth image, and 1 1 convolution operation reduces the number of channels of the spliced feature map by half; The feature map after splicing and fusion is as follows: wherein, represents the feature map learned by the i-th output channel, represents the i-th RGB feature map, represents the i-th depth feature map, represents the number of channels of the RGB image features before fusion, represents the number of channels of the depth features before fusion;​​​ The output of the YOLOv5 network includes the classified target, the detected bounding box position, and the confidence score. The three-dimensional spatial positioning of components based on target detection results specifically includes: Based on the detection frame position output by the YOLOv5 network and the preprocessed depth image, the three-dimensional coordinates of the parts in the camera coordinate system are obtained To: in, , , Represents the three-dimensional coordinates in the camera coordinate system; and Represents the x and y coordinates of the center point of the component in the pixel coordinate system; and Indicates the center pixel coordinates; Indicates pixel depth; Indicates the camera's focal length; Based on the current camera pose, the spatial coordinates of the components in the SLAM world coordinate system are calculated. , in, and It is the rotation and translation matrix from the camera coordinate system to the SLAM world coordinate system; Based on landmarks, the SLAM coordinate system is aligned with the geodetic coordinate system, and the spatial difference between the two is calculated. and , in, and It is the rotation and translation matrix of the marker in the geodetic coordinate system, and is defined as the origin of the geodetic coordinate system in the system. and It is the rotation and translation matrix of ArUco in the SLAM world coordinate system; Based on spatial difference and Calculate the spatial position of the component relative to the marker in the geodetic coordinate system. , .

2. The component detection and positioning method based on multimodal data according to claim 1, characterized in that, The RGB and depth images are acquired by a camera.

3. The component detection and positioning method based on multimodal data according to claim 1, characterized in that, The depth image and RGB image input to the YOLOv5 network are both 640 pixels in height and width.

4. The component detection and positioning method based on multimodal data according to claim 1, characterized in that, The marker is marked with ArUco.

5. An apparatus for component detection and positioning based on multimodal data as described in any one of claims 1-4, characterized in that, The system includes an image acquisition module, an image preprocessing module, a target detection module, and a target localization module. The image acquisition module acquires RGB and depth images of the component. The image preprocessing module performs hole filling on the depth image and aligns the depth image with the RGB image. The target detection module uses a YOLOv5 network, taking the depth and RGB images as inputs to the YOLOv5 network to detect the component. The target localization module performs three-dimensional spatial localization of the component based on the target detection results to obtain its position.

Images