Target positioning method and device, equipment, storage medium and product

Abstract

The invention discloses a target positioning method and device, equipment, a storage medium and a product. The method comprises the steps that a target RGB image and a target depth image of the same target area are acquired; inputting the target RGB image and the target depth image into a pre-constructed double-flow network to extract visual features of the target RGB image and geometric features of the target depth image, and obtaining a prediction image according to fusion features obtained by fusing the visual features and the geometric features; and training a double-flow network by using a preset loss function and the predicted image to obtain a trained double-flow network, and performing target positioning on the to-be-detected image by using the trained double-flow network. According to the embodiment of the invention, the target positioning accuracy can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a target localization method, apparatus, device, storage medium, and product. Background Technology

[0002] LED indicator lights in computer room equipment are typically used to indicate the operating status, connection status, and other information of the equipment, playing an important role in equipment status monitoring and fault diagnosis.

[0003] With the rapid development of deep learning technology, object detection has made great progress. Utilizing deep neural networks to process images and achieve automated LED indicator location can improve the management efficiency and fault handling speed of data center equipment. LED indicator targets are small and densely packed in complex backgrounds; existing solutions relying solely on single-modal information cannot fully utilize image features, resulting in low accuracy in location results. Summary of the Invention

[0004] This application provides a target localization method, apparatus, device, storage medium, and product to solve the problem that the prior art relies only on a single modal information, which cannot make full use of image features, resulting in low accuracy of localization results.

[0005] To achieve the above objectives, embodiments of this application provide a target localization method, including:

[0006] Obtain the target RGB image and target depth map of the same target area;

[0007] The target RGB image and the target depth image are input into a pre-constructed two-stream network to extract the visual features of the target RGB image and the geometric features of the target depth image. The predicted image is obtained by fusing the visual features and the geometric features.

[0008] The dual-stream network is trained using a preset loss function and the predicted image to obtain a trained dual-stream network, which is then used to locate the target in the image to be detected.

[0009] As an improvement to the above scheme, the structure of the dual-stream network includes: a first feature network, a second feature network, a first fusion module, a decoding module, and an output module;

[0010] The first feature network is used to extract the visual features;

[0011] The second feature network is used to extract the geometric features;

[0012] The fusion module is used to fuse the visual features and the geometric features to obtain the fused features;

[0013] The decoding module is used to decode the fused features to obtain a decoded image;

[0014] The output module is used to generate the predicted image based on the decoded image.

[0015] As an improvement to the above scheme, the second feature network includes: a sparse convolution module, a geometric feature extraction module, a multi-scale feature fusion module, a global context awareness module, a deep edge enhancement module, and a second fusion module;

[0016] The sparse convolution module is used to perform convolution operations on the effective region of the target depth map and output the first target depth map;

[0017] The geometric feature extraction module is used to extract geometric features from the first target depth map and output a second target depth map;

[0018] The multi-scale feature fusion module is used to extract and fuse features of different scales from the second target depth map;

[0019] The global context awareness module is used to extract global context information of the second target depth map;

[0020] The depth edge enhancement module is used to extract and enhance the edge information of the second target depth map;

[0021] The second fusion module is used to fuse the information output by the multi-scale feature fusion module, the global context awareness module, and the deep edge enhancement module to obtain the geometric features.

[0022] As an improvement to the above scheme, the loss function includes at least one of the following: cross-entropy loss function, Dice loss function, edge loss function, depth-aware loss function, and composite loss function;

[0023] The cross-entropy loss function is used to compare the difference between the classification results of pixels in the predicted image and the classification results of pixels in the real image.

[0024] The Dice loss function is used to measure the degree of overlap between the positive sample regions of the predicted image and the positive sample regions of the real image.

[0025] The edge loss function is used to enhance the edge information of the depth map;

[0026] The depth-sensing loss function is used to compare the depth value of the predicted image with the depth value of the real image.

[0027] The composite loss function is obtained by weighted summation of at least one of the cross-entropy loss function, Dice loss function, edge loss function, and depth perception loss function.

[0028] As an improvement to the above solution, the step of obtaining the target RGB image and target depth map of the same target region includes:

[0029] Obtain the first RGB image and the first depth image of the same target region;

[0030] The first RGB image and the first depth image are subjected to a first preprocessing to obtain a second RGB image and a second depth image; wherein, the first RGB image and the first depth image are subjected to geometric calibration and image alignment in sequence, and the aligned first RGB image is subjected to joint bilateral filtering interpolation and resampling to match the resolution of the aligned first RGB image;

[0031] If the preset error condition is met, a second preprocessing is performed on the second RGB image and the second depth image, namely the target RGB image and the target depth image; the second preprocessing includes at least one of data standardization and data augmentation.

[0032] As an improvement to the above scheme, the preset error condition includes that the average pixel error of the second RGB image and the second depth image is less than a preset value.

[0033] To achieve the above objectives, embodiments of this application also provide a target positioning device, comprising:

[0034] The acquisition module is used to acquire the target RGB image and target depth image of the same target area;

[0035] An extraction module is used to input the target RGB image and the target depth image into a pre-constructed two-stream network to extract the visual features of the target RGB image and the geometric features of the target depth image, and to obtain a predicted image based on the fusion features obtained by fusing the visual features and the geometric features.

[0036] The training module is used to train the dual-stream network using a preset loss function and the predicted image to obtain a trained dual-stream network, which is then used to locate the target in the image to be detected.

[0037] To achieve the above objectives, embodiments of this application also provide a target positioning device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the target positioning method as described above when executing the computer program.

[0038] To achieve the above objectives, embodiments of this application also provide a computer-readable storage medium, the computer-readable storage medium including a stored computer program; wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the target positioning method as described above.

[0039] To achieve the above objectives, embodiments of this application also provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement the target location method as described above.

[0040] Compared with existing technologies, the target localization method, apparatus, device, storage medium, and product provided in this application embodiment acquire a target RGB image and a target depth map of the same target region; input the target RGB image and the target depth map into a pre-constructed two-stream network to extract visual features of the target RGB image and geometric features of the target depth map, and obtain a predicted image based on the fused features obtained by fusing the visual features and the geometric features; train the two-stream network using a preset loss function and the predicted image to obtain a trained two-stream network, and use the trained two-stream network to perform target localization on the image to be detected. Therefore, this application embodiment, through the two-stream network, fully leverages the feature advantages of both the RGB image and the depth map, making full use of the geometric features of the depth map and the visual features of the RGB image, significantly improving the accuracy and robustness of small target object localization, especially showing better performance in complex scenes. Attached Figure Description

[0041] Figure 1 This is a flowchart of a target localization method provided in an embodiment of this application;

[0042] Figure 2 This is a structural block diagram of a two-stream network provided in an embodiment of this application;

[0043] Figure 3 This is a structural block diagram of a first feature network provided in an embodiment of this application;

[0044] Figure 4 This is a structural block diagram of a second feature network provided in an embodiment of this application;

[0045] Figure 5 This is a structural block diagram of a target positioning device provided in an embodiment of this application;

[0046] Figure 6 This is a structural block diagram of a target positioning device provided in an embodiment of this application. Detailed Implementation

[0047] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0048] See Figure 1 , Figure 1 This is a flowchart of a target localization method provided in an embodiment of this application. The target localization method includes:

[0049] S1. Obtain the target RGB image and target depth image of the same target area;

[0050] S2. Input the target RGB image and the target depth image into a pre-constructed two-stream network to extract the visual features of the target RGB image and the geometric features of the target depth image, and obtain the predicted image based on the fusion features obtained by fusing the visual features and the geometric features.

[0051] S3. The dual-stream network is trained using a preset loss function and the predicted image to obtain a trained dual-stream network, which is then used to locate the target in the image to be detected.

[0052] It's worth noting that a depth camera is used to capture the target depth map, while an RGB camera is used to capture the target RGB image. Because LED indicators are small and emit strong light, a light field camera with anti-light interference capabilities is preferred. This allows the light field camera to capture depth maps of the LED indicator from multiple angles, recording the distance of each pixel to the camera and providing the three-dimensional position of the LED indicator. The RGB camera is preferably a high-resolution camera. Using a high-resolution camera to capture the RGB image of the LED indicator ensures clear recording of its color, brightness, and emission status under various lighting conditions. The RGB camera needs to be sensitive to weak light sources to avoid overexposure or image blurring in low light.

[0053] In this embodiment, a two-stream network is used to process the target RGB image and target depth map of the same target region in parallel. The extracted visual features of the target RGB image and the geometric features of the target depth map are fused to obtain fused features, and then a predicted image is obtained through the fused features. Generally, the predicted image has the same resolution as the target RGB image and target depth map.

[0054] Optionally, the visual features of the extracted target RGB image and the geometric features of the target depth image are fused. Specifically, the visual features and geometric features are concatenated to obtain a fused feature. By directly combining and concatenating the visual features and geometric features into a single feature map for processing, information from different modalities is fully utilized, improving the fusion effect and making it suitable for the direct processing of complex data.

[0055] Optionally, the visual features of the target RGB image include at least one of the following: color, brightness, and texture information. The geometric features of the target depth image include at least one of the following: depth, position, gradient, and surface normal vector, with spatial geometric features being particularly important.

[0056] In this embodiment, the trained dual-stream network can perform high-precision target segmentation and / or target detection to achieve high-precision target localization. For example, the image to be detected is input into the trained dual-stream network to obtain a target segmentation result, which indicates the location of the target object. Similarly, the image to be detected is input into the trained dual-stream network to obtain a target detection result, which indicates the location of the target object. This embodiment ensures that the dual-stream network can fully utilize depth and RGB information to achieve high-precision target detection and segmentation. The image to be detected is the RGB image and depth image of the region to be detected; the target object is the object to be located, such as an LED indicator, an LED indicator in an illuminated state, an LED indicator in a non-illuminated state, etc., which can be set according to actual needs.

[0057] For example, by setting the target object as an LED indicator light that is in an illuminated state, and by using a trained dual-stream network to identify the brightness of the LED indicator lights on the rack equipment, it is possible to quickly detect which devices in the rack are powered on, which can greatly save the search time of maintenance personnel, thereby reducing costs and improving efficiency.

[0058] For example, by setting the target object as an LED indicator light that is not emitting light, and by using a trained dual-stream network to identify the brightness of the LED indicator lights on the rack equipment, it is possible to quickly detect which devices in the rack have lost power, which can greatly save the search time of maintenance personnel, thereby reducing costs and improving efficiency.

[0059] For example, target objects are set as LED indicator lights. For target segmentation, a threshold of 0.8 is set, and pixels below the threshold are classified as background, while pixels above the threshold are classified as LED indicator lights. For target detection, non-maximum suppression is used to remove redundant detection boxes, ensuring that only one optimal detection box is retained for each target object. Then, the detection results are visualized and overlaid on the original image to demonstrate the detection effect of the two-stream network. The detection results are output in a specific format (such as JSON, XML, etc.) for use by downstream tasks or systems.

[0060] In one alternative embodiment, such as Figure 2 The structure of the dual-stream network includes: a first feature network, a second feature network, a first fusion module, a decoding module, and an output module;

[0061] The first feature network is used to extract the visual features;

[0062] Specifically, the first feature network is a convolutional neural network used to extract visual features from the target RGB image. For example, the structure of the first feature network is as follows: Figure 3 As shown, ResNet50 balances computational cost and feature representation capability by inserting an additional 3x3 fine-grained convolutional layer in the second convolutional module. Of course, the first feature network can also be other structures, as long as it can extract the visual features of the target RGB image, such as only including convolutional and pooling modules. No specific restrictions are imposed here.

[0063] The second feature network is used to extract the geometric features;

[0064] Specifically, the second feature network is a convolutional neural network used to extract geometric features, especially spatial geometric features, from the target depth map. The specific structure of the second feature network only needs to be able to extract the geometric features of the depth map; for example, it can include only a geometric feature extraction module. No specific restrictions are imposed here.

[0065] The fusion module is used to fuse the visual features and the geometric features to obtain the fused features;

[0066] Specifically, the visual features of the target RGB image and the geometric features of the target depth image are concatenated along the channel dimension to synthesize a multi-channel feature map. This method can preserve information of each modality, improve the fusion effect, and is suitable for the direct processing of complex data.

[0067] The decoding module is used to decode the fused features to obtain a decoded image;

[0068] Specifically, the decoding module uses upsampling to restore the low-resolution multi-channel feature map to a decoded map with the same resolution as the target RGB image and target depth map, which is then used for subsequent target segmentation and target detection. During the decoding process, skip connections similar to U-Net are used to combine the high-resolution features from the encoder with the low-resolution features from the decoder, ensuring that more detailed information is preserved during the decoding stage and improving segmentation accuracy.

[0069] The output module is used to generate the predicted image based on the decoded image.

[0070] Specifically, the predicted image includes object segmentation results and / or object detection results, i.e., outputting classification information and / or detection boxes for each pixel. For object segmentation tasks, object segmentation results are generated. Specifically, a probability map of the category to which each pixel belongs is generated using the Softmax activation function, ultimately generating object segmentation results for accurate location and identification of target objects. For object detection tasks, object detection results are generated. The output module can generate detection boxes or keypoints for target objects, used for accurate spatial location of target objects.

[0071] In one alternative embodiment, such as Figure 4 The second feature network includes: a sparse convolution module, a geometric feature extraction module, a multi-scale feature fusion module, a global context awareness module, a deep edge enhancement module, and a second fusion module;

[0072] The sparse convolution module is used to perform convolution operations on the effective region of the target depth map and output the first target depth map;

[0073] It's worth noting that some regions in the target depth map may be sparse or missing. Therefore, the sparse convolution module only processes regions containing valid depth information, i.e., the valid regions, thus saving computational resources. The sparse convolution module can effectively handle the sparsity and discontinuity in the depth map, improving computational efficiency and reducing unnecessary computational overhead.

[0074] To expand the receptive field, embodiments of this application combine sparse convolution modules with dilated convolution techniques to capture a wider range of contextual information without increasing computational complexity, which is particularly suitable for small target detection of distant LED indicators.

[0075] The geometric feature extraction module is used to extract geometric features from the first target depth map and output a second target depth map;

[0076] It is worth noting that the geometric feature extraction module extracts the geometric features of the first target depth map, thereby enhancing the network's understanding of the shape and position of objects in three-dimensional space.

[0077] For example, by analyzing the gradient of the first target's depth map, the surface normal vector of each pixel in the scene is calculated. The surface normal vector helps the network better understand the surface structure and object boundaries in 3D space. By calculating the Sobel operator for each pixel in the depth map, a gradient map describing depth changes is generated, which is used to identify the edges and contours of the target object. Depth gradients are particularly important for segmentation tasks, helping the network to better distinguish between the target and the background. The relevant formulas are as follows:

[0078] Sobel operator in the horizontal direction (x-direction):

[0079]

[0080] Sobel operator in the vertical direction (y-direction):

[0081]

[0082] Horizontal gradient:

[0083]

[0084] Where D(u,v) is the depth value of the depth map at pixel (u,v).

[0085] Vertical gradient:

[0086]

[0087] The surface normal vector is obtained from the horizontal and vertical gradients. Specifically, the surface normal vector is obtained by cross-product of the horizontal and vertical gradients.

[0088] Furthermore, based on the horizontal and vertical gradients, the gradient magnitude of each pixel can be calculated to detect edges or regions of significant depth change in the depth map.

[0089]

[0090] The multi-scale feature fusion module is used to extract and fuse features of different scales from the second target depth map;

[0091] It is worth noting that the multi-scale feature fusion module extracts and fuses features of different scales from the second target depth map, ensuring that the second feature network can recognize target objects of various sizes.

[0092] Specifically, multiple convolutional kernels of different scales (such as 3x3, 5x5, and 7x7) are designed to extract detailed and global features of the target object at different levels. Features from different scales are fused to ensure the network can handle both small and large targets simultaneously and maintain high sensitivity to target objects of various sizes. Furthermore, the convolutional kernels can be 3D convolutions to process multi-frame depth map sequences obtained from the data center, capturing the spatiotemporal features of the target object at different scales.

[0093] The global context awareness module is used to extract global context information of the second target depth map;

[0094] It is worth noting that the global context awareness module extracts global context information from the depth map of the second target in order to suppress background information in the depth map of the second target and enhance the features of small targets.

[0095] The global context awareness module extracts global context information from the scene through a global average pooling layer to help the network understand the relative position of targets and their environmental relationships within the scene. This global context information is particularly helpful when multiple similar target objects exist in the scene, enabling the network to perform more accurate identification.

[0096] The depth edge enhancement module is used to extract and enhance the edge information of the second target depth map;

[0097] It is worth noting that the deep edge enhancement module extracts and enhances the edge information in the second target depth map, enabling the second feature network to more accurately identify and segment the target's edges.

[0098] Edge detection is performed on the second target depth map using the Sobel operator, generating an edge map containing edge information. Furthermore, during training, edge information is incorporated into the loss function, forcing the network to focus on edge features in the depth map. This approach effectively improves the network's segmentation accuracy at target boundaries.

[0099] The second fusion module is used to fuse the information output by the multi-scale feature fusion module, the global context awareness module, and the deep edge enhancement module to obtain the geometric features.

[0100] It is worth noting that the multi-scale feature fusion module outputs fused features at different scales, the global context awareness module outputs global context information, and the depth edge enhancement module outputs edge information. The information output by these three modules is fused to obtain the geometric features of the target depth map.

[0101] In one optional embodiment, the loss function includes at least one of the following: cross-entropy loss function, Dice loss function, edge loss function, depth-aware loss function, and composite loss function;

[0102] The cross-entropy loss function is used to compare the difference between the classification results of pixels in the predicted image and the classification results of pixels in the real image.

[0103] The Dice loss function is used to measure the degree of overlap between the positive sample regions of the predicted image and the positive sample regions of the real image.

[0104] The edge loss function is used to enhance the edge information of the depth map;

[0105] The depth-sensing loss function is used to compare the depth value of the predicted image with the depth value of the real image.

[0106] The composite loss function is obtained by weighted summation of at least one of the cross-entropy loss function, Dice loss function, edge loss function, and depth perception loss function.

[0107] In this embodiment, a loss function is used to measure the difference between the output of the two-stream network and the true label, and to guide the optimization of the two-stream network parameters. To ensure that the two-stream network can effectively process the visual features of the target RGB image and the geometric features of the target depth image, this embodiment uses at least one of the following loss functions for training:

[0108] Cross-entropy loss function: Used for pixel-level classification problems, it measures the difference between the predicted classification result and the true classification result for each pixel; that is, the difference between the classification result of the predicted image pixels and the classification result of the real image pixels. For example, in the LED indicator segmentation task, cross-entropy loss is used to compare the difference between the classification probability (such as background or LED indicator) of each pixel output by the two-stream network and the true label of the pixel. Minimizing this difference can improve the segmentation accuracy.

[0109] The cross-entropy loss function is specifically expressed as:

[0110]

[0111] Among them, y i It is the true label of the i-th pixel. is the probability predicted by the two-stream network, and N is the total number of pixels.

[0112] The Dice loss function measures the overlap between positive sample regions (i.e., target object regions) in the predicted image and positive sample regions in the ground truth image, improving the detection accuracy for small targets, especially when the target object region is disproportionately large compared to the background. For example, it measures the overlap between LED indicator regions in the predicted image and those in the ground truth image. The Dice loss effectively handles imbalanced sampling, guiding the model to focus on smaller targets when they occupy a smaller proportion of the overall image area.

[0113] The Dice loss function is specifically expressed as follows:

[0114]

[0115] Where P is the positive sample region of the predicted image, and T is the positive sample region of the real image.

[0116] Edge loss function: This function enhances the network's ability to detect edge information in the depth map, enabling the two-stream network to more accurately detect the edges and shapes of LED indicators. Specifically, by calculating gradient information in the depth map, it guides the network to focus more on edge regions, thereby improving the ability to detect target boundaries. This loss function can be used in conjunction with the gradient image of the depth map.

[0117] The edge loss function is specifically expressed as:

[0118]

[0119] in, and are the gradients of the predicted image and the real image at pixel (i,j), respectively.

[0120] Depth-aware loss function: This function compares the depth values ​​of the predicted image with those of the ground truth image, enabling two-stream networks to place greater emphasis on the geometric information of the depth map. This is particularly beneficial in object detection and segmentation, where depth information is utilized to optimize network performance. The loss function can be designed by comparing the depth values ​​of the predicted image with those of the ground truth image, or by using edge information from the depth map as a supervisory signal. This helps the network better understand 3D structures and reduces errors caused by lighting or background interference.

[0121] The depth-aware loss function is specifically expressed as follows:

[0122]

[0123] Among them, D i It is the depth value of the actual image. is the predicted depth value of the image, and N is the total number of pixels.

[0124] Composite loss function: This is used to combine different loss functions to optimize model performance for different tasks. For example, in the task of detecting LED indicators, cross-entropy loss, Dice loss, edge loss, and depth perception loss are often combined to form a composite loss function, so as to ensure that the two-stream network can be comprehensively optimized when handling different tasks.

[0125] The composite loss function is specifically expressed as follows:

[0126] L total =αL CE +βL Dice +γL Edge +δL Depth

[0127] Where α is the weight coefficient of the cross-entropy loss function, β is the weight coefficient of the Dice loss function, γ is the weight coefficient of the edge loss function, and δ is the weight coefficient of the depth perception loss function.

[0128] In an optional embodiment, obtaining the target RGB image and target depth map of the same target region includes:

[0129] Obtain the first RGB image and the first depth image of the same target region;

[0130] The first RGB image and the first depth image are subjected to a first preprocessing to obtain a second RGB image and a second depth image; wherein, the first RGB image and the first depth image are subjected to geometric calibration and image alignment in sequence, and the aligned first RGB image is subjected to joint bilateral filtering interpolation and resampling to match the resolution of the aligned first RGB image;

[0131] If the preset error condition is met, a second preprocessing is performed on the second RGB image and the second depth image, namely the target RGB image and the target depth image; the second preprocessing includes at least one of data standardization and data augmentation.

[0132] This application embodiment preprocesses the first RGB image and the first depth image to ensure that these data can be effectively utilized by the dual-stream network. Optionally, the first preprocessing includes geometric calibration, image alignment, interpolation, and resampling. The second preprocessing includes data normalization and data augmentation.

[0133] For example, geometric calibration, image alignment, interpolation, and resampling are performed sequentially on the first RGB image and the first depth image.

[0134] Geometric calibration: Geometric calibration is performed on the RGB camera and depth camera to ensure that the coordinate systems of the two cameras are consistent, thereby guaranteeing the image alignment accuracy in subsequent processing. Specifically, a checkerboard pattern is used for calibration to calibrate the intrinsic parameters (such as focal length, principal point position, and distortion parameters) and extrinsic parameters (relative position and rotation) of the RGB camera and depth camera, respectively. These calibration parameters will be used in the geometric alignment process. This embodiment uses the standard camera calibration tools provided by OpenCV to calculate the camera's intrinsic and extrinsic parameters.

[0135] RGB camera intrinsic parameter matrix:

[0136]

[0137] Among them, f rgb_x It is the focal length of the RGB camera in the x-direction, f rgb_y It is the focal length of the RGB camera in the y-direction, (c rgb_x ,c rgb_y () represents the coordinates of the principal point in the first RGB image.

[0138] Depth camera intrinsic parameter matrix:

[0139]

[0140] Among them, f d_x It is the focal length of the depth camera in the x-direction, f d_y It is the focal length of the depth camera in the y-direction, (c d_x ,c d_y ) are the coordinates of the principal point in the first depth map.

[0141] Extrinsic parameter matrix:

[0142]

[0143] Where R is the rotation matrix and T is the translation vector, representing the relative positional relationship between the depth camera and the RGB camera.

[0144] r 11 Let r be the element in the first row and first column of the rotation matrix. 21 Let r be the element in the second row and first column of the rotation matrix. 31 Let r be the element in the third row and first column of the rotation matrix. 12 Let r be the element in the first row and second column of the rotation matrix. 22 Let r be the element in the second row and second column of the rotation matrix. 32 Let r be the element in the third row and second column of the rotation matrix. 13 Let r be the element in the first row and third column of the rotation matrix. 23 Let r be the element in the second row and third column of the rotation matrix. 33The elements in the third row and third column of the rotation matrix represent the camera's rotation method, specifically describing how it rotates from the world coordinate system to the camera coordinate system. For example: the first column vector (r 11 ,r 21 ,r 31 The first column vector (r) represents how the x-axis in the world coordinate system is transformed into the x-axis in the camera coordinate system; the second column vector (r) represents how the x-axis in the world coordinate system is transformed into the x-axis in the camera coordinate system. 21 ,r 22 ,r 23 This describes how the y-axis in the world coordinate system is transformed to the y-axis in the camera coordinate system. The third column vector (r) 31 ,r 32 ,r 33 This refers to how the z-axis in the world coordinate system is transformed into the z-axis in the camera coordinate system.

[0145] t x Let t be the x-coordinate of the camera center in the world coordinate system. y Let t be the y-coordinate of the camera center in the world coordinate system. z Let z be the z-coordinate of the camera center in the world coordinate system.

[0146] Image alignment: Since the first RGB image and the first depth image usually come from different cameras, there is a spatial inconsistency, which requires alignment to ensure that the correspondence of each pixel in the two images is consistent.

[0147] Specifically: Using the extrinsic parameter matrix from the calibration results, the pixels of the first depth map are projected onto the first RGB image. The first depth map is mapped to correspond one-to-one with the pixels of the first RGB image. This requires calculating the projection relationship for each pixel to ensure that the aligned images are in the same spatial space.

[0148] For each pixel in the first depth map, assume the depth value is Z. d Its corresponding 3D coordinates in the camera coordinate system are (X d ,Y d Z d The calculation of this three-dimensional point is as follows:

[0149] First, transform from depth image coordinates to 3D coordinates in the depth camera coordinate system:

[0150]

[0151] Among them, (u d ,v d Z represents the coordinates of a pixel in the first depth map. d It corresponds to the depth value, f d_x It is the focal length of the depth camera in the x-direction, f d_y It is the focal length of the depth camera in the y-direction, (cd_x ,c d_y ) are the coordinates of the principal point in the first depth map.

[0152] The extrinsic parameter matrix [R|T] can be used to represent the 3D points (X, Y, T) in the depth camera coordinate system. d ,Y d Z d Transform to 3D points in the RGB camera coordinate system (X) rgb ,Y rgb Z rgb ):

[0153]

[0154] Where R is the rotation matrix and T is the translation vector.

[0155] Point (X) in the three-dimensional coordinate system rgb ,Y rgb Z rgb ) Pixel coordinates projected onto the RGB image plane (u rgb ,v rgb Perspective projection is performed using the intrinsic parameter matrix of an RGB camera.

[0156]

[0157] This formula allows us to map 3D points from a depth camera onto 2D pixel coordinates in an RGB image, thus aligning the first depth image with the first RGB image.

[0158] Interpolation and resampling: The resolution of depth maps is usually lower than that of RGB images, and there are sparse regions or missing data. Therefore, it is necessary to interpolate and resample the aligned first depth map to match the resolution of the aligned first RGB image.

[0159] Specifically, to preserve edge information, this embodiment uses joint bilateral filtering interpolation to retain key edges and details of the depth map during the interpolation process. When calculating the weighted average of spatial and pixel values, this embodiment uses an aligned first RGB image to guide the interpolation operation of the aligned first depth map, thereby maintaining consistent edges. Based on the resolution of the aligned first RGB image, the aligned depth map is scaled or cropped to match its size. This step ensures that both have the same dimensionality when input to the network. Furthermore, depth map data often contains noise, especially under complex lighting conditions. Joint bilateral filtering smooths noise while preserving edge information, improving image quality. The basic formula for joint bilateral filtering interpolation is as follows:

[0160]

[0161] in, I(q) is the value of the pixel after interpolation, I(q) is the original value of the first depth map after alignment at pixel q, and J(p) and J(q) are the pixel values ​​of the first RGB map after alignment at pixels p and q, respectively. ||pq|| is the Euclidean distance between pixels p and q. The variance parameter that controls the smoothness of the spatial domain. It is a variance parameter that controls the smoothness of the pixel value range. W p It is a normalization factor used to ensure that the weights of the filtering results are normalized.

[0162] This completes the first preprocessing step, resulting in the second RGB image and the second depth image.

[0163] Data standardization: In order to ensure that the data can be effectively processed by the network, the second RGB image and the second depth image need to be standardized so that different types of data have the same numerical range or distribution.

[0164] Specifically, the pixel values ​​of the second RGB image are normalized to the [0,1] range. The numerical range of the second depth image is relatively large, so the depth values ​​of the second depth image need to be logarithmically transformed to compress the numerical range.

[0165] Data augmentation: By using data augmentation techniques, the diversity of training data is increased, thereby improving the robustness of the network to different scenarios.

[0166] Data augmentation includes image rotation, cropping, and brightness adjustment to adapt to different lighting conditions and viewing angles. Specifically, rotation, translation, scaling, and cropping operations are performed simultaneously on the second RGB image and the second depth image to simulate different shooting angles and distances. For the second RGB image, brightness, contrast, and saturation are adjusted to simulate different lighting conditions. During the training phase, simulated noise is injected into the second depth image to enhance the network's robustness to noisy data.

[0167] This completes the second preprocessing step, resulting in the target RGB image and the target depth image.

[0168] In one optional embodiment, the preset error condition includes the average pixel error of the second RGB image and the second depth image being less than a preset value.

[0169] Understandably, after the first preprocessing step, the alignment and interpolation results need to be verified to ensure the correct spatial correspondence between the resulting second RGB image and the second depth map. The alignment effect is quantitatively evaluated by calculating the pixel error after projection. Ideally, the projection error should be close to zero, indicating that the projected points closely match the actual points. To evaluate the alignment quality of the entire image, the formula for calculating the average projection error is as follows:

[0170]

[0171] In the formula, (u d,i v d,i ) represents the coordinates of the i-th pixel in the second depth map, (u rgb,i v rgb,o ) represents the coordinates of the i-th pixel in the second RGB image.

[0172] Compared with the prior art, this application has the following beneficial effects:

[0173] (1) Multimodal data processing: Most existing object detection methods typically rely on only single-modal information. However, the embodiments of this application design a dual-stream network architecture to process RGB images and depth maps separately, making full use of the visual features of RGB images and the geometric features of depth maps. The modal fusion implemented in the first fusion module is more effective, significantly improving the accuracy and robustness of small target object detection, especially in complex scenes, where it performs better and significantly improves the ability to handle complex backgrounds and occlusion.

[0174] (2) Utilizing edge information in multi-level features: Existing technologies often lack adaptability to targets of different scales, resulting in poor performance when detecting targets with large size differences. Furthermore, existing methods underutilize edge information in depth maps, easily leading to misjudgments at boundaries. This application's embodiment, by introducing a multi-scale feature fusion module, can adaptively process targets of different scales, ensuring consistent detection results for both large and small targets. Simultaneously, the depth edge enhancement module further improves the network's recognition ability at depth map edges, enhancing the accurate detection of target boundaries, particularly excelling in scenarios with small targets such as LED indicator lights. This improves the network's robustness in handling multi-scale small targets and complex boundaries, making it suitable for accurate detection in various scenarios.

[0175] (3) Utilizing composite loss function information: Traditional object detection and segmentation methods typically rely on a single loss function, making it difficult to simultaneously optimize detection accuracy, edge segmentation, and the utilization of depth information. However, this application's embodiments design a composite loss function, combining cross-entropy loss, Dice loss, edge loss, and depth-aware loss functions. This allows for simultaneous optimization of different features in both the RGB and depth maps, effectively guiding the network to comprehensively optimize target edges, depth information, and classification accuracy. It is particularly suitable for handling imbalanced small target object detection. Through this multi-dimensional loss optimization, the network exhibits higher accuracy and stability when handling small target object detection and segmentation tasks, especially performing better when dealing with imbalances between the target and background regions.

[0176] (4) Handling invalid information in images: Traditional convolution operations typically process the entire image uniformly, ignoring the sparsity that may exist in the depth map, leading to wasted computational resources and difficulty in handling invalid regions in the depth map. However, the embodiments of this application optimize the processing of sparse regions in the depth map using sparse convolution technology, reducing unnecessary computational overhead and improving network efficiency and response speed. This is very useful for handling scenarios where depth map data is sparse or partially missing, significantly improving computational efficiency while ensuring performance.

[0177] (5) 3D Feature Detection: Many existing methods process depth maps only in two-dimensional planes, lacking effective utilization of three-dimensional spatial geometric information. However, this application's embodiment significantly enhances the network's ability to understand the shape and structure of objects in three-dimensional space through geometric feature extraction modules, such as surface normal vector calculation and depth gradient calculation. This module helps the network better capture the geometric details of small target objects, demonstrating higher accuracy in target boundary detection and spatial localization.

[0178] See Figure 5 , Figure 5 This is a structural block diagram of a target positioning device 10 provided in an embodiment of this application. The target positioning device 10 includes:

[0179] Module 11 is used to acquire the target RGB image and target depth image of the same target area;

[0180] Extraction module 12 is used to input the target RGB image and the target depth image into a pre-constructed two-stream network to extract the visual features of the target RGB image and the geometric features of the target depth image, and to obtain a predicted image based on the fusion features obtained by fusing the visual features and the geometric features.

[0181] Training module 13 is used to train the dual-stream network using a preset loss function and the predicted image to obtain a trained dual-stream network, which is then used to locate the target in the image to be detected.

[0182] Optionally, the structure of the dual-stream network includes: a first feature network, a second feature network, a first fusion module, a decoding module, and an output module;

[0183] The first feature network is used to extract the visual features;

[0184] The second feature network is used to extract the geometric features;

[0185] The fusion module is used to fuse the visual features and the geometric features to obtain the fused features;

[0186] The decoding module is used to decode the fused features to obtain a decoded image;

[0187] The output module is used to generate the predicted image based on the decoded image.

[0188] Optionally, the second feature network includes: a sparse convolution module, a geometric feature extraction module, a multi-scale feature fusion module, a global context-aware module, a deep edge enhancement module, and a second fusion module;

[0189] The sparse convolution module is used to perform convolution operations on the effective region of the target depth map and output the first target depth map;

[0190] The geometric feature extraction module is used to extract geometric features from the first target depth map and output a second target depth map;

[0191] The multi-scale feature fusion module is used to extract and fuse features of different scales from the second target depth map;

[0192] The global context awareness module is used to extract global context information of the second target depth map;

[0193] The depth edge enhancement module is used to extract and enhance the edge information of the second target depth map;

[0194] The second fusion module is used to fuse the information output by the multi-scale feature fusion module, the global context awareness module, and the deep edge enhancement module to obtain the geometric features.

[0195] Optionally, the loss function includes at least one of the following: cross-entropy loss function, Dice loss function, edge loss function, depth-aware loss function, and composite loss function;

[0196] The cross-entropy loss function is used to compare the difference between the classification results of pixels in the predicted image and the classification results of pixels in the real image.

[0197] The Dice loss function is used to measure the degree of overlap between the positive sample regions of the predicted image and the positive sample regions of the real image.

[0198] The edge loss function is used to enhance the edge information of the depth map;

[0199] The depth-sensing loss function is used to compare the depth value of the predicted image with the depth value of the real image.

[0200] The composite loss function is obtained by weighted summation of at least one of the cross-entropy loss function, Dice loss function, edge loss function, and depth perception loss function.

[0201] Optionally, the acquisition module 11 is specifically configured to include:

[0202] Obtain the first RGB image and the first depth image of the same target region;

[0203] The first RGB image and the first depth image are subjected to a first preprocessing to obtain a second RGB image and a second depth image; wherein, the first preprocessing includes at least: performing geometric calibration and image alignment on the first RGB image and the first depth image in sequence, and performing joint bilateral filtering interpolation and resampling on the aligned first RGB image to match the resolution of the aligned first RGB image;

[0204] If the preset error condition is met, a second preprocessing is performed on the second RGB image and the second depth image, namely the target RGB image and the target depth image; the second preprocessing includes at least one of data standardization and data augmentation.

[0205] Optionally, the preset error condition includes the average pixel error of the second RGB image and the second depth image being less than a preset value.

[0206] It is worth noting that the working process of each module in the target positioning device 10 described in this application embodiment can refer to the working process of the target positioning method described in the above embodiment, and will not be repeated here.

[0207] This application provides a target localization device 10 that acquires a target RGB image and a target depth map of the same target region. The target RGB image and the target depth map are input into a pre-constructed two-stream network to extract visual features from the target RGB image and geometric features from the target depth map. A predicted image is obtained based on the fused features obtained by fusing the visual and geometric features. The two-stream network is trained using a preset loss function and the predicted image to obtain a trained two-stream network, which is then used to localize targets in the image to be detected. Therefore, this application embodiment, through a two-stream network, fully leverages the respective feature advantages of the RGB image and the depth map, making full use of the geometric features of the depth map and the visual features of the RGB image, significantly improving the accuracy and robustness of small target localization, especially in complex scenes.

[0208] Furthermore, this application also provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the target positioning method as described in any of the above embodiments.

[0209] Furthermore, this application also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implements the target localization method as described in any of the above embodiments.

[0210] See Figure 6 , Figure 6 This is a structural block diagram of a target positioning device 20 provided in an embodiment of this application. The target positioning device 20 includes: a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, it implements the steps in the above-described target positioning method embodiments. Alternatively, when the processor 21 executes the computer program, it implements the functions of each module / unit in the above-described device embodiments.

[0211] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the target positioning device 20.

[0212] The target positioning device 20 may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will understand that the schematic diagram is merely an example of the target positioning device 20 and does not constitute a limitation on the target positioning device 20. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the target positioning device 20 may also include input / output devices, network access devices, buses, etc.

[0213] The processor 21 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 21 is the control center of the target positioning device 20, connecting all parts of the target positioning device 20 via various interfaces and lines.

[0214] The memory 22 can be used to store the computer programs and / or modules. The processor 21 implements various functions of the target positioning device 20 by running or executing the computer programs and / or modules stored in the memory 22 and calling the data stored in the memory 22. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital card (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0215] If the modules / units integrated in the target positioning device 20 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 21, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0216] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided in this application, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0217] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A target localization method, characterized in that, include: Obtain the target RGB image and target depth map of the same target area; The target RGB image and the target depth image are input into a pre-constructed two-stream network to extract the visual features of the target RGB image and the geometric features of the target depth image. The predicted image is obtained by fusing the visual features and the geometric features. The dual-stream network is trained using a preset loss function and the predicted image to obtain a trained dual-stream network, which is then used to locate the target in the image to be detected.

2. The target localization method as described in claim 1, characterized in that, The structure of a two-stream network includes: a first feature network, a second feature network, a first fusion module, a decoding module, and an output module; The first feature network is used to extract the visual features; The second feature network is used to extract the geometric features; The fusion module is used to fuse the visual features and the geometric features to obtain the fused features; The decoding module is used to decode the fused features to obtain a decoded image; The output module is used to generate the predicted image based on the decoded image.

3. The target localization method as described in claim 2, characterized in that, The second feature network includes: a sparse convolution module, a geometric feature extraction module, a multi-scale feature fusion module, a global context-aware module, a deep edge enhancement module, and a second fusion module; The sparse convolution module is used to perform convolution operations on the effective region of the target depth map and output the first target depth map; The geometric feature extraction module is used to extract geometric features from the first target depth map and output a second target depth map; The multi-scale feature fusion module is used to extract and fuse features of different scales from the second target depth map; The global context awareness module is used to extract global context information of the second target depth map; The depth edge enhancement module is used to extract and enhance the edge information of the second target depth map; The second fusion module is used to fuse the information output by the multi-scale feature fusion module, the global context awareness module, and the deep edge enhancement module to obtain the geometric features.

4. The target localization method as described in claim 1, characterized in that, The loss function includes at least one of the following: cross-entropy loss function, Dice loss function, edge loss function, depth-aware loss function, and composite loss function; The cross-entropy loss function is used to compare the difference between the classification results of pixels in the predicted image and the classification results of pixels in the real image. The Dice loss function is used to measure the degree of overlap between the positive sample regions of the predicted image and the positive sample regions of the real image. The edge loss function is used to enhance the edge information of the depth map; The depth-sensing loss function is used to compare the depth value of the predicted image with the depth value of the real image. The composite loss function is obtained by weighted summation of at least one of the cross-entropy loss function, Dice loss function, edge loss function, and depth perception loss function.

5. The target localization method as described in claim 1, characterized in that, The acquisition of the target RGB image and target depth image of the same target region includes: Obtain the first RGB image and the first depth image of the same target region; The first RGB image and the first depth image are subjected to a first preprocessing to obtain a second RGB image and a second depth image; wherein, the first preprocessing includes at least: performing geometric calibration and image alignment on the first RGB image and the first depth image in sequence, and performing joint bilateral filtering interpolation and resampling on the aligned first RGB image to match the resolution of the aligned first RGB image; If the preset error condition is met, a second preprocessing is performed on the second RGB image and the second depth image, namely the target RGB image and the target depth image; the second preprocessing includes at least one of data standardization and data augmentation.

6. The target localization method as described in claim 5, characterized in that, The preset error condition includes that the average pixel error of the second RGB image and the second depth image is less than a preset value.

7. A target positioning device, characterized in that, include: The acquisition module is used to acquire the target RGB image and target depth image of the same target area; An extraction module is used to input the target RGB image and the target depth image into a pre-constructed two-stream network to extract the visual features of the target RGB image and the geometric features of the target depth image, and to obtain a predicted image based on the fusion features obtained by fusing the visual features and the geometric features. The training module is used to train the dual-stream network using a preset loss function and the predicted image to obtain a trained dual-stream network, which is then used to locate the target in the image to be detected.

8. A target positioning device, characterized in that, The device includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the target localization method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program; wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the target localization method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, It includes a computer program / instruction that, when executed by a processor, implements the target localization method as described in any one of claims 1 to 6.

Images