A robot low-light grabbing detection method based on wavelet domain visual enhancement

Abstract

The present application belongs to the technical field of computer vision and intelligent robot, and particularly relates to a robot low-light grasping detection method based on wavelet domain vision enhancement. In order to solve the problem that the traditional enhancement method will destroy the related features of grasping detection in low-light environment, resulting in low accuracy of grasping detection result, the present application collects low-light images, obtains RGB and depth image information, and labels the grasping frame to form a pair of "low-light image-normal light image-depth image-grasping label information" data pairs, a specific low-light grasping detection data set is made, and a wavelet domain is introduced to implement fusion transformation, which is helpful for noise filtering and model feature learning. In addition, the edge attention module and the feature guide module are used to make the model pay more attention to the extraction of edge information of the grasping image, which is helpful to improve the algorithm accuracy of the grasping detection task.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and intelligent robot technology, specifically relating to a robot low-light grasping detection method based on wavelet domain vision enhancement. Background Technology

[0002] With the rapid development of computer vision and intelligent robotics technologies, low-light grasping and detection technology has demonstrated significant value in various fields. This technology requires the accurate identification, analysis, and processing of images in low-light environments to enable the grasping of target objects under such conditions. In manufacturing, robots are needed to accurately grasp parts in low-light environments, improving the flexibility and efficiency of production lines. In scientific research fields, such as space exploration, research equipment needs to collect samples and data under extremely low light conditions. Therefore, how to achieve accurate grasping and detection under low-light conditions has become a crucial problem that urgently needs to be solved.

[0003] Currently, the traditional approach to low-light grasping and detection problems involves first restoring the quality of the low-light image through low-light image enhancement techniques before proceeding with the normal grasping and detection process. Another approach is to implement grasping and detection in low-light environments through specific extraction methods in an end-to-end manner. Furthermore, if the traditional "enhancement before grasping" strategy is adopted, information beneficial to grasping and detection will be lost during the enhancement process, leading to a sharp decline in the performance of the grasping task.

[0004] To address the aforementioned problems, this invention proposes a low-light grasping detection method for robots based on wavelet domain visual enhancement. This invention employs an end-to-end approach to improve the visual quality of grasping images in low-light environments while ensuring the performance of the grasping task. Summary of the Invention

[0005] To address the problem that traditional enhancement methods in low-light environments destroy features related to grasping detection, resulting in low accuracy of grasping detection results, this invention proposes a robot low-light grasping detection method based on wavelet domain visual enhancement.

[0006] To achieve the above objectives, the present invention employs the following technical solution:

[0007] This invention provides a low-light grasping detection method for robots based on wavelet domain vision enhancement, comprising the following steps:

[0008] Step 1: Acquire paired low-light and normal-light data of the target to be captured in a low-light environment to construct a dataset DS. The dataset DS consists of pairs of "low-light image - normal-light image - depth image - capture annotation information", i.e. in, This represents the i-th low-light image. This represents the i-th normal light image. Let P represent the depth image of the i-th image. i This represents the captured annotation information for the i-th image, including the true category information y. i , i = 1, 2, 3, ..., n, where n represents the number of images in the dataset;

[0009] Step 2: Input the constructed dataset DS into the RGB-depth map fusion enhancement detection model for training. The RGB-depth map fusion enhancement detection model includes an image enhancement branch and a grasping detection branch. The backbone network of the image enhancement branch adopts U-Net to extract local features in the input low-light image and learn global context information. The grasping detection branch fuses the low-light image and the depth map in the wavelet domain, and inputs the fused image and the enhanced image from the image enhancement branch into the edge attention module. The feature guidance module then brings the edge feature information of the two closer together.

[0010] Step 3: Input the acquired low-light image and depth image into the trained RGB-depth map fusion enhancement detection model to obtain the predicted enhanced image and the predicted value of the capture box, thus completing the capture detection task in low-light environment.

[0011] Furthermore, step 1 specifically includes:

[0012] Step 1.1: Acquire low-light and normal-light images, as well as depth images of the target to be captured in a dark room;

[0013] Step 1.2: Mark the capture box on the corresponding normal light image to form a pair of "low light image - normal light image - depth image - capture annotation information";

[0014] Step 1.3: Change the target to be captured, and repeat steps 1.1 and 1.2 until the complete dataset DS is obtained.

[0015] Furthermore, step 2 specifically includes:

[0016] Step 2.1, the U-Net adopts an encoder-decoder structure, and the input low-light image R low The shape is (N,C,H,W). In each convolutional layer, the output is halved in width and height and doubled in channels compared to the previous layer, i.e., the output is (N,2×C,H / 2,W / 2). The encoder consists of a series of convolutional layers, activation layers, and max-pooling layers, as shown in the following equation:

[0017] Y k =MaxPooling(relu(BN(Conv(R)) low )))),k=1,2,3,4(1)

[0018] Where Conv(·) represents a convolutional layer, ReLU(·) represents an activation layer, MaxPooling(·) represents a max-pooling layer, and Y k This represents the feature map output of the k-th layer;

[0019] Add an SE unit to the bottleneck layer, as follows:

[0020] Y BottleNeck =SE(Y4)(2)

[0021] The decoder performs a preliminary restoration of the image, as shown in the following formula:

[0022] M1 = relu(BN(Conv(Y) BottleNeck )))(3)

[0023] The skip connection part is responsible for passing the feature map from the encoder to the decoder part, which is achieved through a concatenation operation, as shown in the following equation:

[0024] C1=[M1,Y4](4)

[0025] Then, repeat the above process to obtain M2, C2, M3, C3, M4, as shown in the following formula:

[0026] M2 = relu(BN(Conv(M1)))(5)

[0027] C2=[M2,Y3](6)

[0028] M3 = relu(BN(Conv(M2)))(7)

[0029] C3 = [M3, Y2](8)

[0030] M4 = relu(BN(Conv(M3)))(9)

[0031] M4 is the final output I. enhance .

[0032] Step 2.2, process the low-light image R respectively. low and depth image D depth Perform discrete wavelet transform, then fuse the two, followed by convolutional block operation to obtain the fused image I. fuse As shown in the following formula:

[0033]

[0034]

[0035]

[0036] Among them, WRGB W is the wavelet coefficient matrix of the low-light image. depth It is the wavelet coefficient matrix of the depth image. Represents the discrete wavelet transform operation; ConvBlock(g) represents a set of convolutional blocks. Indicates a splicing operation; Indicates the inverse wavelet transform;

[0037] Step 2.3, take the image I obtained in step 2.1. enhance Image I obtained in step 2.2 fuse The edge information is fed into the edge attention module to learn edge information and obtain the feature map f. enhance ,f fuse As shown in the following formula:

[0038] f enhance ,f fuse =Edge_attention(I enhance ,I fuse (13)

[0039] Where Edge_attention(g) is the edge attention module;

[0040] Step 2.4, the feature map f obtained in step 2.3 enhance ,f fuse The feature guidance module is used to narrow the distance between the features of the two features, as shown in the following formula:

[0041] f' enhance ,f' fuse =Feature_guided(f enhance ,f fuse (14)

[0042] Among them, Feature_guided(g) is the feature guidance module;

[0043] Step 2.5, take the image I obtained in step 2.2... fuse The predicted bounding box parameters are output as g after a grasp detection head is used to perform the grasp detection task. i and predicted category information

[0044] Step 2.6, the total loss function of the RGB-depth map fusion enhanced detection model is defined as follows:

[0045] loss = α·loss reg +β·loss cls +γloss_f(15)

[0046]

[0047]

[0048]

[0049] Where, loss reg p represents the regression loss. i g i Represent the parameters of the actual bounding box and the predicted bounding box, respectively; loss cls Represents the category loss, yi. Let $\mathbf$ represent the true class information and the predicted class information, respectively, and $\mathbf$ represent the feature loss. Let ||·|| represent the gradients of the enhanced feature map and the fused feature map, respectively. 2 Let α, β, and γ represent the square of the norm, where α, β, and γ are balance factors.

[0050] Step 2.7: When the performance of the RGB-depth map fusion enhancement detection model meets the expected requirements and there is no longer a significant performance improvement, the training of the RGB-depth map fusion enhancement detection model is considered to have converged, that is, the training is complete.

[0051] Furthermore, step 3 specifically includes:

[0052] M(R low D depth )→{I i enhance ,g i}, i = 1, 2, ..., m (19)

[0053] in, G represents the predicted augmented image. i This represents the predicted capture box parameters, where m represents the total number of capture boxes.

[0054] Compared with the prior art, the present invention has the following advantages:

[0055] 1. A specific low-light grasping detection dataset was created for image capture in low-light environments. Low-light images were acquired using a RealSense D435 to obtain RGB and depth map information. Grasp bounding boxes were then labeled, forming pairs of data: "low-light image - normal light image - depth image - grasp annotation information".

[0056] 2. Compared to conventional methods that directly fuse RGB and depth images, this approach introduces wavelet domain fusion transformation, which aids in noise filtering and model feature learning. Furthermore, the use of edge attention and feature guidance modules allows the model to focus more on extracting image edge information, improving the algorithm's accuracy for the grasping and detection task.

[0057] 3. The method of this invention is easy to implement, and its application value is mainly reflected in the following aspects: 1) When assembling at night or in a dark factory, insufficient light can blur the outline of objects, causing errors in grasping and detection by traditional vision systems. This method can more accurately identify grasping points or edges, thereby improving the success rate of grasping. 2) In disaster relief, such as in the ruins after an earthquake or inside buildings after a fire, there is often a serious problem of insufficient lighting. Search and rescue robots can effectively identify obstacles and trapped personnel in extreme lighting environments, improving the success rate of rescue. Attached Figure Description

[0058] Figure 1 This is a flowchart illustrating the overall framework of the present invention;

[0059] Figure 2 This is a basic structural diagram of the model proposed in this invention;

[0060] Figure 3 This includes the captured low-light images, depth images, and the model's grasping and detection results. Detailed Implementation

[0061] To further illustrate the technical solution of the present invention, the present invention will be further described below through embodiments.

[0062] like Figure 1 As shown in this embodiment, a robot low-light grasping detection method based on wavelet domain vision enhancement is characterized by the following steps:

[0063] Step 1: Acquire paired low-light and normal-light data of the target to be captured in a low-light environment to construct a dataset DS. The dataset DS consists of pairs of "low-light image - normal-light image - depth image - capture annotation information", i.e. in, This represents the i-th low-light image. This represents the i-th normal light image. Let P represent the depth image of the i-th image. i This represents the captured annotation information for the i-th image, including the true category information y. i , i = 1, 2, 3, ..., n, where n represents the number of images in the dataset;

[0064] In this embodiment, step 1 is specifically as follows: Step 1.1, in a dark room, a realsense D435i is selected as the image acquisition device to acquire low-light, normal light, and depth images of the target to be captured.

[0065] Step 1.2: Mark the capture box on the corresponding normal light image to form a pair of "low light image - normal light image - depth image - capture annotation information";

[0066] Step 1.3: Change the target to be captured, and repeat steps 1.1 and 1.2 until the complete dataset DS is obtained.

[0067] Step 2: Input the constructed dataset DS into the RGB-Depth Map Fusion Enhancement Detection Model for training. The basic structure of the RGB-Depth Map Fusion Enhancement Detection Model is as follows: Figure 2 As shown, it includes an image enhancement branch and a grasping detection branch. The backbone network of the image enhancement branch adopts U-Net to extract local features in the input low-light image and learn global context information. The grasping detection branch fuses the low-light image and the depth map in the wavelet domain, and inputs the fused image and the enhanced image of the image enhancement branch into the edge attention module. The feature guidance module brings the edge feature information of the two closer together.

[0068] Step 2 of this embodiment is specifically as follows: Step 2.1, the U-Net adopts an encoder-decoder structure, and the input low-light image R low The shape is (N,C,H,W). In each convolutional layer, the output is halved in width and height and doubled in channels compared to the previous layer, i.e., the output is (N,2×C,H / 2,W / 2). The encoder consists of a series of convolutional layers, activation layers, and max-pooling layers, as shown in the following equation:

[0069] Y k =MaxPooling(relu(BN(Conv(R)) low )))),k=1,2,3,4(1)

[0070] Where Conv(·) represents a convolutional layer, ReLU(·) represents an activation layer, MaxPooling(·) represents a max-pooling layer, and Y k This represents the feature map output of the k-th layer;

[0071] Add an SE unit to the bottleneck layer, as follows:

[0072] Y BottleNeck =SE(Y4)(2)

[0073] The decoder performs a preliminary restoration of the image, as shown in the following formula:

[0074] M1 = relu(BN(Conv(Y) BottleNeck )))(3)

[0075] The skip connection part is responsible for passing the feature map from the encoder to the decoder part, which is achieved through a concatenation operation, as shown in the following equation:

[0076] C1=[M1,Y4](4)

[0077] Then, repeat the above process to obtain M2, C2, M3, C3, M4, as shown in the following formula:

[0078] M2 = relu(BN(Conv(M1)))(5)

[0079] C2=[M2,Y3](6)

[0080] M3 = relu(BN(Conv(M2)))(7)

[0081] C3 = [M3, Y2](8)

[0082] M4 = relu(BN(Conv(M3)))(9)

[0083] M4 is the final output I. enhance .

[0084] Step 2.2, process the low-light image R respectively. low and depth image D depth Perform discrete wavelet transform, then fuse the two, followed by convolutional block operation to obtain the fused image I. fuse As shown in the following formula:

[0085]

[0086]

[0087]

[0088] Among them, W RGB W is the wavelet coefficient matrix of the low-light image. depth It is the wavelet coefficient matrix of the depth image. Represents the discrete wavelet transform operation; ConvBlock(g) represents a set of convolutional blocks. Indicates a splicing operation; Indicates the inverse wavelet transform;

[0089] Step 2.3, take the image I obtained in step 2.1. enhance Image I obtained in step 2.2 fuse The edge information is fed into the edge attention module to learn edge information and obtain the feature map f. enhance ,f fuse As shown in the following formula:

[0090] f enhance ,f fuse =Edge_attention(Ienhance ,I fuse (13)

[0091] Where Edge_attention(g) is the edge attention module;

[0092] Step 2.4, the feature map f obtained in step 2.3 enhance ,f fuse The feature guidance module is used to narrow the distance between the features of the two features, as shown in the following formula:

[0093] f' enhance ,f' fuse =Feature_guided(f enhance ,f fuse (14)

[0094] Among them, Feature_guided(g) is the feature guidance module;

[0095] Step 2.5, take the image I obtained in step 2.2... fuse The predicted bounding box parameters are output as g after a grasp detection head is used to perform the grasp detection task. i and predicted category information

[0096] Step 2.6, the total loss function of the RGB-depth map fusion enhanced detection model is defined as follows:

[0097] loss = α·loss reg +β·loss cls +γloss_f(15)

[0098]

[0099]

[0100]

[0101] Where, loss reg p represents the regression loss, used to evaluate the difference in position and size between the predicted and actual crawl boxes. i g i Represent the parameters of the actual bounding box and the predicted bounding box, respectively; loss cls Represents the category loss, used to evaluate the difference between the category of the captured bounding box and the category of the true captured bounding box, y i , These represent the true category information and the predicted category information, respectively. `loss_f` represents the feature loss, primarily used to encourage the model to focus more on edge information extraction. `N` represents the number of images. Let ||·|| represent the gradients of the enhanced feature map and the fused feature map, respectively. 2 Let α, β, and γ represent the square of the norm, where α, β, and γ are balance factors.

[0102] Step 2.7: Divide the dataset DS into training and test sets in an 8:2 ratio. Evaluate the model performance using the validation set. When the performance of the RGB-depth map fusion enhancement detection model meets the expected requirements and there is no longer a significant performance improvement, the training of the RGB-depth map fusion enhancement detection model is considered to have converged, i.e., training is complete.

[0103] Step 3: Input the acquired low-light image and depth image into the trained RGB-depth map fusion enhancement detection model to obtain the predicted enhanced image and the predicted capture box value, thus completing the capture detection task in low-light environment. The low-light image, depth image, and capture detection results of the model in this embodiment are shown below. Figure 3 As shown;

[0104] The entire process is as follows:

[0105] M(R low D depth )→{I i enhance ,g i}, i = 1, 2, ..., m (19)

[0106] in, G represents the predicted augmented image. i This represents the predicted capture box parameters, where m represents the total number of capture boxes.

[0107] The foregoing has shown and described the main features and advantages of the present invention. It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, it is intended that all variations falling within the meaning and scope of equivalents of the claims be included within the present invention.

[0108] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A robot low-light grasping detection method based on wavelet domain vision enhancement, characterized in that, Includes the following steps: Step 1: Acquire paired low-light and normal-light data of the target to be captured in a low-light environment to construct a dataset DS. The dataset DS consists of pairs of "low-light image - normal-light image - depth image - capture annotation information", i.e. ,in, Indicates the first A low-light image, Indicates the first A normal light image, Indicates the first Aspect depth image, Indicates the first Capture and label information for each image, including true category information. , , This indicates the number of images in the dataset; Step 2: Input the constructed dataset DS into the RGB-depth map fusion enhancement detection model for training. The RGB-depth map fusion enhancement detection model includes an image enhancement branch and a grasping detection branch. The backbone network of the image enhancement branch adopts U-Net to extract local features in the input low-light image and learn global context information. The grasping detection branch fuses the low-light image and the depth map in the wavelet domain, and inputs the fused image and the enhanced image from the image enhancement branch into the edge attention module. The feature guidance module then brings the edge feature information of the two closer together. The capture and detection branches respectively handle low-light images. and depth images Perform discrete wavelet transform, then fuse the two, followed by convolutional block operation to obtain the fused image. As shown in the following formula: （10） （11） （12） in, It is the wavelet coefficient matrix of the low-light image. It is the wavelet coefficient matrix of the depth image. This represents the discrete wavelet transform operation; Represents a set of convolutional blocks, Indicates a splicing operation; Indicates the inverse wavelet transform; Image obtained from image enhancement branch With images The edge information is fed into the edge attention module to learn edge information and obtain feature maps. As shown in the following formula: （13） in, For edge attention modules; The obtained feature map The feature guidance module is used to narrow the distance between the features of the two features, as shown in the following formula: （14） in, It is a feature-guided module; Image The predicted bounding box parameters are output after a crawling detection head is used to perform the crawling detection task. and predicted category information ; The total loss function of the RGB-depth map fusion enhanced detection model is defined as follows: （15） （16） （17） （18） in, Indicates regression loss, These represent the actual bounding box parameters and the predicted bounding box parameters, respectively. Indicates category loss. These represent the actual category information and the predicted category information, respectively. Represents feature loss, Indicates the number of images. These represent the gradients of the enhanced feature map and the fused feature map, respectively. Represents the square of the norm. It is a balancing factor; Step 3: Input the acquired low-light image and depth image into the trained RGB-depth map fusion enhancement detection model to obtain the predicted enhanced image and the predicted value of the capture box, thus completing the capture detection task in low-light environment.

2. The robot low-light grasping detection method based on wavelet domain vision enhancement according to claim 1, characterized in that, Step 1 specifically involves: Step 1.1: Acquire low-light and normal-light images, as well as depth images of the target to be captured in a dark room; Step 1.2: Mark the capture box on the corresponding normal light image to form a pair of "low light image - normal light image - depth image - capture annotation information"; Step 1.3: Change the target to be captured, and repeat steps 1.1 and 1.2 until the complete dataset DS is obtained.

3. The robot low-light grasping detection method based on wavelet domain vision enhancement according to claim 1, characterized in that, The image enhancement branch in step 2 specifically includes: The U-Net employs an encoder-decoder structure, taking a low-light image as input. The shape is In each convolutional layer, the output is halved in width and height and doubled in channels compared to the previous layer; that is, the output is... The encoder consists of a series of convolutional layers, activation layers, and max pooling layers, as shown in the following equation: （1） in, Indicates a convolutional layer. Indicates the activation layer. Indicates the max pooling layer. This represents the feature map output of the k-th layer; Add an SE unit to the bottleneck layer, as follows: （2） The decoder performs a preliminary restoration of the image, as shown in the following formula: （3） The skip connection part is responsible for passing the feature map from the encoder to the decoder part, which is achieved through a concatenation operation, as shown in the following equation: （4） Then, repeat the above process to obtain... As shown in the following formula: （5） （6） （7） （8） （9） This is the final output. ; When the performance of the RGB-depth map fusion enhancement detection model meets the expected requirements and there is no longer a significant performance improvement, the training of the RGB-depth map fusion enhancement detection model is considered to have converged, that is, the training is complete.

4. The robot low-light grasping detection method based on wavelet domain vision enhancement according to claim 1, characterized in that, Step 3 specifically involves: （19） in, Represents the predicted enhanced image, Indicates the parameters of the predicted capture box. This indicates the total number of capture boxes.

Images