Robot low-light grabbing detection method based on wavelet domain vision enhancement

Abstract

The invention belongs to the technical field of computer vision and intelligent robots, and particularly relates to a robot low-light grabbing detection method based on wavelet domain vision enhancement. In order to solve the problem that in a low-light environment, a traditional enhancement method destroys related characteristics related to capture detection, and consequently the accuracy rate of a capture detection result is low, the method comprises the steps that a low-light image is collected, RGB and depth map information is obtained, a capture box is marked, and the accuracy rate of the capture detection result is increased; according to the method, a low-light image, a normal-light image, a depth image and capture annotation information are combined to form paired data pairs of the low-light image, the normal-light image, the depth image and the capture annotation information, a specific low-light capture detection data set is made, a wavelet domain is introduced to implement fusion transformation, and noise filtering and model feature learning are facilitated. In addition, an edge attention module and a feature guide module are used to make the model pay more attention to extraction of captured image edge information, and improvement of algorithm precision of a capture detection task is facilitated.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and intelligent robots, and in particular relates to a robot low-light grasping detection method based on wavelet domain vision enhancement. Background Art

[0002] With the rapid development of computer vision and intelligent robot technology, low-light grasping and detection technology has shown important value in many fields. This technology requires accurate recognition, analysis and processing of images in low-light environments, and the realization of grasping operations on target objects in low-light environments. In the manufacturing industry, robots are required to accurately grasp parts in low-light environments to improve the flexibility and efficiency of production lines; in scientific research fields, such as space exploration, scientific research equipment is required to collect samples and data under extremely low light conditions. Therefore, how to achieve accurate grasping and detection under low-light conditions has become an important issue that needs to be solved urgently.

[0003] At present, the traditional method for low-light grasping and detection is to first restore the quality of low-light images through low-light image enhancement technology, and then perform the normal grasping and detection process. Another method is to use a specific extraction method to achieve the grasping and detection process in a low-light environment in an end-to-end manner during the grasping and detection process. In addition, if the traditional "enhance first and then grasp" strategy is adopted, information that is beneficial to grasping and detection will be lost during the enhancement process, resulting in a sharp decline in the performance of the grasping task.

[0004] In view of the above problems, the present invention proposes a robot low-light grasping detection method based on wavelet domain visual enhancement. The invention adopts an end-to-end approach to improve the visual quality of grasped images in low-light environments while ensuring the grasping task performance. Summary of the invention

[0005] In order to solve the problem that traditional enhancement methods in low-light environments will destroy the relevant features of grasping detection and thus lead to low accuracy of grasping detection results, the present invention proposes a robot low-light grasping detection method based on wavelet domain vision enhancement.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] The present invention provides a robot low-light grasping detection method based on wavelet domain visual enhancement, comprising the following steps:

[0008] Step 1: Collect paired low-light and normal-light data of the target to be captured in a low-light environment to construct a dataset DS. The dataset DS consists of pairs of “low-light image-normal-light image-depth image-capture annotation information”, that is, in, represents the i-th low-light image, represents the i-th normal light image, represents the i-th depth image, P i Represents the captured annotation information of the i-th image, including the actual category information y i , i = 1, 2, 3, ···, n, n represents the number of images in the dataset;

[0009] Step 2: Input the constructed dataset DS into the RGB-depth image fusion enhancement detection model for training. The RGB-depth image fusion enhancement detection model includes an image enhancement branch and a grasping detection branch. The backbone network of the image enhancement branch adopts U-Net to extract local features in the input low-light image and learn global context information. The grasping detection branch fuses the low-light image with the depth map in the wavelet domain, and inputs the fused image and the image enhanced by the image enhancement branch into the edge attention module together, and closes the edge feature information of the two in the feature guidance module;

[0010] Step 3: Input the collected low-light image and depth image into the trained RGB-depth image fusion enhancement detection model to obtain the predicted enhanced image and grasping box prediction value, and complete the grasping detection task in the low-light environment.

[0011] Further, the step 1 is specifically as follows:

[0012] Step 1.1, collect low-light, normal-light captured images and depth images of the target to be captured in a dark room;

[0013] Step 1.2, perform grabbing frame annotation on the corresponding normal light image to form a pair of "low light image-normal light image-depth image-grabbed annotation information";

[0014] Step 1.3: Change the target to be grasped and repeat steps 1.1 and 1.2 until a complete data set DS is obtained.

[0015] Further, the step 2 is specifically as follows:

[0016] Step 2.1: The U-Net adopts an encoder-decoder structure and inputs the low-light image R low The shape of is (N, C, H, W). The output of each convolution layer is half the width and height of the previous layer, and the channel is doubled, that is, the output is (N, 2×C, H / 2, W / 2). The encoder is composed of a series of convolutional layers-activation layers-maximum pooling layers, as shown below:

[0017] Y k =MaxPooling(relu(BN(Conv(R low )))), k = 1, 2, 3, 4 (1)

[0018] Among them, Conv(·) represents the convolution layer, relu(·) represents the activation layer, MaxPooling(·) represents the maximum pooling layer, and Y k Represents the feature map output of the kth layer;

[0019] Add SE units in the bottleneck layer as follows:

[0020] Y BottleNeck =SE(Y4)(2)

[0021] The decoder performs a preliminary restoration of the image as follows:

[0022] M1=relu(BN(Conv(Y BottleNeck )))(3)

[0023] The jump connection part is responsible for passing the feature map of the encoder part to the decoder part, which is achieved through the splicing operation as follows:

[0024] C1=[M1,Y4](4)

[0025] After that, repeat the above process to obtain M2, C2, M3, C3, M4, as shown below:

[0026] M2 = relu(BN(Conv(M1)))(5)

[0027] C2=[M2,Y3](6)

[0028] M3 = relu(BN(Conv(M2)))(7)

[0029] C3=[M3,Y2](8)

[0030] M4=relu(BN(Conv(M3)))(9)

[0031] M4 is the final output I enhance .

[0032] Step 2.2: For the low-light image R low and the depth image D depth Perform discrete wavelet transform, then fuse the two, and perform convolution block operation after fusion to obtain the fused image I fuse , as follows:

[0033]

[0034]

[0035]

[0036] Among them, WRGB is the wavelet coefficient matrix of the low-light image, W depth is the wavelet coefficient matrix of the depth image, Represents a discrete wavelet transform operation; ConvBlock(g) represents a set of convolution blocks, Represents a splicing operation; represents inverse wavelet transform;

[0037] Step 2.3: The image I obtained in step 2.1 enhance The image I obtained in step 2.2 fuse Send it to the edge attention module to learn edge information and get the feature map f enhance ,f fuse , as follows:

[0038] f enhance ,f fuse =Edge_attention(I enhance ,I fuse )(13)

[0039] Among them, Edge_attention(g) is the edge attention module;

[0040] Step 2.4: The feature map f obtained in step 2.3 enhance ,f fuse The feature guidance module is sent to shorten the distance between the two features, as shown in the following formula:

[0041] f' enhance ,f' fuse =Feature_guided(f enhance ,f fuse ) (14)

[0042] Among them, Feature_guided(g) is the feature guided module;

[0043] Step 2.5: The image I obtained in step 2.2 fuse The predicted grab box parameters of the grab detection task are output through a grab detection head. i , and predicted category information

[0044] In step 2.6, the total loss function of the RGB-depth image fusion enhanced detection model is defined as follows:

[0045] loss = α·loss reg +β·loss cls +γloss_f(15)

[0046]

[0047]

[0048]

[0049] Among them, loss reg represents the regression loss, p i , g i They represent the real grab box parameters and the predicted grab box parameters, respectively, and loss cls represents the category loss, yi, They represent the real category information and the predicted category information respectively, loss_f represents the feature loss, Respectively represent the gradient of the enhanced feature map and the gradient of the fused feature map, ||·|| 2 represents the square of the norm, α, β, γ are balance factors;

[0050] Step 2.7: When the performance of the RGB-depth image fusion enhanced detection model meets the expected requirements and there is no significant performance improvement, the training of the RGB-depth image fusion enhanced detection model is considered to have converged, that is, the training is completed.

[0051] Further, the step 3 is specifically as follows:

[0052] M(R low ,D depth )→{I i enhance ,g i},i=1,2,···,m(19)

[0053] in, represents the predicted enhanced image, g i represents the predicted grab box parameters, and m represents the total number of grab boxes.

[0054] Compared with the prior art, the present invention has the following advantages:

[0055] 1. For captured images in low-light environments, a specific low-light capture detection dataset was created. Realsense D435 was used to collect low-light images and obtain RGB and depth map information. The capture frames were annotated to form pairs of "low-light image-normal light image-depth image-capture annotation information" data;

[0056] 2. Compared with the general method of directly fusing RGB and depth images, the introduction of wavelet domain to implement fusion transformation is helpful for noise filtering and model feature learning. In addition, the use of edge attention module and feature guidance module makes the model focus more on the extraction of edge information of the captured image, which helps to improve the algorithm accuracy of the capture detection task.

[0057] 3. The method of the present invention is easy to implement, and its application value is mainly reflected in the following aspects: 1) When assembling at night or in a dark factory, the insufficient light environment will make the outline of the object blurred, causing the traditional visual system to easily make errors in grasping detection. Through this method, the grasping point or grasping edge can be identified more accurately, thereby improving the success rate of grasping. 2) In disaster rescue, such as ruins after an earthquake or the interior of a building after a fire, there is usually a serious problem of insufficient lighting. The search and rescue robot can effectively identify obstacles and trapped people in extreme lighting environments, thereby improving the success rate of rescue. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is the overall framework flow chart of the present invention;

[0059] Figure 2 It is the basic structural diagram of the model proposed by the present invention;

[0060] Figure 3 The captured low-light images, depth images, and the model’s captured detection results. DETAILED DESCRIPTION

[0061] In order to further illustrate the technical solution of the present invention, the present invention is further described below through embodiments.

[0062] like Figure 1 As shown, a robot low-light grasping detection method based on wavelet domain visual enhancement in this embodiment is characterized by comprising the following steps:

[0063] Step 1: Collect paired low-light and normal-light data of the target to be captured in a low-light environment to construct a dataset DS. The dataset DS consists of pairs of “low-light image-normal-light image-depth image-capture annotation information”, that is, in, represents the i-th low-light image, represents the i-th normal light image, represents the i-th depth image, P i Represents the captured annotation information of the i-th image, including the actual category information y i , i = 1, 2, 3, ···, n, n represents the number of images in the dataset;

[0064] Step 1 of this embodiment is specifically as follows: Step 1.1, in a dark room, realsense D435i is selected as an image acquisition device to acquire low-light, normal-light captured images and depth images of the target to be captured;

[0065] Step 1.2, perform grabbing frame annotation on the corresponding normal light image to form a pair of "low light image-normal light image-depth image-grabbed annotation information";

[0066] Step 1.3: Change the target to be grasped and repeat steps 1.1 and 1.2 until a complete data set DS is obtained.

[0067] Step 2: Input the constructed dataset DS into the RGB-depth image fusion enhanced detection model for training. The basic structure of the RGB-depth image fusion enhanced detection model is as follows: Figure 2 As shown, it includes an image enhancement branch and a capture detection branch. The backbone network of the image enhancement branch adopts U-Net, which is used to extract local features in the input low-light image and learn global context information. The capture detection branch fuses the low-light image with the depth map in the wavelet domain, and inputs the fused image and the image enhanced by the image enhancement branch into the edge attention module together, and closes the edge feature information of the two in the feature guidance module.

[0068] Step 2 of this embodiment is specifically as follows: Step 2.1: the U-Net adopts an encoder-decoder structure and inputs a low-light image R low The shape of is (N, C, H, W). The output of each convolution layer is half the width and height of the previous layer, and the channel is doubled, that is, the output is (N, 2×C, H / 2, W / 2). The encoder is composed of a series of convolutional layers-activation layers-maximum pooling layers, as shown below:

[0069] Y k =MaxPooling(relu(BN(Conv(R low )))), k = 1, 2, 3, 4 (1)

[0070] Among them, Conv(·) represents the convolution layer, relu(·) represents the activation layer, MaxPooling(·) represents the maximum pooling layer, and Y k Represents the feature map output of the kth layer;

[0071] Add SE units in the bottleneck layer as follows:

[0072] Y BottleNeck =SE(Y4)(2)

[0073] The decoder performs a preliminary restoration of the image as follows:

[0074] M1=relu(BN(Conv(Y BottleNeck )))(3)

[0075] The jump connection part is responsible for passing the feature map of the encoder part to the decoder part, which is achieved through the splicing operation as follows:

[0076] C1=[M1,Y4](4)

[0077] After that, repeat the above process to obtain M2, C2, M3, C3, M4, as shown below:

[0078] M2 = relu(BN(Conv(M1)))(5)

[0079] C2=[M2,Y3](6)

[0080] M3 = relu(BN(Conv(M2)))(7)

[0081] C3=[M3,Y2](8)

[0082] M4=relu(BN(Conv(M3)))(9)

[0083] M4 is the final output I enhance .

[0084] Step 2.2: For the low-light image R low and the depth image D depth Perform discrete wavelet transform, then fuse the two, and perform convolution block operation after fusion to obtain the fused image I fuse , as follows:

[0085]

[0086]

[0087]

[0088] Among them, W RGB is the wavelet coefficient matrix of the low-light image, W depth is the wavelet coefficient matrix of the depth image, Represents a discrete wavelet transform operation; ConvBlock(g) represents a set of convolution blocks, Represents a splicing operation; represents inverse wavelet transform;

[0089] Step 2.3: The image I obtained in step 2.1 enhance The image I obtained in step 2.2 fuse Send it to the edge attention module to learn edge information and get the feature map f enhance ,f fuse , as follows:

[0090] f enhance ,f fuse =Edge_attention(Ienhance ,I fuse )(13)

[0091] Among them, Edge_attention(g) is the edge attention module;

[0092] Step 2.4: The feature map f obtained in step 2.3 enhance ,f fuse The feature guidance module is sent to shorten the distance between the two features, as shown in the following formula:

[0093] f' enhance ,f' fuse =Feature_guided(f enhance ,f fuse ) (14)

[0094] Among them, Feature_guided(g) is the feature guided module;

[0095] Step 2.5: The image I obtained in step 2.2 fuse The predicted grab box parameters of the grab detection task are output through a grab detection head. i , and predicted category information

[0096] In step 2.6, the total loss function of the RGB-depth image fusion enhanced detection model is defined as follows:

[0097] loss = α·loss reg +β·loss cls +γloss_f(15)

[0098]

[0099]

[0100]

[0101] Among them, loss reg represents the regression loss, which is used to evaluate the position and size difference between the predicted grasping box and the real grasping box, p i , g i They represent the real grab box parameters and the predicted grab box parameters, respectively, and loss cls Represents the category loss, which is used to evaluate the difference between the captured box category and the true captured box category, y i , They represent the real category information and the predicted category information respectively. Loss_f represents the feature loss, which is mainly used to make the model focus more on the extraction of edge information. N represents the number of images. Respectively represent the gradient of the enhanced feature map and the gradient of the fused feature map, ||·|| 2 represents the square of the norm, α, β, γ are balance factors;

[0102] Step 2.7, divide the training set and the test set from the dataset DS in a ratio of 8:2. Evaluate the model performance through the validation set. When the performance of the RGB-depth image fusion enhanced detection model meets the expected requirements and there is no significant performance improvement, the training of the RGB-depth image fusion enhanced detection model is considered to have converged, that is, the training is completed.

[0103] Step 3: Input the collected low-light image and depth image into the trained RGB-depth image fusion enhancement detection model to obtain the predicted enhanced image and the predicted value of the grab box, and complete the grab detection task in the low-light environment. The low-light image, depth image, and grab detection results of the model taken in this embodiment are shown in Figure 2. Figure 3 As shown;

[0104] The whole process is:

[0105] M(R low ,D depth )→{I i enhance ,g i},i=1,2,···,m(19)

[0106] in, represents the predicted enhanced image, g i represents the predicted grab box parameters, and m represents the total number of grab boxes.

[0107] The above shows and describes the main features and advantages of the present invention. It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes that fall within the meaning and scope of the equivalent elements of the claims are included in the present invention.

[0108] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A robot low-light grasping detection method based on wavelet domain visual enhancement, characterized in that: The following steps are involved: Step 1: Collect paired low-light and normal-light data of the target to be captured in a low-light environment to construct a dataset DS. The dataset DS consists of pairs of "low-light image-normal-light image-depth image-capture annotation information", that is, in, represents the i-th low-light image, represents the i-th normal light image, represents the i-th depth image, P i Represents the captured annotation information of the i-th image, including the actual category information y i , i = 1, 2, 3, ···, n, n represents the number of images in the dataset; Step 2: Input the constructed dataset DS into the RGB-depth image fusion enhancement detection model for training. The RGB-depth image fusion enhancement detection model includes an image enhancement branch and a grasping detection branch. The backbone network of the image enhancement branch adopts U-Net to extract local features in the input low-light image and learn global context information. The grasping detection branch fuses the low-light image with the depth map in the wavelet domain, and inputs the fused image and the image enhanced by the image enhancement branch into the edge attention module together, and closes the edge feature information of the two in the feature guidance module; Step 3: Input the collected low-light image and depth image into the trained RGB-depth image fusion enhancement detection model to obtain the predicted enhanced image and grasping box prediction value, and complete the grasping detection task in the low-light environment.

2. According to claim 1, a robot low-light grasping detection method based on wavelet domain visual enhancement is characterized in that: The step 1 is specifically as follows: Step 1.1, collect low-light, normal-light captured images and depth images of the target to be captured in a dark room; Step 1.2, perform grabbing frame annotation on the corresponding normal light image to form a pair of "low light image-normal light image-depth image-grab annotation information"; Step 1.3: Change the target to be grasped and repeat steps 1.1 and 1.2 until a complete data set DS is obtained.

3. According to claim 1, a robot low-light grasping detection method based on wavelet domain visual enhancement is characterized in that: The step 2 is specifically as follows: Step 2.1: The U-Net adopts an encoder-decoder structure and inputs the low-light image R low The shape of is (N, C, H, W). The output of each convolution layer is half the width and height of the previous layer, and the channel is doubled, that is, the output is (N, 2×C, H / 2, W / 2). The encoder is composed of a series of convolutional layers-activation layers-maximum pooling layers, as shown below: Y k =MaxPooling(relu(BN(Conv(R low ))))=1,2,3,4 (1) Among them, Conv(·) represents the convolution layer, relu(·) represents the activation layer, MaxPooling(·) represents the maximum pooling layer, and Y k Represents the feature map output of the kth layer; Add SE units in the bottleneck layer as follows: AND BottleNeck =SE(Y4) (2) The decoder performs a preliminary restoration of the image as follows: M1=relu(BN(Conv(Y BottleNeck ))) (3) The jump connection part is responsible for passing the feature map of the encoder part to the decoder part, which is achieved through the splicing operation as follows: C1=[M1,Y4] (4) After that, repeat the above process to obtain M2, C2, M3, C3, M4, as shown below: M2=relu(BN(Conv(M1))) (5) C2=[M2,Y3] (6) M3=relu(BN(Conv(M2))) (7) C3=[M3,Y2] (8) M4=relu(BN(Conv(M3))) (9) M4 is the final output I enhance . Step 2.2: For the low-light image R low and the depth image D depth Perform discrete wavelet transform, then fuse the two, and perform convolution block operation after fusion to obtain the fused image I fuse , as follows: Among them, W RGB is the wavelet coefficient matrix of the low-light image, W depth is the wavelet coefficient matrix of the depth image, W represents the discrete wavelet transform operation; ConvBlock(g) represents a set of convolution blocks, Represents a splicing operation; represents inverse wavelet transform; Step 2.3: The image I obtained in step 2.1 enhance The image I obtained in step 2.2 fuse Send it to the edge attention module to learn edge information and get the feature map f enhance ,f fuse , as follows: f enhance ,f fuse =Edge_attention(I enhance ,I fuse ) (13) Among them, Edge_attention(g) is the edge attention module; Step 2.4: The feature map f obtained in step 2.3 enhance ,f fuse The feature guidance module is sent to shorten the distance between the two features, as shown in the following formula: f′ enhance ,f′ fuse =Feature_guided(f enhance ,f fuse ) (14) Among them, Feature_guided(g) is the feature guided module; Step 2.5: The image I obtained in step 2.2 fuse The predicted grab box parameters of the grab detection task are output through a grab detection head. i , and predicted category information In step 2.6, the total loss function of the RGB-depth image fusion enhanced detection model is defined as follows: loss=α·loss reg +β·loss cls +γloss_f (15) Among them, loss reg represents the regression loss, p i , g i They represent the real grab box parameters and the predicted grab box parameters, respectively, and loss cls represents the category loss, They represent the real category information and the predicted category information respectively, loss_f represents the feature loss, N represents the number of images, They represent the gradient of the enhanced feature map and the gradient of the fused feature map, respectively, and ||g|| 2 represents the square of the norm, α, β, γ are balance factors; Step 2.7: When the performance of the RGB-depth image fusion enhanced detection model meets the expected requirements and there is no significant performance improvement, the training of the RGB-depth image fusion enhanced detection model is considered to have converged, that is, the training is completed.

4. According to claim 1, a robot low-light grasping detection method based on wavelet domain visual enhancement is characterized in that: The step 3 is specifically as follows: in, represents the predicted enhanced image, g i represents the predicted grab box parameters, and m represents the total number of grab boxes.

Images