Foreign matter identification method suitable for narrow dark environment

Abstract

The invention discloses a foreign matter recognition method suitable for a narrow dark environment. A flexible mechanical arm is used for collecting foreign matter videos possibly contained in the narrow dark environment in a simulation body to make a detection data set; introducing a new scale adjustment parameter l and an attenuation item parameter alpha into the image in the data set by using an improved dynamic center surrounding function to extract an illumination image and a reflection image; three-branch parallel feature extraction is carried out on an original image, an illumination image and a reflection image, CBAM is used for feature enhancement, then fusion reconstruction is carried out for enhancement, an improved dynamic SSIM loss function is introduced, a parameter value of a center surround function is adjusted, an optimal fusion image is obtained, and the optimal fusion image is obtained. And the enhanced picture is continuously input into the backbone network, a convolution layer is formed by dual-kernel convolution and a C2f module, convolution in feature fusion uses dual-kernel convolution and a multi-scale attention mechanism is added, and an end-to-end detection network is completed.

Description

Technical Field

[0001] The present invention relates to a method for detecting a foreign object that may exist in a narrow space by a flexible mechanical arm, and more specifically to a method for identifying a foreign object that is suitable for use in a narrow dark environment. Background Art

[0002] With the continuous development of target detection technology, the problem of target detection under conditions such as bad natural weather, low visibility indoors and outdoors, and uneven lighting under low-light fill-in conditions has gradually become a long-term and continuous challenge in the field of machine vision. Effective target detection has important application value in monitoring, security, autonomous driving, robot search and rescue, military reconnaissance and other fields for special circumstances such as uneven fill-in lighting conditions in narrow environments and low visibility in bad weather in natural environments. Effective identification of targets in harsh and uneven lighting environments is not only the key to improving safety and efficiency, but also the core of promoting the advancement of related technologies.

[0003] Among them, the flexible robotic arm needs to carry out related work in a closed and lightless narrow space. When the ultra-redundant flexible robotic arm vision system collects visual information data in a narrow space, there must be active fill-in lighting measures. Due to the influence of ambient light sources, the visual sensor has problems such as overexposure and underexposure. The collected low-light images mainly show problems such as insufficient local lighting, uneven overall lighting, and severe image noise. In its special environment, there may be industrial foreign matter and reflective boric acid crystals of various shapes in a narrow environment. Foreign matter in a narrow dark environment may affect the stability and safety of the robot's work, so an algorithm for identifying foreign matter in a narrow dark environment is particularly important.

[0004] Based on this, the present invention proposes a foreign object recognition method suitable for small dark environments, aiming to overcome the difficulties of image processing under uneven fill light in small spaces through innovative network architecture and processing methods, thereby significantly improving the accuracy and robustness of detection. Summary of the invention

[0005] In order to improve the accuracy of foreign object recognition in a small dark environment, the present invention proposes a foreign object recognition method suitable for a small dark environment, comprising the following steps:

[0006] S1: Use a flexible robotic arm to run in a simulated body to collect videos containing foreign objects, create a data set from the video images and divide it into a training set, a test set, and a validation set;

[0007] S2: Extract the reflection image and illumination image from the images collected by the camera in the flexible robotic arm in a narrow dark environment and a narrow dark environment with uneven fill light. When extracting the illumination image and the reflection image, the improved center surround function is used. The Gaussian distribution scale adjustment parameter l and the square attenuation parameter α are introduced to enhance the improved dynamic SSIM loss function in the network. Construct a new function to iterate the values ​​of two parameters to extract the reflection image and illumination image;

[0008] S3: The extracted illumination image, reflection image and original image are subjected to three-branch feature extraction. The extraction stream is respectively subjected to feature extraction by the common convolution layer and the AGRDB feature extraction module. The extracted features are then enhanced by the CBAM attention mechanism. The enhanced three-branch feature maps are directly added to obtain the comprehensive feature map. Finally, the comprehensive feature map is reconstructed to obtain the enhanced image. The improved dynamic SSIM loss function is introduced to adjust the reflection image and illumination image of the image decomposition.

[0009] Optionally, the specific steps of dynamically adjusting parameters to extract the illumination image and the reflection image in step S2 are as follows:

[0010] S21: The original image I(x,y) is transformed into

[0011] log(I(x,y))=log(L(x,y))+log(R(x,y))

[0012] Separate the illumination component and the reflection component;

[0013] S22: Use the improved Gaussian template to convolve the image I(x,y) to approximately obtain the incident image L(x,y), L(x,y) = F(x,y)*I(x,y), where F(x,y) is the improved center surround function

[0014]

[0015] c is the Gaussian surround scale; F(x, y) is the calculation of the illumination image in space, which is expressed as estimating the illumination in the image by calculating the weighted average of the pixel points in the image and the surrounding area; where l is used to control the scale change table of the Gaussian distribution, α controls the attenuation rate, and the loss function is used to iterate the values ​​of l and α, where η is the learning rate;

[0016] S23: Use R(x,y)=exp(log(I(x,y))-log(L(x,y))) to subtract the illumination image from the original image to obtain the reflected image.

[0017] Optionally, in step S3, the image loss function is introduced to adjust the center surround function parameters, i and α are taken as parameters to be optimized, their values ​​are dynamically adjusted by minimizing the loss function, and the weighted SSIM loss is introduced as the optimization target, and the total loss function is defined as:

[0018]

[0019] in Weight Matrix The improved dynamic SSIM local (x,y),

[0020]

[0021] is the image reconstruction loss, is the image smoothing loss.

[0022] Optionally, in step S3, a three-branch feature extraction flow is used to extract features and perform fusion reconstruction, and the specific steps are as follows:

[0023] S31: The reflection image, illumination image and dark channel image extracted from the narrow dark light image collected by the flexible robot arm camera are input into the feature extraction flow in parallel, which includes a common convolution layer and three adaptive gradient residual dense blocks AGRDB;

[0024] S32: After the three-branch feature extraction, the feature is input into the CBAM attention mechanism for feature enhancement. The enhanced feature maps are added and fused into a comprehensive feature map;

[0025] S33: The fused comprehensive feature map is reconstructed through a reconstruction module consisting of four 3×3 convolution layers with Leaky ReLU activation functions and one 1×1 convolution layer with Tanh activation function to obtain a fused image; the SSIM structural similarity between the fused image and the original image is calculated, and the center-surround function parameters are adjusted using the improved dynamic structural similarity loss function to obtain the illumination map and reflectance map with the best fusion effect, resulting in the best fused image;

[0026] Optionally, the steps of more lightweight detection network in step S4 are as follows:

[0027] S41: The fused image is input into the backbone feature extraction network, where the backbone network uses a lighter dual-core convolution and C2f module as the convolution layer to extract features and improve feature expression capabilities;

[0028] S42: The features extracted at different stages by the backbone network are further processed and integrated. Combined with the characteristics of dual-core convolution, a multi-scale attention mechanism (EMA) is further introduced to enhance the expression of multi-scale features and improve the recognition ability of objects of different scales.

[0029] S43: The processed features are input into the classification and detection branches, and the decoupled classification and regression branches are used to classify and locate the target. Finally, the position offset of the box is calculated through the regression head, and the category probability of each candidate box is output through the classification branch to complete the detection task.

[0030] Optionally, the adaptive gradient residual dense block AGRDB of step S3 includes two 3×3 convolutional layers and one 1×1 ordinary convolutional layer, and adopts a dense connection method to fully integrate the features extracted by each convolutional layer; the residual stream calculates the gradient size of the feature through adaptive gradient, and weights the features of each layer based on the gradient information of adaptive learning; in order to solve the problem of inconsistent channel dimensions, the network uses a 1×1 convolutional layer for dimensional matching at the end.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] (1) The present invention extracts an illumination image and a reflection image from an original image, fuses the illumination image, the reflection image and the original image, adjusts the image lighting information, and enhances the image detail information.

[0033] (2) Compared with other target detection methods based on image enhancement, the present invention reduces the number of parameters while improving the detection accuracy (mAP@50). BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 The present invention is a flow chart of a foreign body recognition method suitable for a narrow and dark environment.

[0035] Figure 2 It is a block diagram of an image enhancement network in a foreign body recognition method suitable for a narrow and dark environment of the present invention.

[0036] Figure 3 It is a structural diagram of an adaptive gradient residual dense block in an image enhancement network in a foreign body recognition method suitable for a small dark environment of the present invention.

[0037] Figure 4 The present invention is a foreign body recognition method detection network block diagram suitable for a small dark environment. DETAILED DESCRIPTION

[0038] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0039] Figure 1 This is a flow chart of a foreign body identification method suitable for a narrow dark environment of the present invention. Figure 1 As shown, a boric acid crystal detection method suitable for a small dark environment specifically includes the following steps:

[0040] S1: Use a flexible robotic arm to run in a simulation body to collect videos containing boric acid crystals, create a dataset from the video images and divide it into training set, test set and validation set;

[0041] S2: Extract the reflection image and illumination image from the images collected by the camera in the flexible robotic arm in low light, uneven fill light, crystal reflection and other environments. When extracting the illumination image and reflection image, the Gaussian distribution scale adjustment parameter l and the square attenuation term parameter α are introduced. The reflection image and illumination image are extracted by iterating the values ​​of the two parameters through the improved dynamic SSIM loss function in the enhanced network;

[0042] S3: The three-branch feature extraction streams of the extracted original image, illumination image and reflection image are respectively subjected to feature extraction by the common convolution layer and the AGRDB feature extraction module, and then the extracted features are enhanced by the CBAM attention mechanism. The enhanced three-branch feature maps are directly added to obtain the comprehensive feature map. Finally, the comprehensive feature map is reconstructed to obtain the enhanced image, and the improved dynamic SSIM loss function is introduced to adjust the reflection image and illumination image of the image decomposition;

[0043] S4: The image obtained by fusing the reflection image and the illumination image extracted from the original image with the original image is input into the lightweight detection network for overall network training to obtain the training model weights, and the training parameters are adjusted to obtain the optimal detection model;

[0044] Figure 2 This is a fusion image network model diagram of a foreign body recognition method suitable for a narrow dark environment of the present invention, which specifically includes the following steps:

[0045] S21: The original image I(x,y) is transformed into

[0046] log(I(x,y))=log(L(x,y))+log(R(x,y))

[0047] Separate the illumination component and the reflection component;

[0048] S22: Use the Gaussian template to convolve the image I(x,y) to approximately obtain the incident image L(x,y), L(x,y) = F(x,y)*I(x,y), where F(x,y) is the improved center surround function

[0049]

[0050] c is the Gaussian surround scale; F(x, y) is the calculation of the illumination image in space, which is expressed as estimating the illumination in the image by calculating the weighted average of the pixel points in the image and the surrounding area; where l is used to control the scale change table of the Gaussian distribution, α controls the attenuation rate, and the loss function is used to iterate the values ​​of l and α, where η is the learning rate;

[0051] S23: Use R(x,y)=exp(log(I(x,y))-log(L(x,y))) to subtract the illumination image from the original image to obtain the reflected image.

[0052] S24: The dark channel image, illumination image and original image extracted from the original image are input into the fusion network in parallel. The shallow features are extracted through a common convolutional layer with a kernel size of 3×3 and an activation function of Leaky corrected linear unit. Then two adaptive gradient residual dense blocks AGRDB are used to extract fine-grained features from the shallow features. Finally, the extracted features are added. Given the input F of AGRDB i , its output F o+1 It can be expressed as:

[0053]

[0054] Where Conv(·) represents a convolutional layer, and Convn(·) represents n cascaded convolutional layers. It is a gradient operator, which is a special convolution operation whose convolution kernel is manually designed. The gradient operator convolves the input features with the high-frequency convolution kernel to extract fine-grained detail information, uses adaptive gradient calculation to calculate the gradient, and dynamically calculates the gradient information of the image through the convolution layer. Here, two convolution layers are used to calculate the gradient in the horizontal and vertical directions respectively. In addition, Represents the sum of elements.

[0055] S25: The features extracted by the three-branch feature extraction stream are fused and reconstructed through the CBAM attention mechanism to achieve feature aggregation and image reconstruction. The adaptive gradient residual dense block aggregates the learnable convolution features and gradient amplitude information. Then, the fused image is reconstructed through the feature fusion and image reconstruction module. The cascade fusion strategy is used to fuse the features of the original image grayscale image and the original image reflectance image, which contains rich fine-grained spatial details. The fusion and reconstruction process is expressed as follows:

[0056]

[0057] I f =R I (F f )

[0058] where С(·) refers to the connection in the channel dimension. Finally, the image reconstructor R I Recover F from fused features f Fusion Image I f

[0059] S26: Calculate the structural similarity SSIM of the original image and the fused image, introduce the improved dynamic SSIM loss function to adjust the center surround function parameters, take l and α as the parameters to be optimized, dynamically adjust their values ​​by minimizing the loss function, introduce weighted SSIM loss as the optimization target, and define the total loss function as:

[0060]

[0061] in Weight Matrix The improved dynamic SSIM local (x,y),

[0062]

[0063] is the image reconstruction loss, is the image smoothing loss. The illumination map and reflectance map are adjusted through the loss function to obtain the optimal fused image.

[0064] Figure 3 It is an adaptive gradient residual dense block in a dark image fusion enhancement network in a foreign body recognition method suitable for a narrow dark environment of the present invention. Specifically, the module deploys three 3×3 convolutional layers and one ordinary convolutional layer, and the kernel size of the convolutional layer is 1×1. In particular, dense connections are introduced in the module to make full use of the features extracted by each convolutional layer. The residual stream uses adaptive gradient operations to calculate the gradient size of the features, and uses a 1×1 regular convolutional layer to eliminate channel dimension differences. Then, the outputs of the main dense stream and the residual gradient stream are added by element-by-element addition to integrate deep features and fine-grained detail features;

[0065] Figure 4 The present invention is a lightweight detection network block diagram of a foreign body recognition method suitable for a narrow dark environment, which specifically includes the following steps:

[0066] S41: The fused image dataset enhanced by the fusion network is input into the backbone feature extraction of target detection. The backbone part consists of a common convolution, 4 dual-core convolution modules, 4 C2f modules and an SPPF module. In the backbone feature extraction part, a series of convolution and deconvolution layers are used to extract features, and residual connections and bottleneck structures are also used to reduce the size of the network and improve performance. Specifically, this part uses C2f and dual-core convolution modules as the basic components of the convolution layer. Compared with the common convolution layer, the dual convolution kernel is used together with the C2f module to have better feature extraction capabilities while reducing the number of parameters. The dual-core convolution uses 3×3 and 1×1 convolution kernels to process the same input feature map channel at the same time, and uses group convolution technology. The input and output feature maps are divided into multiple groups. The convolution filter of each group only processes a part of the corresponding input feature map, and performs group convolution and point-by-point convolution at the same time, and then adds the results.

[0067] S42: The features extracted by the backbone are fused through the path aggregation network PAN-FPN structure with a feature pyramid network. The feature map obtained after feature fusion is input into the multi-scale attention module and then output to the detection and classification module; the multi-scale attention module reshapes some channels to the batch dimension and groups the channel dimension into multiple sub-features to retain the information of each channel and reduce the computational overhead, and uses cross-dimensional interactions to capture pixel-level relationships. The multi-scale attention module EMA inputs the features extracted by the front network into the multi-scale attention module and is first grouped, and then processed by different branches: one branch performs one-dimensional global pooling, and the other performs feature extraction through 3×3 convolution; the output features of the two branches are then modulated by the sigmoid function and normalization operation, and finally merged through the cross-dimensional interaction module to capture the pairwise relationship at the pixel level; after the final sigmoid adjustment, the output feature map is used to enhance or weaken the original input features to obtain the final output. It introduces different structures and techniques to fuse multi-scale feature maps to better capture information of targets of different scales. At the same time, it uses an efficient multi-scale attention module EMA to recalibrate the channel weights in each parallel branch by encoding global information and interact across dimensions to capture pixel-level relationships.

[0068] S43: Input the fused features into the detection and classification module for target detection and classification tasks. This module consists of two parts: the detection branch and the classification branch. The detection head consists of a series of convolutional layers and deconvolutional layers, which are responsible for generating the detection results of the target; the classification head classifies each feature map through global average pooling. The detection and classification module first calculates the offset between the predicted box and the true box, and inputs these offsets into the regression head for loss calculation, so as to obtain the probability that the candidate box belongs to each category. Finally, the optimal detection box is selected as the final detection result by the maximum suppression method.

[0069] The above description is only a preferred embodiment of the present invention, so that those skilled in the art can understand the present invention. However, it should be clear that this is not intended to limit the present invention. All other embodiments obtained by those of ordinary skill in the art without making any creative work, any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention, should fall within the scope of protection of the present invention.

Claims

1. A foreign body recognition method suitable for use in a narrow dark environment, comprising the following steps: S1: Use a flexible robotic arm to run in a simulated body to collect videos containing foreign objects, create a data set from the video images and divide it into a training set, a test set, and a validation set; S2: Extract the reflection image and illumination image from the images collected by the camera in the flexible robotic arm in a narrow dark environment and a narrow dark environment with uneven fill light. When extracting the illumination image and the reflection image, the improved center surround function is used. The Gaussian distribution scale adjustment parameter l and the square attenuation parameter α are introduced to fusion network to improve the dynamic SSIM loss function. Construct a new function to iterate the values ​​of two parameters to extract the reflection image and illumination image; S3: The extracted illumination image, reflection image and original image are subjected to three-branch feature extraction. The extraction stream is respectively subjected to feature extraction by the common convolution layer and the AGRDB feature extraction module. The extracted features are then enhanced by the CBAM attention mechanism. The enhanced three-branch feature maps are directly added to obtain the comprehensive feature map. Finally, the comprehensive feature map is reconstructed to obtain the enhanced image. The improved dynamic SSIM loss function is introduced to adjust the reflection image and illumination image of the image decomposition. S4: The image obtained by fusing the reflection image and illumination image extracted from the original image with the original image is input into the lightweight detection network for overall network training to obtain the training model weights, and the training parameters are adjusted to obtain the optimal detection model, thus completing the foreign object recognition method of the flexible robotic arm in a narrow and dark environment.

2. The foreign body identification method suitable for use in a narrow dark environment according to claim 1, characterized in that: The specific steps of dynamically adjusting parameters to extract illumination image and reflection image in step S2 are as follows: S21: The original image I(x,y) is transformed into log(I(x,y))=log(L(x,y))+log(R(x,y)) Separate the illumination component and the reflection component; S22: Use the Gaussian template to convolve the image I(x,y) to approximately obtain the incident image L(x,y), L(x,y) = F(x,y)*I(x,y), where F(x,y) is the improved center surround function c is the Gaussian surround scale; F(x,y) is the calculation of the illumination image in space, which is expressed as estimating the illumination in the image by calculating the weighted average of the pixel point in the image and the surrounding area; Where l is used to control the scale change table of Gaussian distribution, α controls the attenuation rate, and the loss function is used to iterate the values ​​of l and α. η is the learning rate; S23: Use R(x,y)=exp(log(I(x,y))-log(L(x,y))) to subtract the illumination image from the original image to obtain the reflected image.

3. The foreign body identification method applicable to a narrow dark environment according to claim 1, characterized in that: In step S3, the image loss function is introduced to adjust the center-surround function parameters. l and α are taken as parameters to be optimized, and their values ​​are dynamically adjusted by minimizing the loss function. The weighted SSIM loss is introduced as the optimization target, and the total loss function is defined as: in Weight Matrix The improved dynamic SSIM local (x,y): also is the image reconstruction loss is the image smoothing loss.

4. The foreign body identification method suitable for use in a narrow dark environment according to claim 1, characterized in that: In step S3, a three-branch feature extraction flow is used to extract features and perform fusion reconstruction. The specific steps are as follows: S31: Extract the reflection image and illumination image from the narrow dark light image collected by the flexible robot arm camera, and input the decomposed image and the original image into the feature extraction flow in three branches in parallel, which includes a common convolution layer and three adaptive gradient residual dense blocks AGRDB; S33: After the three-branch feature extraction, the feature is input into the CBAM attention mechanism for feature enhancement, and the enhanced feature maps are added and fused into a comprehensive feature map; S33: The fused comprehensive feature map is reconstructed through a reconstruction module consisting of four 3×3 convolution layers with Leaky ReLU activation functions and one 1×1 convolution layer with Tanh activation function to obtain a fused image; The SSIM structural similarity between the fused image and the original image is calculated, and the center-surround function parameters l and α are adjusted using the improved dynamic structural similarity loss function to obtain the illumination map and reflectance map with the best fusion effect.

5. The foreign body identification method suitable for use in a narrow dark environment according to claim 1, characterized in that: The steps of the robot arm perception detection network in step S4 are as follows: S41: The fused and enhanced image is input into the backbone feature extraction network, where the backbone network uses a lighter dual-core convolution and C2f module as the convolution layer to extract features and improve feature expression capabilities; S42: The features extracted at different stages by the backbone network are further processed and integrated. Combined with the characteristics of dual-core convolution, a multi-scale attention mechanism (EMA) is further introduced to enhance the expression of multi-scale features and improve the recognition ability of objects of different scales. S43: The processed features are input into the classification and detection branches, and the decoupled classification and regression branches are used to classify and locate the target. Finally, the position offset of the box is calculated through the regression head, and the category probability of each candidate box is output through the classification branch to complete the detection task.

6. The foreign body identification method applicable to a narrow dark environment according to claim 4, characterized in that: The adaptive gradient residual dense block AGRDB comprises two 3×3 convolutional layers and one 1×1 ordinary convolutional layer, and adopts a dense connection method to fully integrate the features extracted by each convolutional layer; The residual stream calculates the gradient size of the feature through adaptive gradient, and weights the features of each layer based on the gradient information of adaptive learning; in order to solve the problem of inconsistent channel dimensions, the network uses a 1×1 convolutional layer for dimensional matching at the end.

Images