A foreign matter recognition method suitable for narrow dark environment

Abstract

The application discloses a foreign matter identification method suitable for narrow and dark environments, uses a flexible mechanical arm to collect foreign matter video that may be contained in a narrow and dark light environment in a simulation body to make a detection data set; introduces new scale adjustment parameter l and attenuation term parameter alpha into an improved dynamic center around function to extract an illumination image and a reflection image; extracts three-branch parallel features of an original image, the illumination image and the reflection image, and performs feature enhancement and re-fusion reconstruction by using CBAM for enhancement, introduces an improved dynamic SSIM loss function, adjusts the parameter value of the center around function, obtains an optimal fusion image, continuously inputs the enhanced picture into a backbone network, and completes an end-to-end detection network by using a double-core convolution and a C2f module to form a convolution layer, using double-core convolution in convolution in feature fusion, and adding a multi-scale attention mechanism.

Description

Technical Field

[0001] This invention relates to a method for detecting foreign objects in confined spaces using a flexible robotic arm, and more specifically to a method for identifying foreign objects in confined, dark environments. Background Technology

[0002] With the continuous development of target detection technology, the problem of target detection under adverse weather conditions, low indoor and outdoor visibility, and uneven lighting conditions has gradually become a long-term and ongoing challenge in the field of machine vision. Effective target detection in special environments such as narrow spaces with uneven lighting conditions and low visibility in adverse weather has significant application value in various fields such as surveillance, security, autonomous driving, robotic search and rescue, and military reconnaissance. Effectively identifying targets in harsh and unevenly lit environments is not only key to improving safety and efficiency but also a core driver of related technological advancements.

[0003] Flexible robotic arms need to operate in confined, dark spaces. When a highly redundant flexible robotic arm vision system acquires visual information in such a confined space, active supplemental lighting is essential. Due to the influence of ambient light, vision sensors suffer from overexposure and underexposure, resulting in low-light images characterized by insufficient local illumination, uneven overall lighting, and significant image noise. Furthermore, industrial foreign objects and irregularly shaped borate crystals reflecting light may be present in these confined environments. Foreign objects in small, dark environments can affect the stability and safety of the robotic arm's operation; therefore, a foreign object recognition algorithm for such environments is particularly important.

[0004] Based on this, the present invention proposes a foreign object identification method suitable for confined and dark environments. It aims to overcome the difficulties of image processing under uneven lighting in confined spaces through innovative network architecture and processing methods, thereby significantly improving the accuracy and robustness of detection. Summary of the Invention

[0005] To improve the accuracy of foreign object identification in confined and dark environments, this invention proposes a foreign object identification method suitable for such environments, comprising the following steps:

[0006] S1: Use a flexible robotic arm to run in the simulation to collect videos containing foreign objects, create a dataset from the video images and divide it into training set, test set and validation set;

[0007] S2: Extract the reflection and illumination images from images acquired using a camera in a flexible robotic arm in confined dark environments and in confined dark environments with uneven lighting. When extracting the illumination and reflection images, utilize the improved center-around function. This involves introducing a Gaussian distribution scaling parameter l and a squared decay term parameter α to enhance the improved dynamic SSIM loss function in the network. A new function is constructed to iterate over the values ​​of two parameters to extract the reflectance and illumination images;

[0008] S3: The extracted illumination image, reflectance image, and original image are subjected to three-branch feature extraction. The extraction stream is performed by a regular convolutional layer and an AGRDB feature extraction module, respectively. The extracted features are then enhanced by the CBAM attention mechanism. The enhanced three-branch feature maps are directly added together to obtain a comprehensive feature map. Finally, the comprehensive feature map is reconstructed to obtain the enhanced image. An improved dynamic SSIM loss function is introduced to adjust the reflectance image and illumination image of the image decomposition.

[0009] Optionally, the specific steps for dynamically adjusting parameters to extract the illuminance image and reflectance image in step S2 are as follows:

[0010] S21: The original image I(x,y) is transformed using the logarithm method.

[0011] log(I(x,y))=log(L(x,y))+log(R(x,y))

[0012] Separate the illuminance component and the reflectance component;

[0013] S22: Convolve the image I(x,y) using the improved Gaussian template to approximate the incident image L(x,y), where L(x,y) = F(x,y)*I(x,y), and F(x,y) is the improved center wrap function.

[0014]

[0015] c represents the Gaussian surrounding scale; F(x,y) is the calculation of the illumination image in space, which is expressed as estimating the illumination in the image by calculating the weighted average of the pixels in the image and the surrounding area; where l is used to control the scale change table of the Gaussian distribution, α controls the decay rate, and the values ​​of l and α are iterated using the loss function, where η is the learning rate;

[0016] S23: Use R(x,y)=exp(log(I(x,y))-log(L(x,y))) to subtract the illumination image from the original image to obtain the reflection image.

[0017] Optionally, in step S3, an image loss function is introduced to adjust the center wrap function parameters. This involves using l and α as parameters to be optimized, dynamically adjusting their values ​​by minimizing the loss function, and introducing a weighted SSIM loss as the optimization objective. The total loss function is defined as follows:

[0018]

[0019] in weight matrix Among them, the improved dynamic SSIM local (x,y),

[0020] For image reconstruction loss, Image smoothing loss.

[0021] Optionally, in step S3, a three-branch feature extraction flow is used to extract features and perform fusion reconstruction. The specific steps are as follows:

[0022] S31: The feature extraction stream consists of three parallel branches: the reflection image, the illumination image, and the dark channel image extracted from the narrow, low-light image captured by the flexible robotic arm camera. It includes a common convolutional layer and three adaptive gradient residual dense blocks (AGRDB).

[0023] S32: After the features are extracted through the three-branch feature extraction, they are input into the CBAM attention mechanism for feature enhancement. The enhanced feature maps are then added together and fused into a comprehensive feature map.

[0024] S33: The fused comprehensive feature map is passed through a reconstruction module consisting of four 3×3 convolutional layers with Leaky ReLU activation functions and one 1×1 convolutional layer with Tanh activation function to reconstruct the image, resulting in the fused image; the SSIM structural similarity between the fused image and the original image is calculated, and the center surrounding function parameters are adjusted using the improved dynamic structural similarity loss function to obtain the illumination map and reflectance map with the best fusion effect, thus achieving the best fused image;

[0025] Optionally, the step of making the detection network more lightweight in step S4 is as follows:

[0026] S41: The fused image is input into the backbone feature extraction network, where the backbone network uses a lighter dual-kernel convolution and C2f module as convolutional layers to extract features while improving feature expression capabilities;

[0027] S42: Further process and fuse the features extracted from different stages of the backbone network. Combining the characteristics of dual-kernel convolution, a multi-scale attention mechanism (EMA) is introduced to enhance the expression of multi-scale features and improve the ability to recognize targets at different scales.

[0028] S43: Input the processed features into the classification and detection branches, use the decoupled classification and regression branches to classify and locate the target, finally calculate the position offset of the bounding box through the regression head, and output the class probability of each candidate box through the classification branch, thereby completing the detection task.

[0029] Optionally, the Adaptive Gradient Residual Dense Block (AGRDB) in step S3 includes two 3×3 convolutional layers and one 1×1 ordinary convolutional layer, and adopts a dense connection method to fully integrate the features extracted by each convolutional layer; the residual stream calculates the gradient magnitude of the features through adaptive gradient, and performs weighted adjustment of the features of each layer based on the gradient information learned adaptively; in order to solve the problem of inconsistent channel dimensions, the network uses a 1×1 convolutional layer for dimension matching at the end.

[0030] Compared with the prior art, the beneficial effects of the present invention are:

[0031] (1) The present invention utilizes the extraction of illuminance image and reflection image from the original image, and performs image fusion of illuminance image, reflection image and original image to adjust image lighting information and enhance image detail information.

[0032] (2) Compared with other image enhancement-based target detection methods, this invention improves detection accuracy (mAP@50) while reducing the number of parameters. Attached Figure Description

[0033] Figure 1 This is a flowchart of a foreign object identification method applicable to small, dark environments according to the present invention.

[0034] Figure 2 This is a block diagram of an image enhancement network in a foreign object recognition method applicable to small, dark environments according to the present invention.

[0035] Figure 3 This is a structural diagram of the adaptive gradient residual dense block in the image enhancement network of a foreign object recognition method suitable for small, dark environments according to the present invention.

[0036] Figure 4 This is a network diagram of a foreign object identification method applicable to small, dark environments according to the present invention. Detailed Implementation

[0037] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0038] Figure 1 This is a flowchart of a foreign object identification method applicable to confined, dark environments according to the present invention. For example... Figure 1 As shown, a method for detecting boric acid crystals suitable for confined, dark environments specifically includes the following steps:

[0039] S1: Use a flexible robotic arm to run in the simulation to collect videos containing borate crystals, create a dataset from the video images and divide it into training set, test set and validation set;

[0040] S2: Extract the reflection image and illuminance image from images captured by the camera in the flexible robotic arm in environments such as low light, uneven lighting, and crystal reflection. When extracting the illuminance image and the reflection image, a Gaussian distribution scale adjustment parameter l and a squared decay term parameter α are introduced. The values ​​of the two parameters are iterated by the improved dynamic SSIM loss function in the enhancement network to extract the reflection image and the illuminance image.

[0041] S3: The three-branch feature extraction streams of the extracted original image, illumination image, and reflectance image are extracted by ordinary convolutional layer and AGRDB feature extraction module, respectively. The extracted features are then enhanced by CBAM attention mechanism. The enhanced three-branch feature maps are directly added to obtain a comprehensive feature map. Finally, the comprehensive feature map is reconstructed to obtain the enhanced image. An improved dynamic SSIM loss function is introduced to adjust the reflectance image and illumination image of the image decomposition.

[0042] S4: The enhanced image obtained by fusing the reflection image and illumination image extracted from the original image with the original image is input into a lightweight detection network for overall network training. The training model weights are obtained by training the network and the training parameters are adjusted to obtain the optimal detection model.

[0043] Figure 2 This invention provides a method for identifying foreign objects in confined, dark environments, which incorporates an image fusion network model and includes the following steps:

[0044] S21: The original image I(x,y) is transformed using the logarithm method.

[0045] log(I(x,y))=log(L(x,y))+log(R(x,y))

[0046] Separate the illuminance component and the reflectance component;

[0047] S22: Convolve the image I(x,y) using a Gaussian template to approximate the incident image L(x,y), where L(x,y) = F(x,y)*I(x,y), and F(x,y) is the improved center wrap function.

[0048]

[0049] c represents the Gaussian surrounding scale; F(x,y) is the calculation of the illumination image in space, which is expressed as estimating the illumination in the image by calculating the weighted average of the pixels in the image and the surrounding area; where l is used to control the scale change table of the Gaussian distribution, α controls the decay rate, and the values ​​of l and α are iterated using the loss function, where η is the learning rate;

[0050] S23: Use R(x,y)=exp(log(I(x,y))-log(L(x,y))) to subtract the illumination image from the original image to obtain the reflection image.

[0051] S24: The dark channel image and illumination image extracted from the original image are input in parallel into the fusion network. Shallow features are extracted through ordinary convolutional layers with a kernel size of 3×3 and Leaky-corrected linear units as the activation function. Then, two Adaptive Gradient Residual Dense Blocks (AGRDB) are used to extract fine-grained features from the shallow features. Finally, the extracted features are summed, given the input F of the AGRDB. i Its output F o+1 It can be represented as:

[0052]

[0053] Where Conv(·) represents a convolutional layer, and Convn(·) represents an n-cascaded convolutional layer. This is the gradient operator, a special type of convolution operation with a manually designed kernel. The gradient operator convolves input features with a high-frequency kernel to extract fine-grained details, utilizes adaptive gradient calculation to compute the gradient, and dynamically calculates the image's gradient information through convolutional layers. Here, two convolutional layers are used to calculate the horizontal and vertical gradients respectively. Furthermore, This indicates summation of elements.

[0054] S25: The features extracted by the three-branch feature extraction stream are fused and reconstructed using the CBAM attention mechanism to achieve feature aggregation and image reconstruction. Adaptive gradient residual dense blocks aggregate learnable convolutional features with gradient magnitude information. Then, the fused image is reconstructed through the feature fusion and image reconstruction module. A cascaded fusion strategy is used to fuse features from the original image's grayscale and reflectance maps, containing rich fine-grained spatial details. The fusion and reconstruction process is represented as follows:

[0055]

[0056] I f =R I (F f )

[0057] Where C(·) refers to the connectivity in the channel dimension. Finally, the image reconstructor R... I F is recovered from the fusion features. f Fusion Image I f

[0058] S26: Calculate the structural similarity (SSIM) between the original image and the fused image. An improved dynamic SSIM loss function is introduced to adjust the center-wrap function parameters. This involves using l and α as parameters to be optimized, dynamically adjusting their values ​​by minimizing the loss function. A weighted SSIM loss is introduced as the optimization objective, and the total loss function is defined as:

[0059]

[0060] in weight matrix Among them, the improved dynamic SSIM local (x,y),

[0061] For image reconstruction loss, Image smoothing loss is applied. The optimal fused image is obtained by adjusting the illumination map and reflectance map using the loss function.

[0062] Figure 3 This invention relates to an adaptive gradient residual dense block in a dark image fusion enhancement network for foreign object recognition in confined, dark environments. Specifically, this module deploys three 3×3 convolutional layers and one regular convolutional layer, with a kernel size of 1×1. Dense connections are introduced into the module to fully utilize the features extracted by each convolutional layer. The residual stream uses adaptive gradient operations to calculate the gradient magnitude of the features, and a 1×1 regular convolutional layer is used to eliminate channel dimension differences. Then, the outputs of the main dense stream and the residual gradient stream are added element-wise to integrate deep features and fine-grained detail features.

[0063] Figure 4 This invention provides a lightweight detection network diagram for a foreign object identification method suitable for confined, dark environments, specifically including the following steps:

[0064] S41: The fused image dataset, enhanced by the fusion network, is input into the backbone feature extraction for object detection. The backbone consists of a regular convolution, four dual-kernel convolutional modules, four C2f modules, and one SPPF module. The backbone feature extraction employs a series of convolutional and deconvolutional layers to extract features, while also using residual connections and bottleneck structures to reduce network size and improve performance. Specifically, this part uses C2f and dual-kernel convolutional modules as the basic convolutional layers. Compared to regular convolutional layers, using dual convolutional kernels with C2f modules provides superior feature extraction capabilities while reducing the number of parameters. The dual-kernel convolution uses 3×3 and 1×1 convolutional kernels to process the same input feature map channels simultaneously, and employs group convolution techniques. The input and output feature maps are divided into multiple groups, and the convolutional filter in each group only processes a portion of the corresponding input feature map. Group convolution and pointwise convolution are performed simultaneously, and the results are then summed.

[0065] S42: The features extracted from the backbone are fused through a PAN-FPN (Packet-Packet Network) structure with a feature pyramid network. The resulting feature map is then input into a multi-scale attention module and output to the detection and classification module. The multi-scale attention module reshapes some channels to the batch dimension and groups the channel dimension into multiple sub-features to retain information from each channel and reduce computational overhead. It utilizes cross-dimensional interactions to capture pixel-level relationships. The multi-scale attention module (EMA) first groups the features extracted from the preceding network and then processes them through different branches: one branch performs one-dimensional global pooling, and the other extracts features through 3×3 convolutions. The output features from both branches are then modulated using a sigmoid function and normalization, and finally merged through a cross-dimensional interaction module to capture pixel-level pairwise relationships. After final sigmoid adjustment, the output feature map enhances or weakens the original input features, thus obtaining the final output. It fuses feature maps at multiple scales by introducing different structures and techniques to better capture information about targets at different scales. At the same time, it uses an efficient multi-scale attention module (EMA) to recalibrate the channel weights in each parallel branch by encoding global information and to capture pixel-level relationships through cross-dimensional interactions.

[0066] S43: The fused features are input into the detection and classification module for object detection and classification tasks. This module consists of two parts: a detection branch and a classification branch. The detection head consists of a series of convolutional and deconvolutional layers, responsible for generating the object detection results; the classification head performs classification processing on each feature map through global average pooling. The detection and classification module first calculates the offset between the predicted box and the ground truth box, and inputs these offsets into the regression head for loss calculation, thereby obtaining the probability that the candidate box belongs to each category. Finally, the optimal detection box is selected as the final detection result through the maximum suppression method.

[0067] The above description is merely a preferred embodiment of the present invention, intended to help those skilled in the art understand the invention. However, it should be understood that this is not intended to limit the invention. All other embodiments obtained by those skilled in the art without creative effort, and any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention, should fall within the scope of protection of the present invention.

Claims

1. A method for foreign object identification in confined, dark environments, comprising the following steps: S1: Use a flexible robotic arm to run in the simulation to collect videos containing foreign objects, create a dataset from the video images and divide it into training set, test set and validation set; S2: Extract the reflection and illumination images from images captured by the camera in a flexible robotic arm in confined dark environments and in confined dark environments with uneven lighting. When extracting the illumination and reflection images, utilize the improved center-around function. This involves introducing a Gaussian distribution scaling parameter l and a squared decay term parameter α, and using an improved dynamic SSIM loss function in the fusion network. A new function is constructed to iterate over the values ​​of two parameters to extract the reflectance and illumination images. The specific steps are as follows: S21: The original image I(x,y) is transformed using the logarithm method. log(I(x,y))=log(L(x,y))+log(R(x,y)) Separate the illuminance component and the reflectance component; S22: Convolve the image I(x,y) using a Gaussian template to approximate the incident image L(x,y), where L(x,y) = F(x,y)*I(x,y), and F(x,y) is the improved center wrap function. Where λ is the normalization constant, c is the Gaussian surround scale; F(x,y) is the calculation of the illumination image in space, which means estimating the illumination in the image by calculating the weighted average of the pixels in the image and the surrounding area; Where l controls the scaling of the Gaussian distribution and α controls the decay rate, the values ​​of l and α are iterated using the loss function, where η is the learning rate; S23: Use R(x,y)=exp(log(I(x,y))-log(L(x,y))) to subtract the illumination image from the original image to obtain the reflectance image; S3: The extracted illumination image, reflectance image, and original image are subjected to three-branch feature extraction. The extraction stream is performed by a regular convolutional layer and an AGRDB feature extraction module, respectively. The extracted features are then enhanced by the CBAM attention mechanism. The enhanced three-branch feature maps are directly added together to obtain a comprehensive feature map. Finally, the comprehensive feature map is reconstructed to obtain the enhanced image. An improved dynamic SSIM loss function is introduced to adjust the reflectance image and illumination image of the image decomposition. S4: The enhanced image obtained by fusing the reflection image and illumination image extracted from the original image with the original image is input into a lightweight detection network for overall network training. The training model weights are obtained by training the network, and the training parameters are adjusted to obtain the optimal detection model, thus completing the method for foreign object recognition by a flexible robotic arm in a narrow and dark environment.

2. The method for foreign object identification in confined, dark environments according to claim 1, characterized in that, In step S2, an image loss function is introduced to adjust the center wrap function parameters. This involves using l and α as parameters to be optimized, and dynamically adjusting their values ​​by minimizing the loss function. A weighted SSIM loss is introduced as the optimization objective, and the total loss function is defined as follows: in weight matrix λ1 and λ2 are the loss weights, where the improved dynamic SSIM local (x,y): also For image reconstruction loss, Image smoothing loss.

3. The method for foreign object identification in confined, dark environments according to claim 1, characterized in that, Step S3 uses a three-branch feature extraction flow to extract features and perform fusion reconstruction. The specific steps are as follows: S31: The reflection image and illumination image extracted from the narrow, low-light image acquired by the flexible robotic arm camera are input into the feature extraction stream in three parallel branches along with the original image. This stream includes a common convolutional layer and three adaptive gradient residual dense blocks (AGRDB). S32: After the features are extracted through the three-branch feature extraction, they are input into the CBAM attention mechanism for feature enhancement. The enhanced feature maps are then added together and fused into a comprehensive feature map. S33: The fused comprehensive feature map is passed through a reconstruction module consisting of four 3×3 convolutional layers with Leaky ReLU activation functions and one 1×1 convolutional layer with Tanh activation function to reconstruct the image, resulting in the fused image. The SSIM structural similarity between the fused image and the original image is calculated. The center surround function parameters l and α are adjusted using the improved dynamic structural similarity loss function to obtain the illumination map and reflectance map with the best fusion effect.

4. The method for foreign object identification in confined, dark environments according to claim 1, characterized in that, The steps for detecting the network in step S4 are as follows: S41: The fused and enhanced image is input into the backbone feature extraction network, where the backbone network uses a lighter dual-kernel convolution and C2f module as convolutional layers for feature extraction while improving feature representation capability; S42: Further process and fuse the features extracted from different stages of the backbone network. Combining the characteristics of dual-kernel convolution, a multi-scale attention mechanism (EMA) is introduced to enhance the expression of multi-scale features and improve the ability to recognize targets at different scales. S43: Input the processed features into the classification and detection branches, use the decoupled classification and regression branches to classify and locate the target, finally calculate the position offset of the bounding box through the regression head, and output the class probability of each candidate box through the classification branch, thereby completing the detection task.

5. The method for foreign object identification in confined, dark environments according to claim 3, characterized in that, The Adaptive Gradient Residual Dense Block (AGRDB) contains two 3×3 convolutional layers and one 1×1 ordinary convolutional layer, and adopts a dense connection method to fully integrate the features extracted by each convolutional layer. The residual flow calculates the gradient magnitude of the features through adaptive gradient and performs weighted adjustment of the features of each layer based on the gradient information learned adaptively. In order to solve the problem of inconsistent channel dimensions, the network uses a 1×1 convolutional layer for dimension matching at the end.

Images