A micro-mark target detection method and system for industrial environment

Abstract

The present application provides a kind of micro mark target detection method and system for industrial environment, the method is: using preliminary feature extraction to the image to be detected by preprocessing network, obtain preliminary feature map, preliminary feature map is divided into regions, and the position information coding of each region is carried out;The coded preliminary feature map is extracted in feature extraction branch network Local detailed feature information, extract global information in global expression branch network, and the detailed feature information and global information of local are interacted and information purification fusion, and the purified fusion information is used as the feature layer to be predicted;The feature layer to be predicted is predicted, and the prediction result is excluded by confidence screening Unreliable result, then through non-maximum suppression to obtain the position information of micro mark.The micro mark target detection method for industrial environment has high small target detection capability, can accurately locate the position information of micro mark, and improves the efficiency of micro mark detection.

Description

Technical Field

[0001] This invention relates to the field of target detection, and more specifically to a method and system for detecting tiny marker targets in industrial environments. Background Technology

[0002] Object detection aims to identify all objects of interest in an image and determine their location and category using detection algorithms. In detection tasks, object features serve as a crucial basis for the detection results.

[0003] Traditional object detection methods typically acquire deep feature maps of image patches for detection and then use subsequent filtering to obtain correct classification and location information. However, in industrial environments, small markers have limited available features and require high localization accuracy, making it difficult for traditional object detection methods to meet the practical needs of such scenarios. Summary of the Invention

[0004] In order to overcome the shortcomings of the existing technology, the purpose of this invention is to provide a method and system for detecting tiny marker targets in industrial environments.

[0005] To achieve the above-mentioned objectives of this invention, this invention provides a method for detecting tiny marker targets in industrial environments, comprising the following steps: Acquire the image to be detected; A parallel multi-feature information extraction, fusion and purification network is constructed, which includes a preprocessing network, a feature extraction branch network and a global expression branch network. A preprocessing network is used to perform preliminary feature extraction on the image to be detected, resulting in a preliminary feature map. The preliminary feature map is then divided into regions, and the location information of each region is encoded. The encoded preliminary feature map is used to extract local detailed feature information in the feature extraction branch network and global information in the global expression branch network. The local detailed feature information and global information are then interacted and purified and fused. The purified fused information is used as the feature layer to be predicted. The prediction is performed on the feature layer to be predicted. Unreliable results are filtered out by confidence level. Then, the location information of the tiny markers is obtained by non-maximum suppression.

[0006] This method for detecting minute markers in industrial environments has high small target detection capability and can accurately locate the position information of minute markers, thus improving the efficiency of minute marker detection.

[0007] The preferred scheme of this method for detecting tiny markers in industrial environments includes the following specific steps for obtaining the feature layer to be predicted: a. Extract detailed local feature information from the complete encoded preliminary feature map in the feature extraction branch network and retain detailed small target features within the local area to obtain a local feature information map; b. Divide the complete encoded preliminary feature map into a series of patches and feed them into the global expression branch network to obtain global feature information, thus obtaining a global feature information map; c. Perform global average pooling on the patches containing global feature information. After processing, reassemble the patches into a complete feature map based on their position encoding information. Interact with the local detailed feature information and the global feature information so that the local detailed feature information can contain the global feature information, and the global feature information can also contain the local detailed feature information. d. Based on the feature information after interaction, introduce preliminary feature maps to construct context branches and enrich the fusion information; e. Perform channel filtering and spatial filtering purification on the fused information respectively, and then fuse the information after channel filtering and spatial filtering purification to obtain the feature layer to be predicted. f. First, divide the global feature information map output by the global representation branch network into a series of patches, and then repeat steps a, b, c, d, and e to obtain a new feature layer to be predicted.

[0008] This preferred scheme extracts detailed local and global feature information, retains the initial feature information to construct contextual information branches, and then performs interactive processing on the detailed local and global feature information to fuse the two types of information. It also introduces complete initial information for contextual information interaction to obtain the fused information most conducive to small target detection. Then, it purifies the fused information to eliminate redundant and conflicting information brought about by the fusion of features at different scales, thereby improving the feature representation ability. The purified fused information is used as the input for the next information extraction and one of the bases for target detection. The process of information extraction, fusion and purification is repeated to obtain deeper fused and purified information for detection.

[0009] Step e specifically involves: In channel filtering, the fused information is processed by both global pooling and average pooling, and the global features of the image are obtained after convolution and addition. In channel filtering, the relative weights of each position in the feature map with respect to the channel are normalized using Softmax. Finally, the channel filtering and the filtering information obtained from the channel filtering are integrated to eliminate redundant and conflicting information, resulting in the feature layer to be predicted.

[0010] The preferred embodiment of this method for detecting minute markers in industrial environments involves obtaining the final location information by linking the acquired location information: The link detection is performed using the peak signal-to-noise ratio of PSNR and the SSIM structural similarity of the edge region and the surrounding connected region of the location information as indicators.

[0011] Where m is the maximum pixel value of the image, mse is the mean square error of the edge region and its connected regions, l(x,y) is the brightness similarity information, c(x,y) is the contrast similarity information, s(x,y) is the structural similarity information, β1, β2, β3, β4 are the weight hyperparameters, and L ink The standard value for whether a connection is established is used to determine whether the surrounding connection area belongs to that tiny marker. When L... ink When the value is greater than the set threshold, it indicates that the surrounding connected area belongs to this tiny flag. ink If the value is less than the set threshold, it indicates that the surrounding connected area does not belong to this tiny marker.

[0012] This preferred approach improves the accuracy of location information by connecting the predicted region to its surrounding connected regions through detection.

[0013] The preferred scheme of this method for detecting small marker targets in industrial environments is as follows: before detection begins, the parallel multi-feature information extraction, fusion and purification network is trained. The training steps are as follows: first, the feature extraction branch network and the global expression branch network are pre-trained separately, and then the parallel multi-feature information extraction, fusion and purification network is trained as a whole.

[0014] When training the feature extraction branch network, a reward model is added to guide the network to approximate the correct feature location, obtain its encoding information, and then use the encoding information of the region to obtain the coordinates. The detection network trained by the feature extraction branch network then performs regression to obtain accurate localization and classification.

[0015] When training the global representation branch network, a decoder module is introduced. The decoder network is trained using target query and Hungarian matching. Several blank query boxes are input into the decoder module. These query boxes match the features at that location based on the feature information extracted by the encoder module in the global representation branch network and obtain a classification. Then, the network is trained using the Hungarian matching algorithm, making the loss function L... t Minimum.

[0016] When training the parallel multi-feature information extraction, fusion and purification network as a whole, regression prediction is performed based on the two purified feature layers to be predicted, and the prediction box loss, confidence loss and classification loss are obtained.

[0017] In this preferred scheme, the network undergoes pre-training, which introduces a reward module and a Decoder module for enhanced training. Based on the pre-training results, the entire network is trained, and the network's detection performance is improved by using the loss of latent spatial distribution, prediction confidence, and object classification.

[0018] This application also proposes a micro-marker target detection system, including a processor and a memory, which are communicatively connected. The memory stores at least one executable instruction that causes the processor to perform operations corresponding to the aforementioned micro-marker target detection method for industrial environments. This micro-marker target detection system possesses all the advantages of the aforementioned micro-marker target detection method for industrial environments.

[0019] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0020] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a flowchart illustrating a method for detecting tiny markers in industrial environments. Figure 2 Block diagram of the parallel multi-feature information extraction, fusion and purification network principle; Figure 3 This is a block diagram illustrating the principle of the feature extraction module in a CNN (Convolutional Neural Network). Figure 4 This is a block diagram illustrating the principle of the key multi-head self-attention module in the Transformer branch network; Figure 5 The detection result image obtained based on the feature map. Detailed Implementation

[0021] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0022] In the description of this invention, unless otherwise specified and limited, it should be noted that the terms "installation", "connection" and "linking" should be interpreted broadly. For example, they can refer to mechanical or electrical connections, or internal connections between two components. They can be direct connections or indirect connections through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0023] like Figures 1 to 5 As shown, this invention provides an embodiment of a method for detecting tiny marker targets in industrial environments, comprising the following steps: Obtain the image to be detected.

[0024] In this embodiment, an industrial camera is used to capture actual images of an industrial scene to obtain the required raw image. The image is then compressed to a specified size and used as the network input image, i.e., the image to be detected.

[0025] A parallel multi-feature information extraction, fusion, and purification network is constructed, comprising a preprocessing network, a feature extraction branch network, and a global representation branch network. In this embodiment, the feature extraction branch network is preferably, but not limited to, a CNN convolutional neural network, and the global representation branch network is preferably, but not limited to, a Transformer branch network.

[0026] A preprocessing network is used to perform preliminary feature extraction on the image to be detected, obtaining a preliminary feature map, which reduces the computational load of subsequent networks. The preliminary feature map is then divided into regions, and the location information of each region is encoded. In this embodiment, the preprocessing network is preferably, but not limited to, a deformable convolutional network.

[0027] The encoded preliminary feature map is used to extract local detailed feature information in the feature extraction branch network and global information in the global representation branch network. The local detailed feature information and global information are then interacted and purified and fused. The purified fused information is used as the feature layer to be predicted.

[0028] In this embodiment, the specific steps for obtaining the feature layer to be predicted are as follows: a. Extract detailed local features from the initial encoded feature map using the feature extraction branch network, retaining detailed small target features within the local region, to obtain a local feature information map. Specifically: In a CNN (Convolutional Neural Network), the encoded preliminary feature map is first subjected to simple convolution, and then fed into the feature extraction module M-conact to obtain detailed small target features within local regions and retain easily overlooked small target features. The M-conact module is a stacked module based on the CBL (Convolutional Black-Blank Layer) module, which consists of convolution, normalization, and activation functions. Figure 3 As shown, this enhances feature extraction capabilities. The M-conact module takes feature map F1 as input, and after CBL processing, F1 yields a new feature map F2. This process is repeated to obtain feature maps F3, F4, F5, etc. Feature maps F1, F2, F3, F4, and F5 are stacked to deepen the convolutional neural network and extract more specific features while preserving initial feature information, preventing small target features from disappearing as the network deepens. The stacked result is then processed again with CBL to obtain the local feature information map to be interacted with. Furthermore, the M-conact module effectively maintains the network depth through its skip connections, enabling the CNN convolutional neural network to extract more effective and accurate local feature information.

[0029] Meanwhile, the Transformer branch segments the initial feature map based on the positional encoding information, and sends the segmented patches with positional information into the Encoder network for self-attention interaction, so that each patch can contain information from other patches, thereby obtaining a global information representation.

[0030] b. The completed, pre-encoded feature map is divided into a series of patches and fed into the global representation branch network to obtain global feature information, resulting in a global feature information map. Specifically: In the Transformer branch network, the encoded feature layer is divided into multiple small patches (P1, P2, P3, ..., Pn) based on the encoded information. These patches are then fed into the Encoder module for self-attention operations, allowing each patch to learn from the others and contain information about the relationships between all patches, thus obtaining global information about the image. The Encoder module is a crucial component of the Transformer branch. It performs standardization and pooling on the input patches, followed by multi-head self-attention operations between patches to ensure each patch has extensive relational information. Then, pooling and standardization branches are stacked based on residual-like operations on the original input to obtain patches with global information. After standardization and MLP processing, the patches are transformed to the input size. From this point on, the input and output sizes are the same, allowing the operation to be repeated multiple times to obtain more accurate and complete global information.

[0031] c. Perform global average pooling on patches containing global feature information. After processing, reconstruct the patches into complete feature maps based on their positional encoding information. Interact between local detailed feature information and global feature information, ensuring that local detailed feature information contains global feature information, and vice versa. Because the feature maps obtained by the CNN convolutional neural network and the Transformer branch network differ, downsample the local feature map of the CNN convolutional neural network and upsample the global feature map of the Transformer branch network before interaction.

[0032] d. Based on the interactive feature information, a preliminary feature map is introduced to construct a context branch, enriching the fusion information. Here, the initial feature map is the preceding text, and the interactive feature information is the following text. The specific operation of constructing the context branch is to compress the initial feature map to the size of the current feature map, then superimpose it with the interactive feature information, and finally use a standard convolution to modify the number of channels after superposition for easier subsequent processing.

[0033] e. Perform channel filtering and spatial filtering purification on the fused information separately, and then fuse the purified information to obtain the feature layer to be predicted. In this embodiment, the Clean-up purification module is used for information purification.

[0034] Specifically, in channel filtering purification, global pooling and average pooling are used to process the fused information separately, and the results are obtained by convolutional summation to obtain more refined global image features. In spatial filtering purification, softmax normalization is used to normalize the relative weights of each position in the fused information with respect to the channel. Finally, the filtering information obtained from channel filtering purification and spatial filtering purification is fused to eliminate redundant and conflicting information, improve the accuracy of feature representation, and obtain a feature layer more conducive to small object detection as the feature layer to be predicted.

[0035] Next, in order to obtain more specific feature information, the multi-branch operation is repeated to obtain a new feature layer to be predicted.

[0036] Specifically, step f involves first re-dividing the global feature information map output by the global representation branch network (i.e., the Transformer branch network) into a series of patches. Then, steps a, b, c, d, and e are repeated to obtain new feature layers to be predicted. After multiple repetitions, multiple feature layers to be predicted are obtained.

[0037] The multiple feature layers to be predicted are predicted, and the prediction results are filtered out by confidence level to eliminate unreliable results. Then, the location information of the tiny markers is obtained by non-maximum suppression.

[0038] Specifically, it includes the following steps: 1. Obtaining Predicted Boxes and Confidence Scores. Because CNN (Convolutional Neural Network) and Transformer (Branch Network) operate differently in feature extraction, their focus on specific features differs, and this difference persists even after feature interaction. To ensure the accuracy of object detection, predictions are performed on multiple feature layers after the interaction. At this point, preliminary prediction results for several small markers have been obtained.

[0039] 2. Confidence screening. To obtain accurate prediction results, it is first necessary to remove some low-confidence results.

[0040] 3. Non-maximum suppression. Non-maximum suppression selects the highest-scoring bounding boxes within a given region that belong to the same category. Its main operation involves sorting the selected bounding boxes by confidence level, and then performing an IoU operation on the other bounding boxes (B) with the highest-scoring bounding box (A).

[0041] ,in, This indicates the size of the overlapping area between the two predicted bounding boxes. This represents the sum of the two predicted bounding boxes. The calculated IoU value is compared to a set threshold; if it exceeds the threshold, predicted box B is removed. This process is repeated to filter all predicted boxes, yielding the final category detection results.

[0042] Since the area surrounding the predicted location always contains some useful information that is often overlooked, this information is crucial for improving detection accuracy. Therefore, this embodiment also has a preferred solution, namely... The location information obtained through nonmaximum suppression is used to obtain the final location information of the tiny marker through link detection. During detection, the surrounding area is detected based on the location information of the target.

[0043] Link detection is performed using the peak signal-to-noise ratio of PSNR and the structural similarity of SSIM in the edge region and the surrounding connected region of the location information as indicators.

[0044] Where m is the maximum pixel value of the image, mse is the mean square error of the edge region and its connected regions, l(x,y) is the brightness similarity information, c(x,y) is the contrast similarity information, s(x,y) is the structural similarity information. l(x,y), c(x,y), and s(x,y) all need to be calculated based on the pixel values ​​and individual pixels of the image. Existing methods exist for calculating these values, which will not be detailed here. β1, β2, β3, and β4 are weight hyperparameters. ink The standard value for whether a connection is established is used to determine whether the surrounding connection area belongs to this tiny marker. When L...ink When the value is greater than the set threshold, it indicates that the surrounding connected area belongs to this tiny flag. ink If the value is less than the set threshold, it indicates that the surrounding connected area does not belong to this tiny marker.

[0045] The embodiments in this application train the parallel multi-feature information extraction, fusion and purification network before detection begins. The training steps are as follows: first, pre-train the feature extraction branch network and the global expression branch network respectively, and then train the parallel multi-feature information extraction, fusion and purification network as a whole.

[0046] When training the feature extraction branch network, a reward model is added to guide the network to approximate the correct feature location, obtain its encoding information, and then use the encoding information of the region to obtain the coordinates. The detection network trained by the feature extraction branch network then performs regression to obtain accurate localization and classification.

[0047] The reward model determines the location of the target object by acquiring feature information through network convolution, and scores the acquired results based on prior information, inducing the network to continuously strengthen its ability to extract effective information. Its approximation process involves solving the loss function. Minimal process, Where P represents the actual location information. To predict location information, For the extracted feature information, Here is the bootstrap function for the reward module, where x, y, w, and h are the coordinates of the top-left corner and the width and height of the prediction box. The coordinates of the top left corner and the width and height of the actual bounding box are given, and n is the number of predicted bounding boxes in the image. The number of predicted bounding boxes is equal to the number of tiny markers detected. Based on the location information approximated by the reward model and the initial feature information, the network is further trained. This stage mainly involves solving the loss function L. C The smallest process. L C It contains all the parameters required for object detection, namely the bounding box regression loss, confidence loss, and class loss.

[0048] ; in These are the intersection-union loss, predicted bounding box regression loss, confidence loss, and category loss, respectively. b represents the ground truth bounding box and the predicted bounding box, d() represents the minimum Euclidean distance, o represents the prediction confidence, and sig represents the normalization process. Weight hyperparameters set manually. Types representing tiny marks, This represents the true category of the detected entity.

[0049] When training the global representation branch network, a decoder module is introduced. The decoder network is trained using target query and Hungarian matching. Several blank query boxes are input into the decoder module. Based on the feature information extracted by the encoder, the query boxes are matched with the features at that position and classified. Then, the network is trained using the Hungarian matching algorithm, making the loss function L... t Minimum.

[0050]

[0051] in This consists of a set of real bounding boxes and their classification information. For the set of matched query boxes and the predicted category, To determine the cost of optimal pairwise matching between the predicted and ground truth boxes using the Hungarian algorithm, the correlation between the predicted and ground truth boxes and category prediction are considered. The probability of a certain class. argmin is the loss between the predicted bounding box and the ground truth bounding box, and it is the minimum function.

[0052] When training the parallel multi-feature information extraction, fusion and purification network as a whole, regression prediction is performed based on the two purified feature layers to be predicted, and the prediction box loss, confidence loss and classification loss are obtained.

[0053]

[0054] in Let the loss function be the first cleansing feature layer. The loss function for the second cleansing feature layer is... Let be the total loss function of the entire network. Weight hyperparameters are set manually.

[0055] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0056] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for detecting tiny marker targets in industrial environments, characterized in that, Includes the following steps: Acquire the image to be detected; A parallel multi-feature information extraction, fusion and purification network is constructed, which includes a preprocessing network, a feature extraction branch network and a global expression branch network. A preprocessing network is used to perform preliminary feature extraction on the image to be detected, resulting in a preliminary feature map. The preliminary feature map is then divided into regions, and the location information of each region is encoded. The encoded preliminary feature map is used to extract local detailed feature information in the feature extraction branch network and global information in the global expression branch network. The local detailed feature information and global information are then interacted and purified and fused. The purified fused information is used as the feature layer to be predicted. The specific steps are as follows: a. Extract detailed local feature information from the complete encoded preliminary feature map in the feature extraction branch network and retain detailed small target features within the local area to obtain a local feature information map; b. Divide the complete encoded preliminary feature map into a series of patches and feed them into the global expression branch network to obtain global feature information, thus obtaining a global feature information map; c. Perform global average pooling on the patches containing global feature information. After processing, reassemble the patches into a complete feature map based on their position encoding information. Interact with the local detailed feature information and the global feature information so that the local detailed feature information can contain the global feature information, and the global feature information can also contain the local detailed feature information. d. Based on the feature information after interaction, introduce preliminary feature maps to construct context branches and enrich the fusion information; e. Perform channel filtering and spatial filtering purification on the fused information respectively, and then fuse the information after channel filtering and spatial filtering purification to obtain the feature layer to be predicted. f. First, divide the global feature information map output by the global representation branch network into a series of patches, and then repeat steps a, b, c, d, and e to obtain a new feature layer to be predicted. The prediction is performed on the feature layer to be predicted. Unreliable results are filtered out by confidence level. Then, the location information of the tiny markers is obtained by non-maximum suppression.

2. The method for detecting minute marker targets in industrial environments according to claim 1, characterized in that, Step e specifically involves: In channel filtering, the fused information is processed by both global pooling and average pooling, and the global features of the image are obtained after convolution and addition. In channel filtering, the relative weights of each position in the feature map with respect to the channel are normalized using Softmax. Finally, the channel filtering and the filtering information obtained from the channel filtering are integrated to eliminate redundant and conflicting information, resulting in the feature layer to be predicted.

3. The method for detecting minute marker targets in industrial environments according to claim 1, characterized in that, The acquired location information is used to obtain the final location information through link detection: The link detection is performed using the peak signal-to-noise ratio of PSNR and the SSIM structural similarity of the edge region and the surrounding connected region of the location information as indicators. Where m is the maximum pixel value of the image, mse is the mean square error of the edge region and its connected regions, l(x,y) is the brightness similarity information, c(x,y) is the contrast similarity information, s(x,y) is the structural similarity information, β1, β2, β3, β4 are the weight hyperparameters, and L ink The standard value for whether a connection is established is used to determine whether the surrounding connection area belongs to that tiny marker. When L... ink When the value is greater than the set threshold, it indicates that the surrounding connected area belongs to this tiny flag. ink If the value is less than the set threshold, it indicates that the surrounding connected area does not belong to this tiny marker.

4. The method for detecting minute marker targets in industrial environments according to claim 1, characterized in that, Before detection begins, the parallel multi-feature information extraction, fusion and purification network is trained. The training steps are as follows: first, the feature extraction branch network and the global representation branch network are pre-trained separately, and then the parallel multi-feature information extraction, fusion and purification network is trained as a whole.

5. The method for detecting minute marker targets in industrial environments according to claim 4, characterized in that, When training the feature extraction branch network, a reward model is added to guide the network to approximate the correct feature location, obtain its encoding information, and then use the encoding information of the region to obtain the coordinates. The detection network trained by the feature extraction branch network then performs regression to obtain accurate localization and classification.

6. The method for detecting minute marker targets in industrial environments according to claim 4, characterized in that, When training the global representation branch network, a decoder module is introduced. The decoder network is trained using target query and Hungarian matching. Several blank query boxes are input into the decoder module. These query boxes match the features at that location based on the feature information extracted by the encoder module in the global representation branch network and obtain a classification. Then, the network is trained using the Hungarian matching algorithm, making the loss function L... t Minimum.

7. The method for detecting minute marker targets in industrial environments according to claim 5, characterized in that, When training the parallel multi-feature information extraction, fusion and purification network as a whole, regression prediction is performed based on the two purified feature layers to be predicted, and the prediction box loss, confidence loss and classification loss are obtained.

8. A system for detecting minute marker targets, characterized in that, The device includes a processor and a memory, which are communicatively connected. The memory is used to store at least one executable instruction that causes the processor to perform an operation corresponding to the method for detecting tiny marker targets in an industrial environment as described in any one of claims 1-7.

Images