A small target real-time identification method based on image
By fusing multi-level features through deep convolutional neural networks, the problem of low efficiency in traditional manual inspection has been solved, achieving high-precision and rapid defect detection and improving the product quality of industrial production.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-27
- Publication Date
- 2026-03-24
AI Technical Summary
Traditional manual visual inspection methods are inefficient, have a high omission rate, and are difficult to effectively identify small defects and foreign objects in industrial production, thus affecting product quality.
A real-time small target recognition method based on deep convolutional neural networks is adopted. By extracting features from the backbone network, using a multi-scale merging module and a multi-dimensional attention module, multi-level features are fused to improve the accuracy and speed of defect detection.
It achieves high-precision and rapid defect detection, improves product yield, and is suitable for batch processing in industrial environments.
Smart Images
Figure CN116310595B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of detection, and relates to a small target real-time identification method based on images. BACKGROUND
[0002] Controlling product quality in an industrial environment is a crucial link on the production line, but when cotton, tobacco or fruit crops are batched and picked, mixed with small foreign matter, it will affect the taste quality of the product; during fabric embryo production, due to limited technical level and improper operation, fabric texture has defect points that will affect the quality of the next production link. Overcoming small target defect detection in industrial production can further improve product quality.
[0003] Traditional industrial small target detection controls product quality by manual naked eye, picks out small defects and foreign matter, but this method is low in efficiency and high in omission rate, which is not conducive to long-term batch processing on the industrial production line.
[0004] The product quality control method based on machine vision is widely used in agricultural pest control, industrial production and other manufacturing fields, and can detect small target defects of products and improve the yield rate. The powerful deep learning method has strong nonlinear fitting properties and can integrate the recognition and detection capabilities of different categories, different shapes and different scales into the network model, and has good generalization. Therefore, designing a product defect small target foreign matter identification system with high accuracy and fast detection speed has high industrial value. SUMMARY
[0005] Therefore, the purpose of the present application is to provide a small target real-time identification method based on images.
[0006] To achieve the above purpose, the present application provides the following technical scheme:
[0007] A small target real-time identification method based on images, comprising the following steps:
[0008] S1: Collect product defect data photos, make data sets required by the network model, and divide the training set and the test set;
[0009] S2: Input the image data into the deep convolutional neural network, extract the features through the backbone network, and obtain five-level features;
[0010] S3: Use the multi-scale merging module to fuse the five-level features into three-level features of high, medium and low categories;
[0011] S4: The three-level features with different characteristics are input into the multi-dimensional attention module to capture the channel and spatial dimension feature relationship, and the three-level features are fused into one-level features in the head network to enhance and calibrate the input information;
[0012] S5: The head network performs convolution calculation on the enhanced features and the remaining network layer features, the incoming probe head obtains the bounding box and object category information, and classification is performed according to the defect detection result.
[0013] Optionally, the S1 specifically includes the following steps:
[0014] S11: Selecting a defect image including stains, hooks, broken lines, holes and color errors as a data set sample;
[0015] S12: Using image translation, rotation, mirroring, cropping, scaling, image blurring and image splicing to expand the data set sample;
[0016] S12: Using the image labeling tool LabelImg to label the foreign object position coordinates and classification in the image, and converting the obtained data into the format of the PASCALVOC2012 data set, and dividing the data set into a training set and a test set in a ratio of 7:3.
[0017] Optionally, the S2 specifically includes the following steps:
[0018] S21: The input image is subjected to feature extraction by the backbone network and subjected to convolution calculation by multiple modules; including three convolution layers in the first stage, one convolution layer and a multi-connection module in the second stage, transition modules and multi-connection modules in the third, fourth and fifth stages, and an SPPCSPC module with a CSP structure;
[0019] S22: The second and third convolution layers in the first stage, the output feature maps of the second, third and fifth stages construct a five-level feature pyramid;
[0020] S23: The feature maps output by the fourth stage and the SPPCSPC module are directly transmitted to the head network.
[0021] Optionally, the S3 specifically includes the following steps:
[0022] S31: The five-level feature pyramid is transmitted to a multi-scale merging module, adjacent features are fused to form multi-scale merging nodes, and the nodes are distributed in a V shape;
[0023] S32: C l+1 Up-sampling is performed by using a bilinear interpolation method, C l-1 Channel and spatial information interaction is performed by using a Fcous down-sampling method, C l+1 and C l-1 are reshaped to have the same size as C l , three adjacent features C l+1 , C l , C l-1Top-down and bottom-up fusion are respectively performed, wherein l represents a network layer number, 2≤l≤4;
[0024] S33: After feature fusion at the multi-scale merging node, transmission to the bottleneck layer, and aggregation of features by the plurality of cross-scale fusion nodes into low, medium and high three types of feature mapping.
[0025] Optionally, the S4 specifically includes the following steps:
[0026] S41: The high-class feature H is used to strengthen the information of the channel dimension, and is connected in series with the input data B so that the number of channels is the same, and global average pooling and global standard deviation pooling operations are used to collect global signal channel features;
[0027] S42: A channel weight vector m is generated by a fully connected layer calculation, and the parameter after a sigmoid operation is represented as:
[0028] m=S i (A c )
[0029] wherein A c is a feature connected with the high-class feature and the input data, {A c ,H,input}∈R C×H×W ; S represents an activation function, and a channel attention feature map at the i-th position is calculated and represented as:
[0030]
[0031] wherein z i ∈R C has different representations at different pooling operations, and represents information of the i-th channel;
[0032] The weight parameter is multiplied by the input data, A c The number of channels is consistent with the input data, and the final output CR is obtained:
[0033] CR=mB;
[0034] S43: The medium-class feature M has rich semantic information, the CR feature is spliced with the medium-class feature, and maximum pooling and average pooling operations are used to obtain spatial information features;
[0035] S44: A spatial weight vector w is obtained by a sigmoid activation function, a feature is weighted by using a training parameter β, and an element pair of the weighted medium-class feature and the weighted channel feature is summed, and is represented as:
[0036] E=w(β·M+(1-β)·CR)
[0037] S45: Convolution operation is performed on the low-class feature L with accurate structure information to generate a vector with a channel number of 1, wherein the height and width are unchanged;
[0038] S46: The bias feature of the level is obtained through a sigmoid activation function, and is added to the previous branch feature E to obtain the final first-level feature:
[0039] F = E + S (Conv (L))
[0040] S47: The features are aggregated into the first-level feature through the multi-dimensional attention module of the head network, and the features of the fourth stage network layer and the features of the SPPCSPC module are also transmitted to the head network as three-layer enhanced features.
[0041] Optionally, the S5 specifically comprises the following steps:
[0042] S51: The three-scale feature layers are subjected to multiple convolution operations in the head network, and are transmitted to the RepConv module, wherein the RepConv module comprises a plurality of residual structures;
[0043] S52: The feature information is transmitted to the detection head, and there are three prior boundary boxes and detection results for each feature point;
[0044] S53: Classification is performed according to the feature mapping and the results are output, and the types of defects detected and recognized include stains, hooks, broken lines, holes and color errors.
[0045] The beneficial effects of the present application are:
[0046] Other advantages, objects and features of the present application will be set forth in part in the following specification, and in part will become apparent to those skilled in the art from the examination of the following specification, or can be learned from practice of the present application. The objects and other advantages of the present application can be realized and attained by the below description. BRIEF DESCRIPTION OF DRAWINGS
[0047] In order to make the objects, technical solutions and advantages of the present application clearer, the preferred detailed description of the present application will be combined with the drawings, wherein:
[0048] Figure 1 It is a flowchart;
[0049] Figure 2 It is the basic algorithm structure of the high-precision small target real-time identification model;
[0050] Figure 3 It is the basic structure of SPPCSPC;
[0051] Figure 4is a Multi_Concat basic structure;
[0052] Figure 5 is a Transition basic structure;
[0053] Figure 6 is a multi-scale merging module basic structure;
[0054] Figure 7 is a multi-dimensional attention module basic structure. DETAILED DESCRIPTION
[0055] The present application will be described in greater detail by way of specific embodiments, which should not be construed as limiting the present application to those specific embodiments. Other advantages and benefits of the present application will be apparent to those skilled in the art upon reading the description of the embodiments. The present application can be implemented or applied in other different embodiments, and the details in the description can be modified or changed based on different views and applications without departing from the spirit of the present application. It should be noted that the drawings provided in the following embodiments only schematically illustrate the basic concept of the present application, and the features in the following embodiments and examples can be combined with each other without conflict.
[0056] The drawings are only used for exemplary illustration, and the representation is only a schematic diagram, not a physical diagram, and should not be understood as a limitation of the present application. In order to better illustrate the embodiments of the present application, some components in the drawings may be omitted, enlarged or reduced, and do not represent the actual size of the product. It is understandable to those skilled in the art that some well-known structures and their descriptions in the drawings may be omitted.
[0057] The same or similar reference numerals in the drawings of the embodiments of the present application correspond to the same or similar components; in the description of the present application, it should be understood that if the terms "upper", "lower", "left", "right", "front", "back" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, therefore the terms describing the positional relationship in the drawings are only used for exemplary illustration, and should not be understood as a limitation of the present application. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.
[0058] The present application takes YoloV7, which has better speed and accuracy performance in the YOLO series, as the basic network, in order to facilitate the rapid identification of the production line in the industrial environment. The overall architecture is as follows Figure 1As shown, the method includes: collecting defect image samples from the product production line to create a dataset for training; building a defect detection model and setting corresponding parameters; training the detection model based on the dataset; inputting the image to be detected into the training model for detection analysis, and performing statistical analysis and post-processing on the detection results; the target detection model includes a backbone network, a multi-scale merging module, and a multi-dimensional attention module; this invention utilizes the multi-scale merging module and the multi-dimensional attention module to effectively aggregate multi-level feature mappings and achieve effective semantic information enhancement and supplementation based on the structural characteristics of features at different levels, thereby improving the accuracy and efficiency of small target defect recognition.
[0059] The specific steps are as follows:
[0060] S1: Collect photos of product defects, create the dataset needed for the network model, and divide it into training and test sets.
[0061] S1 specifically includes:
[0062] S11: Select images containing defects such as stains, snags, broken threads, holes, and miscolors as dataset samples;
[0063] S12: The dataset samples are augmented using methods such as image translation, rotation, mirroring, cropping, scaling, image blurring, and image stitching;
[0064] S13: Use the image annotation tool LabelImg to annotate the location coordinates and classification of foreign objects in the image, and convert the obtained data into the format of the PASCALVOC2012 dataset. Divide the dataset into training and test sets in a 7:3 ratio.
[0065] S2: Input the image data into a deep convolutional neural network, extract features through the backbone network, and obtain five levels of features.
[0066] S2 specifically includes:
[0067] S21: The input image (640*640) undergoes feature extraction via the backbone network, followed by convolutional operations across multiple modules, such as... Figure 2 As shown. It includes three convolutional layers in the first stage, one convolutional layer and multiple connection modules in the second stage, transition modules and multiple connection modules in the third, fourth, and fifth stages, and an SPC / SPC module with a CSP structure, as shown. Figure 3 As shown;
[0068] S22: The second and third convolutional layers in the first stage, and the feature maps output from the second, third and fifth stages, construct a five-level feature pyramid;
[0069] S23: The feature maps output by the fourth stage and the SPPCSPC module are directly transmitted to the head network.
[0070] S3: The multi-scale merging module is used to fuse the five-level features into three-level features: high-class, medium-class, and low-class.
[0071] S3 specifically includes:
[0072] S31: The five-level feature pyramid is transmitted to the multi-scale merging module, and adjacent features are fused to form multi-scale merging nodes, which are distributed in a V-shape.
[0073] S32:C l+1 Upsampling is performed using bilinear interpolation, C l-1 Using the Fcous downsampling method for channel and spatial information exchange, C l+1 and C l-1 Reshaped into C l For the same scale, three adjacent features C l+1 C l C l-1 The fusion is performed from top to bottom and from bottom to top, respectively, where l represents the network layer number, 2≤l≤4;
[0074] Figure 4 This is the basic structure of Multi_Concat; Figure 5 This is the basic structure of Transition.
[0075] S33: After feature fusion at the multi-scale merging node, the data is transmitted to the bottleneck layer, such as... Figure 6 As shown, multiple cross-scale fusion nodes eventually aggregate features into three categories: low, medium, and high feature maps.
[0076] S4: The multi-dimensional attention module, which takes three levels of features with different characteristics as input, captures multi-dimensional feature relationships. In the head network, the three levels of features are fused into one level of features to further enhance and calibrate the input information.
[0077] S4 specifically includes:
[0078] S41: High-class feature H is used to enhance the information of the channel dimension. It is concatenated with the input data B to make the number of channels the same. Global average pooling and global standard deviation pooling operations are used to collect global channel feature information.
[0079] S42: The channel weight vector m is generated through the fully connected layer. The parameters after the sigmoid operation can be expressed as follows:
[0080] m = S i (A c )
[0081] Where Ac It is a feature that connects high-class features and input data, {A c H,input}∈R C×H×W S represents the activation function. The channel attention feature map calculated at position i can then be represented as:
[0082]
[0083] Where z i ∈R C It has different representations in different pooling operations, representing the information of the i-th channel.
[0084] like Figure 7 As shown, multiplying the weight parameters by the input data, A c The number of channels remains consistent with the input data to obtain the final output CR, which can be represented as:
[0085] CR = mB
[0086] With the above structure, channel attention can be selectively focused on important features, thereby suppressing unnecessary features.
[0087] S43: The mid-class feature M has rich semantic information. The CR feature is concatenated with the mid-class feature, and the spatial information feature is obtained by using max pooling and average pooling operations.
[0088] S44: Obtain the spatial weight vector w through the sigmoid activation function, use the training parameter β to weight the features, and finally sum the weighted class features and channel features element by element. This can be expressed as:
[0089] E=w(β·M+(1-β)·CR)
[0090] S45: Perform a convolution operation on the low-class feature L with accurate structural information to generate a vector with 1 channel, where the height and width remain unchanged;
[0091] S46: Obtain the bias feature of this level through the sigmoid activation function, and add it to the feature E of the previous branch to obtain the final first-level feature:
[0092] F = E + S(Conv(L))
[0093] S47: The multi-dimensional attention module of the head network aggregates the features into first-level features. At the same time, the features of the fourth-stage network layer and the features of the SPPCSPC module are also transmitted to the head network as third-layer enhanced features.
[0094] S5: Perform convolution calculations on the enhanced features and features of the remaining network layers in the head network, and input them into the probe to obtain detection boxes and object category information.
[0095] S5 specifically includes:
[0096] S51: The feature layers of three scales undergo multiple convolution operations in the head network and are then passed to the RepConv module, which contains multiple residual structures.
[0097] S52: Feature information is transmitted to the probe head. Each feature point has three prior bounding boxes and detection results.
[0098] S53: Classify and output the results based on the feature map. The types of defects detected include: stains, snags, broken threads, holes, and color errors.
[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.
Claims
1. A real-time image-based method for small target recognition, characterized in that: The method includes the following steps: S1: Collect photos of product defects, create the dataset needed for the network model, and divide it into training and test sets; S2: Input the image data into a deep convolutional neural network, and extract features through the backbone network to obtain five levels of features; S2 specifically includes the following steps: S21: The input image undergoes feature extraction through the backbone network and convolutional computation through multiple modules; including three convolutional layers in the first stage, one convolutional layer and multiple connection modules in the second stage, transition modules and multiple connection modules in the third, fourth and fifth stages, and the SPPCSPC module with CSP structure; S22: The second and third convolutional layers in the first stage, and the feature maps output from the second, third and fifth stages construct a five-level feature pyramid; S23: The feature maps output by the fourth stage and the SPPCSPC module are directly transmitted to the head network; S3: The five-level features are fused into three-level features—high-class, medium-class, and low-class—using a multi-scale merging module; S3 specifically includes the following steps: S31: The five-level feature pyramid is transmitted to the multi-scale merging module, and adjacent features are fused to form multi-scale merging nodes, which are distributed in a V-shape. S32: Upsampling is performed using bilinear interpolation. Using the Focus downsampling method for channel and spatial information exchange. and Reshaped into Same scale size, three adjacent features , , The fusion was performed separately from top-down and bottom-up approaches, among which... l Indicates the network layer ordinal number. ; S33: After feature fusion at the multi-scale merging node, the data is transmitted to the bottleneck layer, where multiple cross-scale fusion nodes aggregate the features into three categories: low, medium, and high feature maps. S4: Inputting tertiary features with different characteristics into a multi-dimensional attention module captures the relationship between channel and spatial dimension features. The head network then fuses the tertiary features into a primary feature to enhance and calibrate the input information. S4 specifically includes the following steps: S41: High-class features H Used to enhance information in the channel dimension, in conjunction with the input data. By concatenating channels to ensure the same number of channels, global average pooling and global standard deviation pooling operations are used to collect global channel feature information. S42: Generate channel weight vectors through fully connected layers. m The parameters after using the sigmoid operation are expressed as follows: in It is a feature that connects high-class features and input data. ; Represents the activation function, and calculates the... The channel attention feature map at each position is represented as follows: in It has different representations for different pooling operations, representing the first... Information from each channel; Multiply the weight parameters by the input data. The number of channels must be consistent with the input data to obtain the final output. CR : ； S43: Medium-class features M With rich semantic information, CR features are concatenated with mid-class features, and spatial information features are obtained by using max pooling and average pooling operations; S44: Obtain the spatial weight vector using the sigmoid activation function. Using training parameters Weighted features, calculated by summing the weighted categorical features and weighted channel features for each element, are expressed as follows: S45: Low-class features with accurate structural information L Perform a convolution operation to generate a vector with 1 channel, where the height and width remain unchanged; S46: Obtain the bias features of the current level through the sigmoid activation function, and compare them with the features of the previous branch. The summation yields the final first-level feature: S47: The multi-dimensional attention module of the head network aggregates the features into first-level features. At the same time, the features of the fourth-stage network layer and the features of the SPPCSPC module are also transmitted to the head network as third-layer enhanced features. S5: Convolutional calculations are performed on the enhanced features and features of the remaining network layers in the head network, which are then fed into the probe to obtain detection boxes and object category information. Classification is performed based on the defect detection results.
2. The image-based real-time target recognition method according to claim 1, characterized in that: S1 specifically includes the following steps: S11: Select defective images including stains, snags, broken threads, holes, and miscolors as dataset samples; S12: The dataset samples are augmented using methods such as image translation, rotation, mirroring, cropping, scaling, image blurring, and image stitching; S12: Use the image annotation tool LabelImg to annotate the location coordinates and classification of foreign objects in the image, and convert the obtained data into the format of the PASCAL VOC2012 dataset. Divide the dataset into training and test sets in a 7:3 ratio.
3. The image-based real-time target recognition method according to claim 1, characterized in that: S5 specifically includes the following steps: S51: The feature layers of the three scales undergo multiple convolution operations in the head network and are then transmitted to the RepConv module, which contains several residual structures. S52: Feature information is transmitted to the detector head. Each feature point has three prior bounding boxes and the detection result. S53: Classify and output the results based on the feature map. The types of defects detected include: stains, snags, broken threads, holes, and miscolors.
Citation Information
Patent Citations
Image target detection method based on double attention mechanism
CN114863236A