Small target detection method based on feature entanglement

Abstract

The invention discloses a small target detection method based on feature entanglement, and relates to the technical field of image detection. The objective of the invention is to solve the problems that a background and a foreground are difficult to effectively distinguish and the detection precision of a small-size target is low in an existing small target detection method based on deep learning. According to the invention, a target detection method based on deep learning is adopted to carry out BTM antenna equipment bolt detection. Specifically, a feature entanglement module is constructed and introduced into a trunk part of a Faster RCNN target detection network, so that a small target detection network based on feature entanglement is obtained. The network can realize the high-precision detection of the BTM antenna equipment bolt with a small size, reduces the risk of missing detection of a small target, effectively guarantees the operation safety, and greatly improves the vehicle detection efficiency.

Description

Technical Field

[0001] The invention belongs to the technical field of image detection. Background Art

[0002] The BTM (Transponder Transmission Unit) antenna equipment is the core equipment for EMU onboard equipment to receive transponder information. The EMU must undergo a first-level maintenance inspection at the end of each operation or within 48 hours, and conduct a comprehensive inspection of the key components and loose bolts on the bottom of the vehicle to ensure the safe and stable operation of the EMU. As an important part of the EMU operation control equipment, the BTM antenna equipment is fixed to the EMU vehicle end bottom plate by multiple bolts. It has the important task of judging the road condition information ahead of the train. Whether its installation is firm is directly related to the driving safety of the EMU train. The loosening of BTM bolts is mostly due to the gradual loosening of the bolts due to vibration and stress during the long-term operation of the EMU train, which eventually leads to the falling off of the bolts. In the first-level maintenance and inspection project, it is manifested as the falling off of the bolts and the sticking of glue in the original position. Therefore, the detection of the loosening of the bolts of the EMU BTM antenna equipment is the focus of the first-level maintenance operation of the EMU train.

[0003] Figure 1 The following figure shows the situation of BTM bolts loose and glued (red frame) and BTM bolts in normal condition (green frame). It can be seen from the figure that there is no obvious difference in shape and color between bolt glue and normal bolts, and it accounts for a small proportion in a single inspection image, and is not much different from the background color, which brings certain difficulties to manual inspection operations.

[0004] Initially, the vehicle inspection workshop used manual labor to inspect the faults of the vehicles to be inspected, which could easily lead to missed inspections of loose bolts with small shapes and low color difference from the background due to fatigue of the inspection workers. Therefore, a small target detection method based on deep learning with high precision and high efficiency came into being.

[0005] Small objects account for a small proportion of pixels in an image and have a texture similar to that of the background. Therefore, how to accurately extract the spatial structure information of small objects and accurately map semantic information such as color and texture has become a research focus.

[0006] In order to effectively distinguish the background and foreground of the target and improve the ability to detect small targets, there are many attempts in the prior art, such as establishing deep and shallow feature interactions between the codec structure and learning the characteristic elements contained in the image from multiple angles. Relying on the codec structure, many target detection methods have been improved and optimized (application number 1: CN202210780211.6; application number 2: CN202011059872.7). These networks contain encoders, decoders, and jump connections connecting the deep and shallow layers of the network. This method fuses the deep and shallow features of the network through an asymmetric feature aggregation method, proving the effectiveness of cross-layer feature fusion in retaining small target features. Subsequently, feature fusion measurement has been widely used in small target detection tasks. Some methods (application number: CN116883714A) promote the transmission of more complete features to the decoder by improving the jump connection to achieve adaptive enhancement of deep and shallow feature information.

[0007] The above method can improve the accuracy and stability of feature transmission by learning the spatial and semantic information contained in the features from the overall level through deep and shallow layer connections and feature fusion. However, it is impossible to learn the spatial structure and detailed color and texture information contained in the input features separately. To a certain extent, it will ignore the association of different attribute information and lack the ability to encode the comprehensive attributes of the target. There is a possibility of inaccurate positioning of small targets and errors in mapping color and texture information. When applied to the first-level maintenance of EMU trains for BTM antenna equipment bolt inspection, it is difficult to effectively distinguish the texture information of the bolts and the bottom plate, and there is a problem of low detection accuracy for small bolts, which seriously affects driving safety. Therefore, it is particularly important to establish an effective interaction path in the image domain to learn and enhance the potential association of the distribution of spatial features such as structural contours and semantic features such as detailed colors. Summary of the invention

[0008] The present invention aims to solve the problem that the existing small target detection method based on deep learning is difficult to effectively distinguish the background and foreground, and has low detection accuracy for small targets. A small target detection method based on feature entanglement is now provided.

[0009] A small target detection method based on feature entanglement comprises: inputting an image to be detected into a small target detection network based on feature entanglement to detect small targets in the image to be detected;

[0010] The small target detection network based on feature entanglement includes:

[0011] A feature entanglement module, used for extracting features of the image to be inspected;

[0012] RPN region proposal network, used to screen candidate boxes in the features using a CNN convolutional neural network;

[0013] Roi Pooling layer, which divides each candidate box into multiple sub-regions, performs a maximum pooling operation on each sub-region, obtains a feature vector of fixed length, and maps the feature vector to the target region of the image to be inspected;

[0014] The classification and regression network includes a classification task and a regression task. The classification task uses Softmax to classify the target in the candidate frame. The regression task uses a fully connected layer to compress the channel and correct the position information and size information of the candidate frame.

[0015] Furthermore, the feature entanglement module includes: a spatial feature preservation branch, a semantic feature enhancement branch and a feature fusion unit;

[0016] The spatial feature preserving branch and the semantic feature enhancing branch are connected in parallel to each other, and the feature fusion unit fuses the output of the spatial feature preserving branch and the output of the semantic feature enhancing branch by multiplication to obtain the features of the image to be inspected.

[0017] Furthermore, the above semantic feature enhancement branch:

[0018] A 1×1 convolution is used to expand the input features to a feature vector with 1024 channels;

[0019] The 1024-dimensional feature vector is processed using depthwise convolutions with convolution kernels of 1×1, 3×3, and 5×5, respectively, to obtain three sets of output results;

[0020] The three groups of output results are standardized respectively by using batch normalization and ReLU activation function, and the three groups of processing results are spliced;

[0021] Global average pooling is used to spatially compress the concatenated features, and a fully connected layer is used to extract the semantic features of each channel.

[0022] Furthermore, the above 1×1 convolution is used to expand its input features to a feature vector with 1024 channels, expressed as:

[0023]

[0024] in, is the input feature of the feature entanglement module, Conv 1×1 (·) indicates 1×1 convolution processing, It is the feature vector output by 1×1 convolution;

[0025] The three groups of output results are expressed as:

[0026]

[0027] in, Indicates the depth convolution processing with a convolution kernel of k×k. is the output result after deep convolution processing, k=1,3,5;

[0028] The three groups of output results are standardized by batch normalization and ReLU activation function, respectively, and the expressions are:

[0029]

[0030] Among them, F Norm (·) represents batch normalization, σ(·) is the ReLU activation function, It is the output feature of each branch after batch normalization and ReLU activation function processing;

[0031] The three groups of processing results are concatenated, and the expression is:

[0032]

[0033] in, is the concatenated feature, F p (·) indicates feature concatenation operation;

[0034] The global average pooling is used to compress the concatenated features spatially, and the fully connected layer is used to extract the semantic features of each channel. The expression is:

[0035]

[0036] Among them, GAP(·) and FC(·) represent global average pooling and fully connected layer processing, respectively. The output feature of the branch is enhanced for the semantic feature.

[0037] Furthermore, the above feature fusion unit:

[0038] The semantic features output by the semantic feature enhancement branch and the input features of the feature entanglement module transmitted by the spatial feature retention branch are multiplied and fused, and then the multiplication and fusion results are processed using 1×1 convolution to obtain features with the same number of input feature channels.

[0039] Furthermore, the expression of the above feature fusion unit is:

[0040]

[0041] Among them, Y c is the output feature of the feature fusion unit, is the input feature of the feature entanglement module, The output feature of the branch is enhanced for the semantic feature, Conv 1×1 (·) indicates 1×1 convolution processing.

[0042] Furthermore, the above RPN region proposal network:

[0043] The anchor with the largest overlap area ratio with the true candidate region is recorded as the foreground;

[0044] Among the remaining anchors, the anchors whose overlapping area ratio with the true candidate region is greater than 0.7 are recorded as foreground, the anchors whose overlapping area ratio with the true candidate region is less than 0.3 are recorded as background, and the rest of the anchors are discarded.

[0045] Furthermore, the loss function expression of the above small target detection network based on feature entanglement is:

[0046] L=L cls +L box ,

[0047] Among them, L is the total loss, L cls is the loss value of the category to which each candidate box belongs, L box The loss value of the coordinate value of the rectangular box of each candidate box.

[0048] Furthermore, the above-mentioned image to be inspected is: an inspection image of the bottom of the EMU obtained by moving the on-track linear array camera.

[0049] Furthermore, if the small target detection network based on feature entanglement does not detect the "trouble" category, it is determined that the image to be inspected is normal and the BTM antenna device has no bolt detachment.

[0050] If the small target detection network based on feature entanglement detects the "trouble" category, it is determined that the bolts of the BTM antenna device in the image to be inspected are loose.

[0051] The beneficial effects of the present invention are as follows:

[0052] 1. The present invention uses a deep learning-based target detection method to perform primary maintenance inspection tasks, which can avoid the problems of low labor efficiency and poor safety of manual vehicle inspection, effectively ensure operational safety, and improve operational efficiency.

[0053] 2. The present invention uses a feature entanglement module constructed by multi-scale deep convolution to adaptively interact spatial and semantic features in a multiplicative manner, which can achieve precise alignment of the spatial structure and color texture of small targets and multi-scale targets, and reduce the possibility of target color confusion.

[0054] 3. The present invention is applied to the task of detecting loose bolt faults, which can effectively distinguish foreground targets and background texture information, and reduce the probability of missed detection and false detection of small targets such as loose bolts.

[0055] 4. The feature entanglement module described in the present invention can modify the hyperparameters and the number of stacking layers to adapt to other fault detection tasks and has a certain generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a schematic diagram of bolt falling off;

[0057] Figure 2 Flowchart for model training;

[0058] Figure 3 This is the basic network structure diagram of Faster RCNN;

[0059] Figure 4 It is the structural diagram of the characteristic entanglement module;

[0060] Figure 5 This is a flow chart of the fault identification procedure. DETAILED DESCRIPTION

[0061] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0062] Specific implementation method 1: The input features of the target detection network contain information of different levels of components. According to the different attributes of the information contained, they can be divided into two categories: spatial features representing two-dimensional spatial position and contour information, and semantic information representing one-dimensional color texture information of the target. Spatial features are features that contain the spatial position and contour structure information of the target and have a certain length and width. Semantic features are features that contain the color texture information of the target and ignore the spatial structure information, with a length and width of 1×1. The true degree of coordination between the two directly determines the accuracy of target detection. The current method adopts the same branch to deepen or widen the network to obtain multi-level feature information of the image. Although it has a certain effect on maintaining contextual features, there is a possibility of feature transfer loss when deepening or widening the network. The recognition ability of targets with similar color textures to the background and small targets is limited, which can easily lead to confusion between spatial structure and detail color, and is unable to perform the task of detecting bolt loss in BTM antenna equipment.

[0063] To address this problem, this implementation combines the advantages of multi-dimensional convolutional neural networks and shortcut connections in local spatial information extraction and retention, and one-dimensional convolutional neural networks in global information modeling and color semantic information extraction to construct a feature entanglement module. The feature entanglement module is introduced into the backbone of the Faster RCNN target detection network to obtain a small target detection network based on feature entanglement. The network relies on multi-dimensional and one-dimensional convolutions to model the deep pixel differences and correlation levels of features in an interactive entanglement manner in the image domain, and can achieve accurate transmission of two-dimensional spatial features and one-dimensional semantic features. Furthermore, this implementation also applies the small target detection network based on feature entanglement to the first-level maintenance operations of EMU trains, which can effectively distinguish the bolts of BTM antenna equipment from the bottom plate background, can achieve accurate detection of small targets, reduce the risk of missed detection of small targets, and effectively ensure operational safety, greatly improving inspection efficiency.

[0064] Specifically, refer to Figures 1 to 5 Specifically describing this embodiment, the small target detection method based on feature entanglement described in this embodiment includes:

[0065] S1. Establish a sample data set

[0066] The process of constructing a sample data set includes three parts: data collection, data cleaning, and data augmentation.

[0067] 1.1 Data Collection

[0068] The original data set is acquired by the vehicle-mounted linear array camera. The data in the original data set is the first-level maintenance vehicle image with a resolution of 5000*4096 without data amplification processing. The details are as follows:

[0069] (1) Image acquisition: The on-track linear array camera is used to take pictures and collect images of the EMU vehicle end, vehicle body, and the bottom of the bogie as the data set production material;

[0070] (2) Acquisition of BTM bolt-off fault materials: Collect images of first-level maintenance vehicles and construct BTM bolt-off fault materials through PS and selection of real faults;

[0071] (3) Sample dataset preparation: The BTM bolt loosening area is marked as the "trouble" category as the positive sample; the area with normal bolts after cropping is marked as the "normal" category and set as the negative sample. After the positive and negative samples are marked, the original dataset is formed.

[0072] 1.2 Data Cleaning

[0073] Data cleaning is to mark, check and clean the images in the original data set that are mislabeled, missing, or do not meet the high-precision training requirements due to background interference, and then reconstruct the original data set. The details are as follows:

[0074] (1) Training phase: The augmented dataset is cut into a size of 1024 × 1024 and used as the target detection network training set. The BTM bolt loosening detection training task is performed to obtain the model weights and configuration files.

[0075] (2) Use the model configuration file obtained in (1) to infer the data set. If the confidence of the BTM bolt loosening category obtained by inference is less than 0.5, it is determined that the image cannot meet the high-precision training requirements, and the image data and corresponding xml tags are deleted. At the same time, the image is reviewed to analyze the root cause of its low confidence;

[0076] (3) Use the model configuration weights obtained in (1) to infer the augmented data set. If the IOU between the true value and the predicted value is greater than 0.3, but the categories are inconsistent, then it is determined that the image corresponding to the label is wrong, and the label should be checked and re-labeled if necessary.

[0077] 1.3 Data Augmentation

[0078] Data augmentation refers to the use of random data augmentation methods on the basis of the reconstructed original data set, and the two types of labels, "trouble" and "normal", are obtained after a 3-fold data augmentation operation. Specific methods include horizontal image flipping, brightness adjustment, histogram equalization, adding Gaussian noise, adding rain and snow weather, etc. The above methods can create new data for more scenes, enhance the generalization ability of the detection network, and reduce the probability of model overfitting. At the same time, the use of a variety of data augmentation methods to construct a data set can meet the application background of railway data for vehicle inspection projects in all weather and any weather scenarios.

[0079] At this point, the sample data set required for the small target detection network based on feature entanglement has been completed and divided according to the ratio of "training set: test set: verification set = 7:2:1" to meet the needs of model training, model testing reasoning and verification.

[0080] S2. Model training

[0081] The model is built and trained based on the data set after data augmentation and data cleaning. The basic process is as follows Figure 2 This embodiment uses the Faster RCNN target detection network including the feature entanglement module to detect the loosening of BTM bolts in the first-level maintenance vehicle image. The basic network structure of Faster RCNN is as follows: Figure 3 shown.

[0082] 2.1 CNN feature extraction

[0083] In the target detection task, it is particularly important to ensure the consistent transmission of the spatial structure information of the input features and the accurate mapping of semantic information such as the foreground and background color texture. Faster RCNN uses the backbone network part as the target detection network feature extractor to obtain the approximate area of ​​the foreground target from the inspection image to provide accurate classification and detection to RPN and Roi Pooling. This architecture uses residual units as the backbone network to extract features, which can well solve the feature redundancy caused by the stacking of convolutional layers. However, the current method uses the same branch to process the input features from the overall level, and cannot perform targeted extraction and enhancement of the spatial and semantic information in the input features. When applied to the bolt detachment detection task of BTM antenna equipment, due to the small size of the bolts and the similar color to the base plate texture, the current method will ignore the inherent correlation between the spatial structure and color semantic information in the image to a certain extent, resulting in low network detection accuracy and the risk of missed detection.

[0084] To this end, this embodiment constructs a feature entanglement module including two parallel branches, a spatial feature preservation branch and a semantic feature enhancement branch, to specifically enhance and process the two-dimensional spatial structure and one-dimensional color semantic information in the input features. The structure is as follows: Figure 4 Furthermore, the residual unit in Faster RCNN is replaced by the feature entanglement module, which is used as the backbone network to perform feature extraction tasks, so as to achieve the minimum deviation transmission of two-dimensional spatial structure information and the accurate mapping of one-dimensional color texture and other semantic information, thereby improving the small target detection capability.

[0085] The feature entanglement module is divided into two branches: spatial feature preservation and semantic feature enhancement. The two branches focus on maintaining information such as spatial structure and accurately mapping semantic features such as color texture. The outputs of the two are adaptively fused by multiplication, which can achieve accurate alignment of spatial position and color texture information in the image. The details are as follows:

[0086] (1) Spatial feature preservation branch

[0087] The spatial feature preserving branch is a shortcut connection between the input and output of the feature entanglement module. It uses a jump connection to artificially skip the lower-layer neuron connection, so that a direct connection is established between the deep and shallow layers of the feature matrix. It can effectively make up for the loss of spatial feature transmission caused by the deepening of the network, and play a role in maintaining the consistency of spatial structure information between the deep and shallow layers.

[0088] (2) Semantic feature enhancement branch

[0089] The semantic feature enhancement branch is used to replace the residual unit in the Faster RCNN backbone network. Through the learning process of compressing the two-dimensional space and strengthening the one-dimensional semantics, it is possible to learn the distribution of multi-level semantic information such as color and texture, and effectively distinguish the foreground and background information of small targets. The whole process can be disassembled as follows:

[0090] Small targets have less feature information, and increasing the number of feature channels helps the model capture more subtle changes, which is crucial for distinguishing small targets from similar background information. Therefore, this implementation uses 1×1 convolution to uniformly expand the input features to a 1024-dimensional feature vector with a channel number. It can be expressed as follows:

[0091]

[0092] In the formula, is the feature with the number of channels c input into the backbone network; Conv 1×1 (·) represents 1×1 convolution processing, which normalizes features with different channel numbers to 1024 without changing the feature size; It is the 1×1 convolution output feature.

[0093] The output features of the unified channel number The deep convolutions with convolution kernels of 1×1, 3×3, and 5×5 are used for processing. This method uses deep convolutions with the same number of channels and different convolution kernel sizes to extract features in parallel. It can extract richer multi-scale spatial information and detailed texture information without significantly increasing the amount of calculation, thereby better understanding the context of small targets in semantics and more accurately locating small targets in space. The process can be expressed as follows:

[0094]

[0095] In the formula, Indicates deep convolution processing, k×k represents convolution kernel 1×1, 3×3, 5×5; The feature of using a deep convolution with a convolution kernel of k×k to output 1024 channels. In It is divided into three branches, which use deep convolutions with k=1, 3, and 5 to extract features at different receptive field levels, which can effectively capture the local detail texture and spatial structure information of the image.

[0096] To help the network stabilize training, batch normalization and ReLU activation functions are used to optimize the features of parallel deep convolution outputs. Standardization is performed. The process can be expressed as:

[0097]

[0098] In the formula, F Norm (·) is batch normalization processing, σ(·) is the ReLU activation function, The output features of each branch are processed by batch normalization and ReLU activation function.

[0099] The features obtained by parallel processing of multi-scale deep convolution Splicing is performed to improve the model's adaptability to objects of different sizes. The process can be expressed as:

[0100]

[0101] In the formula, Features obtained by depth convolution of convolution kernels of different sizes; is the feature after splicing; F p (·) indicates feature concatenation operation.

[0102] Use global average pooling to concatenate features Perform spatial compression and use the fully connected layer to extract semantic features such as detail texture contained in each channel. The process can be expressed as:

[0103]

[0104] In the formula, GAP(·) and FC(·) represent global average pooling and fully connected layer processing respectively; Output features for this stage. In this part, global average pooling calculates the average value of features pixel by pixel, which can capture global feature information and reduce the influence of position information on the expression of semantic features. Furthermore, a fully connected layer is used to establish the intrinsic association of channels, learn the semantic expression of each channel, and then achieve accurate perception of semantic features.

[0105] (3) Feature Fusion

[0106] The spatial feature preservation branch and the semantic feature enhancement branch can achieve accurate extraction of spatial position and color texture, but weaken the spatial consistency and semantic association within the features, and there is a possibility of target positioning deviation or color distortion. Therefore, this stage fuses the two in a multiplicative manner to achieve effective interaction and learning of spatial structure information and color texture semantic information. The process can be expressed as:

[0107]

[0108] In the formula, the input features containing rich spatial structure and shallow information are Features with rich color, texture, semantics and deep information Multiply and fuse them, and use 1×1 convolution to obtain feature Y with the same number of channels as the input feature c In this process, the internal correlation of different attribute features can be analyzed, and spatial consistency transfer and semantic precision mapping can be implemented, thereby achieving precise extraction and positioning of small objects and objects of different sizes.

[0109] In summary, the spatial feature preservation branch and the semantic feature enhancement branch extract different elements of the same target in the image from the spatial and semantic levels, which can enhance the spatial and semantic features in a targeted manner. The output features of the two branches are fused in the form of multiplication, which can learn the target foreground and background texture information on the basis of ensuring the minimum deviation transmission of the spatial structure, thus laying the foundation for the accurate generation of small and multi-scale target candidate areas in Faster RCNN and the accurate positioning and classification of targets.

[0110] 2.2RPN Region Proposal Network

[0111] The RPN region proposal network takes the backbone network output as input and uses the CNN convolutional neural network to determine all possible candidate boxes. The main process is as follows:

[0112] (1) For each marked true candidate region, the anchor with the largest overlapping area ratio is recorded as the foreground;

[0113] (2) For the remaining anchors in (1), if the overlap ratio between them and a certain marker box is greater than 0.7, they are recorded as foreground; if the overlap ratio between them and any marker box is less than 0.3, they are recorded as background;

[0114] (3) The remaining anchors in the above two steps and the anchors that cross the image boundary are discarded.

[0115] 2.3Roi Pooling Layer

[0116] Roi Pooling uses the candidate box filtered by RPN as input to divide each candidate region into multiple sub-regions, and performs a maximum pooling operation on each sub-region to obtain a feature vector of fixed length, thereby mapping it to the area where the target is located in the input image.

[0117] 2.4 Classification and Regression Networks

[0118] For classification tasks, Faster RCNN uses Softmax instead of support vector machine for target classification. This method can convert the model output into probability distribution, so as to better map the relative relationship between multiple categories. For the bolt fall-off detection task, using Softmax can give a quantitative index of whether the target in the candidate box is classified as foreground or background, which more clearly reflects the pros and cons of the model's recognition ability for each category.

[0119] For regression tasks, Faster RCNN uses a fully connected layer to correct the position and size information of the candidate box to obtain a more accurate target detection box. For the bolt-off detection task, this part uses a fully connected layer to compress the channel to enhance the perception of spatial features, thereby enhancing the positioning accuracy of the bolt target detection box in the input image.

[0120] Compare the network output prediction image with the true label image and calculate the loss value:

[0121] L=L cls +L box ,

[0122] Where, L is the total loss; L cls The loss value of the category to which each candidate box belongs is calculated by the cross entropy loss function; L box The loss value of the coordinate value of the rectangular box of each candidate box is calculated by the L1 loss function and the position information and size information of the candidate box in the input image is corrected.

[0123] Furthermore, the SGD-Momentum optimizer with fast speed and low oscillation rate is used for weight optimization during the model training process. During the model iteration process, the loss function and optimizer guide the model to update the weight coefficient, thereby completing this training iteration. The detection model repeats the above process at a set learning rate, completes a fixed number of iterations for all vehicle images, and only updates at a lower weight of the loss function until the optimal weight coefficient is found and the result is output to complete the training, thereby obtaining the model configuration file and conducting the bolt fall-off fault detection test process.

[0124] S3, BTM antenna equipment loose bolt fault identification process

[0125] like Figure 5 As shown, the overall process can be broken down as follows:

[0126] Step 1: Obtaining inspection images and creating datasets

[0127] (1) Use the on-track linear array camera to obtain the bottom inspection image of the EMU and obtain the vehicle model information.

[0128] (2) Through PS and selecting real faults, the loose bolt material of BTM antenna equipment is constructed on the vehicle passing image, and after data amplification and data cleaning, a data set suitable for detection model training and inference is obtained.

[0129] Step 2: Load the model and detect and locate the loosened area of ​​the BTM bolts

[0130] According to the vehicle model information and recognition instructions, a small target detection network based on feature entanglement is loaded to detect whether there is a fault pattern of loose bolts of the BTM antenna equipment in the image to be inspected.

[0131] Step 3: Fault determination and alarm

[0132] (1) If the “trouble” category is not detected, the image is considered normal and no bolts have fallen off. The next level 1 repair image detection task is continued.

[0133] (2) If the "trouble" category is detected, it is determined that the BTM antenna device bolts are loose in the vehicle image, and a fault message is uploaded. The inspection personnel will perform the next step of processing or manual inspection according to the fault message.

[0134] Step 4: Update the inspection image and conduct a new round of inspection

[0135] After completing the fault alarm determination and message upload for this round of vehicle inspection, update the vehicle inspection image. Repeat steps 1 to 3 to conduct a new round of BTM antenna equipment bolt loosening detection until the entire vehicle inspection task is completed.

[0136] This implementation method aims to solve the problem that a single branch is difficult to learn the spatial features and semantic features in the image without bias, resulting in mapping deviations for small targets, and proposes a feature entanglement module. This module establishes entangled learning of different attribute features in the process of hierarchical splitting and reorganization of feature elements to study the relationship between spatial and semantic features, and realize the precise fusion of two-dimensional spatial structure and one-dimensional color semantic features. Specifically, the feature entanglement module is composed of a spatial feature preservation branch and a semantic feature enhancement branch in parallel. The spatial feature preservation branch is responsible for the extraction of spatial structure information; the semantic feature enhancement branch is responsible for the enhancement of semantic information such as texture and color of the target. The outputs of the two branches are interactively entangled in a multiplication manner, which can realize the precise mapping of semantic features such as the detail color of the target foreground and background on the basis of the consistent transmission of deep and shallow spatial structure features, avoid the confusion of spatial structure and detail color in the process of small target extraction, and improve the detection accuracy of small targets and multi-scale targets.

[0137] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that the features described in conjunction with a single embodiment may be used in other described embodiments.

Claims

1. A small target detection method based on feature entanglement, characterized in that: include: Inputting the image to be inspected into a small target detection network based on feature entanglement to detect small targets in the image to be inspected; The small target detection network based on feature entanglement includes: A feature entanglement module, used for extracting features of the image to be inspected; RPN region proposal network, used to screen candidate boxes in the features using a CNN convolutional neural network; Roi Pooling layer, which divides each candidate box into multiple sub-regions, performs a maximum pooling operation on each sub-region, obtains a feature vector of fixed length, and maps the feature vector to the target region of the image to be inspected; The classification and regression network includes a classification task and a regression task. The classification task uses Softmax to classify the target in the candidate frame. The regression task uses a fully connected layer to compress the channel and correct the position information and size information of the candidate frame.

2. The small target detection method based on feature entanglement according to claim 1, characterized in that: The feature entanglement module includes: a spatial feature preservation branch, a semantic feature enhancement branch and a feature fusion unit; The spatial feature preserving branch and the semantic feature enhancing branch are connected in parallel to each other, and the feature fusion unit fuses the output of the spatial feature preserving branch and the output of the semantic feature enhancing branch by multiplication to obtain the features of the image to be inspected.

3. The small target detection method based on feature entanglement according to claim 2, characterized in that: The semantic feature enhancement branch: A 1×1 convolution is used to expand the input features to a feature vector with 1024 channels; The 1024-dimensional feature vector is processed using depthwise convolutions with convolution kernels of 1×1, 3×3, and 5×5, respectively, to obtain three sets of output results; The three groups of output results are standardized respectively by using batch normalization and ReLU activation function, and the three groups of processing results are spliced; Global average pooling is used to spatially compress the concatenated features, and a fully connected layer is used to extract the semantic features of each channel.

4. The small target detection method based on feature entanglement according to claim 3 is characterized in that: The 1×1 convolution is used to expand the input features to a feature vector with 1024 channels, expressed as: in, is the input feature of the feature entanglement module, Conv 1×1 (·) indicates 1×1 convolution processing, It is the feature vector output by 1×1 convolution; The three groups of output results are expressed as: in, Indicates the depth convolution processing with a convolution kernel of k×k. is the output result after deep convolution processing, k=1,3,5; The three groups of output results are standardized by batch normalization and ReLU activation function, respectively, and the expressions are: Among them, F Norm (·) represents batch normalization, σ(·) is the ReLU activation function, It is the output feature of each branch after batch normalization and ReLU activation function processing; The three groups of processing results are concatenated, and the expression is: in, is the concatenated feature, F p (·) indicates feature concatenation operation; The global average pooling is used to compress the concatenated features spatially, and the fully connected layer is used to extract the semantic features of each channel. The expression is: Among them, GAP(·) and FC(·) represent global average pooling and fully connected layer processing, respectively. The output feature of the branch is enhanced for the semantic feature.

5. The small target detection method based on feature entanglement according to claim 3 or 4, characterized in that: Feature fusion unit: The semantic features output by the semantic feature enhancement branch and the input features of the feature entanglement module transmitted by the spatial feature retention branch are multiplied and fused, and then the multiplication and fusion results are processed using 1×1 convolution to obtain features with the same number of input feature channels.

6. The small target detection method based on feature entanglement according to claim 5, characterized in that: The expression of the feature fusion unit is: Among them, Y c is the output feature of the feature fusion unit, is the input feature of the feature entanglement module, The output feature of the branch is enhanced for the semantic feature, Conv 1×1 (·) indicates 1×1 convolution processing.

7. The small target detection method based on feature entanglement according to claim 1, characterized in that: The RPN region proposal network: The anchor with the largest overlap area ratio with the true candidate region is recorded as the foreground; Among the remaining anchors, the anchors whose overlapping area ratio with the true candidate region is greater than 0.7 are recorded as foreground, the anchors whose overlapping area ratio with the true candidate region is less than 0.3 are recorded as background, and the rest of the anchors are discarded.

8. The small target detection method based on feature entanglement according to claim 1, characterized in that: The loss function expression of the small target detection network based on feature entanglement is: L=L cls +L box , Among them, L is the total loss, L cls is the loss value of the category to which each candidate box belongs, L box The loss value of the coordinate value of the rectangular box of each candidate box.

9. The small target detection method based on feature entanglement according to claim 1, characterized in that: The image to be inspected is: an inspection image of the bottom of the EMU obtained by moving the on-track linear array camera.

10. The small target detection method based on feature entanglement according to claim 9, characterized in that: If the small target detection network based on feature entanglement does not detect the "trouble" category, it is determined that the image to be inspected is normal and the BTM antenna device has no bolt falling off. If the small target detection network based on feature entanglement detects the "trouble" category, it is determined that the bolts of the BTM antenna device in the image to be inspected are loose.

Images