Rotating target identification method

Abstract

The invention relates to a rotating target identification method, belongs to the technical field of ray image target detection, and solves the problems of high resource consumption and low accuracy during rotating target prediction in the prior art. The identification method comprises the following steps: preprocessing a to-be-identified image to obtain a to-be-detected image; inputting the to-be-detected image into a pre-trained rotating image detection model to obtain a prediction category and prediction rotating bounding box information of each target in the to-be-detected image; and calculating four corner point coordinates of each target according to the predicted rotation bounding box information of each target, and connecting the four corner point coordinates of each target as a predicted rotation bounding box of each target. The method achieves the quick prediction of the rotating target, and improves the recognition precision of the rotating image.

Description

Technical Field

[0001] The present invention relates to the technical field of radiographic image target detection, and in particular to a rotating target recognition method. Background Art

[0002] With the development of society and economic prosperity, the frequency of people's travel and the scale of express logistics are constantly expanding. Security inspection plays a vital role in maintaining public safety. At present, X-ray scanning is a commonly used technical means in security inspection. It can penetrate the surface of objects and help security inspectors identify potential prohibited items. However, this process not only places high demands on the attention and judgment of security inspectors, but also makes it difficult to maintain long-term concentration. Frequent personnel rotation also increases the burden on human resources.

[0003] In recent years, with the rise of deep learning technology, contraband detection has also made breakthrough progress. The deep learning model can automatically extract image features and learn complex patterns, significantly improving the accuracy and robustness of detection. This not only brings new solutions to security inspections, but also provides a more solid guarantee for public safety.

[0004] However, current X-ray target detection methods mainly use horizontal rectangular frames to constrain objects, ignoring the problem of irregular placement of objects. Although existing methods perform well in detecting objects with normal postures, they have large errors in detecting irregularly placed objects and objects with serious overlapping interference. Usually, instance segmentation can avoid this problem, but the computational complexity of the segmentation model increases significantly, more computing resources are required, and it takes more time on the same hardware, making it difficult to use in practice.

[0005] Therefore, there is an urgent need for a technical solution that can achieve rapid detection of X-ray images using a small amount of computing resources. Summary of the invention

[0006] In view of the above analysis, an embodiment of the present invention aims to provide a rotating target recognition method to solve the problems of high resource consumption and low accuracy in rotating target prediction in the prior art.

[0007] An embodiment of the present invention provides a rotating target recognition method, the recognition method comprising:

[0008] Preprocess the image to be identified to obtain the image to be detected;

[0009] Input the image to be detected into a pre-trained rotation image detection model to obtain the predicted category and predicted rotation bounding box information of each target in the image to be detected; the predicted rotation bounding box information includes the predicted center point coordinates, predicted covariance matrix and predicted rotation angle of the rotation bounding box; wherein the rotation image detection model is trained based on an improved YOLOv8 neural network structure, and the improved YOLOv8 neural network structure includes a dual-stream attention module and a terminal feature enhancement module, the dual-stream attention module is used to enhance channel information and feature information when the original backbone network generates a feature map, and the terminal feature enhancement module is used to enhance the features of the feature map output by the original neck network;

[0010] The coordinates of the four corner points of each target are calculated according to the predicted rotation bounding box information of each target, and the coordinates of the four corner points of each target are connected as the predicted rotation bounding box of each target.

[0011] Based on the further improvement of the above recognition method, the rotation image detection model includes a backbone network, a neck network and a detection head;

[0012] The backbone network includes an original backbone network and a dual-stream attention module; the original backbone network is used to extract features from the input image to be detected, obtain the third feature map, the fourth feature map and the fifth feature map and output them to the neck network; the dual-stream attention module enhances channel information and feature information when the original backbone network extracts features to generate the third feature map to the fifth feature map;

[0013] The neck network includes an original neck network and a terminal feature enhancement module; the original neck network is used to perform feature fusion on the third feature map, the fourth feature map and the fifth feature map to obtain the second fused feature map, the third fused feature map and the fourth fused feature map and output them to the terminal feature enhancement module; the terminal feature enhancement module is used to perform feature enhancement on the second fused feature map, the third fused feature map and the fourth fused feature map respectively to obtain the enhanced fifth fused feature map, the sixth fused feature map and the seventh fused feature map and output them to the detection head;

[0014] The detection head is used to predict the predicted category and predicted rotation bounding box information of each target in the image to be detected according to the fifth fused feature map, the sixth fused feature map and the seventh fused feature map.

[0015] Based on the further improvement of the above recognition method, the backbone network includes a first convolution module, a first feature extraction layer, a second feature extraction layer, a third feature extraction layer and a fourth feature extraction layer connected in sequence, and outputs a first feature map, a second feature map, a third feature map, a fourth feature map and a fifth feature map respectively;

[0016] The first feature extraction layer, the second feature extraction layer and the third feature extraction layer have the same structure, and all include a second convolution module and a first residual convolution module connected in sequence; the fourth feature extraction layer includes a second convolution module, a first residual convolution module, a dual-stream attention module and a spatial pyramid pooling fast module SPPF connected in sequence.

[0017] Based on the further improvement of the above recognition method, the dual-stream attention module includes a left branch, a right branch and a first matrix addition layer;

[0018] The left branch is used to enhance the channel information of the input sixth feature map, obtain the eighth feature map and input it to the first matrix addition layer;

[0019] The right branch is used to enhance the feature information of the input sixth feature map, obtain the ninth feature map and input it into the first matrix addition layer;

[0020] The eighth feature map and the ninth feature map are added in the first matrix addition layer to obtain the seventh feature map.

[0021] Based on the further improvement of the above recognition method, the left branch includes a maximum pooling layer, an average pooling layer, a first fully connected layer, a second fully connected layer, a second matrix addition layer, an activation function layer and a multiplication module;

[0022] The sixth feature map is subjected to the maximum pooling layer and the first fully connected layer to obtain the tenth feature map, and the sixth feature map is subjected to the average pooling layer and the second fully connected layer to obtain the eleventh feature map;

[0023] The tenth feature map and the eleventh feature map are added in the second matrix addition layer, and the added feature maps are passed through the activation function layer to obtain the twelfth feature map;

[0024] The twelfth feature map and the sixth feature map are multiplied in the multiplication module to obtain the eighth feature map.

[0025] Based on the further improvement of the above recognition method, the right branch includes a maximum pooling layer, an average pooling layer, a matrix concatenation layer, a convolution layer, an activation function layer and a multiplication module;

[0026] The sixth feature map is passed through the maximum pooling layer to obtain the thirteenth feature map, and the sixth feature map is passed through the average pooling layer to obtain the fourteenth feature map;

[0027] The thirteenth feature map and the fourteenth feature map are spliced ​​in the matrix splicing layer, and the spliced ​​feature maps are sequentially passed through the convolution layer and the activation function layer to obtain the fifteenth feature map;

[0028] The fifteenth feature map and the sixth feature map are multiplied in the multiplication module to obtain a ninth feature map.

[0029] Based on the further improvement of the above identification method, the terminal feature enhancement module includes three identical structures: a first terminal feature enhancement module, a second terminal feature enhancement module and a third terminal feature enhancement module;

[0030] The first terminal feature enhancement module, the second terminal feature enhancement module and the third terminal feature enhancement module respectively enhance the features of the input second fusion feature map, the third fusion feature map and the fourth fusion feature map, and output the fifth fusion feature map, the sixth fusion feature map and the seventh fusion feature map.

[0031] Based on the further improvement of the above recognition method, the first terminal feature enhancement module, the second terminal feature enhancement module or the third terminal feature enhancement module all include a maximum pooling layer, a third fully connected layer, a first activation function layer, a fourth fully connected layer, a second activation function layer and a multiplication module;

[0032] The input feature map is sequentially passed through the maximum pooling layer, the third fully connected layer, the first activation function layer, the fourth fully connected layer, and the second activation function layer to obtain the intermediate feature map;

[0033] The intermediate feature map and the input feature map are multiplied in the multiplication module to obtain the output feature map.

[0034] Based on the further improvement of the above identification method, the preprocessing includes one or more of the following operations:

[0035] Grayscale;

[0036] Binarization;

[0037] Normalization;

[0038] Zoom;

[0039] Cutting;

[0040] Sharpening;

[0041] Filtering.

[0042] Based on the further improvement of the above recognition method, the coordinates of the four corner points of each target are calculated by the following formula:

[0043]

[0044]

[0045] in, Represents the predicted center point coordinates of each target, represents the prediction covariance matrix of each target, θ pred represents the predicted rotation angle of each object.

[0046] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0047] 1. The image to be detected is predicted by the rotation image detection model to obtain the predicted category and predicted rotation bounding box of each target in the image to be detected. The rotation bounding box can effectively constrain each target in the image, reduce the background part of each target, make the bounding box of each target clearer, and improve the recognition accuracy of each target in the image;

[0048] 2. Based on the YOLOv8 neural network structure, a dual-stream attention module is added to enhance the channel information and feature information when the original backbone network generates feature maps, and a terminal feature enhancement module is added to enhance the feature maps output by the original neck network, thereby achieving accurate recognition of rotating targets, avoiding the use of instance segmentation models to increase additional computing workload, and requiring fewer resources.

[0049] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0051] Figure 1 A schematic diagram of a flow chart of a rotating target recognition method provided by an embodiment of the present invention;

[0052] Figure 2 A schematic diagram of using a traditional rectangular frame to constrain irregularly placed objects provided by an embodiment of the present invention;

[0053] Figure 3 A schematic diagram of using a traditional rectangular frame to constrain multiple similar objects placed at an angle provided by an embodiment of the present invention;

[0054] Figure 4 A schematic diagram of using a rotating bounding box to constrain irregularly placed objects provided by an embodiment of the present invention;

[0055] Figure 5 A schematic diagram of using a rotating bounding box to constrain multiple similar objects that are placed tilted, provided in an embodiment of the present invention;

[0056] Figure 6A schematic diagram of the structure of a rotation image detection model provided by an embodiment of the present invention;

[0057] Figure 7 A schematic diagram of the structure of a dual-stream attention module provided in an embodiment of the present invention;

[0058] Figure 8 A schematic diagram of the structure of a first terminal feature enhancement module, a second terminal feature enhancement module or a third terminal feature enhancement module provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0060] A specific embodiment of the present invention discloses a rotating target recognition method, such as Figure 1 As shown, the identification method includes:

[0061] Step S1: preprocess the image to be identified to obtain the image to be detected;

[0062] Step S2: input the image to be detected into a pre-trained rotation image detection model to obtain the predicted category and predicted rotation bounding box information of each target in the image to be detected; the predicted rotation bounding box information includes the predicted center point coordinates, predicted covariance matrix and predicted rotation angle of the rotation bounding box; wherein the rotation image detection model is trained based on an improved YOLOv8 neural network structure, the improved YOLOv8 neural network structure includes a dual-stream attention module and a terminal feature enhancement module, the dual-stream attention module is used to enhance channel information and feature information when the original backbone network generates a feature map, and the terminal feature enhancement module is used to enhance the feature of the feature map output by the original neck network;

[0063] Step S3: Calculate the coordinates of the four corner points of each target according to the predicted rotated bounding box information of each target, and connect the coordinates of the four corner points of each target as the predicted rotated bounding box of each target.

[0064] Specifically, Figure 1 As shown, in step S1, a preprocessing operation is performed on the image to be identified, and the image to be detected is obtained after the preprocessing operation.

[0065] Preferably, the preprocessing includes one or more of the following operations:

[0066] Grayscale;

[0067] Binarization;

[0068] Normalization;

[0069] Zoom;

[0070] Cutting;

[0071] Sharpening;

[0072] Filtering.

[0073] Specifically, grayscale is the process of converting a color image into a grayscale image, which helps reduce the complexity of image processing and storage requirements. Binarization is the process of converting an image into an image containing only two pixel values, which is used to simplify image data and make the image easier to further analyze and process. Normalization is the process of scaling the pixel values ​​of an image to a specific range, such as scaling to a range of 0 to 1. The scale of image data can be adjusted to improve the efficiency and accuracy of analysis and modeling. Scaling and cropping are the process of adjusting the size and content of an image. Scaling changes the resolution and pixel density of an image, while cropping changes the composition and focus of an image, helping to adjust the image to meet analysis and processing requirements. Sharpening is the process of enhancing the edges and contours in an image to make the image look clearer and more vivid. Filtering is the process of processing an image in the frequency domain or spatial domain, adjusting the frequency components of an image to improve image quality or extract useful information to enhance or suppress certain features in the image.

[0074] Specifically, Figure 1 As shown, in step S1, the image to be identified is preprocessed to obtain the image to be detected. In step S2, the image to be detected is input into the rotation image detection model. The rotation image detection model predicts the predicted category and predicted rotation bounding box information of each target in the image to be detected based on the image to be detected.

[0075] Specifically, the predicted category is the category corresponding to the target, and the predicted rotation bounding box information includes the predicted center point coordinates, the predicted covariance matrix, and the predicted rotation angle of the rotation bounding box of the target.

[0076] Specifically, the rotated image detection model is trained based on the improved YOLOv8 neural network structure. The improved YOLOv8 neural network structure adds a dual-stream attention module and a terminal feature enhancement module on the basis of the YOLOv8 neural network structure.

[0077] Specifically, the dual-stream attention module acts on the original backbone network. When the original backbone network extracts the features of the image to be detected, the channel information and feature information of the feature map generated in the middle are enhanced, the feature map is focused, and the recognition accuracy of the image to be detected is improved.

[0078] Specifically, the terminal feature enhancement module acts on the original neck network to further enhance the feature map output by the original neck network to the detection head, thereby improving the recognition accuracy of the image to be detected.

[0079] It is worth noting that if Figure 2 As shown in the figure, irregularly placed objects will cause too much background in the rectangular bounding box when constrained by traditional rectangular boxes. Figure 3 As shown in the figure, when multiple similar objects are placed at an angle, using traditional rectangular boxes to constrain them will cause serious box overlap. Figure 2 and Figure 3 As shown in the figure, if a rectangular bounding box is used as the constraint box of each target for an obliquely placed object, the interference of each target will increase, which will make it difficult to identify the target and the recognition accuracy will be low.

[0080] like Figure 4 and Figure 5 As shown, when the present invention uses a rotating bounding box to constrain irregularly placed objects and multiple similar objects placed at an angle, the background portion of each target can be reduced, and the intersection portion of two overlapping targets can be reduced, so that each target is effectively constrained and the bounding box of each target is clearer.

[0081] It is worth noting that by predicting the image to be detected by the rotation image detection model to obtain the rotation bounding box of each target in the image to be detected, the interference of each target in the image to be detected can be effectively reduced, and the detection speed and recognition accuracy of each target can be improved.

[0082] Specifically, Figure 6 As shown, the rotation image detection model includes a backbone network, a neck network and a detection head;

[0083] The backbone network includes an original backbone network and a dual-stream attention module; the original backbone network is used to extract features from the input image to be detected, obtain the third feature map, the fourth feature map and the fifth feature map and output them to the neck network; the dual-stream attention module enhances channel information and feature information when the original backbone network extracts features to generate the third feature map to the fifth feature map;

[0084] The neck network includes an original neck network and a terminal feature enhancement module; the original neck network is used to perform feature fusion on the third feature map, the fourth feature map and the fifth feature map to obtain the second fused feature map, the third fused feature map and the fourth fused feature map and output them to the terminal feature enhancement module; the terminal feature enhancement module is used to perform feature enhancement on the second fused feature map, the third fused feature map and the fourth fused feature map respectively to obtain the enhanced fifth fused feature map, the sixth fused feature map and the seventh fused feature map and output them to the detection head;

[0085] The detection head is used to predict the predicted category and predicted rotation bounding box information of each target in the image to be detected according to the fifth fused feature map, the sixth fused feature map and the seventh fused feature map.

[0086] Specifically, Figure 6 As shown, the backbone network extracts features from the image to be detected input to the rotation image detection model to obtain the first feature map P1, the second feature map P2, the third feature map P3, the fourth feature map P4 and the fifth feature map P5. The original backbone network extracts features from the input image to be detected, and the dual-stream attention module enhances the channel information and feature information when the original backbone network extracts features to generate the third to fifth feature maps.

[0087] Preferably, if Figure 6 As shown, the backbone network includes a first convolution module, a first feature extraction layer, a second feature extraction layer, a third feature extraction layer and a fourth feature extraction layer connected in sequence, and outputs a first feature map, a second feature map, a third feature map, a fourth feature map and a fifth feature map respectively;

[0088] The first feature extraction layer, the second feature extraction layer and the third feature extraction layer have the same structure, and all include a second convolution module and a first residual convolution module connected in sequence; the fourth feature extraction layer includes a second convolution module, a first residual convolution module, a dual-stream attention module and a spatial pyramid pooling fast module SPPF connected in sequence.

[0089] Specifically, Figure 6 As shown, the structures of the first feature extraction layer, the second feature extraction layer and the third feature extraction layer are the same, and all include a second convolution module and a first residual convolution module connected in sequence; the fourth feature extraction layer includes a second convolution module, a first residual convolution module, a dual-stream attention module and a spatial pyramid pooling fast module SPPF connected in sequence.

[0090] It is worth mentioning that the dual-stream attention module acts on the fourth feature extraction layer to enhance the deep feature map generated by the backbone network.

[0091] Specifically, Figure 6 As shown, the image to be detected passes through the first convolution module to generate a first feature map P1, the first feature map P1 passes through the first feature extraction layer to generate a second feature map P2, the second feature map passes through the second feature extraction layer to generate a third feature map P3, and the third feature map P3 passes through the third feature extraction layer to generate a fourth feature map P4.

[0092] Specifically, when the fourth feature map P4 passes through the fourth feature extraction layer, it passes through the second convolution module and the first residual convolution module in sequence to obtain the sixth feature map P6. The sixth feature map P6 is enhanced with the channel information and feature information by the dual-stream attention module to obtain the seventh feature map P7. The seventh feature map P7 is passed through the spatial pyramid pooling fast module SPPF to obtain the fifth feature map P5 and output it to the neck network.

[0093] For example, Figure 6 As shown in the figure, the size of the image to be detected input to the rotation image detection model is 640*640*3. After the convolution operation, batch normalization operation and Swish activation function operation of the first convolution module, a feature map of 320*320*64 is obtained as the first feature map P1; the first feature map P1 is subjected to convolution operation, batch normalization operation and Swish activation function operation by the second convolution module of the first feature extraction layer, and the residual convolution operation is performed by the first residual convolution module to obtain a feature map of 160*160*128 as the second feature map P2; the second feature map P2 is subjected to convolution operation, batch normalization operation and Swish activation function operation by the second convolution module of the second feature extraction layer, and the residual convolution operation is performed by the first residual convolution module to obtain a feature map of 80*80*256 as the second feature map P 3; The third feature map P3 is subjected to convolution operation, batch normalization operation and Swish activation function operation by the second convolution module of the third feature extraction layer, and is subjected to residual convolution operation by the first residual convolution module to obtain a 40*40*512 feature map as the fourth feature map P4; The fourth feature map P4 is subjected to convolution operation, batch normalization operation and Swish activation function operation by the second convolution module of the fourth feature extraction layer, and is subjected to residual convolution operation by the first residual convolution module to obtain a 20*20*512 feature map as the sixth feature map P6; The sixth feature map P6 is subjected to channel information and feature information enhancement by the dual-stream attention module to obtain a 20*20*512 feature map as the seventh feature map P7; The seventh feature map P7 is subjected to SPPF to obtain a 20*20*512 feature map as the fifth feature map.

[0094] Preferably, if Figure 7 As shown, the dual-stream attention module includes a left branch, a right branch and a first matrix addition layer;

[0095] The left branch is used to enhance the channel information of the input sixth feature map, obtain the eighth feature map and input it to the first matrix addition layer;

[0096] The right branch is used to enhance the feature information of the input sixth feature map, obtain the ninth feature map and input it into the first matrix addition layer;

[0097] The eighth feature map and the ninth feature map are added in the first matrix addition layer to obtain the seventh feature map.

[0098] Specifically, in the dual-stream attention module, the left branch enhances the channel information of the sixth feature map, the right branch enhances the feature information of the sixth feature map, and the first matrix addition layer adds the eighth feature map and the ninth feature map to obtain the seventh feature map.

[0099] Preferably, if Figure 7 As shown, the left branch includes a maximum pooling layer, an average pooling layer, a first fully connected layer, a second fully connected layer, a second matrix addition layer, an activation function layer and a multiplication module;

[0100] The sixth feature map is subjected to the maximum pooling layer and the first fully connected layer to obtain the tenth feature map, and the sixth feature map is subjected to the average pooling layer and the second fully connected layer to obtain the eleventh feature map;

[0101] The tenth feature map and the eleventh feature map are added in the second matrix addition layer, and the added feature maps are passed through the activation function layer to obtain the twelfth feature map;

[0102] The twelfth feature map and the sixth feature map are multiplied in the multiplication module to obtain the eighth feature map.

[0103] Specifically, the maximum pooling layer and the average pooling layer in the left branch are pooling operations performed on the channel axis of the sixth feature map. Exemplarily, the sixth feature map P6 input to the left branch is width*height*channel, expressed as H1*W1*C1, then the output dimension after passing through the maximum pooling layer and the average pooling layer is 1*1*C1, and the subsequent first fully connected layer, second fully connected layer, second matrix addition layer, and activation function layer do not change the feature dimension. Finally, the input sixth feature map P6, which is H1*W1*C1, is multiplied by the multiplication module with the twelfth feature map output by the activation function layer. Figure 1 *1*C1, and multiply them to obtain the enhanced eighth feature map P8, which is H1*W1*C1.

[0104] Preferably, if Figure 7 As shown, the right branch includes a maximum pooling layer, an average pooling layer, a matrix concatenation layer, a convolution layer, an activation function layer and a multiplication module;

[0105] The sixth feature map is passed through the maximum pooling layer to obtain the thirteenth feature map, and the sixth feature map is passed through the average pooling layer to obtain the fourteenth feature map;

[0106] The thirteenth feature map and the fourteenth feature map are spliced ​​in the matrix splicing layer, and the spliced ​​feature maps are sequentially passed through the convolution layer and the activation function layer to obtain the fifteenth feature map;

[0107] The fifteenth feature map and the sixth feature map are multiplied in the multiplication module to obtain a ninth feature map.

[0108] Specifically, the maximum pooling layer and the average pooling layer in the right branch are pooling operations performed on the feature axis of the sixth feature map. Exemplarily, the sixth feature map P6 input to the right branch is width*height*channel, expressed as H1*W1*C1, then the output dimension after maximum pooling and average pooling is H1*W1*1, the subsequent matrix splicing layer splices the dimension to H1*W1*2, and the convolution layer reduces the dimension to H1*W1*1, the activation function layer does not change the dimension, and finally the multiplication module is used to multiply the input sixth feature map P6, which is H1*W1*C1, with the fifteenth feature map P15, which is H1*W1*1, output by the activation function layer, to obtain the enhanced ninth feature map, which is H1*W1*C1.

[0109] Specifically, the activation function layers in the left branch and the right branch are Sigmod activation function layers.

[0110] It is worth mentioning that the left branch and the right side of the dual-stream attention module respectively enhance the channel information and feature information of the input sixth feature map P6, and finally obtain the final enhanced seventh feature map P7 through the first matrix addition layer, thereby improving the feature extraction ability of the rotated image detection model and further improving the recognition accuracy of the image to be detected.

[0111] Specifically, Figure 7 As shown, the backbone network inputs the obtained third feature map P3, fourth feature map P4 and fifth feature map P5 into the neck network, the fifth feature map P5 is spliced ​​and fused with the fourth feature map through the first sampling fusion layer to obtain the first fused feature map S1; the first fused feature map S1 is spliced ​​and fused with the third feature map P3 through the second sampling fusion layer to obtain the second fused feature map S2; the second fused feature map S2 is spliced ​​and fused with the first fused feature map S1 through the first convolution fusion layer to obtain the third fused feature map S3; the third fused feature map S3 is spliced ​​and fused with the fifth feature map P5 through the second convolution fusion layer to obtain the fourth fused feature map S4.

[0112] Specifically, the first sampling fusion layer and the second sampling fusion layer have the same structure, both including an upsampling layer, a splicing layer and a second residual convolution module connected in sequence; the first convolution fusion layer and the second convolution fusion layer have the same structure, both including a third convolution module, a splicing layer and a third residual convolution module connected in sequence.

[0113] For example, Figure 6As shown, the third feature map P3 is 80*80*256, the fourth feature map P4 is 40*40*512, and the fifth feature map P5 is 20*20*512; the fifth feature map P5 is subjected to the upsampling layer of the first sampling fusion layer to obtain a feature map 40*40*512, which is input into the splicing layer, and is spliced ​​with the fourth feature map P4 in the splicing layer to obtain a feature map 40*40*512, and is subjected to the second residual convolution module to obtain a feature map 40*40*512 as the first fused feature map S1; similarly, the first fused feature map S1 is subjected to the upsampling layer of the second sampling fusion layer to obtain a feature map 80*80*512, which is input into the splicing layer, and is spliced ​​with the third feature map P3 in the splicing layer to obtain a feature map 80*80*768, and is subjected to the second residual convolution module to obtain a feature map 80*80*256 as the second fused feature map S2. Specifically, the second fused feature map S2 passes through the third convolution module of the first convolution fusion layer to obtain a feature map 40*40*256, which is input into the splicing layer, and is spliced ​​with the first fused feature map S1 in the splicing layer to obtain a feature map 40*40*768, and passes through the third residual convolution module to obtain a feature map 40*40*512, which is used as the third fused feature map S3; similarly, the third fused feature map S3 passes through the third convolution module of the second convolution fusion layer to obtain a feature map 20*20*512, which is input into the splicing layer, and is spliced ​​with the fifth feature map P5 in the splicing layer to obtain a feature map 20*20*1024, and passes through the third residual convolution module to obtain a feature map 20*20*512, which is used as the fourth fused feature map S4.

[0114] Preferably, if Figure 6 As shown, the terminal feature enhancement module includes three terminal feature enhancement modules with the same structure: a first terminal feature enhancement module, a second terminal feature enhancement module and a third terminal feature enhancement module;

[0115] The first terminal feature enhancement module, the second terminal feature enhancement module and the third terminal feature enhancement module respectively enhance the features of the input second fusion feature map, the third fusion feature map and the fourth fusion feature map, and output the fifth fusion feature map, the sixth fusion feature map and the seventh fusion feature map.

[0116] Specifically, the first end feature enhancement module, the second end feature enhancement module, and the third end feature enhancement module do not change the size of the input feature map. Figure 6 As shown, the second fused feature map S2, the third fused feature map S3 and the fourth fused feature map S4 are respectively subjected to the first terminal feature enhancement module, the second terminal feature enhancement module and the third terminal feature enhancement module to obtain the fifth fused feature map S5, the sixth fused feature map S6 and the seventh fused feature map S7.

[0117] Preferably, if Figure 8As shown, the first terminal feature enhancement module, the second terminal feature enhancement module or the third terminal feature enhancement module all include a maximum pooling layer, a third fully connected layer, a first activation function layer, a fourth fully connected layer, a second activation function layer and a multiplication module;

[0118] The input feature map passes through the maximum pooling layer, the third fully connected layer, the first activation function layer, the fourth fully connected layer and the second activation function layer in sequence to obtain the intermediate feature map;

[0119] The intermediate feature map and the input feature map are multiplied in the multiplication module to obtain the output feature map.

[0120] Specifically, the maximum pooling layer of the first terminal feature enhancement module, the second terminal feature enhancement module, or the third terminal feature enhancement module is a pooling operation performed on the channel axis of the input feature map. For example, the input feature map is expressed as H2*W2*C2, then the output dimension after maximum pooling is 1*1*C2, and the subsequent third fully connected layer, the first activation function layer, the fourth fully connected layer, and the second activation function layer do not change the size of the feature map to obtain the intermediate feature map. Figure 1 *1*C2, the feature map H2*W2*C2 input in the multiplication module and the intermediate feature Figure 1 *1*C2 are multiplied to obtain the enhanced feature map H2*W2*C2, that is, the output feature map is H2*W2*C2.

[0121] It is worth noting that the first terminal feature enhancement module, the second terminal feature enhancement module or the third terminal feature enhancement module enhances the feature map input to the detection head, thereby improving the recognition accuracy of the rotated image detection model for the image to be detected.

[0122] Specifically, in the first terminal feature enhancement module, the second terminal feature enhancement module or the third terminal feature enhancement module, the first activation function layer is a Swish activation function layer, and the second activation function layer is a Sigmod activation function layer.

[0123] Specifically, Figure 6 As shown, the fifth fused feature map S5, the sixth fused feature map S6 and the seventh fused feature map S7 are input to the detection head, and are predicted by the first prediction layer, the second prediction layer and the third prediction layer respectively, and all the prediction results are analyzed in the post-processing layer to obtain the predicted category and predicted rotated bounding box information of each target in the image to be detected.

[0124] Specifically, Figure 1 As shown, in step S3, the coordinates of the four corner points of each target are calculated according to the predicted rotated bounding box information of each target, and the coordinates of the four corner points of each target are connected as the predicted rotated bounding box of each target.

[0125] Preferably, the coordinates of the four corner points of each target are calculated by the following formula:

[0126]

[0127] in, Represents the predicted center point coordinates of each target, represents the prediction covariance matrix of each target, θ pred represents the predicted rotation angle of each object.

[0128] Specifically, in step S3, the coordinates of the four corner points are calculated based on the predicted center point coordinates, predicted covariance matrix and predicted rotation angle of each target. top-final ,y top-final )、(x right-final ,y right-final )、(x bottom-final ,y bottom-final ) and (x left-final ,y left-final ) represents the coordinates of the four corner points.

[0129] Specifically, when training the rotation image detection model, the category of each target in the preprocessed radiographic image is annotated to determine the category label of each target in a one-hot label format, such as [0,1,0,0,0,0], indicating that all targets include a total of 6 categories, and this target belongs to the second category.

[0130] Specifically, Figure 4 and Figure 5 As shown, the coordinates of the four corner points of the rotated bounding box refer to the coordinates of the four corner points of the rotated bounding box. After marking each target, the coordinates of the four corner points and the category label of each target can be determined.

[0131] Specifically, according to the coordinates of the four corner points of each target, the center point coordinates and the covariance matrix of the rotation bounding box of each target are calculated as the center point coordinate label and the covariance matrix label of each target.

[0132] Specifically, the center point coordinates and covariance matrix of the rotation bounding box of each target are calculated by the following formula:

[0133]

[0134] Among them, (x0, y0) represents the coordinates of the center point, (x1, y1), (x2, y2), (x3, y3) and (x4, y4) represent the coordinates of the four corner points, ∑ represents the covariance matrix, k = 1, 2, 3, 4.

[0135] Specifically, the category label, center point coordinate label and covariance matrix label of each target in each radiation image are obtained, and a radiation image and the category labels, center point coordinate labels and covariance matrix labels of each target in the image are used as a training sample to obtain several training samples, and the several training samples are divided into a training set and a validation set.

[0136] Specifically, the training set is used to train the rotation image detection model for a preset number of iterations to obtain a neural network model for each iteration. At the same time, the validation set is used to evaluate the neural network model for each iteration to obtain an evaluation result for each iteration. The evaluation results of the neural network models of all iterations are compared, and the neural network model corresponding to the optimal evaluation result is used as the trained rotation image detection model.

[0137] Exemplarily, the preset number of iterations is set to 100 in advance, 100 neural network models are obtained, and corresponding evaluation results are obtained at the same time. The 100 evaluation results are compared, and the neural network model with the best evaluation result is selected as the trained rotated image detection model.

[0138] Specifically, at the beginning of each iteration process, multiple training samples are randomly selected from the training set, and the multiple training samples are divided into multiple groups. For example, 10,000 training samples are selected and divided into 10 groups, with 1,000 training samples in each group. The training order of the 10 groups is determined, and the rotation image detection model is trained using each group of training samples in turn.

[0139] Specifically, after each group of training samples is input into the rotation image detection model, the predicted categories, predicted center point coordinates and predicted covariance matrix of all targets in each group of training samples are obtained, the loss function value of each group of training samples is calculated, and the weight parameters of the rotation image detection model are updated according to the loss function value. The updated rotation image detection model is used as the rotation image detection model for the next group of training samples, and the training process of the next group of training samples is started until all groups of training samples are trained.

[0140] Specifically, determine the category labels and predicted categories of all targets in each group of training samples, and calculate the category loss value using the following formula:

[0141]

[0142] Among them, L BCE represents the category loss value, M represents the number of all targets in all training samples of the current group, N represents the number of categories, and y i，j represents the predicted category value of the i-th category of the j-th target, p i，j Represents the class label value of the i-th class of the j-th target.

[0143] Specifically, determine the center point coordinate labels and covariance matrix labels of all targets in each group of training samples, as well as the predicted center point coordinates and predicted covariance matrix, and calculate the rotation bounding box loss value using the following formula:

[0144]

[0145] Among them, L ProbIoU represents the rotation bounding box loss value, BD represents the distance function, represents the predicted center point coordinates of the jth target, represents the predicted covariance matrix of the jth target, The center point coordinate label of the jth target, Denotes the covariance matrix label of the j-th target.

[0146] After obtaining the category loss value and rotation bounding box loss value of each group of training samples, the loss function value is calculated by the following formula:

[0147] L=ω1L BCE +ω2L ProbIoU ;

[0148] 1 = ω1 + ω2;

[0149] Among them, L represents the loss function value, ω1 represents the preset category weight value, and ω2 represents the preset rotation bounding box weight value.

[0150] Specifically, the loss function value corresponding to each group of training samples is calculated, and the weight parameters of the rotation image detection model are updated according to the preset gradient descent training method to obtain the updated rotation image detection model.

[0151] After all groups of training samples have completed training, the rotation image detection model obtained by the last group of training samples is evaluated using the validation set to obtain the evaluation results of this iteration.

[0152] Preferably, the preset gradient descent training method is any one of the following:

[0153] Stochastic gradient descent;

[0154] Adam gradient descent algorithm;

[0155] Batch Gradient Descent.

[0156] Specifically, the stochastic gradient descent (SGD) method has high computational efficiency, strong adaptability, and can jump out of the local minimum, which helps the rotation image detection model jump out of the local minimum and find the optimal solution.

[0157] Specifically, the Adam gradient descent algorithm combines the characteristics of the momentum gradient descent method and the RMSProp algorithm. It updates the model parameters through adaptive learning rate and second-order moment estimation, which can accelerate the convergence speed of the rotation image detection model in the gradient descent process and avoid falling into the local optimal solution.

[0158] Specifically, batch gradient descent has stable convergence and can obtain a stable rotation image detection model more quickly.

[0159] Specifically, after obtaining the evaluation result of this iteration, the rotation image detection model obtained in this iteration process is used to start the training process of the next iteration until the number of iterations meets the preset number of iterations. The evaluation results of the preset number of iterations are compared, and the rotation image detection model corresponding to the best evaluation result is selected as the trained rotation image detection model.

[0160] Compared with the prior art, a method for identifying a rotating target provided by an embodiment of the present invention predicts an image to be detected by a rotating image detection model to obtain a predicted category and a predicted rotating bounding box of each target in the image to be detected. The rotating bounding box can effectively constrain each target in the image, reduce the background part in each target, make the bounding box of each target clearer, and improve the recognition accuracy of each target in the image; at the same time, on the basis of the YOLOv8 neural network structure, a dual-stream attention module is added to enhance the channel information and feature information when the original backbone network generates a feature map, and a terminal feature enhancement module is added to enhance the feature map output by the original neck network, thereby achieving accurate recognition of the rotating target, avoiding the use of an instance segmentation model to increase the additional computing workload, and requiring fewer resources.

[0161] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0162] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A rotating target recognition method, characterized in that: The identification method comprises: Preprocess the image to be identified to obtain the image to be detected; Input the image to be detected into a pre-trained rotation image detection model to obtain the predicted category and predicted rotation bounding box information of each target in the image to be detected; the predicted rotation bounding box information includes the predicted center point coordinates, predicted covariance matrix and predicted rotation angle of the rotation bounding box; wherein the rotation image detection model is trained based on an improved YOLOv8 neural network structure, and the improved YOLOv8 neural network structure includes a dual-stream attention module and a terminal feature enhancement module, the dual-stream attention module is used to enhance channel information and feature information when the original backbone network generates a feature map, and the terminal feature enhancement module is used to enhance the features of the feature map output by the original neck network; The coordinates of the four corner points of each target are calculated according to the predicted rotation bounding box information of each target, and the coordinates of the four corner points of each target are connected as the predicted rotation bounding box of each target.

2. The identification method according to claim 1, characterized in that: The rotation image detection model includes a backbone network, a neck network and a detection head; The backbone network includes an original backbone network and a dual-stream attention module; the original backbone network is used to extract features from the input image to be detected, obtain the third feature map, the fourth feature map and the fifth feature map and output them to the neck network; The dual-stream attention module enhances the channel information and feature information when the original backbone network performs feature extraction to generate the third to fifth feature maps; The neck network includes the original neck network and the terminal feature enhancement module; The original neck network is used to perform feature fusion on the third feature map, the fourth feature map and the fifth feature map to obtain the second fused feature map, the third fused feature map and the fourth fused feature map and output them to the terminal feature enhancement module; The terminal feature enhancement module is used to enhance the features of the second fusion feature map, the third fusion feature map and the fourth fusion feature map respectively, obtain the enhanced fifth fusion feature map, the sixth fusion feature map and the seventh fusion feature map and output them to the detection head; The detection head is used to predict the predicted category and predicted rotation bounding box information of each target in the image to be detected according to the fifth fused feature map, the sixth fused feature map and the seventh fused feature map.

3. The identification method according to claim 2, characterized in that: The backbone network includes a first convolution module, a first feature extraction layer, a second feature extraction layer, a third feature extraction layer and a fourth feature extraction layer connected in sequence, and outputs a first feature map, a second feature map, a third feature map, a fourth feature map and a fifth feature map respectively; The first feature extraction layer, the second feature extraction layer and the third feature extraction layer have the same structure, and all include a second convolution module and a first residual convolution module connected in sequence; the fourth feature extraction layer includes a second convolution module, a first residual convolution module, a dual-stream attention module and a spatial pyramid pooling fast module SPPF connected in sequence.

4. The identification method according to claim 3, characterized in that: The dual-stream attention module includes a left branch, a right branch and a first matrix addition layer; The left branch is used to enhance the channel information of the input sixth feature map, obtain the eighth feature map and input it into the first matrix addition layer; The right branch is used to enhance the feature information of the input sixth feature map, obtain the ninth feature map and input it into the first matrix addition layer; The eighth feature map and the ninth feature map are added in the first matrix addition layer to obtain the seventh feature map.

5. The identification method according to claim 4, characterized in that: The left branch includes a maximum pooling layer, an average pooling layer, a first fully connected layer, a second fully connected layer, a second matrix addition layer, an activation function layer and a multiplication module; The sixth feature map is subjected to the maximum pooling layer and the first fully connected layer to obtain the tenth feature map, and the sixth feature map is subjected to the average pooling layer and the second fully connected layer to obtain the eleventh feature map; The tenth feature map and the eleventh feature map are added in the second matrix addition layer, and the added feature maps are passed through the activation function layer to obtain the twelfth feature map; The twelfth feature map and the sixth feature map are multiplied in the multiplication module to obtain the eighth feature map.

6. The identification method according to claim 4, characterized in that: The right branch includes a maximum pooling layer, an average pooling layer, a matrix splicing layer, a convolution layer, an activation function layer and a multiplication module; The sixth feature map is passed through the maximum pooling layer to obtain the thirteenth feature map, and the sixth feature map is passed through the average pooling layer to obtain the fourteenth feature map; The thirteenth feature map and the fourteenth feature map are spliced ​​in the matrix splicing layer, and the spliced ​​feature maps are sequentially passed through the convolution layer and the activation function layer to obtain the fifteenth feature map; The fifteenth feature map and the sixth feature map are multiplied in the multiplication module to obtain a ninth feature map.

7. The identification method according to claim 2, characterized in that: The terminal feature enhancement module includes three identical structures: a first terminal feature enhancement module, a second terminal feature enhancement module and a third terminal feature enhancement module; The first terminal feature enhancement module, the second terminal feature enhancement module and the third terminal feature enhancement module respectively enhance the features of the input second fusion feature map, the third fusion feature map and the fourth fusion feature map, and output the fifth fusion feature map, the sixth fusion feature map and the seventh fusion feature map.

8. The identification method according to claim 7, characterized in that: The first terminal feature enhancement module, the second terminal feature enhancement module or the third terminal feature enhancement module all include a maximum pooling layer, a third fully connected layer, a first activation function layer, a fourth fully connected layer, a second activation function layer and a multiplication module; The input feature map is sequentially passed through the maximum pooling layer, the third fully connected layer, the first activation function layer, the fourth fully connected layer, and the second activation function layer to obtain the intermediate feature map; The intermediate feature map and the input feature map are multiplied in the multiplication module to obtain the output feature map.

9. The identification method according to claim 1, characterized in that: The preprocessing includes one or more of the following operations: Grayscale; Binarization; Normalization; Zoom; Cutting; Sharpening; Filtering.

10. The identification method according to claim 1, characterized in that: The coordinates of the four corner points of each target are calculated using the following formula: in, Represents the predicted center point coordinates of each target, represents the prediction covariance matrix of each target, θ pred represents the predicted rotation angle of each object.

Images