Target recognition method based on fused semantic features

Abstract

The invention discloses a target recognition method based on fused semantic features, and the method comprises the steps: obtaining a target infrared image and a maneuvering type corresponding to the target infrared image, and enabling the target infrared image to comprise a target; inputting the target infrared image and the maneuvering category into a target recognition network to obtain a first maneuvering semantic feature, a shallow image feature and a deep image feature; performing feature fusion on the first maneuvering semantic feature and the shallow image feature to obtain a first image fusion feature; performing feature fusion on the first maneuvering semantic feature and the deep image feature to obtain a second image fusion feature; performing up-sampling and splicing operation on the first image fusion feature and the second image fusion feature to obtain multi-scale feature representation; and inputting the multi-scale feature representation into an output head for convolution operation to obtain a target category and an anti-interference identification result. Therefore, the accuracy of target identification is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target recognition, and in particular to a target recognition method based on fused semantic features. BACKGROUND

[0002] In an infrared target recognition task, an aerial infrared target often appears to be partially or largely occluded for a short time due to changes in the maneuvering posture and interference, resulting in incomplete target features of the image and making it difficult for the algorithm to stably recognize the target. In the presence of interference, the form and part of the features of the target have high similarity, which also leads to an increase in the misidentification and missed identification of the foregoing recognition algorithm and network. Therefore, there is an urgent need for a target recognition method that can effectively deal with the problems of unstable recognition and decreased accuracy caused by partial occlusion of the target and high similarity between the target and the interference in a complex surface source interference scene. SUMMARY

[0003] In view of this, the present application provides a target recognition method based on fused semantic features, which is used to solve the problems of unstable recognition and decreased accuracy caused by partial occlusion of the target and high similarity between the target and the interference in a complex surface source interference scene in the prior art.

[0004] The object of the present application can be achieved by the following technical solutions: The first aspect of the present application provides a target recognition method based on fused semantic features, comprising: obtaining a target infrared image and a maneuvering category corresponding to the target infrared image, the target infrared image containing a target; inputting the target infrared image and the maneuvering category into a target recognition network to obtain a first maneuvering semantic feature, a shallow image feature, and a deep image feature; performing feature fusion on the first maneuvering semantic feature and the shallow image feature to obtain a first image fusion feature; performing feature fusion on the first maneuvering semantic feature and the deep image feature to obtain a second image fusion feature; performing upsampling and splicing operations on the first image fusion feature and the second image fusion feature to obtain a multi-scale feature representation; inputting the multi-scale feature representation into an output head to perform convolution operations to obtain a target category and an anti-interference recognition result.

[0005] In an optional embodiment, the target recognition network includes a learnable maneuvering semantic feature space, an ERBlock module, and a SimSPFF module, the target infrared image and the maneuvering category are input into the target recognition network to obtain the first maneuvering semantic feature, the shallow image feature, and the deep image feature, comprising: determining the first maneuvering semantic feature corresponding to the maneuvering category in the learnable maneuvering semantic feature space according to the maneuvering category; obtain the shallow image feature through the ERBlock module; obtain the deep image feature through the SimSPFF module.

[0006] In an optional embodiment, according to the maneuver category, the first maneuver semantic feature corresponding to the maneuver category is determined in the learnable maneuver semantic feature space, comprising: According to the maneuver category, the first maneuver semantic feature corresponding to the maneuver category is determined in the learnable maneuver semantic feature space through the following formula:

[0007] wherein, is the first maneuver semantic feature corresponding to the maneuver category, is the maneuver category, is the learnable maneuver semantic feature space, , represents the real number field, is the number of maneuver categories, is the number of rows of the feature matrix corresponding to the target infrared image, is the number of columns of the feature matrix corresponding to the target infrared image, represents the operation of extracting the first maneuver semantic feature from the learnable maneuver semantic feature space according to the maneuver category.

[0008] In an optional embodiment, the target recognition network further comprises a UAFFM, which performs feature fusion on the first maneuver semantic feature and the shallow image feature to obtain a first image fusion feature, comprising: The UAFFM performs feature fusion on the first maneuver semantic feature and the shallow image feature to obtain the first image fusion feature.

[0009] In an optional embodiment, the UAFFM comprises a convolution layer, a spatial attention unit, a channel attention unit and a Fourier unit, the UAFFM performs feature fusion on the first maneuver semantic feature and the shallow image feature to obtain the first image fusion feature, comprising: The convolution layer is used to adjust the number of channels of the first maneuver semantic feature to the same as the number of channels of the shallow image feature, to obtain a second maneuver semantic feature; The second maneuver semantic feature is up-sampled to adjust the size of the second maneuver semantic feature to the same as the size of the shallow image feature, to obtain a third maneuver semantic feature; The third maneuver semantic feature and the shallow image feature are input into the spatial attention unit to obtain a first spatial attention weight and a second spatial attention weight, and the sum of the first spatial attention weight and the second spatial attention weight is 1; The third mobile semantic feature and the shallow image feature are input into the channel attention unit to obtain a first channel attention weight and a second channel attention weight, and a sum value of the first channel attention weight and the second channel attention weight is 1; The product of the third mobile semantic feature and the first spatial attention weight, the product of the third mobile semantic feature and the first channel attention weight, the product of the shallow image feature and the second spatial attention weight, and the product of the shallow image feature and the second channel attention weight are added to obtain an attention weighted feature; The attention weighted feature is input into the Fourier unit to obtain a first image fusion feature.

[0010] In an optional embodiment, inputting the attention weighted feature into the Fourier unit to obtain the first image fusion feature includes: The attention weighted feature is input into the Fourier unit to perform a Fourier transform operation on the attention weighted feature to obtain a first frequency domain attention weighted feature; The first frequency domain attention weighted feature is processed in a frequency domain to obtain a second frequency domain attention weighted feature; An inverse Fourier transform operation is performed on the second frequency domain attention weighted feature to obtain the first image fusion feature.

[0011] In an optional embodiment, the frequency domain processing includes at least one of the following operations: A convolution operation, a batch normalization operation, and an activation function application operation.

[0012] In an optional embodiment, the target recognition network further includes a CASFM, and the CASFM is configured to perform feature fusion on the first mobile semantic feature and the deep image feature to obtain a second image fusion feature, including: The CASFM is configured to perform feature fusion on the first mobile semantic feature and the deep image feature to obtain the second image fusion feature.

[0013] In an optional embodiment, the CASFM includes a convolution layer, a first cross-attention unit, and a second cross-attention unit, the first cross-attention unit includes a first key weight matrix, a first query weight matrix, and a first value weight matrix, the second cross-attention unit includes a second key weight matrix, a second query weight matrix, and a second value weight matrix, and the CASFM is configured to perform feature fusion on the first mobile semantic feature and the deep image feature to obtain the second image fusion feature, including: The convolution layer is configured to adjust a channel number of the first mobile semantic feature to be the same as a channel number of the deep image feature to obtain a fourth mobile semantic feature; The fourth mobile semantic feature is input into the first key weight matrix, the first query weight matrix, and the first value weight matrix respectively to obtain a first key vector, a first query vector, and a first value vector; The deep image feature is input into the second key weight matrix, the second query weight matrix, and the second value weight matrix respectively to obtain a second key vector, a second query vector, and a second value vector; The deep image feature is taken as a query feature, and cross-attention calculation is performed based on the second query vector, the first key vector, and the first value vector by using a scaled dot-product attention mechanism formula to obtain a first attention output feature; The fourth mobile semantic feature is taken as a query feature, and cross-attention calculation is performed based on the first query vector, the second key vector, and the second value vector by using a scaled dot-product attention mechanism formula to obtain a second attention output feature; The first attention output feature and the fourth mobile semantic feature are added to obtain a third image fusion feature; The second attention output feature and the deep image feature are added to obtain a fourth image fusion feature; The third image fusion feature and the fourth image fusion feature are spliced to obtain a second image fusion feature.

[0014] In an optional embodiment, the CASFM further includes a convolution block unit, and further includes: The second image fusion feature is subjected to a stacked convolution operation by the convolution block unit.

[0015] The second aspect of the application provides a target recognition device based on fusion semantic features, which includes: An acquisition module is configured to acquire a target infrared image and a mobile category corresponding to the target infrared image, and the target infrared image contains a target. A first input module is configured to input the target infrared image and the mobile category into a target recognition network to obtain a first mobile semantic feature, a shallow image feature, and a deep image feature. A first feature fusion module is configured to perform feature fusion on the first mobile semantic feature and the shallow image feature to obtain a first image fusion feature. A second feature fusion module is configured to perform feature fusion on the first mobile semantic feature and the deep image feature to obtain a second image fusion feature. A sampling and splicing module is configured to perform upsampling and splicing operations on the first image fusion feature and the second image fusion feature to obtain a multi-scale feature representation. A second input module is configured to input the multi-scale feature representation into an output head to perform convolution operations to obtain a target category and an anti-interference recognition result.

[0016] The third aspect of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the method of the first aspect.

[0017] The fourth aspect of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method of the first aspect.

[0018] Compared with the prior art, the target recognition method based on fused semantic features provided by the present application acquires a target infrared image and a corresponding maneuvering category of the target infrared image, the target infrared image containing a target; inputs the target infrared image and the maneuvering category into a target recognition network to obtain first maneuvering semantic features, shallow image features and deep image features; fuses the first maneuvering semantic features and the shallow image features to obtain first image fusion features; fuses the first maneuvering semantic features and the deep image features to obtain second image fusion features; performs upsampling and splicing operations on the first image fusion features and the second image fusion features to obtain multi-scale feature representations; inputs the multi-scale feature representations into an output head to perform convolution operations to obtain a target category and an anti-interference recognition result. In this way, the accuracy of target recognition is improved. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0020] Figure 1 A flowchart of the target recognition method based on fused semantic features provided by the embodiments of the present application; Figure 2 A flowchart of inputting the maneuvering category into the learnable maneuvering semantic feature space to obtain the first maneuvering semantic features provided by the embodiments of the present application; Figure 3 A flowchart of fusing the first maneuvering semantic features and the shallow image features by the UAFFM to obtain the first image fusion features provided by the embodiments of the present application; Figure 4 A flowchart of fusing the first maneuvering semantic features and the deep image features by the CASFM to obtain the second image fusion features provided by the embodiments of the present application; Figure 5A structural block diagram of a target recognition device based on fused semantic features provided by an embodiment of the present application is provided. Figure 6 A structural block diagram of an electronic device for implementing a target recognition method based on fused semantic features provided by an embodiment of the present application is provided. DETAILED DESCRIPTION

[0021] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative work should fall within the scope of protection of the present application.

[0022] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product, or device.

[0023] It should be understood that in the embodiments of the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" is only a description of the association relationship between the associated objects, which means that there can be three relationships, for example, A and / or B can represent the three cases of A alone, A and B together, and B alone. The character " / " generally represents an "or" relationship between the associated objects. "Including A, B, and / or C" means including any one or any two or three of A, B, and C.

[0024] It should be understood that in the embodiments of the present application, "B corresponding to A", "B corresponding to A", "A corresponding to B", or "B corresponding to A" means that B is associated with A, and B can be determined according to A. Determining B according to A does not mean that B is determined only according to A, but B can also be determined according to A and / or other information.

[0025] To solve the technical problems existing in the related art, the embodiments of the present application provide a target recognition method based on fused semantic features.

[0026] The target recognition method based on fused semantic features provided in the embodiments of the present application can be executed by an electronic device, which can be a terminal or a server. The terminal can be a terminal device such as a smartphone, a tablet computer, or a laptop computer. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. It is understandable that the present application does not specifically limit the execution entity of the target recognition method based on fused semantic features.

[0027] The technical solution of the present application is described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments described below are used to explain the technical solution of the present application and are not intended to be used as limitations for actual use.

[0028] In order to solve the technical problems existing in the related art, the embodiment of the present application provides a target recognition method based on fusion semantic features, such as Figure 1 As shown, Figure 1 This is a flowchart of a method for object recognition based on fusion semantic features provided in an embodiment of the present application. It should be noted that the steps shown in the flowchart can be performed in a different logical order than that shown in the flowchart of the method. The method may include the following steps S101 to S106.

[0029] Step S101: Acquire a target infrared image and a maneuver category corresponding to the target infrared image.

[0030] It should be noted that the target infrared image is a thermal radiation image captured by an infrared imaging device, which includes the target. The target infrared image can be represented in matrix form, where the value of each pixel reflects the thermal radiation intensity at the corresponding position.

[0031] The maneuver category is the result of classifying the maneuver performed by the target. The maneuver category can be manually pre-labeled or determined based on a preset classification model, which is not limited in this application.

[0032] Step S102: Input the target infrared image and maneuver category into the target recognition network to obtain the first maneuver semantic feature, shallow image feature and deep image feature.

[0033] In an alternative embodiment, the target recognition network comprises a learnable maneuver semantic feature space, an ERBlock (Enhanced Relationship Block) module and a SimSPFF (Simplified Spatial Pyramid Pooling-Fast) module, the target infrared image and the maneuver category are input into the target recognition network to obtain a first maneuver semantic feature, a shallow image feature and a deep image feature, and the method comprises the following steps: determining the first maneuver semantic feature corresponding to the maneuver category in the learnable maneuver semantic feature space according to the maneuver category; obtaining the shallow image feature through the ERBlock module; and obtaining the deep image feature through the SimSPFF module.

[0034] In a specific embodiment, the spatial dimension size of the learnable maneuver semantic feature space is 7, corresponding to 6 maneuver categories of the target and an un-recognized maneuver category, and the 6 maneuver categories can be level flight, left maneuver, right maneuver, jump-up, roll and dive. Of course, the maneuver category can also be another category, which is not limited in the present application. In a more specific embodiment, as shown in Figure 2 Figure 2 The present application provides a flowchart for inputting the maneuver category into the learnable maneuver semantic feature space to obtain the first maneuver semantic feature.

[0035] The learnable maneuver semantic feature space can be continuously optimized to obtain more robust first maneuver semantic features.

[0036] In a more specific embodiment, the first maneuver semantic feature corresponding to the maneuver category in the learnable maneuver semantic feature space is determined according to the maneuver category, comprising: The first maneuver semantic feature corresponding to the maneuver category in the learnable maneuver semantic feature space is determined according to the maneuver category by the following formula: (1) wherein, is the first maneuver semantic feature corresponding to the maneuver category, is the maneuver category, is the learnable maneuver semantic feature space, , represents the real number field, is the number of maneuver categories, is the number of rows of the feature matrix corresponding to the target infrared image, is the number of columns of the feature matrix corresponding to the target infrared image, represents the operation of extracting the first maneuver semantic feature from the learnable maneuver semantic feature space according to the maneuver category.​

[0037] In a more specific embodiment, the ERBlock module is suitable for shallow or middle-level image feature extraction, which can enhance the relationship modeling between the target and the background in the target infrared image. The SimSPFF module is suitable for deep image feature extraction, which helps to improve the recognition ability of targets of different distances and sizes in the target infrared image. It should be noted that shallow image features, middle-level image features and deep image features are features extracted at different levels when the convolutional neural network processes the image. Shallow image features are usually located in the initial layers of the convolutional neural network, and deep image features are usually located at the end of the convolutional neural network.

[0038] Step S103: performing feature fusion on the first motor semantic feature and the shallow image feature to obtain a first image fusion feature.

[0039] In an optional embodiment, the object recognition network further includes a Unified Attention Fourier Fusion Module (UAFFM), which performs feature fusion on the first motor semantic feature and the shallow image feature to obtain a first image fusion feature. Specifically, the following steps are included: The first motor semantic feature and shallow image feature are fused through UAFFM to obtain the first image fusion feature.

[0040] In a specific embodiment, Figure 3 As shown, Figure 3 A flow chart of a process for fusing the first motor semantic feature and the shallow image feature by UAFFM to obtain the first image fusion feature is provided in the embodiment of the present application. Figure 3 In [1], UAFFM includes convolutional layers, spatial attention units, channel attention units, and Fourier units. UAFFM fuses the first motor semantic features and shallow image features to obtain the first image fusion features. Specifically, the following steps are included: The channel number of the first mobile semantic feature is adjusted to be the same as the channel number of the shallow image feature by using a convolutional layer to obtain a second mobile semantic feature; the size of the second mobile semantic feature is adjusted to be the same as the size of the shallow image feature by upsampling the second mobile semantic feature to obtain a third mobile semantic feature; the third mobile semantic feature and the shallow image feature are input into a spatial attention unit to obtain a first spatial attention weight and a second spatial attention weight, and the sum of the first spatial attention weight and the second spatial attention weight is 1; the third mobile semantic feature and the shallow image feature are input into a channel attention unit to obtain a first channel attention weight and a second channel attention weight, and the sum of the first channel attention weight and the second channel attention weight is 1; the product of the third mobile semantic feature and the first spatial attention weight, the product of the third mobile semantic feature and the first channel attention weight, the product of the shallow image feature and the second spatial attention weight, and the product of the shallow image feature and the second channel attention weight are added to obtain an attention weighted feature; and the attention weighted feature is input into a Fourier unit to obtain a first image fusion feature.

[0041] In a more specific embodiment, the size of the second mobile semantic feature is adjusted to be the same as the size of the shallow image feature by upsampling the second mobile semantic feature to obtain a third mobile semantic feature, specifically comprising the following steps: the UAFFM adjusts the size of the second mobile semantic feature to be the same as the size of the shallow image feature by upsampling the second mobile semantic feature through linear interpolation to obtain the third mobile semantic feature.

[0042] In a more specific embodiment, the size of the second mobile semantic feature is adjusted to be the same as the size of the shallow image feature by upsampling the second mobile semantic feature to obtain a third mobile semantic feature, specifically comprising the following steps: the UAFFM adjusts the size of the second mobile semantic feature to be the same as the size of the shallow image feature by upsampling the second mobile semantic feature through linear interpolation to obtain the third mobile semantic feature. (2) wherein, is the third mobile semantic feature, denotes an upsampling operation, denotes the second mobile semantic feature.

[0043] In a more specific embodiment, the third mobile semantic feature and the shallow image feature are input into a spatial attention unit to obtain a first spatial attention weight by the following formula: (3) wherein, is the first spatial attention weight, denotes spatial attention, is the third mobile semantic feature, is the shallow image feature.

[0044] In a more specific embodiment, the third mobile semantic feature and the shallow image feature are input into the channel attention unit to obtain the first channel attention weight by the following formula: (4) wherein, is the first channel attention weight, denotes channel attention, is the third mobile semantic feature, is the shallow image feature.

[0045] In a more specific embodiment, the product of the third mobile semantic feature and the first spatial attention weight, the product of the third mobile semantic feature and the first channel attention weight, the product of the shallow image feature and the second spatial attention weight, and the product of the shallow image feature and the second channel attention weight are added to obtain the attention weighted feature by the following formula: (5) wherein, is the attention weighted feature, is the third mobile semantic feature, is the first spatial attention weight, is the shallow image feature, is the first channel attention weight.

[0046] In a more specific embodiment, the attention weighted feature is input into the Fourier unit to obtain the first image fusion feature by the following formula: (6) wherein, is the first image fusion feature, is the attention weighted feature, denotes Fourier transform.

[0047] In a more specific embodiment, the attention weighted feature is input into the Fourier unit to obtain the first image fusion feature, specifically comprising the following operations: the attention weighted feature is input into the Fourier unit, and a Fourier transform operation is performed on the attention weighted feature to obtain a first frequency domain attention weighted feature; the first frequency domain attention weighted feature is processed in the frequency domain to obtain a second frequency domain attention weighted feature; an inverse Fourier transform operation is performed on the second frequency domain attention weighted feature to obtain the first image fusion feature.

[0048] In a more specific embodiment, the frequency domain processing comprises at least one of the following operations: convolution operation, batch normalization operation, and application of an activation function operation.

[0049] In a more specific embodiment, the first frequency domain attention weighted feature is subjected to frequency domain processing to obtain a second frequency domain attention weighted feature, specifically including the following steps: The convolution operation, the batch normalization operation and the activation function operation are sequentially performed on the first frequency domain attention weighted feature.

[0050] Step S104: The first mobile semantic feature and the deep image feature are subjected to feature fusion to obtain a second image fusion feature.

[0051] In an optional embodiment, the target recognition network further includes a CASFM (Cross Attention Stacked Fusion Module) for subjecting the first mobile semantic feature and the deep image feature to feature fusion to obtain the second image fusion feature, including: The first mobile semantic feature and the deep image feature are subjected to feature fusion by the CASFM to obtain the second image fusion feature.

[0052] In a specific embodiment, the CASFM includes a convolution layer, a first cross attention unit and a second cross attention unit, the first cross attention unit includes a first key weight matrix, a first query weight matrix and a first value weight matrix, the second cross attention unit includes a second key weight matrix, a second query weight matrix and a second value weight matrix, the first mobile semantic feature and the deep image feature are subjected to feature fusion by the CASFM to obtain the second image fusion feature, specifically including the following steps: the convolution layer is used to adjust the channel number of the first mobile semantic feature to be the same as the channel number of the deep image feature to obtain a fourth mobile semantic feature; the fourth mobile semantic feature is input into the first key weight matrix, the first query weight matrix and the first value weight matrix respectively to obtain a first key vector, a first query vector and a first value vector; the deep image feature is input into the second key weight matrix, the second query weight matrix and the second value weight matrix respectively to obtain a second key vector, a second query vector and a second value vector; the deep image feature is taken as a query feature, cross attention calculation is performed based on the second query vector, the first key vector and the first value vector by using a scaled dot-product attention mechanism formula to obtain a first attention output feature; the fourth mobile semantic feature is taken as a query feature, cross attention calculation is performed based on the first query vector, the second key vector and the second value vector by using the scaled dot-product attention mechanism formula to obtain a second attention output feature; the first attention output feature and the fourth mobile semantic feature are added to obtain a third image fusion feature; the second attention output feature and the deep image feature are added to obtain a fourth image fusion feature; the third image fusion feature and the fourth image fusion feature are spliced to obtain the second image fusion feature. As shown in Figure 4 Figure 4 ​A flowchart of an embodiment of the present application is provided for fusing first motor semantic features and deep image features through CASFM to obtain second image fusion features.

[0053] In a more specific embodiment, the fourth motorized semantic feature is input into the first key weight matrix, the first query weight matrix, and the first value weight matrix, respectively, to obtain a first key vector, a first query vector, and a first value vector; the deep image feature is input into the second key weight matrix, the second query weight matrix, and the second value weight matrix, respectively, to obtain a second key vector, a second query vector, and a second value vector, which can be expressed by the following expression: (7) in, is the first query vector, is the first bond vector, is the first value vector, is the fourth mobile semantic feature, The first query matrix, is the first bond matrix, is the first value matrix, is the second query vector, is the second key vector, is the second value vector, is the deep image feature, The second query matrix, is the second bond matrix, is the second value matrix.

[0054] In a more specific embodiment, the deep image feature is used as the query feature, and the scaled dot product attention mechanism formula is used to perform cross attention calculation based on the second query vector, the first key vector, and the first value vector to obtain the first attention output feature through the following formula: (8) in, is the first attention output feature, is the first bond vector The transpose of is the first bond vector Dimensions, is the second query vector, is the first value vector.

[0055] In a more specific embodiment, the fourth motorized semantic feature is used as the query feature, and a cross-attention calculation is performed based on the first query vector, the second key vector, and the second value vector using the scaled dot product attention mechanism formula to obtain the second attention output feature through the following formula: (9) in, Output features for the second attention, is the second key vector The transpose of is the second key vector Dimensions, is the first query vector, is the second value vector.

[0056] In a more specific embodiment, the first attention output feature and the fourth motorized semantic feature are added together to obtain a third image fusion feature; the second attention output feature and the deep image feature are added together to obtain a fourth image fusion feature; and the third image fusion feature and the fourth image fusion feature are concatenated to obtain a second image fusion feature, using the following formula: (10) in, is the second image fusion feature, is the splicing function, is a standard convolution operation, is the first attention output feature, is the fourth mobile semantic feature, Output features for the second attention, is the deep image feature.

[0057] In another more specific embodiment, the CASFM further includes a convolution block unit, further including: The second image fusion feature is subjected to a stacked convolution operation through a convolution block unit.

[0058] In a more specific embodiment, the convolutional block unit includes a convolutional layer, a dropout layer, a batch normalization layer, and an activation layer. The convolutional layer enables the network to better capture the complex relationship between deep image features and motor features, and extract detailed features from the fused features. The dropout layer introduces randomness, reduces the strong dependency between features, and effectively prevents overfitting. Batch normalization improves training stability. The final activation layer uses the LeakyReLU (LeakyRectified Linear Uni) function, which can alleviate the "zero gradient problem" of the linear rectifier unit in the negative region and enhance the network's nonlinear fitting ability.

[0059] In a more specific embodiment, a stacked convolution operation is performed on the second image fusion feature using the following formula: (11) (12) (13) wherein, is a second image fusion feature after a stacked convolution operation, and is an intermediate feature map, denotes a batch normalization operation, denotes a regularization technique, is a standard convolution operation, is a second image fusion feature.

[0060] Step S105: performing upsampling and concatenation operations on the first image fusion feature and the second image fusion feature to obtain a multi-scale feature representation.

[0061] In an optional embodiment, the first image fusion feature and the second image fusion feature are upsampled by means of bilinear interpolation or deconvolution.

[0062] In this step, the new feature map obtained after upsampling and concatenation operations contains information from different scales, i.e., a multi-scale feature representation. The multi-scale feature representation combines global information at low resolution and local details at high resolution, which helps the model learn complex patterns and structures more effectively.

[0063] Step S106: inputting the multi-scale feature representation into an output head for convolution operation to obtain a target class and an anti-interference recognition result.

[0064] It should be noted that the output head is designed according to the task requirements and application scenarios.

[0065] In an optional embodiment, the output head includes at least one of the following components: a convolution layer, an activation function, a batch normalization, a global pooling, and a fully connected layer.

[0066] In an optional embodiment, the anti-interference recognition result includes at least one of the following: accuracy, stability, robustness, and generalization ability.

[0067] It also needs to be explained that the experimental results show that, after introducing the UAFFM module, the precision of the algorithm is improved from 90.4% to 94.3% by 3.9%, indicating that the UAFFM module effectively enhances the fusion of shallow image features and maneuvering features by introducing spatial attention, channel attention and Fourier module, so that the model can more accurately detect the target position and improve the precision. The recall rate is improved from 81.7% to 82.4% by 0.7%, which shows that the UAFFM module has certain improvement on the recall ability of the network. After adding the CASFM module to the basic network, the Precision of the network is improved by 3.1%, and the Recall of the network is improved by 1.6%, indicating that the design of cross-attention stacking in the CASFM module can more fully capture the complex correlation between maneuvering features and deep image features, and improve the detection accuracy of the network. When the UAFFM module and the CASFM module are added to the basic network at the same time, the Precision is improved to 95.1%, and the Recall is improved to 84.6%. Experiments show that the UAFFM module and the CASFM module effectively improve the anti-interference recognition performance of the network on infrared images through the fusion of maneuvering semantic features and image features.

[0068] Corresponding to the target recognition method based on fused semantic features provided by the embodiments of the present application, the embodiments of the present application also provide a target recognition device based on fused semantic features, as shown in Figure 5 The target recognition device based on fused semantic features includes: The acquisition module 501 is configured to acquire a target infrared image and a maneuvering category corresponding to the target infrared image, and the target infrared image contains a target. The first input module 502 is configured to input the target infrared image and the maneuvering category into a target recognition network to obtain a first maneuvering semantic feature, a shallow image feature and a deep image feature. The first feature fusion module 503 is configured to perform feature fusion on the first maneuvering semantic feature and the shallow image feature to obtain a first image fusion feature. The second feature fusion module 504 is configured to perform feature fusion on the first maneuvering semantic feature and the deep image feature to obtain a second image fusion feature. The sampling and splicing module 505 is configured to perform upsampling and splicing operations on the first image fusion feature and the second image fusion feature to obtain a multi-scale feature representation. The second input module 506 is configured to input the multi-scale feature representation into an output head to perform convolution operation to obtain a target category and an anti-interference recognition result.

[0069] In an optional embodiment, the target recognition network includes a learnable maneuvering semantic feature space, an ERBlock module and a SimSPFF module, and the first input module is specifically configured to: According to the maneuver category, a first maneuver semantic feature corresponding to the maneuver category is determined in a learnable maneuver semantic feature space. The shallow image feature is obtained through the ERBlock module. The deep image feature is obtained through the SimSPFF module.

[0070] In an optional embodiment, the first input module is specifically configured to: According to the maneuver category, a first maneuver semantic feature corresponding to the maneuver category is determined in a learnable maneuver semantic feature space through the following formula:

[0071] wherein, is the first maneuver semantic feature corresponding to the maneuver category, is the maneuver category, is the learnable maneuver semantic feature space, , represents a real number field, is the number of maneuver categories, is the number of rows of the feature matrix corresponding to the target infrared image, is the number of columns of the feature matrix corresponding to the target infrared image, represents an operation of extracting the first maneuver semantic feature from the learnable maneuver semantic feature space according to the maneuver category.

[0072] In an optional embodiment, the target recognition network further comprises a UAFFM, and the first feature fusion module is specifically configured to: The first maneuver semantic feature and the shallow image feature are fused through the UAFFM to obtain a first image fusion feature.

[0073] In an optional embodiment, the UAFFM comprises a convolution layer, a spatial attention unit, a channel attention unit and a Fourier unit, and the first feature fusion module is specifically configured to: The channel number of the first maneuver semantic feature is adjusted to be the same as the channel number of the shallow image feature through the convolution layer to obtain a second maneuver semantic feature; The second maneuver semantic feature is up-sampled to adjust the size of the second maneuver semantic feature to be the same as the size of the shallow image feature to obtain a third maneuver semantic feature; The third maneuver semantic feature and the shallow image feature are input into the spatial attention unit to obtain a first spatial attention weight and a second spatial attention weight, and the sum of the first spatial attention weight and the second spatial attention weight is 1; The third mobile semantic feature and the shallow image feature are input into the channel attention unit to obtain a first channel attention weight and a second channel attention weight, and a sum value of the first channel attention weight and the second channel attention weight is 1; The product of the third mobile semantic feature and the first spatial attention weight, the product of the third mobile semantic feature and the first channel attention weight, the product of the shallow image feature and the second spatial attention weight, and the product of the shallow image feature and the second channel attention weight are added to obtain an attention weighted feature; The attention weighted feature is input into the Fourier unit to obtain a first image fusion feature.

[0074] In an optional embodiment, the first feature fusion module is specifically configured to: The attention weighted feature is input into the Fourier unit to perform a Fourier transform operation on the attention weighted feature to obtain a first frequency domain attention weighted feature; The first frequency domain attention weighted feature is subjected to frequency domain processing to obtain a second frequency domain attention weighted feature; An inverse Fourier transform operation is performed on the second frequency domain attention weighted feature to obtain the first image fusion feature.

[0075] In an optional embodiment, the frequency domain processing includes at least one of the following operations: A convolution operation, a batch normalization operation, and an activation function application operation.

[0076] In an optional embodiment, the target recognition network further includes a CASFM and a second feature fusion module, and the second feature fusion module is specifically configured to: The first mobile semantic feature and the deep image feature are subjected to feature fusion by the CASFM to obtain a second image fusion feature.

[0077] In an optional embodiment, the CASFM includes a convolution layer, a first cross-attention unit, and a second cross-attention unit, the first cross-attention unit includes a first key weight matrix, a first query weight matrix, and a first value weight matrix, the second cross-attention unit includes a second key weight matrix, a second query weight matrix, and a second value weight matrix, and the second feature fusion module is specifically configured to: The convolution layer is used to adjust a channel number of the first mobile semantic feature to be the same as a channel number of the deep image feature to obtain a fourth mobile semantic feature; The fourth mobile semantic feature is input into the first key weight matrix, the first query weight matrix, and the first value weight matrix respectively to obtain a first key vector, a first query vector, and a first value vector; The deep image features are respectively input into the second key weight matrix, the second query weight matrix and the second value weight matrix to obtain a second key vector, a second query vector and a second value vector; The deep image features are taken as query features, and cross attention calculation is performed based on the second query vector, the first key vector and the first value vector by using a scaled dot-product attention mechanism formula to obtain a first attention output feature; The fourth maneuvering semantic feature is taken as a query feature, and cross attention calculation is performed based on the first query vector, the second key vector and the second value vector by using a scaled dot-product attention mechanism formula to obtain a second attention output feature; The first attention output feature and the fourth maneuvering semantic feature are added to obtain a third image fusion feature; The second attention output feature and the deep image feature are added to obtain a fourth image fusion feature; The third image fusion feature and the fourth image fusion feature are spliced to obtain a second image fusion feature.

[0078] In an optional embodiment, the CASFM further includes a convolution block unit, and further includes: The second image fusion feature is subjected to a stacked convolution operation by the convolution block unit.

[0079] Corresponding to the target recognition method based on fused semantic features provided in the embodiments of the present application, the embodiments of the present application further provide an electronic device for executing the target recognition method based on fused semantic features, as shown in Figure 6 The electronic device includes a processor 601 and a memory 602 for storing a program of the target recognition method based on fused semantic features. After the device is powered on and the program of the target recognition method based on fused semantic features is run by the processor, the following steps are performed: An infrared image of a target and a maneuvering category corresponding to the infrared image of the target are acquired, and the infrared image of the target contains the target; The infrared image of the target and the maneuvering category are input into a target recognition network to obtain a first maneuvering semantic feature, a shallow image feature and a deep image feature; The first maneuvering semantic feature and the shallow image feature are subjected to feature fusion to obtain a first image fusion feature; The first maneuvering semantic feature and the deep image feature are subjected to feature fusion to obtain a second image fusion feature; The first image fusion feature and the second image fusion feature are subjected to upsampling and splicing operations to obtain a multi-scale feature representation; The multi-scale feature representation is input into an output head to perform convolution operation to obtain a target category and an anti-interference recognition result.

[0080] Corresponding to the target recognition method based on fused semantic features provided in the embodiments of the present application, the embodiments of the present application also provide a computer readable storage medium, which stores a program of the target recognition method based on fused semantic features, and the program is run by a processor to perform the following steps: obtaining a target infrared image and a maneuver category corresponding to the target infrared image, the target infrared image containing a target; inputting the target infrared image and the maneuver category into a target recognition network to obtain a first maneuver semantic feature, a shallow image feature and a deep image feature; performing feature fusion on the first maneuver semantic feature and the shallow image feature to obtain a first image fusion feature; performing feature fusion on the first maneuver semantic feature and the deep image feature to obtain a second image fusion feature; performing upsampling and splicing operations on the first image fusion feature and the second image fusion feature to obtain a multi-scale feature representation; inputting the multi-scale feature representation into an output head to perform convolution operation to obtain a target category and an anti-interference recognition result.

[0081] Corresponding to the target recognition method based on fused semantic features provided in the embodiments of the present application, the embodiments of the present application also provide a computer readable storage medium, which stores a program of the target recognition method based on fused semantic features, and the program is run by a processor to perform the following steps: obtaining a target infrared image and a maneuver category corresponding to the target infrared image, the target infrared image containing a target; inputting the target infrared image and the maneuver category into a target recognition network to obtain a first maneuver semantic feature, a shallow image feature and a deep image feature; performing feature fusion on the first maneuver semantic feature and the shallow image feature to obtain a first image fusion feature; performing feature fusion on the first maneuver semantic feature and the deep image feature to obtain a second image fusion feature; performing upsampling and splicing operations on the first image fusion feature and the second image fusion feature to obtain a multi-scale feature representation; inputting the multi-scale feature representation into an output head to perform convolution operation to obtain a target category and an anti-interference recognition result.

[0082] It should be noted that the detailed description of the target recognition device based on fused semantic features, the electronic device, the computer readable storage medium and the computer program provided in the embodiments of the present application can refer to the related description of the target recognition method based on fused semantic features provided in the embodiments of the present application, which will not be repeated here.

[0083] Although the present application is disclosed with reference to the preferred embodiments above, it is not intended to limit the present application, and any person skilled in the art can make possible variations and modifications without departing from the spirit and scope of the present application. Therefore, the scope of the present application should be defined by the appended claims.

[0084] In one typical configuration, the electronic device includes one or more processors (Central Processing Unit), input / output interfaces, network interfaces, and memory.

[0085] The memory can include non-persistent memory, Random Access Memory (RAM), and / or non-volatile memory, such as Read Only Memory (ROM) or flash memory, in a computer readable medium. The memory is an example of computer readable media.

[0086] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (Programmable Analog Module), static random access memory (Static Random Access Memory), dynamic random access memory (Dynamic Random Access Memory), other types of random access memory (Random Access Memory), read only memory (Read Only Memory), electrically erasable programmable read only memory (Electrically Erasable Programmable Read Only Memory), flash memory or other memory technologies, compact disc read only memory (Compact Disc Read-Only Memory), digital video disc (Digital Video Disc) or other optical storage, magnetic cassette tape, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer readable media does not include non-transitory computer readable media (transitory media), such as modulated data signals and carriers.

[0087] Those skilled in the art will appreciate that embodiments of the present application can be readily used as a method, a system or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage and so forth) embodying computer-readable program code.

[0088] Although the present application has been disclosed in its preferred embodiments with reference to the accompanying drawings, it is to be understood that the application is not limited to those precise embodiments, and that changes and modifications can be effected therein by one of ordinary skill in the art without departing from the scope of the application.

Claims

1. A target recognition method based on fusion semantic features, characterized in that: include: Acquire a target infrared image and a maneuver category corresponding to the target infrared image, wherein the target infrared image contains a target; Inputting the target infrared image and the maneuver category into a target recognition network to obtain a first maneuver semantic feature, a shallow image feature, and a deep image feature; performing feature fusion on the first motor semantic feature and the shallow image feature to obtain a first image fusion feature; Performing feature fusion on the first motor semantic feature and the deep image feature to obtain a second image fusion feature; performing upsampling and concatenation operations on the first image fusion feature and the second image fusion feature to obtain a multi-scale feature representation; The multi-scale feature representation is input into the output head for convolution operation to obtain the target category and anti-interference recognition results.

2. The object recognition method based on fusion semantic features according to claim 1 is characterized in that: The target recognition network includes a learnable maneuvering semantic feature space, an ERBlock module, and a SimSPFF module. The target infrared image and the maneuvering category are input into the target recognition network to obtain a first maneuvering semantic feature, a shallow image feature, and a deep image feature, including: determining, according to the maneuver category, a first maneuver semantic feature corresponding to the maneuver category in the learnable maneuver semantic feature space; Obtain shallow image features through the ERBlock module; The deep image features are obtained through the SimSPFF module.

3. The object recognition method based on fusion semantic features according to claim 2 is characterized in that: The determining, according to the maneuver category, a first maneuver semantic feature corresponding to the maneuver category in the learnable maneuver semantic feature space includes: According to the maneuver category, a first maneuver semantic feature corresponding to the maneuver category is determined in the learnable maneuver semantic feature space by the following formula: Among them, the is the first maneuver semantic feature corresponding to the maneuver category, For the maneuver category, the is a learnable mobile semantic feature space, , represents the real number field, is the number of motor categories, is the number of rows of the feature matrix corresponding to the target infrared image, is the number of columns of the feature matrix corresponding to the target infrared image, It represents an operation of extracting the first maneuver semantic feature from the learnable maneuver semantic feature space according to the maneuver category.

4. The object recognition method based on fusion semantic features according to claim 1, characterized in that: The target recognition network further includes a UAFFM, and the feature fusion of the first maneuver semantic feature and the shallow image feature to obtain a first image fusion feature includes: The first motor semantic feature and the shallow image feature are fused through the UAFFM to obtain a first image fusion feature.

5. The object recognition method based on fusion semantic features according to claim 4 is characterized in that: The UAFFM includes a convolutional layer, a spatial attention unit, a channel attention unit, and a Fourier unit. The UAFFM performs feature fusion on the first motor semantic feature and the shallow image feature to obtain a first image fusion feature, including: Using the convolutional layer, the number of channels of the first maneuvering semantic feature is adjusted to be the same as the number of channels of the shallow image feature, thereby obtaining a second maneuvering semantic feature; Upsampling the second maneuvering semantic feature to adjust the size of the second maneuvering semantic feature to the same size as the shallow image feature to obtain a third maneuvering semantic feature; Inputting the third motorized semantic feature and the shallow image feature into a spatial attention unit to obtain a first spatial attention weight and a second spatial attention weight, wherein the sum of the first spatial attention weight and the second spatial attention weight is 1; Inputting the third motorized semantic feature and the shallow image feature into a channel attention unit to obtain a first channel attention weight and a second channel attention weight, wherein the sum of the first channel attention weight and the second channel attention weight is 1; Adding the product of the third motor semantic feature and the first spatial attention weight, the product of the third motor semantic feature and the first channel attention weight, the product of the shallow image feature and the second spatial attention weight, and the product of the shallow image feature and the second channel attention weight to obtain an attention weighted feature; The attention weighted feature is input into the Fourier unit to obtain a first image fusion feature.

6. The object recognition method based on fusion semantic features according to claim 5 is characterized in that: Inputting the attention weighted feature into the Fourier unit to obtain a first image fusion feature includes: Inputting the attention weighted feature into the Fourier unit, performing a Fourier transform operation on the attention weighted feature to obtain a first frequency domain attention weighted feature; Performing frequency domain processing on the first frequency domain attention weighted feature to obtain a second frequency domain attention weighted feature; An inverse Fourier transform operation is performed on the second frequency-domain attention weighted feature to obtain a first image fusion feature.

7. The object recognition method based on fusion semantic features according to claim 6 is characterized in that: The frequency domain processing includes at least one of the following operations: Convolution operation, batch normalization operation, and application of activation function operation.

8. The object recognition method based on fusion semantic features according to claim 1 is characterized in that: The target recognition network further includes CASFM, and the feature fusion of the first maneuver semantic feature and the deep image feature to obtain a second image fusion feature includes: The first motor semantic feature and the deep image feature are fused by the CASFM to obtain a second image fusion feature.

9. The object recognition method based on fusion semantic features according to claim 8, characterized in that: The CASFM includes a convolutional layer, a first cross-attention unit and a second cross-attention unit, the first cross-attention unit includes a first key weight matrix, a first query weight matrix and a first value weight matrix, the second cross-attention unit includes a second key weight matrix, a second query weight matrix and a second value weight matrix, and the first motor semantic feature and the deep image feature are subjected to feature fusion by the CASFM to obtain a second image fusion feature, including: Using the convolutional layer, the number of channels of the first motor semantic feature is adjusted to be the same as the number of channels of the deep image feature, thereby obtaining a fourth motor semantic feature; Inputting the fourth maneuver semantic feature into the first key weight matrix, the first query weight matrix, and the first value weight matrix respectively to obtain a first key vector, a first query vector, and a first value vector; Inputting the deep image features into the second key weight matrix, the second query weight matrix and the second value weight matrix respectively to obtain a second key vector, a second query vector and a second value vector; Using the deep image feature as a query feature, performing a cross attention calculation based on the second query vector, the first key vector, and the first value vector using a scaled dot product attention mechanism formula to obtain a first attention output feature; Using the fourth motorized semantic feature as a query feature, performing a cross-attention calculation based on the first query vector, the second key vector, and the second value vector using a scaled dot product attention mechanism formula to obtain a second attention output feature; Adding the first attention output feature and the fourth motorized semantic feature to obtain a third image fusion feature; Adding the second attention output feature and the deep image feature to obtain a fourth image fusion feature; The third image fusion feature and the fourth image fusion feature are spliced ​​together to obtain a second image fusion feature.

10. The object recognition method based on fusion semantic features according to claim 8, characterized in that: The CASFM also includes a convolution block unit, further comprising: A stacked convolution operation is performed on the second image fusion feature through a convolution block unit.

Images