Target recognition method, system and device based on multi-modal data and storage medium

Abstract

The application relates to the technical field of target identification, and specifically provides a target identification method, system and device based on multi-modal data and a storage medium, which comprises the following steps: collecting radar echo data and infrared images by using radars and infrared sensors at the same position, associating the radar echo data and the infrared images of the same target, obtaining target radar echo data and target infrared images, converting the target radar echo data into a range-Doppler graph, extracting radar features from the range-Doppler graph by using a convolutional neural network, extracting infrared features from the target infrared images, obtaining environmental parameters and a quality evaluation coefficient of the target infrared images, generating feature confidence according to the environmental parameters and the quality evaluation coefficient, fusing the radar features and the infrared features according to the feature confidence, and determining the target type according to the fused features. The application effectively overcomes the limitation that single radar identification is disturbed by complex environments, and greatly improves the accuracy and reliability of target identification.

Description

Technical Field

[0001] This invention belongs to the field of target recognition technology, specifically relating to a target recognition method, system, device, and storage medium based on multimodal data. Background Technology

[0002] Radar, as a commonly used means of target identification, plays an important role in many fields. However, when using radar for target identification in high-speed aircraft, it faces many challenges. Due to the complex environment during high-speed flight, there are various interference factors, such as electromagnetic interference and clutter, which leads to low accuracy in radar identification. Summary of the Invention

[0003] In view of the above-mentioned shortcomings of the prior art, the present invention provides a target recognition method, system, device and storage medium based on multimodal data to solve the above-mentioned technical problems.

[0004] In a first aspect, the present invention provides a target recognition method based on multimodal data, comprising:

[0005] Radar echo data and infrared images are collected using radar and infrared sensors at the same location, and the radar echo data and infrared images of the same target are correlated to obtain the target radar echo data and target infrared image.

[0006] Convert the target radar echo data into a range-Doppler image;

[0007] Radar features are extracted from the range-Doppler image using a convolutional neural network, and infrared features are extracted from the target infrared image.

[0008] Obtain environmental parameters and quality evaluation coefficients of the target infrared image, and generate feature confidence scores based on the environmental parameters and the quality evaluation coefficients;

[0009] Radar and infrared features are fused based on feature confidence levels, and the target type is determined based on the fused features.

[0010] In an optional implementation, radar echo data and infrared images are acquired using radar and infrared sensors at the same location, and the radar echo data and infrared images of the same target are correlated to obtain target radar echo data and target infrared images, including:

[0011] The radar acquires radar echo data from multiple measurement points, and the infrared sensor acquires infrared images from multiple measurement points, with the radar and infrared sensor installed at the same location.

[0012] The radar position parameters and infrared position parameters of each measurement point are obtained. The radar position parameters include the azimuth and elevation angles relative to the radar, and the infrared position parameters include the azimuth and elevation angles relative to the infrared sensor.

[0013] Calculate the spherical distance between the radar position parameter of each measurement point in the radar echo data and the infrared position parameter of the measurement point collected by the infrared sensor. Then, take the measurement point corresponding to the radar position parameter with the smallest spherical distance and the infrared position parameter as the matching measurement point. Determine the radar echo data and infrared image corresponding to the matching measurement point as the target radar echo data and target infrared image of the same target.

[0014] In an optional implementation, converting the target radar echo data into a range-Doppler image includes:

[0015] The target radar echo data is pulse-compressed by convolution operation to obtain the compressed echo signal;

[0016] The frequency spectrum is obtained by performing a Fourier transform on the compressed echo signal of each distance cell;

[0017] Take the amplitude of the frequency spectrum, move the zero-frequency component to the center of the spectrum, and obtain the range-Doppler map of the target.

[0018] In one alternative implementation, the convolutional neural network includes four convolutional layers, eight residual modules, and a global pooling layer.

[0019] In one optional implementation, acquiring environmental parameters and quality evaluation coefficients of the target infrared image, generating confidence levels of radar features based on the environmental parameters, and generating confidence levels of the infrared features based on the quality evaluation coefficients, includes:

[0020] Obtain environmental parameters and convert them into quantified environmental parameter values;

[0021] The infrared image of the target is subjected to quality detection to obtain a quality evaluation coefficient;

[0022] The quantified values ​​of the environmental parameters and the quality evaluation coefficients are input into a multilayer perceptron to obtain the confidence levels of the radar features and the infrared features.

[0023] The multilayer perceptron includes a first branch, a second branch, and a confidence calculation module. Both the first branch and the second branch are composed of fully connected layers and ReLU layers.

[0024] The first branch is used to perform a linear transformation on the radar features to obtain the radar confidence vector;

[0025] The second branch is used to perform a linear transformation on the infrared features to obtain the infrared confidence vector;

[0026] The confidence calculation module is used to concatenate the radar confidence vector and the infrared confidence vector into a combined vector, and obtain the feature confidence by performing dimensionality reduction on the combined vector.

[0027] In an optional implementation, radar and infrared features are fused based on feature confidence levels, and the target type is determined based on the fused features, including:

[0028] Radar feature weights and infrared feature weights are generated based on the feature confidence levels.

[0029] The dot product of radar features and radar feature weights is denoted as the radar feature vector, and the dot product of infrared features and infrared feature weights is denoted as the infrared feature vector.

[0030] Obtain the low-rank mode factor of radar features and the low-rank mode factor of infrared features;

[0031] The fused feature vector is calculated based on the radar feature vector, the low-rank mode factor of the radar feature, the infrared feature vector, and the low-rank mode factor of the infrared feature.

[0032] The target type is determined using a classifier based on the fused feature vector.

[0033] In an optional implementation, generating radar feature weights and infrared feature weights based on the feature confidence level includes:

[0034] The feature confidence scores are normalized using norm to obtain the confidence coefficients.

[0035] Use the confidence coefficient as the weight of the infrared feature;

[0036] The difference between 1 and the confidence coefficient is used as the radar feature weight.

[0037] Secondly, the present invention provides a target recognition system based on multimodal data, comprising:

[0038] The data matching module is used to collect radar echo data and infrared images using radar and infrared sensors at the same location, and to associate the radar echo data and infrared images of the same target to obtain the target radar echo data and the target infrared image.

[0039] The preprocessing module is used to convert the target radar echo data into a range-Doppler image;

[0040] The feature extraction module is used to extract radar features from the range-Doppler image using a convolutional neural network and to extract infrared features from the target infrared image;

[0041] The confidence calculation module is used to obtain environmental parameters and quality evaluation coefficients of the target infrared image, and generate feature confidence based on the environmental parameters and the quality evaluation coefficients.

[0042] The feature processing module is used to fuse radar features and infrared features based on feature confidence levels, and to determine the target type based on the fused features.

[0043] Thirdly, a device is provided, comprising:

[0044] Memory, used to store target recognition programs based on multimodal data;

[0045] A processor is configured to implement the steps of the multimodal data-based target recognition method as provided in the first aspect when executing the multimodal data-based target recognition program.

[0046] Fourthly, a computer-readable storage medium is provided, on which a target recognition program based on multimodal data is stored, wherein when the target recognition program based on multimodal data is executed by a processor, the target recognition program based on multimodal data implements the steps of the target recognition method based on multimodal data provided in the first aspect.

[0047] The beneficial effects of this invention lie in the fact that the target recognition method, system, device, and storage medium based on multimodal data provided by this invention utilize radar and infrared sensors at the same location to collect radar echo data and infrared images, and determine the relevant data of the same target, thereby realizing the utilization of multimodal data. After converting the radar echo data into a range-Doppler image, a convolutional neural network is used to extract radar features and infrared features respectively. Simultaneously, environmental parameters and infrared image quality evaluation coefficients are combined to generate feature confidence scores, thereby fusing the two types of features. This multimodal data fusion method effectively overcomes the limitations of single radar recognition due to interference from complex environments, greatly improving the accuracy and reliability of target recognition.

[0048] Furthermore, the design principle of this invention is reliable, the structure is simple, and it has a very wide range of application prospects. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a schematic flowchart of a method according to an embodiment of the present invention.

[0051] Figure 2 This is a schematic diagram illustrating the principle of a method according to an embodiment of the present invention.

[0052] Figure 3 This is a ResNet18 network structure diagram of a method according to an embodiment of the present invention.

[0053] Figure 4 This is a residual module structure diagram of a method according to an embodiment of the present invention.

[0054] Figure 5 This is a flowchart of an infrared image self-test method according to an embodiment of the present invention.

[0055] Figure 6 This is a flowchart illustrating the confidence calculation process of a method according to an embodiment of the present invention.

[0056] Figure 7 This is a schematic diagram illustrating the feature fusion of a method according to an embodiment of the present invention.

[0057] Figure 8 This is a schematic block diagram of a system according to an embodiment of the present invention.

[0058] Figure 9 This is a schematic diagram of the structure of a device provided in an embodiment of the present invention. Detailed Implementation

[0059] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0060] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this application and in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0061] The target recognition method for multimodal data provided in this embodiment of the invention is executed by a computer device, and correspondingly, the target recognition system for multimodal data runs in the computer device.

[0062] Figure 1 This is a schematic flowchart illustrating a method according to an embodiment of the present invention. Wherein, Figure 1 The executing entity can be a target recognition system based on multimodal data. Depending on different requirements, the order of the steps in this flowchart can be changed, and some can be omitted.

[0063] like Figure 1As shown, the method includes:

[0064] S1. Use radar and infrared sensors at the same location to collect radar echo data and infrared images, and associate the radar echo data and infrared images of the same target to obtain the target radar echo data and target infrared image;

[0065] S2. Convert the target radar echo data into a range-Doppler image;

[0066] S3. Use a convolutional neural network to extract radar features from the range-Doppler image and extract infrared features from the target infrared image;

[0067] S4. Obtain environmental parameters and quality evaluation coefficients of the target infrared image, and generate feature confidence based on the environmental parameters and the quality evaluation coefficients;

[0068] S5. Fuse radar and infrared features based on feature confidence levels, and determine the target type based on the fused features.

[0069] Please refer to Figure 2 The system can be divided into a self-detection module and a feature fusion module. In the self-detection module, the input infrared image and radar data are first evaluated for quality. Next, the quantified indicators based on the evaluation system are encoded, and data confidence is learned through a network. Finally, the learned confidence is fed into the feature fusion module, adjusting the sensor's feature vectors based on the confidence level. In the feature extraction module, a CNN network is first used to extract features from the processed radar RD image and infrared image. Then, the obtained feature vectors are adjusted using the confidence from the detection module, allowing for more reliable information to be used in subsequent fusion. Next, an LMF network is used to interact with the heterogeneous feature vectors, obtaining the interdependencies between features, reducing redundancy, and adaptively fusing the feature vectors. Finally, a decoder is used to classify the features and identify the target type.

[0070] In one embodiment of the present invention, based on step S1, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0071] Multiple radar echo data points are acquired using radar, and infrared images of multiple measurement points are acquired using infrared sensors, with the radar and infrared sensors mounted in the same location. Radar position parameters and infrared position parameters are obtained for each measurement point. The radar position parameters include azimuth and elevation angles relative to the radar, and the infrared position parameters include azimuth and elevation angles relative to the infrared sensor. The spherical distance between the radar position parameter of each measurement point in the radar echo data and the infrared position parameter of the measurement point acquired by the infrared sensor is calculated. The measurement point corresponding to the radar position parameter and infrared position parameter with the smallest spherical distance is selected as the matching measurement point. The radar echo data and infrared image corresponding to the matching measurement point are then identified as the target radar echo data and target infrared image of the same target.

[0072] In a specific example, data registration is performed based on angle measurements from radar and infrared sensors, assuming that the radar and infrared sensors are installed in the same location. Represents radar measurement data, This represents infrared measurement data, where each point... and These are the azimuth and elevation angles for radar and infrared, respectively. For each radar measurement point... Calculate its relationship with all infrared measurement points. The spherical distance between them is:

[0073]

[0074] in The radar measurement of the first The point and infrared measurement The spherical distance between points, for each radar measurement point Select the nearest infrared measurement point. :

[0075]

[0076] in Indicates the distance to the radar measurement point The most recent infrared measurement point. Based on this matching criterion, the best match is selected from radar data and infrared data at the same time, thereby associating the radar data and infrared data with the same target.

[0077] In one embodiment of the present invention, based on step S2, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0078] The target radar echo data is pulse-compressed by convolution operation to obtain compressed echo signals; Fourier transform is performed on the compressed echo signals of each range cell to obtain the frequency spectrum; the amplitude of the frequency spectrum is taken, and the zero-frequency component is moved to the center of the spectrum to obtain the range-Doppler map of the target.

[0079] Considering the heterogeneity between radar echo data and infrared image data, and that the radar signal and infrared signal of the target are independent, directly fusing the raw data of the two sensors will result in weak feature correlation. Therefore, we consider performing a preliminary transformation on the radar echo data to obtain a representation that is correlated with the infrared image before feature fusion.

[0080] A radar range-Doppler image is an image that describes the reflection characteristics of a target object under radar wave illumination, and is commonly used in radar signal processing and target detection. Different targets have significantly different moving speeds, which will manifest as large differences in the Doppler frequency domain. Furthermore, different targets also have significant differences in physical size, thus resulting in significant differences in Doppler frequency shift, Doppler broadening, and the range cells they occupy.

[0081] For radar echo signals, pulse compression is first required to improve range resolution. Let the transmitted signal be... The received signal is The pulse-compressed echo signal is obtained through convolution operation. :

[0082]

[0083] After pulse compression, the resulting echo signal has a shorter pulse width in the time domain, accurately reflecting the target's distance. Then, the echo signal of each range unit... The frequency spectrum is obtained by performing a Fourier transform. :

[0084]

[0085] in, For Doppler frequency shift, This represents the Fourier transform. Then, the amplitude is taken, the zero-frequency component is moved to the center of the spectrum, and the RD diagram of the target is plotted.

[0086] In one embodiment of the present invention, based on step S3, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0087] The ResNet18 network was selected as the feature extraction head to extract features from both radar RD images and infrared images. The ResNet18 network consists of four convolutional layers, eight residual blocks, and one global pooling layer. The overall network structure is as follows: Figure 3 As shown. The residual block structure is as follows. Figure 4 As shown.

[0088] The ResNet network effectively addresses the vanishing gradient and degradation problems by introducing residual connections. The residual module mainly consists of two convolutional layers, batch normalization, and an activation function. The specific implementation is as follows: Input feature map After passing through the first convolutional layer, the output is obtained through batch normalization and a non-linear activation function. :

[0089]

[0090] in Represents two-dimensional convolution. This represents the weights of the first convolutional kernel. The output is then obtained after a second convolution and batch normalization. :

[0091]

[0092] in This represents the weights of the second-layer convolutional kernel. Finally, residual connections are implemented, taking the initial input feature map... With the output of the second convolution layer The final output of the residual block is obtained by connecting the components and then performing nonlinear activation. :

[0093]

[0094] The final feature vector is obtained after passing through four residual blocks. Therefore, the entire feature extraction module can be represented as:

[0095]

[0096] in, Indicates the first One residual block, This represents the original input image.

[0097] In one embodiment of the present invention, based on step S4, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0098] S401. Obtain environmental parameters and convert the environmental parameters into quantified values ​​of environmental parameters.

[0099] Among the environmental parameters, such as sea state, the higher the sea state, the larger the waves. The undulation of the waves and the unevenness of the sea surface can cause changes in the reflection path of radar signals, creating a multipath effect that reduces the clarity and accuracy of the echo signal. It can also generate more scattering, causing interference to the radar.

[0100] Since the sea state level itself is already quantified using numerical values, when inputting the data into the network, the sea state level at the time of data collection is directly input as a separate one-dimensional piece of information.

[0101] S402. Perform quality detection on the target infrared image to obtain a quality evaluation coefficient.

[0102] Please refer to Figure 5 The image is first subjected to brightness calculation. If it is not within the normal brightness range, it means that the image contains most of the brightness information and cannot provide information such as the position, texture and contour of the target itself. Therefore, the image is discarded and its features are no longer used. If it is within the normal brightness range, the contrast is then calculated and the calculated contrast is used as a one-dimensional parameter input into the network.

[0103] In the brightness judgment module, the average brightness of the image is first calculated, and then the image is judged based on a brightness threshold. When the overall brightness of the image is too high or too low, it indicates that the image may be overexposed or underexposed. In this case, the effective information in the image is greatly reduced, affecting the subsequent learning of the network, so the image is directly discarded. In the contrast calculation module, the image is first Gaussian smoothed to reduce the impact of noise on subsequent calculations. Then, the Sobel operator is used to calculate the gradient of each pixel in the image.

[0104]

[0105] in This represents the gradient of the pixel in the horizontal direction. This represents the gradient of the pixel in the vertical direction. Indicates the position in the image The pixel value at that location. After obtaining the gradient, the magnitude and direction of the gradient are calculated:

[0106]

[0107]

[0108] in This represents the magnitude of the gradient. The gradient direction is then represented. Non-maximum suppression is then performed along the gradient direction to identify pixels with local maximum gradient magnitudes. Simultaneously, double thresholding is applied to classify these pixels: pixels with gradient magnitudes greater than the higher threshold are classified as strong edges; pixels with gradient magnitudes between the lower and higher thresholds are classified as weak edges; and pixels with gradient magnitudes below the lower threshold are classified as non-edges. Finally, weak edge points are filtered: if a weak edge point is adjacent to a strong edge point, it is retained; otherwise, it is removed. This yields all reliable edge points. The average gradient magnitude of these points is then used to obtain the contrast between the target and background in the image.

[0109]

[0110] in This indicates the contrast of the infrared image. This represents the gradient magnitude value of reliable edge points selected through the above process. Inputting the calculated contrast as one-dimensional information into the network can better describe the reliability of the infrared image.

[0111] After obtaining the edge and contrast values, further analysis is needed to determine if the image is subject to point or area interference, such as that caused by infrared decoys. Based on the characteristics of area source interference, the combustion unit forms an infrared radiation field with a certain intensity and area, generating multiple point or area radiation sources. Therefore, the image edges are morphologically dilated, and then connected component analysis is performed on the dilated image to calculate the total number of connected components. A large number of connected components indicates that the image has been affected by point or area source interference, and the confidence level should be lowered. Combining the above processes, the image quality evaluation coefficient is:

[0112]

[0113] in This is the image quality evaluation coefficient. This is the connected component control coefficient, which controls the degree of influence of the number of connected components on the image quality evaluation coefficient. This represents the number of connected components.

[0114] S403. Input the quantified values ​​of the environmental parameters and the quality evaluation coefficients into the multilayer perceptron to obtain the confidence levels of the radar features and the infrared features.

[0115] The sea state level and infrared image quality evaluation coefficients will be converted into confidence scores to adjust sensor features through this module. Considering that these two parameters are closely related to sensor data, this application uses a multilayer perceptron to deeply couple the relationship between parameters and features. The weights and biases of the multilayer perceptron are updated and iterated in the network, allowing the network to learn the contribution of each feature in the two feature vectors to the recognition task under different conditions. The input to the confidence score calculation module is the sea state level. Infrared image quality evaluation coefficient It consists of two parts, and its structure is as follows: Figure 6 As shown. First, the input is linearly encoded and nonlinearly activated through the information encoding module, so that the information encoding module consists of two identical branches, each branch consisting of a fully connected layer and a... Layered structure, output and Represented as:

[0116]

[0117] in , , and These represent the weights and biases for the linear transformation of sea state grade and infrared image contrast, respectively. This step transforms the feature dimension from 1 to [a value missing]. The dimension of the feature vector is calculated; then, the two vectors after linear transformation are concatenated to obtain a combined vector. This combined vector is then reduced to the dimension of the feature vector using a dimension reduction module. This confidence vector can interact with the feature vector, dynamically adjusting the confidence level of each feature. The output of the confidence calculation module is shown below. It can be represented as:

[0118]

[0119] in and The weights and biases for the dimensionality reduction module.

[0120] In one embodiment of the present invention, based on step S5, a possible embodiment will be given below, and its specific implementation will be described in a non-limiting manner.

[0121] Radar feature weights and infrared feature weights are generated based on the feature confidence levels.

[0122] The dot product of radar features and radar feature weights is denoted as the radar feature vector, and the dot product of infrared features and infrared feature weights is denoted as the infrared feature vector. The low-rank mode factors of radar features and infrared features are obtained. The fused feature vector is calculated based on the radar feature vector, the low-rank mode factors of radar features, the infrared feature vector, and the low-rank mode factors of infrared features. The target type is determined using a classifier based on the fused feature vector.

[0123] Existing methods generally employ feature concatenation as the basic fusion strategy, which essentially involves linearly connecting the feature vectors output by each modal encoder. However, such methods only achieve shallow feature combination and fail to effectively model the nonlinear interaction relationships between cross-modal features, leading to the loss of potential correlation information. To overcome this limitation, this application introduces an outer product interaction mechanism, which achieves high-order feature interaction modeling by constructing a bimodal feature tensor product. Compared to traditional concatenation methods, outer product operations can explicitly capture the second-order correlation between feature dimensions, establishing a mathematical representation of cross-modal feature associations. Compared to the current mainstream cross-attention mechanism, this method has a unique theoretical advantage: when the reliability of sensor data experiences asymmetric degradation (such as radar signals under severe sea conditions), the cross-attention mechanism may lead to erroneous association learning due to the sensitivity of the weight allocation mechanism to noise, while outer product operations, by maintaining the symmetry of feature interactions, can effectively mitigate the impact of single-modal data degradation on fusion performance.

[0124] The disadvantage of outer products lies in their computational complexity and storage cost. Therefore, this paper employs a low-rank matrix factorization (LMF) fusion network, which can effectively reduce computational overhead and storage costs. The fusion network is as follows: Figure 7 As shown.

[0125] The feature confidence is first normalized to a norm, and then adjusted based on the infrared features and radar feature vectors from the encoder:

[0126] ,

[0127] ,

[0128] ,

[0129] in, For vectors of Norm, This represents the normalized confidence vector. The adjusted feature vector is input into the LMF fusion network. Since the outer product needs to be calculated and then transformed into a one-dimensional feature vector for downstream tasks, for a traditional outer product fusion network, we have:

[0130]

[0131]

[0132] For the weight matrix Introducing a low-rank matrix, it is represented as:

[0133]

[0134] in and The first and second infrared and radar features are respectively There are several low-rank mode factors. Further derivation yields:

[0135]

[0136] in" "" represents the dot product between vectors. This transformation decouples the fusion of two modes, transforming the complex computation of calculating the outer product of feature vectors and converting the outer product feature fusion matrix into a feature fusion vector into a direct calculation of the dot product between the feature vector and the mode factor, thus reducing computational overhead. Furthermore, the computational complexity decreases while computational efficiency increases significantly. Simultaneously, performing the outer product operation on feature vectors allows for simple and effective interaction between each feature in the vector, ensuring the network can better capture the relationships between features.

[0137] Furthermore, the low-rank mode factors are obtained by performing low-rank decomposition on the eigenvectors. Low-rank decomposition is the process of decomposing a high-dimensional data matrix into low-rank matrices and sparse matrices. This can be achieved through various methods, such as Robust Principal Component Analysis (RPCA) and Augmented Lagrange Multiplier Method (ALM). In this application, low-rank decomposition is performed on radar and infrared eigenvectors respectively to obtain their respective low-rank mode factors and sparse components.

[0138] In multimodal fusion, to further improve computational efficiency and fusion results, the weight tensor can be decomposed into mode-specific low-rank factors. This method avoids explicitly creating high-dimensional tensors, thereby reducing memory overhead and computational complexity. For low-rank mode factors of radar and infrared features, mode-specific factorization methods can be used to further extract low-rank factors related to their respective modes.

[0139] f U Inputting the data into a support vector machine yields the target type, such as "ship".

[0140] In some embodiments, the target recognition system based on multimodal data may include multiple functional modules composed of computer program segments. The computer programs for each program segment in the target recognition system based on multimodal data may be stored in the memory of a computer device and executed by at least one processor to perform (see details). Figure 1 (Description) The function of target recognition based on multimodal data.

[0141] In this embodiment, the target recognition system based on multimodal data can be divided into multiple functional modules according to the functions it performs, such as... Figure 8 As shown. The module referred to in this invention is a series of computer program segments that can be executed by at least one processor and perform a fixed function, and is stored in memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0142] The data matching module is used to collect radar echo data and infrared images using radar and infrared sensors at the same location, and to associate the radar echo data and infrared images of the same target to obtain the target radar echo data and the target infrared image.

[0143] The preprocessing module is used to convert the target radar echo data into a range-Doppler image;

[0144] The feature extraction module is used to extract radar features from the range-Doppler image using a convolutional neural network and to extract infrared features from the target infrared image;

[0145] The confidence calculation module is used to obtain environmental parameters and quality evaluation coefficients of the target infrared image, and generate feature confidence based on the environmental parameters and the quality evaluation coefficients.

[0146] The feature processing module is used to fuse radar features and infrared features based on feature confidence levels, and to determine the target type based on the fused features.

[0147] Figure 9The target recognition method based on multimodal data provided in the embodiments of this application can be applied to devices. Those skilled in the art will understand that the device structure involved in the embodiments of this invention does not constitute a limitation on the device. A device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. In the embodiments of this invention, the device includes, but is not limited to, laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown in this application, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments described and / or claimed in this application.

[0148] The device 900 may include a processor 910, a memory 920, and a communication unit 930. These components communicate via one or more buses. Those skilled in the art will understand that the server structure shown in the figure does not constitute a limitation of the present invention. It may be a bus topology or a star topology, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0149] The memory 920 can be used to store execution instructions of the processor 910. The memory 920 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. When the execution instructions in the memory 920 are executed by the processor 910, the device 900 is able to perform some or all of the steps in the above method embodiments.

[0150] The processor 910 serves as the control center of the storage device, connecting various parts of the electronic device via various interfaces and lines. It executes software programs and / or modules stored in the memory 920, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of integrated circuits (ICs), such as a single packaged IC or multiple packaged ICs with the same or different functions connected together. For example, the processor 910 may consist only of a central processing unit (CPU). In this embodiment of the invention, the CPU may have a single processing core or include multiple processing cores.

[0151] The communication unit 930 is used to establish a communication channel, enabling the storage device to communicate with other devices. It can receive user data sent by other devices or send user data to other devices.

[0152] The present invention also provides a computer storage medium, wherein the computer storage medium may store a program, which, when executed, may include some or all of the steps provided in the embodiments of the present invention. The storage medium may be a magnetic disk, an optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0153] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or any other medium capable of storing program code. It includes several instructions to cause a computer device (which may be a personal computer, a server, or a second device, network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0154] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

[0155] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or modules may be electrical, mechanical, or other forms.

[0156] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0157] In addition, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0158] Although the present invention has been described in detail with reference to the accompanying drawings and preferred embodiments, the present invention is not limited thereto. Various equivalent modifications or substitutions can be made to the embodiments of the present invention by those skilled in the art without departing from the spirit and essence of the invention, and such modifications or substitutions should all be within the scope of the present invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should also be covered within the protection scope of the present invention.

Claims

1. A target recognition method based on multimodal data, characterized in that, include: Radar echo data and infrared images are collected using radar and infrared sensors at the same location, and the radar echo data and infrared images of the same target are correlated to obtain the target radar echo data and target infrared image. Convert the target radar echo data into a range-Doppler image; Radar features are extracted from the range-Doppler image using a convolutional neural network, and infrared features are extracted from the target infrared image. Obtain environmental parameters and quality evaluation coefficients of the target infrared image, and generate feature confidence scores based on the environmental parameters and the quality evaluation coefficients; Radar and infrared features are fused based on feature confidence levels, and the target type is determined based on the fused features. This involves using radar and infrared sensors at the same location to collect radar echo data and infrared images, and correlating the radar echo data and infrared images of the same target to obtain target radar echo data and target infrared images, including: The radar acquires radar echo data from multiple measurement points, and the infrared sensor acquires infrared images from multiple measurement points, with the radar and infrared sensor installed at the same location. The radar position parameters and infrared position parameters of each measurement point are obtained. The radar position parameters include the azimuth and elevation angles relative to the radar, and the infrared position parameters include the azimuth and elevation angles relative to the infrared sensor. Calculate the spherical distance between the radar position parameter of each measurement point in the radar echo data and the infrared position parameter of the measurement point collected by the infrared sensor, and take the measurement point corresponding to the radar position parameter with the smallest spherical distance and the infrared position parameter as the matching measurement point. Then, determine the radar echo data and infrared image corresponding to the matching measurement point as the target radar echo data and target infrared image of the same target. Acquire environmental parameters and quality assessment coefficients of the target infrared image, and generate feature confidence scores based on the environmental parameters and the quality assessment coefficients, including: Obtain environmental parameters and convert them into quantified environmental parameter values; The infrared image of the target is subjected to quality detection to obtain a quality evaluation coefficient; The quantified values ​​of the environmental parameters and the quality evaluation coefficients are input into a multilayer perceptron to obtain the confidence levels of the radar features and the infrared features. The multilayer perceptron includes a first branch, a second branch, and a confidence calculation module. Both the first branch and the second branch are composed of fully connected layers and ReLU layers. The first branch is used to perform a linear transformation on the radar features to obtain the radar confidence vector; The second branch is used to perform a linear transformation on the infrared features to obtain the infrared confidence vector; The confidence calculation module is used to concatenate the radar confidence vector and the infrared confidence vector into a combined vector, and obtain the feature confidence by performing dimensionality reduction on the combined vector; The radar and infrared features are fused based on feature confidence levels, including: Radar feature weights and infrared feature weights are generated based on the feature confidence levels. The dot product of radar features and radar feature weights is denoted as the radar feature vector, and the dot product of infrared features and infrared feature weights is denoted as the infrared feature vector. Obtain the low-rank mode factor of radar features and the low-rank mode factor of infrared features; The fused feature vector is calculated based on the radar feature vector, the low-rank mode factor of the radar feature, the infrared feature vector, and the low-rank mode factor of the infrared feature. Based on the feature confidence level, radar feature weights and infrared feature weights are generated, including: The feature confidence scores are normalized using norm to obtain the confidence coefficients. Use the confidence coefficient as the weight of the infrared feature; The difference between 1 and the confidence coefficient is used as the radar feature weight.

2. The method according to claim 1, characterized in that, Converting the target radar echo data into a range-Doppler image includes: The target radar echo data is pulse-compressed by convolution operation to obtain the compressed echo signal; The frequency spectrum is obtained by performing a Fourier transform on the compressed echo signal of each distance cell; Take the amplitude of the frequency spectrum, move the zero-frequency component to the center of the spectrum, and obtain the range-Doppler map of the target.

3. The method according to claim 1, characterized in that, The convolutional neural network includes four convolutional layers, eight residual modules, and one global pooling layer.

4. The method according to claim 1, characterized in that, The target type is determined based on the fused features, including: The target type is determined using a classifier based on the fused feature vector.

5. A target recognition system based on multimodal data, characterized in that, include: The data matching module is used to collect radar echo data and infrared images using radar and infrared sensors at the same location, and to associate the radar echo data and infrared images of the same target to obtain the target radar echo data and the target infrared image. A preprocessing module is used to convert the target radar echo data into a range-Doppler image; The feature extraction module is used to extract radar features from the range-Doppler image using a convolutional neural network and to extract infrared features from the target infrared image; The confidence calculation module is used to obtain environmental parameters and quality evaluation coefficients of the target infrared image, and generate feature confidence based on the environmental parameters and the quality evaluation coefficients. The feature processing module is used to fuse radar features and infrared features based on feature confidence levels, and to determine the target type based on the fused features. This involves using radar and infrared sensors at the same location to collect radar echo data and infrared images, and correlating the radar echo data and infrared images of the same target to obtain target radar echo data and target infrared images, including: The radar acquires radar echo data from multiple measurement points, and the infrared sensor acquires infrared images from multiple measurement points, with the radar and infrared sensor installed at the same location. The radar position parameters and infrared position parameters of each measurement point are obtained. The radar position parameters include the azimuth and elevation angles relative to the radar, and the infrared position parameters include the azimuth and elevation angles relative to the infrared sensor. Calculate the spherical distance between the radar position parameter of each measurement point in the radar echo data and the infrared position parameter of the measurement point collected by the infrared sensor, and take the measurement point corresponding to the radar position parameter with the smallest spherical distance and the infrared position parameter as the matching measurement point. Then, determine the radar echo data and infrared image corresponding to the matching measurement point as the target radar echo data and target infrared image of the same target. Acquire environmental parameters and quality assessment coefficients of the target infrared image, and generate feature confidence scores based on the environmental parameters and the quality assessment coefficients, including: Obtain environmental parameters and convert them into quantified environmental parameter values; The infrared image of the target is subjected to quality detection to obtain a quality evaluation coefficient; The quantified values ​​of the environmental parameters and the quality evaluation coefficients are input into a multilayer perceptron to obtain the confidence levels of the radar features and the infrared features. The multilayer perceptron includes a first branch, a second branch, and a confidence calculation module. Both the first branch and the second branch are composed of fully connected layers and ReLU layers. The first branch is used to perform a linear transformation on the radar features to obtain the radar confidence vector; The second branch is used to perform a linear transformation on the infrared features to obtain the infrared confidence vector; The confidence calculation module is used to concatenate the radar confidence vector and the infrared confidence vector into a combined vector, and obtain the feature confidence by performing dimensionality reduction on the combined vector; The radar and infrared features are fused based on feature confidence levels, including: Radar feature weights and infrared feature weights are generated based on the feature confidence levels. The dot product of radar features and radar feature weights is denoted as the radar feature vector, and the dot product of infrared features and infrared feature weights is denoted as the infrared feature vector. Obtain the low-rank mode factor of radar features and the low-rank mode factor of infrared features; The fused feature vector is calculated based on the radar feature vector, the low-rank mode factor of the radar feature, the infrared feature vector, and the low-rank mode factor of the infrared feature. Based on the feature confidence level, radar feature weights and infrared feature weights are generated, including: The feature confidence scores are normalized using norm to obtain the confidence coefficients. Use the confidence coefficient as the weight of the infrared feature; The difference between 1 and the confidence coefficient is used as the radar feature weight.

6. A device, characterized in that, include: Memory, used to store target recognition programs based on multimodal data; A processor, configured to implement the steps of the target recognition method based on multimodal data as described in any one of claims 1-4 when executing the target recognition program based on multimodal data.

7. A computer-readable storage medium storing a computer program, characterized in that, The readable storage medium stores a target recognition program based on multimodal data, which, when executed by a processor, implements the steps of the target recognition method based on multimodal data as described in any one of claims 1-4.

Images