Target recognition system based on active-passive hybrid imaging and recognition method thereof

Abstract

This application discloses a target recognition system and method based on hybrid active-passive imaging. On one hand, it encodes waveforms of multiple echo signals acquired by multiple millimeter-wave radars using a first convolutional neural network to obtain multiple first feature matrices, and constructs these first feature matrices into a three-dimensional first feature map along the channel dimension. On the other hand, it encodes multiple images acquired by multiple cameras using a second convolutional neural network to obtain second feature matrices, and arranges these second feature matrices along the channel dimension to construct a three-dimensional second feature map. Then, it fuses the first and second feature maps to identify the target. This system integrates active imaging from multiple millimeter-wave radars and passive imaging from multiple cameras to perceive the target from different perspectives, thereby improving the accuracy of target recognition.

Description

Technical Field

[0001] This application relates to the field of target recognition, and more specifically, to a target recognition system and method based on hybrid active and passive imaging. Background Technology

[0002] With the development of artificial intelligence technology, smart cameras have emerged. Compared with traditional cameras, smart cameras are like having a brain installed. They can analyze and process images, such as facial recognition and object recognition, which provides new technical support for security checks in public places.

[0003] However, in practical applications, the security capabilities of smart cameras in public places fall short of expectations. This is because the source images captured by the cameras are often obstructed, and background information can interfere with the image. Furthermore, smart cameras encounter problems in object recognition, such as decreased recognition accuracy and misidentification due to changes in posture or clothing.

[0004] Therefore, a novel target object recognition scheme for public places is needed. Summary of the Invention

[0005] To address the aforementioned technical problems, this application is proposed. Embodiments of this application provide a target recognition system and method based on hybrid active-passive imaging, which integrates active imaging from multiple millimeter-wave radars and passive imaging from multiple cameras to perceive the target object from different perspectives, thereby improving the accuracy of target recognition.

[0006] According to one aspect of this application, a target recognition system based on hybrid active-passive imaging is provided, comprising: an active-passive data acquisition unit, configured to acquire multiple echo signals from a target object acquired by multiple millimeter-wave radars with different viewing angles, and multiple images of the target object acquired by multiple cameras with different viewing angles; an active data encoding unit, configured to pass the waveforms of each echo signal in the multiple echo signals through a first convolutional neural network to obtain multiple first feature matrices, and to arrange the multiple first feature matrices along the channel dimension to obtain a first feature map; and a passive data encoding unit, configured to pass each image in the multiple images through a second convolutional neural network. The system comprises: a first feature map and a second feature map; a feature map fusion unit for fusing the first feature map and the second feature map to obtain a classification feature map; a feature map clustering balance correction unit for correcting the classification feature map based on clustering balance entropy parameters to obtain a corrected classification feature map, wherein the clustering balance entropy parameter correction is based on the clustering results of feature values ​​at each position in the classification feature map; and a recognition result generation unit for passing the corrected classification feature map through a classifier to obtain a classification result, wherein the classification result is used to indicate whether the target being tested is a preset object.

[0007] In the above target recognition system, each layer of the first convolutional neural network performs the following on the input data during the forward propagation of the layer: convolution processing on the input data to generate a convolutional feature map; mean pooling along the channel dimension on the convolutional feature map to generate a pooled feature map; and nonlinear activation on the feature values ​​at each position in the pooled feature map to generate an activation feature map; wherein, the output of the last layer of the first convolutional neural network is the first feature matrix.

[0008] In the above target recognition system, each layer of the second convolutional neural network performs the following operations on the input data during the forward propagation of the layer: convolution processing is performed on the input data using a convolutional kernel to generate a convolutional feature map; mean pooling is performed on the convolutional feature map along the channel dimension to generate a pooled feature map; and nonlinear activation is performed on the feature values ​​at each position in the pooled feature map to generate an activation feature map; wherein, the output of the last layer of the second convolutional neural network is the second feature matrix.

[0009] In the above-described target recognition system, the feature map fusion unit is further configured to fuse the first feature map and the second feature map using the following formula to generate the classification feature map; wherein, the formula is: in, For the classification feature map, This is the first feature map. For the second feature map, "" indicates that the elements at corresponding positions in the first feature map and the second feature map are added together. and The weighting parameter is used to control the balance between the first feature map and the second feature map in the classification feature map.

[0010] In the aforementioned target recognition system, the Greater than 0.5, the Less than 0.5, and the .

[0011] In the above-mentioned target recognition system, the feature map clustering balance correction unit is further used to correct the classification feature map based on the clustering balance entropy parameter using the following formula to obtain the corrected classification feature map; wherein, the formula is: in The feature values ​​at each position in the classification feature map are... The feature values ​​at each position in the corrected classification feature map are... It is the ratio of the number of cluster eigenvalues ​​to the total number of eigenvalues.

[0012] In the above target recognition system, the feature map clustering balance correction unit includes: a clustering subunit, used to calculate the absolute value of the difference between feature values ​​at each position in the classification feature map to cluster the classification feature map to obtain the clustering result.

[0013] In the aforementioned target recognition system, the recognition result generation unit includes: a fully connected encoding subunit, which encodes the corrected classification feature map using multiple fully connected layers of the classifier to transform the corrected classification feature map into a one-dimensional classification feature vector; a soft maximum subunit, which inputs the one-dimensional classification feature vector into a Softmax classification function to obtain the probability value of the one-dimensional classification feature vector belonging to each target object label; and a result generation subunit, which determines the classification result by assigning the target object label corresponding to the one with the largest probability value.

[0014] According to another aspect of this application, a target recognition method based on hybrid active-passive imaging is provided, comprising: acquiring multiple echo signals from a target being measured, collected by multiple millimeter-wave radars with different perspectives, and multiple images of the target being measured, collected by multiple cameras with different perspectives; passing the waveforms of each echo signal in the multiple echo signals through a first convolutional neural network to obtain multiple first feature matrices, and arranging the multiple first feature matrices along the channel dimension to obtain a first feature map; passing each image in the multiple images through a second convolutional neural network to obtain multiple second feature matrices, and arranging the multiple second feature matrices along the channel dimension to obtain a second feature map; fusing the first feature map and the second feature map to obtain a classification feature map; performing clustering balance entropy parameter correction on the classification feature map to obtain a corrected classification feature map, wherein the clustering balance entropy parameter correction is based on the clustering results of feature values ​​at each position in the classification feature map; and passing the corrected classification feature map through a classifier to obtain a classification result, the classification result being used to indicate whether the target being measured is a preset object.

[0015] In the above-mentioned target recognition method based on hybrid active-passive imaging, the classification feature map is corrected based on clustering equilibrium entropy parameters to obtain a corrected classification feature map. The clustering equilibrium entropy parameter correction is based on the clustering results of feature values ​​at various positions in the classification feature map, including: fusing the first feature map and the second feature map using the following formula to generate the classification feature map; wherein the formula is: in, For the classification feature map, This is the first feature map. For the second feature map, "" indicates that the elements at corresponding positions in the first feature map and the second feature map are added together. and This refers to a weighting parameter used to control the balance between the first feature map and the second feature map in the classification feature map. According to another aspect of this application, an electronic device is provided, comprising: a processor; and a memory storing computer program instructions, which, when executed by the processor, cause the processor to perform the target recognition method based on hybrid active-passive imaging described above.

[0016] According to another aspect of this application, a computer-readable medium is provided having computer program instructions stored thereon, which, when executed by a processor, cause the processor to perform the target recognition method based on hybrid active-passive imaging as described above.

[0017] Compared with the prior art, this application provides a target recognition system and method based on active and passive hybrid imaging, which integrates active imaging from multiple millimeter-wave radars and passive imaging from multiple cameras to perceive the target object from different perspectives, thereby improving the accuracy of target recognition. Attached Figure Description

[0018] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0019] Figure 1 This is an application diagram of a target recognition system based on active-passive hybrid imaging, which is a preferred embodiment of the present invention.

[0020] Figure 2 This is a block diagram of a target recognition system based on active-passive hybrid imaging, which is a preferred embodiment of the present invention.

[0021] Figure 3 This is a flowchart of a target recognition method based on active-passive hybrid imaging, according to a preferred embodiment of the present invention.

[0022] Figure 4 This is a framework diagram of a target recognition method based on hybrid active and passive imaging according to a preferred embodiment of the present invention.

[0023] Figure 5 This is a flowchart illustrating the processing of echo signals using a first convolutional neural network in a target recognition method based on hybrid active-passive imaging, according to a preferred embodiment of the present invention.

[0024] Figure 6 This is a flowchart illustrating the image processing method using a second convolutional neural network in a preferred embodiment of the target recognition method based on hybrid active-passive imaging of the present invention.

[0025] Figure 7 This is a block diagram of an electronic device according to a preferred embodiment of the present invention. Detailed Implementation

[0026] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0027] Scenario Overview: In recent years, the development of millimeter-wave imaging technology has provided new technological possibilities for target object identification in public places. Compared with ordinary cameras, millimeter-wave imaging technology has strong penetration through clothing, higher imaging resolution, and greater advantages in radiation safety protection, making it suitable for human body security checks. Therefore, the inventors of this application attempt to integrate millimeter-wave imaging (active imaging) and ordinary camera imaging (passive imaging) to construct a new target identification scheme for public places, thereby providing new technical support for crime prevention in public places.

[0028] Specifically, in order to acquire richer data about the target being measured, the technical solution of this application considers that multiple cameras and multiple millimeter-wave radars are usually deployed in public places. Therefore, it first acquires multiple echo signals from the target being measured collected by multiple millimeter-wave radars with different perspectives, as well as multiple images of the target being measured collected by multiple cameras with different perspectives. It should be understood that the millimeter-wave radars with different perspectives can acquire information about the target being measured from multiple different perspectives, and the cameras with different perspectives can acquire information about the target being measured from different perspectives. This allows for target perception of the target object from multiple perspectives. Furthermore, there is a correlation between the information acquired by the millimeter-wave radars with different perspectives and the information acquired by the cameras with different perspectives; fully utilizing this information can improve the accuracy of target recognition.

[0029] Accordingly, considering the superior performance of convolutional neural networks (CNNs) in image processing, CNNs are used to encode the waveforms of each echo signal in the plurality of echo signals to obtain a plurality of first feature matrices, and CNNs are used to encode each image in the plurality of images to obtain a plurality of second feature matrices. Then, the plurality of first feature matrices and the plurality of second feature matrices are arranged along the channel dimension to obtain a first feature map and a second feature map. It should be understood that the first feature map preserves the differences between information acquired by millimeter-wave radar from different viewpoints in the channel dimension, and the second feature map preserves the differences between information acquired by the camera at different times in the channel dimension.

[0030] Next, the first feature map and the second feature map are fused in a high-dimensional feature space. For example, the first feature map and the second feature map are fused by calculating the positional addition of the first feature map and the second feature map to obtain a classification feature map.

[0031] Since the first feature map corresponds to the high-resolution semantics of millimeter-wave images, its fusion with the second feature map may lead to an imbalance in the class confidence of feature value samples inside and outside the cluster. Therefore, by using the class cross-entropy mechanism to penalize and compensate for the degree of imbalance in confidence between classes, a bidirectional constraint on the information entropy dimension can be applied to the feature value samples to avoid the loss of class distribution balance caused by the fusion of features with different resolution semantics.

[0032] Specifically, cluster balance entropy parameter correction is performed on the classification feature map. This involves first clustering the feature values ​​at each location of the classification feature map, for example, based on the numerical or absolute values ​​of the feature values, to obtain the cluster size within each cluster. ) and eigenvalues ​​outside clusters The correction is as follows: in, This is the ratio of the number of clustered feature values ​​to the total number of feature values. This improves the accuracy of target recognition, thereby enhancing security in public places.

[0033] Figure 1 The illustration shows an application scenario of the target recognition system based on hybrid active and passive imaging according to this application. Figure 1 As shown, in this application scenario, multiple millimeter-wave radars from different perspectives (e.g.) Figure 1 The H in the sample is the target being measured (e.g., the target being measured). Figure 1 Multiple echo signals from B in the image are transmitted through multiple cameras with different perspectives (e.g., ...). Figure 1 The J) acquires multiple images of the target under test, and then inputs the multiple echo signals and the multiple images into a server (e.g., Figure 1 In the case of S), the server is capable of target perception of the object under test from different perspectives based on the multiple echo signals and the multiple images, so as to identify the target under test.

[0034] After introducing the basic principles of this application, various non-limiting embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0035] Exemplary system: Figure 2 The figure shows a block diagram of a target recognition system 100 based on hybrid active and passive imaging according to an embodiment of this application.

[0036] like Figure 2As shown, the target recognition system 100 based on active-passive hybrid imaging includes: an active-passive data acquisition unit 10, an active data encoding unit 20, a passive data encoding unit 30, a feature map fusion unit 40, a feature map clustering balance correction unit 50, and a recognition result generation unit 60. The active-passive data acquisition unit 10 is used to acquire multiple echo signals from the target object collected by multiple millimeter-wave radars with different perspectives, and multiple images of the target object collected by multiple cameras with different perspectives. The active data encoding unit 20 is used to pass the waveforms of each echo signal in the multiple echo signals through a first convolutional neural network to obtain multiple first feature matrices, and arrange the multiple first feature matrices along the channel dimension to obtain a first feature map. The passive data encoding unit 30 is used to pass each image in the multiple images through a second convolutional neural network to obtain multiple second feature matrices, and arrange the multiple first feature matrices along the channel dimension to obtain a first feature map. The second feature matrix is ​​arranged along the channel dimension to obtain a second feature map; the feature map fusion unit 40 is used to fuse the first feature map and the second feature map to obtain a classification feature map; the feature map clustering balance correction unit 50 is used to perform clustering balance entropy parameter correction on the classification feature map to obtain a corrected classification feature map, wherein the clustering balance entropy parameter correction is based on the clustering results of the feature values ​​at each position in the classification feature map; and the recognition result generation unit 60 is used to pass the corrected classification feature map through a classifier to obtain a classification result, wherein the classification result is used to indicate whether the target being tested is a preset object.

[0037] During the operation of the identification system, the active and passive data acquisition unit 10 first acquires multiple echo signals from the target object collected by multiple millimeter-wave radars with different viewing angles, as well as multiple images of the target object collected by multiple cameras with different viewing angles. In this application, in order to acquire richer data about the target object, multiple cameras and multiple millimeter-wave radars are deployed to obtain multiple echo signals and multiple images of the target object from multiple different viewing angles.

[0038] First, multiple echo signals from the target object are acquired by multiple millimeter-wave radars with different viewing angles, and multiple images of the target object are acquired by multiple cameras with different viewing angles. It should be understood that the millimeter-wave radars with different viewing angles can acquire information about the target object from multiple different perspectives, and the cameras with different viewing angles can acquire information about the target object from different perspectives. This allows for target perception from multiple perspectives, and furthermore, there is a correlation between the information acquired by the millimeter-wave radars and the information acquired by the cameras, which, when fully utilized, can improve the accuracy of target recognition.

[0039] Preferably, the millimeter-wave radar is a passive imaging type radar, which uses the millimeter waves emitted by the target itself for imaging. Optionally, the millimeter-wave radar is an active imaging type radar, which emits a millimeter-wave source to the object to be detected and detects the field strength reflected by the object for imaging. Optionally, the plurality of millimeter-wave radars includes both passive and active imaging types. Specifically, the frequency of the millimeter-wave radar is in the band between 30 GHz and 300 GHz, which is between infrared and microwave in the spectrum. This spectrum has a certain degree of penetration into most non-metallic objects compared to visible light and infrared, while possessing a resolution that microwaves do not have.

[0040] In the process where the active data encoding unit 20 passes the waveforms of each of the multiple echo signals through a first convolutional neural network to obtain multiple first feature matrices, the encoding process of the waveforms of each echo signal by the first convolutional neural network includes: First, each layer of the first convolutional neural network performs convolution processing on the waveforms of the echo signals using two-dimensional convolutional kernels to obtain convolutional feature maps, wherein the convolutional feature maps include multiple feature matrices along the channel dimension, and the number of the multiple feature matrices is consistent with the number of the two-dimensional convolutional kernels. Next, the layer of the first convolutional neural network performs mean pooling processing along the channel dimension to obtain pooled feature maps. Then, the layer of the first convolutional neural network performs nonlinear activation on the pooled feature maps to obtain activation feature maps, wherein the nonlinear activation function can be a ReLU activation function, a Sigmoid activation function, etc. That is, each layer of the first convolutional neural network performs convolution, pooling and activation processing on the input data. In this way, during the transfer between layers, the waveform of the echo signal is subjected to deep learning so that the last layer of the first convolutional neural network outputs the first feature matrix.

[0041] In the process of the passive data encoding unit 30 passing each image in the plurality of images through the second convolutional neural network to obtain multiple second feature matrices, the encoding process of the second convolutional neural network for each image in the plurality of images includes: First, each layer of the second convolutional neural network uses a two-dimensional convolutional kernel to convolve the waveform of the image to obtain a convolutional feature map, wherein the convolutional feature map includes multiple feature matrices along the channel dimension, and the number of the multiple feature matrices is consistent with the number of the two-dimensional convolutional kernels. Next, the layer of the second convolutional neural network performs mean pooling along the channel dimension to obtain a pooling feature map. Then, the layer of the second convolutional neural network performs nonlinear activation on the pooling feature map to obtain an activation feature map, wherein the nonlinear activation function can be a ReLU activation function, a Sigmoid activation function, etc. That is, each layer of the second convolutional neural network performs convolution processing, pooling processing, and activation processing on the input data respectively. In this way, in the layer-to-layer transfer, deep learning is performed on the image so that the last layer of the second convolutional neural network outputs the second feature matrix.

[0042] In this application, the excellent performance of convolutional neural networks in image processing is fully considered. Using convolutional neural networks to process the information of the target acquired by millimeter-wave radar and the information of the target acquired by camera can give full play to the advantages of convolutional neural networks in image processing and improve recognition accuracy.

[0043] Specifically, the feature map fusion unit 40 is used to fuse the first feature map and the second feature map to obtain a classification feature map, specifically using the following formula to fuse the first feature map and the second feature map to generate the classification feature map; wherein, the formula is: in, For the classification feature map, This is the first feature map. For the second feature map, "" indicates that the elements at corresponding positions in the first feature map and the second feature map are added together. and The weighting parameter is used to control the balance between the first feature map and the second feature map in the classification feature map.

[0044] That is, specifically, by using weighted parameters and By controlling the balance between the first feature map and the second feature map in the classification feature map, a balance can be achieved between millimeter-wave active imaging features and camera passive imaging features in the classification feature map, resulting in a more accurate recognition effect. Furthermore, the... Greater than 0.5, the Less than 0.5, and the In other words, in this application, during the process of determining the classification feature map using the first feature map and the second feature map, the first feature map, corresponding to multiple echo signals from the target acquired by the millimeter-wave radar, will occupy a larger proportion. Compared to ordinary cameras, millimeter-wave imaging technology has strong penetration through clothing and higher imaging resolution. Increasing the proportion of the first feature map can better improve the resolution of the obtained classification feature map, especially solving problems such as target occlusion.

[0045] Specifically, the first feature map and the second feature map are fused in a high-dimensional feature space. For example, the first feature map and the second feature map are fused by calculating the position-point addition of the first feature map and the second feature map to obtain a classification feature map.

[0046] The feature map clustering balance correction unit 50 is used to perform clustering balance entropy parameter correction on the classification feature map to obtain a corrected classification feature map, wherein the clustering balance entropy parameter correction is based on the clustering results of the feature values ​​at each position in the classification feature map.

[0047] Specifically, the feature map clustering balance correction unit 50 performs clustering balance entropy parameter correction on the classification feature map using the following formula to obtain the corrected classification feature map; wherein, the formula is: in The feature values ​​at each position in the classification feature map are... The feature values ​​at each position in the corrected classification feature map are... It is the ratio of the number of cluster eigenvalues ​​to the total number of eigenvalues.

[0048] Since the first feature map corresponds to the high-resolution semantics of millimeter-wave images, its fusion with the second feature map may lead to an imbalance in the class confidence of feature value samples inside and outside the cluster. Therefore, by using a class cross-entropy mechanism to penalize and compensate for the imbalance in confidence between classes, a bidirectional constraint on the information entropy dimension can be applied to the feature value samples to avoid the loss of class distribution balance caused by the fusion of features with different resolution semantics, thereby improving the accuracy of target recognition and enhancing the safety of public places.

[0049] Furthermore, the feature map clustering balance correction unit 50 includes a clustering subunit 51, which is used to calculate the absolute value of the difference between feature values ​​at each position in the classification feature map to cluster the classification feature map and obtain the clustering result.

[0050] The clustering subunit 51 clusters the classification feature map based on the absolute value of the difference between the feature values ​​at each position in the classification feature map to obtain the clustering result. This can weaken the imbalance of class confidence between feature value samples inside and outside the cluster and improve the accuracy of the clustering result.

[0051] The recognition result generation unit 60 includes: a fully connected encoding subunit 61, which encodes the corrected classification feature map using multiple fully connected layers of the classifier to convert the corrected classification feature map into a one-dimensional classification feature vector; a soft maximum subunit 62, which inputs the one-dimensional classification feature vector into a Softmax classification function to obtain the probability value of the one-dimensional classification feature vector belonging to each target object label; and a result generation subunit 63, which determines the classification result by the target object label corresponding to the one with the largest probability value.

[0052] Specifically, in this embodiment of the application, the process of obtaining a classification result by passing the classification feature map through a classifier includes: processing the classification feature map using the classifier with the following formula to generate the classification result; wherein, the formula is: ,in This indicates that the classification feature map is projected into a classification feature vector. and Here are the weight matrices for each fully connected layer. to This represents the bias matrix of each fully connected layer.

[0053] As described above, the target recognition system 100 based on active-passive hybrid imaging according to the embodiments of this application can be implemented in various wireless terminals, such as servers used to identify the object under test. In one example, the target recognition system 100 based on active-passive hybrid imaging according to the embodiments of this application can be integrated into a wireless terminal as a software module and / or a hardware module. For example, the target recognition system 100 based on active-passive hybrid imaging can be a software module in the operating system of the wireless terminal, or it can be an application developed for the wireless terminal; of course, the target recognition method based on active-passive hybrid imaging can also be one of the many hardware modules of the wireless terminal.

[0054] Alternatively, in another example, the target recognition system 100 based on hybrid active and passive imaging and the wireless terminal may also be separate devices, and the target recognition system 100 based on hybrid active and passive imaging may be connected to the wireless terminal via wired and / or wireless networks and transmit interactive information in accordance with an agreed data format.

[0055] Exemplary method: Figure 3The illustration shows a flowchart of a target recognition method based on hybrid active and passive imaging according to an embodiment of this application.

[0056] like Figure 3 As shown, the target recognition method based on hybrid active-passive imaging includes: S101, acquiring multiple echo signals from the target object collected by multiple millimeter-wave radars with different perspectives and multiple images of the target object collected by multiple cameras with different perspectives; S102, passing the waveforms of each echo signal in the multiple echo signals through a first convolutional neural network to obtain multiple first feature matrices, and arranging the multiple first feature matrices along the channel dimension to obtain a first feature map; S103, passing each image in the multiple images through a second convolutional neural network to obtain multiple second feature matrices, and arranging the multiple second feature matrices along the channel dimension to obtain a second feature map; S104, fusing the first feature map and the second feature map to obtain a classification feature map; S105, performing clustering balance entropy parameter correction on the classification feature map to obtain a corrected classification feature map, wherein the clustering balance entropy parameter correction is based on the clustering results of the feature values ​​at each position in the classification feature map; and S106, passing the corrected classification feature map through a classifier to obtain a classification result, the classification result being used to indicate whether the target object is a preset object.

[0057] refer to Figure 5 In step S102 above, the waveforms of each echo signal in the plurality of echo signals are respectively passed through a first convolutional neural network to obtain a plurality of first feature matrices, including: S1021, using the convolution kernel of the first convolutional neural network to perform convolution processing on the input data to generate a convolutional feature map; S1022, using the convolutional layer of the first convolutional neural network to perform global mean pooling along the channel dimension on the convolutional feature map to generate a pooled feature matrix; and S1023, using the activation layer of the first convolutional neural network to perform nonlinear activation on the feature values ​​at each position in the pooled feature matrix to generate an activation feature map; S1024, using the last layer of the first convolutional neural network to output the first feature matrix.

[0058] refer to Figure 6In step S103 above, each of the multiple images is passed through a second convolutional neural network to obtain multiple second feature matrices, including: S1031, using the convolution kernel of the second convolutional neural network to perform convolution processing on the input data to generate a convolutional feature map; S1032, using the convolutional layer of the second convolutional neural network to perform global mean pooling along the channel dimension on the convolutional feature map to generate a pooled feature matrix; and S1033, using the activation layer of the second convolutional neural network to perform nonlinear activation on the feature values ​​at each position in the pooled feature matrix to generate an activation feature map; S1034, using the output of the last layer of the second convolutional neural network as the second feature matrix.

[0059] In step S104 above, when fusing the first feature map and the second feature map to obtain the classification feature map, the first feature map and the second feature map are fused using the following formula to generate the classification feature map; wherein, the formula is: in, For the classification feature map, This is the first feature map. For the second feature map, "" indicates that the elements at corresponding positions in the first feature map and the second feature map are added together. and These are weighting parameters used to control the balance between the first feature map and the second feature map in the classification feature map. Greater than 0.5, the Less than 0.5, and the .

[0060] In step S105 above, the classification feature map is corrected based on the clustering equilibrium entropy parameter to obtain the corrected classification feature map. The correction based on the clustering equilibrium entropy parameter is based on the clustering results of the feature values ​​at each position in the classification feature map, and the classification feature map is corrected based on the clustering equilibrium entropy parameter to obtain the corrected classification feature map using the following formula.

[0061] The formula is as follows: in The feature values ​​at each position in the classification feature map are... The feature values ​​at each position in the corrected classification feature map are... It is the ratio of the number of cluster eigenvalues ​​to the total number of eigenvalues.

[0062] Step S106 above, which involves passing the corrected classification feature map through a classifier to obtain a classification result, wherein the classification result is used to indicate whether the target being tested is a preset object, further includes: S1061, encoding the corrected classification feature map using multiple fully connected layers of the classifier to convert the corrected classification feature map into a one-dimensional classification feature vector; S1061, inputting the one-dimensional classification feature vector into a Softmax classification function to obtain the probability value of the one-dimensional classification feature vector belonging to each target object label; and S1061, determining the classification result by identifying the target object label corresponding to the one with the largest probability value.

[0063] Here, those skilled in the art will understand that the specific functions and operations of each step in the above-described target recognition method based on hybrid active and passive imaging have been referenced above. Figure 1 The target recognition system based on hybrid active and passive imaging has been described in detail, and therefore, its repeated description will be omitted.

[0064] Exemplary electronic device: Below, refer to Figure 7 This describes an electronic device according to embodiments of the present application.

[0065] Figure 7 A block diagram of an electronic device according to an embodiment of this application is illustrated.

[0066] like Figure 7 As shown, the electronic device 200 includes one or more processors 11 and memory 12.

[0067] The processor 11 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 200 to perform desired functions.

[0068] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the program instructions to implement the target recognition method based on active-passive hybrid imaging of the various embodiments of this application described above, and / or other desired functions. Various contents such as millimeter-wave radar and classification feature maps may also be stored in the computer-readable storage medium.

[0069] In one example, the electronic device 200 may also include an input device 13 and an output device 14, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0070] The input device 13 may include, for example, a keyboard, a mouse, etc.

[0071] The output device 14 can output various information to the outside, including target recognition results. The output device 14 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0072] Of course, for the sake of simplicity, Figure 6 Only some of the components of the electronic device 200 relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device 200 may include any other suitable components depending on the specific application.

[0073] Exemplary computer program products and computer-readable storage media: In addition to the methods and apparatus described above, embodiments of this application may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the target recognition method based on active-passive hybrid imaging according to various embodiments of this application as described in the "Exemplary Methods" section of this specification.

[0074] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0075] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the target recognition method based on active-passive hybrid imaging according to various embodiments of this application as described in the "Exemplary Methods" section above.

[0076] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0077] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0078] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0079] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0080] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0081] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A target recognition system based on hybrid active-passive imaging, characterized in that, It includes: an active and passive data acquisition unit, used to acquire multiple echo signals from the target being measured collected by multiple millimeter-wave radars with different perspectives, and multiple images of the target being measured collected by multiple cameras with different perspectives; and an active data encoding unit, used to pass the waveform of each echo signal in the multiple echo signals through a first convolutional neural network to obtain multiple first feature matrices, and to arrange the multiple first feature matrices along the channel dimension to obtain a first feature map. A passive data encoding unit is used to pass each of the plurality of images through a second convolutional neural network to obtain a plurality of second feature matrices, and to arrange the plurality of second feature matrices along the channel dimension to obtain a second feature map; The system includes a feature map fusion unit for fusing the first feature map and the second feature map to obtain a classification feature map; a feature map clustering balance correction unit for correcting the classification feature map based on a clustering balance entropy parameter to obtain a corrected classification feature map, wherein the clustering balance entropy parameter correction is based on the clustering results of the feature values ​​at each position in the classification feature map; and a recognition result generation unit for passing the corrected classification feature map through a classifier to obtain a classification result, wherein the classification result is used to indicate whether the target being tested is a preset object.

2. The target recognition system based on hybrid active-passive imaging according to claim 1, wherein, Each layer of the first convolutional neural network performs the following operations on the input data during the forward propagation of the layer: convolution processing on the input data to generate a convolutional feature map; mean pooling along the channel dimension on the convolutional feature map to generate a pooled feature map; and nonlinear activation on the feature values ​​at each position in the pooled feature map to generate an activation feature map; wherein, the output of the last layer of the first convolutional neural network is the first feature matrix.

3. The target recognition system based on hybrid active-passive imaging according to claim 1, wherein, Each layer of the second convolutional neural network performs the following operations on the input data during the forward propagation of the layer: convolution processing is performed on the input data using a convolutional kernel to generate a convolutional feature map; mean pooling is performed on the convolutional feature map along the channel dimension to generate a pooled feature map; and nonlinear activation is performed on the feature values ​​at each position in the pooled feature map to generate an activation feature map; wherein, the output of the last layer of the second convolutional neural network is the second feature matrix.

4. The target recognition system based on hybrid active-passive imaging according to claim 1, wherein, The feature map fusion unit is further configured to fuse the first feature map and the second feature map using the following formula to generate the classification feature map; wherein the formula is: in, For the classification feature map, This is the first feature map. For the second feature map, "" indicates that the elements at corresponding positions in the first feature map and the second feature map are added together. and The weighting parameter is used to control the balance between the first feature map and the second feature map in the classification feature map.

5. The target recognition system based on hybrid active-passive imaging according to claim 4, wherein, The Greater than 0.5, the Less than 0.5, and the .

6. The target recognition system based on hybrid active-passive imaging according to claim 5, wherein, The feature map clustering balance correction unit is further used to correct the classification feature map based on the clustering balance entropy parameter using the following formula to obtain the corrected classification feature map; The formula is as follows: in The feature values ​​at each position in the classification feature map are... The feature values ​​at each position in the corrected classification feature map are... It is the ratio of the number of cluster eigenvalues ​​to the total number of eigenvalues.

7. The target recognition system based on hybrid active-passive imaging according to claim 6, wherein, The feature map clustering balance correction unit includes: a clustering subunit, used to calculate the absolute value of the difference between feature values ​​at each position in the classification feature map to cluster the classification feature map and obtain the clustering result.

8. The target recognition system based on hybrid active-passive imaging according to claim 7, wherein, The recognition result generation unit includes: a fully connected encoding subunit, which encodes the corrected classification feature map using multiple fully connected layers of the classifier to transform the corrected classification feature map into a one-dimensional classification feature vector; a soft maximum subunit, which inputs the one-dimensional classification feature vector into a Softmax classification function to obtain the probability value of the one-dimensional classification feature vector belonging to each target object label; and a result generation subunit, which determines the classification result by assigning the target object label corresponding to the one with the largest probability value.

9. A target recognition method based on hybrid active-passive imaging, characterized in that, include: Acquire multiple echo signals from the target being measured by multiple millimeter-wave radars with different perspectives, and multiple images of the target being measured by multiple cameras with different perspectives; The waveforms of each echo signal in the plurality of echo signals are respectively passed through a first convolutional neural network to obtain a plurality of first feature matrices, and the plurality of first feature matrices are arranged along the channel dimension to obtain a first feature map; each image in the plurality of images is respectively passed through a second convolutional neural network to obtain a plurality of second feature matrices, and the plurality of second feature matrices are arranged along the channel dimension to obtain a second feature map; the first feature map and the second feature map are fused to obtain a classification feature map; the classification feature map is corrected based on clustering balance entropy parameters to obtain a corrected classification feature map, wherein the clustering balance entropy parameter correction is based on the clustering results of the feature values ​​at each position in the classification feature map; and the corrected classification feature map is passed through a classifier to obtain a classification result, the classification result being used to indicate whether the target being tested is a preset object.

10. The target recognition method based on hybrid active-passive imaging according to claim 9, wherein, The classification feature map is corrected based on clustering equilibrium entropy parameters to obtain a corrected classification feature map. The clustering equilibrium entropy parameter correction is based on the clustering results of feature values ​​at various positions in the classification feature map, and includes fusing the first feature map and the second feature map using the following formula to generate the classification feature map; wherein the formula is: in, For the classification feature map, This is the first feature map. For the second feature map, "" indicates that the elements at corresponding positions in the first feature map and the second feature map are added together. and The weighting parameter is used to control the balance between the first feature map and the second feature map in the classification feature map.

Images