Target detection method based on FPGA (Field Programmable Gate Array) multi-modal feature fusion and related device

Abstract

The embodiment of the invention provides a target detection method based on FPGA (Field Programmable Gate Array) multi-modal feature fusion and a related device, which realize real-time extraction and fusion of multi-modal features by utilizing the high parallel processing capability of the FPGA, meet the requirement of high real-time application, and remarkably reduce the overall power consumption of a system through the efficient computing capability and low power consumption characteristic of the FPGA. A Bayesian feature fusion method is adopted, the relation between multi-modal features is fully utilized, the accuracy and robustness of target detection are improved, meanwhile, feature information of different sensors is compared and optimized in real time through a suppression-return mechanism, in the fusion process, a system can continuously compare and optimize the feature information, and the fusion accuracy and robustness are improved. And the characteristics extracted by the infrared image are consistent with the data acquired by the millimeter wave radar. In this way, it can be ensured that the fused feature information is more accurate and reliable. Based on this, the accuracy and robustness of target detection can be improved, and different application scenes can be flexibly adapted.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of target detection technology, and in particular to a target detection method and related devices based on FPGA multi-modal feature fusion. Background Art

[0002] Traditional target detection methods usually rely on a single sensor to obtain data. However, a single sensor often cannot provide reliable and accurate target information in complex environments. Therefore, how to improve the reliability and accuracy of target detection has become a technical problem that needs to be solved in order to meet the development requirements of intelligent monitoring, autonomous driving and other fields. Summary of the invention

[0003] The embodiment of the present invention provides a target detection method and related devices based on FPGA multimodal feature fusion, which can improve the accuracy and robustness of target detection through efficient feature extraction and fusion mechanism.

[0004] In a first aspect, an embodiment of the present invention provides a target detection method based on FPGA multimodal feature fusion, comprising:

[0005] Acquire target infrared image data from infrared imaging equipment through FPGA system;

[0006] Acquire target millimeter-wave radar data from the millimeter-wave radar through the FPGA system;

[0007] Performing multi-level feature extraction on the target infrared image data to obtain multiple target infrared image features;

[0008] Normalizing the target infrared image features to obtain normalized target infrared image features;

[0009] Extracting target millimeter-wave radar features based on the target millimeter-wave radar data;

[0010] Performing Bayesian feature fusion on the normalized target infrared image features and the target millimeter wave radar features to generate a global saliency map;

[0011] The normalized target infrared image features are compared with the target millimeter-wave radar features. When the normalized target infrared image features are inconsistent with the target millimeter-wave radar features, the global saliency map is corrected through a suppression-return mechanism until the target millimeter-wave radar features are consistent with the normalized target infrared image features. When the normalized target infrared image features are consistent with the target millimeter-wave radar features, the target detection result is output.

[0012] In some embodiments, the multi-level feature extraction is performed on the target infrared image data to obtain multiple target infrared image features, including:

[0013] De-noising the target infrared image data to obtain first infrared image data;

[0014] performing contrast enhancement on the first infrared image data to obtain second infrared image data;

[0015] Multi-level feature extraction is performed on the second infrared image data to obtain multiple target infrared image features.

[0016] In some embodiments, the performing multi-level feature extraction on the second infrared image data to obtain multiple target infrared image features includes:

[0017] Performing grayscale and smoothing filtering processing on the second infrared image data to obtain a third infrared image;

[0018] Extracting image edge information from the third infrared image using a Canny edge detection algorithm;

[0019] The histogram of the third infrared image in the local gradient direction is calculated by the HOG algorithm to generate a plurality of target infrared image feature vectors.

[0020] In some embodiments, the normalizing the target infrared image features to obtain the normalized target infrared image features includes:

[0021] Calculating the mean and standard deviation of each target infrared image feature;

[0022] Subtract the mean from the feature value of each target infrared image feature to obtain each target feature vector;

[0023] Each of the target feature vectors is divided by the standard deviation to obtain multi-normalized target infrared image features.

[0024] In some embodiments, extracting target millimeter-wave radar features based on the target millimeter-wave radar data includes:

[0025] Filtering the target millimeter-wave radar data to obtain first millimeter-wave radar data;

[0026] quantizing the first millimeter-wave radar data to obtain second millimeter-wave radar data;

[0027] performing noise reduction on the second millimeter-wave radar data to obtain third millimeter-wave radar data;

[0028] Perform feature extraction on the third millimeter-wave radar data to obtain target millimeter-wave radar features.

[0029] In some embodiments, performing Bayesian feature fusion on the normalized target infrared image features and the target millimeter wave radar features to generate a global saliency map includes:

[0030] Performing Bayesian feature fusion on the normalized target infrared image features and the target millimeter wave radar features to obtain fused data;

[0031] Performing saliency calculation on the fused data to obtain a saliency map;

[0032] Dividing the saliency map into salient areas and non-salient areas according to a preset threshold;

[0033] Dilation and erosion processing is performed on the salient area and the non-salient area to generate a global salient map.

[0034] In some embodiments, performing saliency calculation on the fused data to obtain a saliency map includes:

[0035] determining a fused image based on the fused data;

[0036] Converting the fused image to the frequency domain by Fourier transform to obtain the frequency spectrum characteristics of the fused image;

[0037] A saliency calculation is performed according to the frequency spectrum features of the fused image to obtain a saliency map.

[0038] In a second aspect, an embodiment of the present invention further provides a target detection device based on FPGA multimodal feature fusion, the device comprising:

[0039] A first acquisition module is used to acquire target infrared image data from an infrared imaging device through an FPGA system;

[0040] The second acquisition module is used to acquire target millimeter-wave radar data from the millimeter-wave radar through the FPGA system;

[0041] A first extraction module is used to perform multi-level feature extraction on the target infrared image data to obtain multiple target infrared image features;

[0042] A standardization module, used for normalizing the target infrared image features to obtain normalized target infrared image features;

[0043] A second extraction module is used to extract target millimeter-wave radar features based on the target millimeter-wave radar data;

[0044] A fusion module, used for performing Bayesian feature fusion on the normalized target infrared image features and the target millimeter wave radar features to generate a global saliency map;

[0045] The output module is used to compare the normalized target infrared image features with the target millimeter-wave radar features. When the normalized target infrared image features are inconsistent with the target millimeter-wave radar features, the global saliency map is corrected by a suppression-return mechanism until the target millimeter-wave radar features are consistent with the normalized target infrared image features; when the normalized target infrared image features are consistent with the target millimeter-wave radar features, the target detection result is output.

[0046] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the target detection method based on FPGA multimodal feature fusion as described in the first aspect when executing the computer program.

[0047] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the target detection method based on FPGA multimodal feature fusion as described in the first aspect.

[0048] According to the target detection method and related device based on FPGA multimodal feature fusion provided by the embodiment of the present invention, the target detection method based on FPGA multimodal feature fusion includes: obtaining target infrared image data from an infrared imaging device through an FPGA system; obtaining target millimeter-wave radar data from a millimeter-wave radar through an FPGA system; performing multi-level feature extraction on the target infrared image data to obtain multiple target infrared image features; normalizing the target infrared image features to obtain normalized target infrared image features; extracting target millimeter-wave radar features based on target millimeter-wave radar data; performing Bayesian feature fusion on the normalized target infrared image features and the target millimeter-wave radar features to generate a global saliency map; comparing the normalized target infrared image features with the target millimeter-wave radar features, and in the case where the normalized target infrared image features are inconsistent with the target millimeter-wave radar features, correcting the global saliency map through a suppression-return mechanism until the target millimeter-wave radar features are consistent with the normalized target infrared image features; and outputting the target detection result when the normalized target infrared image features are consistent with the target millimeter-wave radar features. In the process of target feature extraction of the infrared image acquired at the front end by the FPGA system, multiple features such as the shape, size, etc. of the target are considered. These feature extraction modules process and analyze the image to identify and extract the key features of the target. Then, the obtained features will be integrated and normalized, and then, through the Bayesian feature fusion method, the system will fuse the feature information from different sources to generate a global saliency map. This global saliency map can more comprehensively describe the characteristics and state of the target, improving the accuracy and robustness of target detection. At the same time, the millimeter wave radar end will also obtain target data, such as the distance, speed and other information of the target. These data may be different from the features extracted from the infrared image, so further comparison and optimization are required. When inconsistency occurs, the global saliency map is corrected by the suppression-return mechanism. During the fusion process, the system will continuously compare and optimize the feature information until the features extracted from the infrared image and the data obtained by the millimeter wave radar are consistent. In this way, it can be ensured that the fused feature information is more accurate and reliable. Based on this, the embodiment of the present invention can improve the accuracy and robustness of target detection through efficient feature extraction and fusion mechanisms. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a flow chart of a target detection method based on FPGA multi-modal feature fusion provided by an embodiment of the present invention;

[0050] Figure 2 is a feature information fusion flow chart provided by an embodiment of the present invention;

[0051] Figure 3It is a schematic diagram of the FPGA structure provided by an embodiment of the present invention;

[0052] Figure 4 is a Bayesian feature fusion flow chart provided by an embodiment of the present invention;

[0053] Figure 5 is a flowchart of generating a global saliency map provided by an embodiment of the present invention;

[0054] Figure 6 It is a schematic diagram of a target detection device based on FPGA multi-modal feature fusion provided by an embodiment of the present invention;

[0055] Figure 7 is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0057] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the following drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0058] In the embodiments of the present invention, the words "further", "exemplarily" or "optionally" are used to indicate examples, illustrations or descriptions, and should not be interpreted as being more preferred or more advantageous than other embodiments or designs. The use of the words "further", "exemplarily" or "optionally" is intended to present related concepts in a specific way.

[0059] First, several terms involved in the present invention are analyzed:

[0060] FPGA (Field Programmable Gate Array): is a product further developed on the basis of programmable devices such as PAL (Programmable Array Logic) and GAL (General Array Logic). It appears as a semi-custom circuit in the field of Application Specific Integrated Circuit (ASIC), which not only solves the shortcomings of custom circuits, but also overcomes the shortcomings of the limited number of gate circuits of the original programmable devices.

[0061] Infrared imaging technology: is a promising high-tech. Electromagnetic waves longer than 0.78 microns are located outside the red part of the visible light spectrum and are called infrared rays, also known as infrared radiation. It refers to electromagnetic waves with a wavelength of 0.78-1000 microns, of which the part with a wavelength of 0.78-2.0 microns is called near infrared, and the part with a wavelength of 2.0-1000 microns is called thermal infrared. In nature, all objects can radiate infrared rays, so using a detector to measure the infrared difference between the target itself and the background can obtain infrared images formed by different thermal infrared rays. Infrared imaging is a technology that uses infrared radiation to obtain visible plane images of scenes. The basic principle is to record the difference (distribution state) of infrared energy reflected or radiated by the scene or different parts of the scene, and convert it into a two-dimensional visible image in a certain way.

[0062] Millimeter wave radar: It is a radar that works in the millimeter wave band (millimeter wave) for detection. Usually millimeter waves refer to the frequency domain of 30 to 300GHz (wavelength of 1 to 10mm). The wavelength of millimeter waves is between microwaves and centimeter waves, so millimeter wave radar has some advantages of microwave radar and photoelectric radar. Compared with centimeter wave seekers, millimeter wave seekers have the characteristics of small size, light weight and high spatial resolution. Compared with optical seekers such as infrared, laser, and television, millimeter wave seekers have strong ability to penetrate fog, smoke, and dust, and have the characteristics of all-weather (except heavy rainy days) and all-day. In addition, the anti-interference and anti-stealth capabilities of millimeter wave seekers are also better than other microwave seekers. Millimeter wave radar can distinguish and identify very small targets, and can identify multiple targets at the same time; it has imaging capabilities, small size, good mobility and concealment, and strong survivability on the battlefield.

[0063] In order to more conveniently describe the working principle of the embodiment of the present invention later, an introduction to relevant technical scenarios is first given below.

[0064] Traditional target detection methods usually rely on a single sensor to obtain data. However, a single sensor often cannot provide reliable and accurate target information in complex environments. Therefore, how to improve the reliability and accuracy of target detection has become a technical problem that needs to be solved in order to meet the development requirements of intelligent monitoring, autonomous driving and other fields.

[0065] Based on this, the present invention provides a target detection method and related device based on FPGA multimodal feature fusion. Among them, the target detection method based on FPGA multimodal feature fusion includes: obtaining target infrared image data from an infrared imaging device through an FPGA system; obtaining target millimeter-wave radar data from a millimeter-wave radar through an FPGA system; performing multi-level feature extraction on the target infrared image data to obtain multiple target infrared image features; performing normalization processing on the target infrared image features to obtain normalized target infrared image features; extracting target millimeter-wave radar features based on target millimeter-wave radar data; performing Bayesian feature fusion on the normalized target infrared image features and the target millimeter-wave radar features to generate a global saliency map; comparing the normalized target infrared image features with the target millimeter-wave radar features, and in the case where the normalized target infrared image features are inconsistent with the target millimeter-wave radar features, correcting the global saliency map through a suppression-return mechanism until the target millimeter-wave radar features are consistent with the normalized target infrared image features; and outputting the target detection result when the normalized target infrared image features are consistent with the target millimeter-wave radar features. In the process of target feature extraction of the infrared image acquired at the front end by the FPGA system, multiple features such as the shape, size, etc. of the target will be considered. These feature extraction modules process and analyze the image to identify and extract the key features of the target. Then, the obtained features will be integrated and normalized, and then, through the Bayesian feature fusion method, the system will fuse the feature information from different sources to generate a global saliency map. This global saliency map can more comprehensively describe the characteristics and state of the target, improving the accuracy and robustness of target detection. At the same time, the millimeter wave radar end will also obtain target data, such as the distance, speed and other information of the target. These data may be different from the features extracted from the infrared image, so further comparison and optimization are required. When inconsistency occurs, the global saliency map is corrected by the suppression-return mechanism. During the fusion process, the system will continuously compare and optimize the feature information until the features extracted from the infrared image and the data obtained by the millimeter wave radar are consistent. In this way, it can be ensured that the fused feature information is more accurate and reliable. Based on this, the embodiment of the present invention can improve the accuracy and robustness of target detection through efficient feature extraction and fusion mechanisms.

[0066] The embodiments of the present invention are further described below in conjunction with the accompanying drawings.

[0067] like Figure 1 As shown, Figure 1 It is a flowchart of a target detection method based on FPGA multimodal feature fusion provided by an embodiment of the present invention. The target detection method based on FPGA multimodal feature fusion may include but is not limited to steps S101 to S107.

[0068] Step S101: Acquire target infrared image data from an infrared imaging device through an FPGA system;

[0069] Step S102: Acquire target millimeter-wave radar data from the millimeter-wave radar through the FPGA system;

[0070] Step S103: performing multi-level feature extraction on the target infrared image data to obtain multiple target infrared image features;

[0071] Step S104: normalizing the target infrared image features to obtain normalized target infrared image features;

[0072] Step S105: extracting target millimeter-wave radar features based on target millimeter-wave radar data;

[0073] Step S106: performing Bayesian feature fusion on the normalized target infrared image features and the target millimeter wave radar features to generate a global saliency map;

[0074] Step S107: Compare the normalized target infrared image features with the target millimeter-wave radar features. If the normalized target infrared image features are inconsistent with the target millimeter-wave radar features, correct the global saliency map through the suppression-return mechanism until the target millimeter-wave radar features are consistent with the normalized target infrared image features. If the normalized target infrared image features are consistent with the target millimeter-wave radar features, output the target detection result.

[0075] It can be understood that a field programmable gate array (FPGA) system is used for front-end data acquisition to obtain target image data from the infrared imaging device, while collecting data information such as target distance and speed provided by the millimeter wave radar.

[0076] It is understandable that multi-level feature extraction is performed on infrared images, including but not limited to visual features such as the shape and size of the target.

[0077] It is understandable that the extracted feature information is normalized to ensure the uniformity of different sensor data and the consistency of the comparison basis.

[0078] It can be understood that the Bayesian theory is used to fuse the feature information of infrared images and millimeter-wave radar to generate a global saliency map, which combines the advantages of the two sensor data and enhances the comprehensiveness of the target description.

[0079] It can be understood that the infrared image features and millimeter-wave radar data are compared in real time. When the information of the two is inconsistent, the fusion strategy is dynamically adjusted through the suppression-return mechanism until the data reaches consistency, ensuring the accuracy and reliability of feature fusion.

[0080] like Figure 2 As shown in the figure, first, during the target feature extraction process of the infrared image acquired by the front end, the FPGA system will consider multiple features such as the shape, size, etc. of the target. These feature extraction modules will process and analyze the image to identify and extract the key features of the target. Then, the obtained features will be integrated and normalized. Subsequently, through the Bayesian feature fusion method, the system will fuse the feature information from different sources to generate a global saliency map. This global saliency map can more comprehensively describe the characteristics and state of the target, improving the accuracy and robustness of target detection. At the same time, the millimeter-wave radar end will also obtain target data, such as the distance, speed and other information of the target. These data may be different from the features extracted from the infrared image, so further comparison and optimization are required. When inconsistency occurs, the global saliency map is corrected by the suppression-return mechanism. During the fusion process, the system will continuously compare and optimize the feature information until the features extracted from the infrared image and the data obtained by the millimeter-wave radar are consistent. In this way, it can be ensured that the fused feature information is more accurate and reliable.

[0081] In one embodiment, multi-level feature extraction is performed on the target infrared image data to obtain multiple target infrared image features, including:

[0082] De-noising the target infrared image data to obtain first infrared image data;

[0083] performing contrast enhancement on the first infrared image data to obtain second infrared image data;

[0084] Multi-level feature extraction is performed on the second infrared image data to obtain multiple target infrared image features.

[0085] In one embodiment, multi-level feature extraction is performed on the second infrared image data to obtain multiple target infrared image features, including:

[0086] gray-scaling and smoothing filtering are performed on the second infrared image data to obtain a third infrared image;

[0087] The image edge information is extracted from the third infrared image using the Canny edge detection algorithm;

[0088] The histogram of the third infrared image in the local gradient direction is calculated by the HOG algorithm to generate multiple target infrared image feature vectors.

[0089] In one embodiment, the target infrared image features are normalized to obtain normalized target infrared image features, including:

[0090] Calculate the mean and standard deviation of each target infrared image feature;

[0091] Subtract the mean from the eigenvalue of each target infrared image feature to obtain each target eigenvector;

[0092] Each target feature vector is divided by the standard deviation to obtain multi-normalized target infrared image features, thereby ensuring the comparability between different features.

[0093] In one embodiment, extracting target millimeter-wave radar features based on target millimeter-wave radar data includes:

[0094] Filtering the target millimeter-wave radar data to obtain first millimeter-wave radar data;

[0095] quantizing the first millimeter-wave radar data to obtain second millimeter-wave radar data;

[0096] De-noising the second millimeter-wave radar data to obtain third millimeter-wave radar data;

[0097] Feature extraction is performed on the third millimeter-wave radar data to obtain target millimeter-wave radar features.

[0098] In one embodiment, the normalized target infrared image features and the target millimeter wave radar features are subjected to Bayesian feature fusion to generate a global saliency map, including:

[0099] Perform Bayesian feature fusion on the normalized target infrared image features and target millimeter wave radar features to obtain fused data;

[0100] Perform saliency calculation on the fused data to obtain a saliency map;

[0101] Divide the saliency map into salient areas and non-salient areas according to a preset threshold;

[0102] The salient and non-salient regions are dilated and eroded to generate a global saliency map.

[0103] In one embodiment, saliency calculation is performed on the fused data to obtain a saliency map, including:

[0104] determining a fused image based on the fused data;

[0105] The fused image is converted into the frequency domain through Fourier transform to obtain the frequency spectrum characteristics of the fused image;

[0106] The saliency is calculated according to the spectral features of the fused image to obtain a saliency map.

[0107] In one embodiment, the above steps can be integrated into various modules and then embedded into the FPGA system. Figure 3 shown.

[0108] (1) Front-end data collection:

[0109] The infrared data is input to the infrared image input module through the MIPI interface. This module performs preliminary processing on the input infrared data, such as denoising and contrast enhancement, to prepare for subsequent feature extraction.

[0110] The millimeter wave data is input to the ADC sampling module through the 4-channel ADC interface. The millimeter wave data is collected and digitized in the ADC sampling module. The ADC sampling module performs filtering, quantization and noise reduction inside to obtain high-quality millimeter wave radar data information.

[0111] (2) Feature extraction:

[0112] The infrared image input module passes the processed data to the feature extraction module. The feature extraction module uses Canny edge detection and HOG algorithm to extract the key feature vectors in the image.

[0113] First, the transmitted image is grayed and smoothed, then the Canny edge detection algorithm is used to extract the edge information in the image, and finally the HOG algorithm is used to calculate the histogram of the local gradient direction of the image, and finally the feature vector is generated.

[0114] (3) Standardization:

[0115] The extracted feature vector is passed to the normalization module. In the normalization module, standardization and normalization are performed to eliminate the dimension differences and noise effects in the feature data. First, the mean m of the feature vector is calculated. v With standard deviation σ v .

[0116]

[0117] Subtract the mean m from each eigenvalue v Get the new feature vector This eliminates the mean bias between features. Each eigenvalue in is divided by the standard deviation σ v , which is the standardization operation, so as to ensure the comparability between different features.

[0118] (4) Bayesian feature fusion

[0119] The Bayesian feature fusion module is the core part of the system, responsible for fusing infrared data and millimeter wave data. This module is based on the Bayesian law. The specific process is as follows Figure 4 shown.

[0120] First, based on prior knowledge, the prior probability distribution of each type of target is set, and then the likelihood function of each type of target is calculated based on the feature data output by the feature extraction module. Then, the posterior probability is calculated according to the Bayesian formula of equations (1-3) and (1-4), and the probability of the target category is updated.

[0121]

[0122] Where:

[0123] P(target|feature): posterior probability, the probability that the target is recognized under the condition that the feature is observed;

[0124] P(feature|target): likelihood probability, the probability of the corresponding feature occurring under the condition that the target is identified;

[0125] P (target): the prior probability of the target, which is generally a probability value obtained from past experience and analysis. It is also the first step in the flowchart, setting the prior probability;

[0126] P(feature): marginal probability of the feature, which is the probability of the feature occurring without considering whether the target is recognized or not.

[0127] After the calculation is completed, the maximum a posteriori probability criterion is used according to the posterior probability to decide the final detection result.

[0128] (5) Feedback and display

[0129] After completing the first four steps, the output results can be passed to the global saliency map module. The function of the global saliency map module is divided into three processes as follows: Figure 5 shown.

[0130] Multi-scale feature fusion aims to further fuse the feature vectors of infrared data processed by the front end with the data transmitted by the millimeter-wave radar. The fusion method is based on the Bayesian criterion. First, the saliency of the fused data is calculated to analyze the eigenvalues ​​of each area of ​​the image. The image is converted to the frequency domain through Fourier transform, and the spectral characteristics of the image are analyzed to calculate the saliency. Then, according to actual needs, the appropriate threshold is selected to divide the saliency map into salient areas and non-salient areas, and through operations such as expansion and corrosion, the complete saliency information of the target is finally extracted.

[0131] The fused results are connected to the suppression-return module. This module continuously receives the data transmitted by the millimeter-wave radar and the fused feature data for further comparison and optimization. When inconsistent data is detected, the global saliency map is corrected through the suppression-return mechanism; if the data is consistent, the final target detection result is output.

[0132] Based on this, the present invention proposes a target detection method based on FPGA multimodal feature fusion, which aims to improve real-time performance, reduce power consumption, optimize feature fusion effects, introduce dynamic optimization mechanisms, and enhance the scalability and flexibility of the system. The high parallel processing capability of FPGA is used to realize the real-time extraction and fusion of multimodal features, meet the needs of high real-time applications, and significantly reduce the overall power consumption of the system through its efficient computing power and low power consumption characteristics. The Bayesian feature fusion method is adopted to make full use of the relationship between multimodal features to improve the accuracy and robustness of target detection. At the same time, the feature information of different sensors is compared and optimized in real time through the suppression-return mechanism to ensure the consistency and reliability of the fusion results. With the programmable characteristics of FPGA, the system can flexibly adapt to different application scenarios, improve adaptability and scalability, thereby overcoming the shortcomings of traditional systems and making it more widely applicable and higher performance in practical applications (such as intelligent monitoring, autonomous driving, etc.).

[0133] In addition, if Figure 6 As shown, one embodiment of the present invention further discloses a target detection device based on FPGA multi-modal feature fusion, the device comprising:

[0134] A first acquisition module 110 is used to acquire target infrared image data from an infrared imaging device through an FPGA system;

[0135] A second acquisition module 120 is used to acquire target millimeter-wave radar data from the millimeter-wave radar through the FPGA system;

[0136] The first extraction module 130 is used to perform multi-level feature extraction on the target infrared image data to obtain multiple target infrared image features;

[0137] The standardization module 140 is used to perform standardization processing on the target infrared image features to obtain the standardized target infrared image features;

[0138] A second extraction module 150 is used to extract target millimeter-wave radar features based on target millimeter-wave radar data;

[0139] A fusion module 160 is used to perform Bayesian feature fusion on the normalized target infrared image features and the target millimeter wave radar features to generate a global saliency map;

[0140] The output module 170 is used to compare the normalized target infrared image features with the target millimeter-wave radar features. When the normalized target infrared image features are inconsistent with the target millimeter-wave radar features, the global saliency map is corrected by the suppression-return mechanism until the target millimeter-wave radar features are consistent with the normalized target infrared image features; when the normalized target infrared image features are consistent with the target millimeter-wave radar features, the target detection result is output.

[0141] The lithium-ion battery capacity prediction device of the embodiment of the present invention is used to execute the target detection method based on FPGA multimodal feature fusion in the above embodiment. Its specific processing process is the same as the target detection method based on FPGA multimodal feature fusion in the above embodiment, and will not be repeated here.

[0142] In addition, if Figure 7 As shown, an embodiment of the present invention also discloses an electronic device, including: at least one processor 210; at least one memory 220, for storing at least one program; when the at least one program is executed by the at least one processor 210, a target detection method based on FPGA multimodal feature fusion as in any of the previous embodiments is implemented.

[0143] In addition, an embodiment of the present invention further discloses a computer-readable storage medium, which stores computer-executable instructions for executing a target detection method based on FPGA multimodal feature fusion as in any of the previous embodiments.

[0144] The system architecture and application scenarios described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art will appreciate that with the evolution of the system architecture and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are equally applicable to similar technical problems.

[0145] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0146] In hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0147] The terms "component", "module", "system", etc. used in this specification are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, or a computer. By way of illustration, both applications running on a computing device and a computing device can be components. One or more components may reside in a process or an execution thread, and a component may be located on a computer or distributed between two or more computers. In addition, these components may be executed from various computer-readable media having various data structures stored thereon. Components may communicate, for example, through local or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, or a network, such as the Internet interacting with other systems through signals).

Claims

1. A target detection method based on FPGA multimodal feature fusion, characterized in that: include: Acquire target infrared image data from infrared imaging equipment through FPGA system; Acquire target millimeter-wave radar data from the millimeter-wave radar through the FPGA system; Performing multi-level feature extraction on the target infrared image data to obtain multiple target infrared image features; Normalizing the target infrared image features to obtain normalized target infrared image features; Extracting target millimeter-wave radar features based on the target millimeter-wave radar data; Performing Bayesian feature fusion on the normalized target infrared image features and the target millimeter wave radar features to generate a global saliency map; Comparing the normalized target infrared image features with the target millimeter-wave radar features, and in the case where the normalized target infrared image features are inconsistent with the target millimeter-wave radar features, correcting the global saliency map through a suppression-return mechanism until the target millimeter-wave radar features are consistent with the normalized target infrared image features; When the normalized target infrared image features are consistent with the target millimeter wave radar features, the target detection result is output.

2. The target detection method based on FPGA multimodal feature fusion according to claim 1 is characterized in that: The multi-level feature extraction is performed on the target infrared image data to obtain multiple target infrared image features, including: De-noising the target infrared image data to obtain first infrared image data; performing contrast enhancement on the first infrared image data to obtain second infrared image data; Multi-level feature extraction is performed on the second infrared image data to obtain multiple target infrared image features.

3. The target detection method based on FPGA multimodal feature fusion according to claim 2 is characterized in that: The performing multi-level feature extraction on the second infrared image data to obtain a plurality of target infrared image features includes: Performing grayscale and smoothing filtering processing on the second infrared image data to obtain a third infrared image; Extracting image edge information from the third infrared image using a Canny edge detection algorithm; The histogram of the third infrared image in the local gradient direction is calculated by the HOG algorithm to generate a plurality of target infrared image feature vectors.

4. The target detection method based on FPGA multimodal feature fusion according to claim 1, characterized in that: The step of normalizing the target infrared image features to obtain normalized target infrared image features includes: Calculating the mean and standard deviation of each target infrared image feature; Subtract the mean from the feature value of each target infrared image feature to obtain each target feature vector; Each of the target feature vectors is divided by the standard deviation to obtain multi-normalized target infrared image features.

5. The target detection method based on FPGA multimodal feature fusion according to claim 1, characterized in that: The extracting and obtaining target millimeter-wave radar features based on the target millimeter-wave radar data includes: Filtering the target millimeter-wave radar data to obtain first millimeter-wave radar data; quantizing the first millimeter-wave radar data to obtain second millimeter-wave radar data; performing noise reduction on the second millimeter-wave radar data to obtain third millimeter-wave radar data; Perform feature extraction on the third millimeter-wave radar data to obtain target millimeter-wave radar features.

6. The target detection method based on FPGA multimodal feature fusion according to claim 1, characterized in that: The step of performing Bayesian feature fusion on the normalized target infrared image features and the target millimeter wave radar features to generate a global saliency map includes: Performing Bayesian feature fusion on the normalized target infrared image features and the target millimeter wave radar features to obtain fused data; Performing saliency calculation on the fused data to obtain a saliency map; Dividing the saliency map into salient areas and non-salient areas according to a preset threshold; The salient area and the non-salient area are subjected to dilation and erosion processing to generate a global salient map.

7. The target detection method based on FPGA multimodal feature fusion according to claim 6 is characterized in that: The performing saliency calculation on the fused data to obtain a saliency map includes: determining a fused image based on the fused data; Converting the fused image to the frequency domain by Fourier transform to obtain the frequency spectrum characteristics of the fused image; A saliency calculation is performed according to the frequency spectrum features of the fused image to obtain a saliency map.

8. A target detection device based on FPGA multimodal feature fusion, characterized in that: The device comprises: A first acquisition module is used to acquire target infrared image data from an infrared imaging device through an FPGA system; The second acquisition module is used to acquire target millimeter-wave radar data from the millimeter-wave radar through the FPGA system; A first extraction module is used to perform multi-level feature extraction on the target infrared image data to obtain multiple target infrared image features; A standardization module, used for normalizing the target infrared image features to obtain normalized target infrared image features; A second extraction module is used to extract target millimeter-wave radar features based on the target millimeter-wave radar data; A fusion module, used for performing Bayesian feature fusion on the normalized target infrared image features and the target millimeter wave radar features to generate a global saliency map; The output module is used to compare the normalized target infrared image features with the target millimeter-wave radar features. When the normalized target infrared image features are inconsistent with the target millimeter-wave radar features, the global saliency map is corrected by a suppression-return mechanism until the target millimeter-wave radar features are consistent with the normalized target infrared image features; when the normalized target infrared image features are consistent with the target millimeter-wave radar features, the target detection result is output.

9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the target detection method based on FPGA multimodal feature fusion as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the target detection method based on FPGA multi-modal feature fusion as described in any one of claims 1 to 7.

Images