Radio frequency detection and identification method and device fusing low-light and thermal imaging

Abstract

The invention discloses a radio frequency detection and identification method and device fusing low-light and thermal imaging, and relates to the technical field of detection and identification, and the method comprises the steps: obtaining the angle information of a radio frequency signal through an array antenna, and generating the preliminary spatial signal distribution of a target region; performing image alignment fusion by combining the low-light night vision image and the thermal imaging information to generate a preliminary fusion image; spatial signal distribution is mapped to a fused image coordinate, and a spatial corresponding relation between a radio frequency signal and an image is established; fusing the radio frequency signal features and the image features based on the corresponding relation to generate joint features; and target positioning and labeling are realized by using the combined features. The technical problem of low target identification and positioning precision in a complex environment in the prior art is solved, and the technical effect of improving the target identification and positioning precision is achieved through more comprehensive information acquisition and data fusion.

Description

Technical Field

[0001] The present application relates to the field of detection and identification technology, and in particular to a radio frequency detection and identification method and device integrating low-light and thermal imaging. Background Art

[0002] In complex environments, target detection and recognition face challenges of insufficient accuracy and reliability. Traditional single data source detection methods, such as relying solely on RF signals or image data, are often limited by environmental noise, multipath interference, weak light, or unclear temperature differences, resulting in poor detection results. Although low-light images can provide target edge and structure information in low-light environments, and thermal imaging has advantages when temperature differences are significant, they have obvious limitations when used alone. For example, low-light images are less effective in extremely weak light environments, and thermal imaging has difficulty distinguishing targets with unclear temperature differences. Therefore, it is difficult for existing technologies to comprehensively utilize multi-dimensional information to achieve high-precision target recognition and positioning in complex environments. There is an urgent need for a solution that can integrate the advantages of multiple data sources to make up for the shortcomings of a single method.

[0003] At present, relevant technologies still have the technical problem of low target recognition and positioning accuracy in complex environments. Summary of the invention

[0004] The present application solves the technical problem of low target recognition and positioning accuracy in complex environments in the prior art by providing a radio frequency detection and identification method and device that integrates low-light and thermal imaging.

[0005] The present application provides a radio frequency detection and identification method integrating low light and thermal imaging, including:

[0006] The invention uses an array antenna to obtain angle information of a radio frequency signal and construct a preliminary spatial signal distribution of a target area; obtains image data, wherein the image data includes a low-light-level night vision recognition image and thermal imaging area information, and uses the low-light-level night vision recognition image and thermal imaging area information to perform image alignment and fusion to obtain a preliminary fused image; maps the preliminary spatial signal distribution to the position coordinates of the preliminary fused image and establishes a spatial correspondence between the radio frequency signal and the image data; based on the spatial correspondence, extracts radio frequency signal features and preliminary fused image features and performs feature fusion to obtain joint features; and detects and identifies objects for positioning and labeling based on the joint features.

[0007] The present application also provides a radio frequency detection and identification device integrating low light and thermal imaging, including:

[0008] A preliminary spatial signal distribution construction module, the preliminary spatial signal distribution construction module is used to use the array antenna to obtain the angle information of the radio frequency signal, and construct the preliminary spatial signal distribution of the target area; an image data acquisition module, the image data acquisition module is used to obtain image data, the image data includes a low-light night vision recognition image and thermal imaging area information, and the low-light night vision recognition image and thermal imaging area information are used to perform image alignment and fusion to obtain a preliminary fused image; a spatial correspondence construction module, the spatial correspondence construction module is used to map the preliminary spatial signal distribution to the position coordinates of the preliminary fused image, and establish a spatial correspondence between the radio frequency signal and the image data; a joint feature acquisition module, the joint feature acquisition module is used to fuse the extracted radio frequency signal features with the preliminary fused image features according to the spatial correspondence to obtain a joint feature; a detection identification object positioning module, the detection identification object positioning module is used to locate and mark the detection identification object according to the joint feature.

[0009] The RF detection and identification method and device for integrating low-light and thermal imaging proposed in this application first obtain the RF signal angle information through the array antenna to generate the preliminary spatial signal distribution of the target area; combine the low-light night vision image and thermal imaging information to perform image alignment and fusion to generate a preliminary fused image; map the spatial signal distribution to the fused image coordinates to establish a spatial correspondence between the RF signal and the image; based on the correspondence, fuse the RF signal features and the image features to generate joint features; use the joint features to realize target positioning and labeling, thereby achieving the technical effect of improving target recognition and positioning accuracy through more comprehensive information acquisition and data fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solution of the embodiment of the present disclosure, the accompanying drawings of the embodiment of the present disclosure will be briefly introduced below. A flow chart is used in the present application to illustrate the operations performed by the device according to the embodiment of the present application. It should be understood that the previous or following operations are not necessarily performed precisely in order. On the contrary, various steps can be processed in reverse order or simultaneously as needed. At the same time, other operations can also be added to these processes, or one or more operations can be removed from these processes.

[0011] Figure 1 A schematic diagram of the flow of a radio frequency detection and identification method integrating low light and thermal imaging provided in an embodiment of the present application;

[0012] Figure 2 A schematic diagram of the structure of a radio frequency detection and identification device integrating low-light and thermal imaging provided in an embodiment of the present application.

[0013] Explanation of the reference numerals: preliminary spatial signal distribution construction module 10 , image data acquisition module 20 , spatial correspondence relationship construction module 30 , joint feature acquisition module 40 , detection identification object positioning module 50 . DETAILED DESCRIPTION

[0014] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below.

[0015] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0016] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict, and the terms "first\second" involved are merely to distinguish similar objects and do not represent a specific ordering of objects. The terms "including" and "having" and any variations are intended to cover non-exclusive inclusions, for example, a process, method, device, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products, or devices. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by technicians in the technical field of this application. The terms used herein are for the purpose of describing the embodiments of the present application only.

[0017] The present application embodiment provides a radio frequency detection and identification method integrating low light and thermal imaging, such as Figure 1 As shown, the method includes:

[0018] Step S100, using the array antenna to obtain the angle information of the radio frequency signal, and constructing the preliminary spatial signal distribution of the target area. Specifically, the radio frequency signal in the target area is received by the array antenna, the signal amplitude and phase information of each antenna unit is obtained, and the input angle of the signal, including the azimuth and elevation angle, is determined by the direction vector calculation method to characterize the spatial direction characteristics of the signal. According to the calculated direction vector, the radio frequency signal is mapped from the angle information to the three-dimensional space coordinate system, and the position of the array antenna is used as the origin. The spatial coordinate points are generated in combination with the triangular geometric relationship to form a point cloud distribution of the radio frequency signal in the target area. Subsequently, the preliminary spatial signal distribution is analyzed for characteristics, including changes in signal strength distribution, aggregation areas in the target direction, and elimination of noise signals, and finally a preliminary spatial signal distribution that intuitively reflects the characteristics of the radio frequency signal in the target area is constructed, which provides basic support for the subsequent fusion of signal and image features and target detection and positioning.

[0019] In a possible implementation, the angle information of the radio frequency signal is obtained by using an array antenna to construct a preliminary spatial signal distribution of the target area. Step S100 further includes step S110, obtaining the amplitude and phase of the radio frequency signal of each antenna unit in the array. Specifically, the amplitude and phase of the radio frequency signal of each antenna unit are obtained by the array antenna to fully reflect the scattering characteristics and spatial position of the target. The amplitude of the radio frequency signal represents the signal strength, which is used to reflect the scattering characteristics or reflection strength of the target. The amplitude is affected by factors such as the target material, surface characteristics and propagation path loss; the phase represents the relative information of the signal arrival time, reflects the difference in the propagation path of the signal, and is a key parameter for inferring the spatial position of the target. Each antenna unit synchronously collects amplitude and phase data, and measures the amplitude value and phase difference respectively through the power detection circuit and the digital signal processing module to form a complete radio frequency signal feature data set. During the acquisition process, calibration is performed through a standard signal source, and filtering and denoising technology are combined to ensure the accuracy and consistency of the data. Finally, the amplitude and phase information together provide a key basis for the preliminary spatial signal distribution of the target area, laying the foundation for subsequent target positioning and characteristic analysis.

[0020] Step S120, based on the RF signal amplitude and phase of each antenna unit, calculate the direction vector of the signal input angle. Specifically, based on the RF signal amplitude and phase of each antenna unit, calculating the direction vector of the signal input angle is an important step in determining the spatial direction of the target signal. First, the RF signal amplitude is used to reflect the scattering intensity and reflection characteristics of the target, and the phase information is combined to represent the time difference of signal propagation. According to the geometric layout of the antenna array, the phase difference of the signal received by each antenna unit is analyzed, and a mathematical model of the signal propagation direction is established. The direction vector is calculated by formula, and the azimuth and elevation angle of the signal are derived by combining the array factor model and the beamforming algorithm. At the same time, the direction vector is optimized by multi-dimensional calibration and filtering to eliminate the influence of multipath effects and environmental interference. Finally, the direction vector provides the propagation direction of the target signal in three-dimensional space, which is used to construct the spatial signal distribution of the target area, providing key support for subsequent target positioning and feature fusion.

[0021] Step S130, according to the direction vector of the signal input angle, the signal is mapped to the three-dimensional space coordinate system to obtain the preliminary spatial signal distribution. Specifically, according to the direction vector of the signal input angle, the signal is mapped to the three-dimensional space coordinate system. First, according to the azimuth and pitch angle information in the direction vector, it is projected to the three-dimensional space coordinate system using the trigonometric function formula to establish the signal space distribution framework of the target area. Specifically, the xxx, yyy, and zzz coordinates in the three-dimensional coordinate system are calculated to correspond to the distribution positions of the signals in the horizontal plane, vertical direction, and depth, respectively, and the signal strength rrr is used as the modulus of the direction vector. Then, the receiving signal direction of each antenna unit in the antenna array is mapped one by one, and each signal point is integrated to generate a preliminary spatial signal distribution map of the target area. This distribution map intuitively reflects the strength and directional characteristics of the signal in the target area. Finally, the signal distribution is optimized using an interpolation algorithm to eliminate noise and redundant signal interference, thereby obtaining a smooth and continuous spatial signal distribution, which provides an accurate spatial reference frame for the subsequent fusion of radio frequency signals and image features and target detection and positioning.

[0022] Step S200, image data is obtained, the image data includes a low-light-level night vision recognition image and thermal imaging area information, and the low-light-level night vision recognition image and thermal imaging area information are used to perform image alignment and fusion to obtain a preliminary fused image. Specifically, image data of the target area is obtained through a low-light-level night vision device and a thermal imaging device, wherein the low-light-level night vision recognition image captures the detailed edge features of the target object and is suitable for target shape recognition in a low-light environment; the thermal imaging area information provides the temperature distribution and hot spot area characteristics of the target, reflecting the thermal radiation in the environment. Next, the images are spatially aligned based on the optical axis information of the two devices, key points are extracted and feature point pairs are found using the nearest neighbor matching algorithm, and the wrong matching points are eliminated by combining the random sampling point fitting transformation model. After calculating the best homography matrix, the thermal imaging image is perspective transformed to achieve spatial alignment with the low-light-level night vision image. Subsequently, the aligned low-light-level night vision image is enhanced with high frequency details to extract edge and structural features, and low-frequency brightness features are extracted from the thermal imaging image to obtain temperature distribution and hot spot information, and the two types of features are fused based on the spatial alignment relationship to generate a preliminary fused image. Finally, the fused image contains both the edge shape information and temperature characteristics of the target, providing a complete data basis for the subsequent fusion of RF signals and image features.

[0023] In a possible implementation, image data is obtained, the image data includes a low-light-level night vision recognition image and thermal imaging area information, and the low-light-level night vision recognition image and thermal imaging area information are used to perform image alignment and fusion to obtain a preliminary fused image. Step S200 further includes step S210, based on the optical axis of the image acquisition device, the low-light-level night vision recognition image and the thermal imaging area information are spatially aligned. Specifically, spatially aligning the low-light-level night vision recognition image and the thermal imaging area information based on the optical axis information of the image acquisition device is a key step in realizing multi-source data fusion. First, the key point information is extracted from the two images using a feature extraction algorithm to identify significant features such as the edges and corners of the object. Then, the key points of the two images are matched using a nearest neighbor matching algorithm to form a preliminary matching point pair, and the random sampling consensus algorithm (RANSAC) is used to optimize the matching results, eliminate erroneous matching points, and ensure the reliability of the matching points. Next, based on the optimized matching point pair, the optimal homography matrix is ​​calculated, and the thermal imaging area information is perspective transformed so that it is aligned with the low-light-level night vision image in the same spatial coordinate system. Through this processing, the two images can be accurately superimposed in space, laying the foundation for subsequent feature fusion and target recognition, and improving the accuracy and reliability of detection and recognition.

[0024] Step S220, high-frequency details are added to the low-light-level night vision recognition image to extract edge information and structural features of the object. Specifically, high-frequency details are added to the low-light-level night vision recognition image. First, image preprocessing is performed to remove noise and smooth the image to provide a stable input for subsequent processing. Subsequently, the high-frequency components in the image are extracted using methods such as the Laplace operator or wavelet transform to enhance the edge and texture details of the object. By weighted superposition of the high-frequency components with the original image, the clarity of the image is significantly improved, making the edges sharper. Next, algorithms such as Canny edge detection are used to extract the edge contour of the object, and morphological operations are combined to optimize the continuity and integrity of the edge, while geometric analysis methods are used to extract the main structural features of the target. The final enhanced image not only has clear edge information, but also contains the complete structural features of the object, laying a solid foundation for subsequent target recognition and feature fusion.

[0025] Step S230, extract low-frequency brightness features from the thermal imaging area information to obtain temperature distribution and hot spot area information. Specifically, the process of extracting low-frequency brightness features from the thermal imaging area information mainly includes brightness decomposition and feature extraction to obtain the temperature distribution and hot spot area information of the target area. First, by low-pass filtering (such as Gaussian filtering) on ​​the thermal imaging image, high-frequency noise and texture details are smoothed and removed, and only low-frequency brightness information is retained to highlight the overall temperature distribution characteristics. Subsequently, the low-frequency brightness data is processed, the temperature intensity value of each pixel is calculated, a temperature distribution map is generated, and the brightness value is converted into a corresponding temperature range. Next, the hot spot area is extracted from the temperature distribution map using a threshold segmentation method, and pixels with brightness values ​​higher than a set threshold are marked as hot spots, and feature information such as their boundaries, positions, areas, and temperature ranges are determined through connectivity analysis. Finally, these extracted temperature distribution and hot spot information are output for subsequent feature fusion and target recognition, providing accurate input data for further analysis and processing.

[0026] Step S240, according to the spatial alignment relationship, the edge information and structural features of the object are feature fused with the temperature distribution and hot spot area information to obtain the preliminary fused image. Specifically, according to the spatial alignment relationship, the process of feature fusion of the edge information and structural features of the object in the low-light night vision image with the temperature distribution and hot spot area information in the thermal imaging image includes the following steps. First, through the spatial alignment relationship, the data in the low-light night vision image and the thermal imaging image are mapped to the same coordinate system to ensure that each feature point corresponds accurately in space. Then, the Gaussian pyramid method is used to perform multi-scale decomposition on the two image features, and the information is divided into low-frequency components and high-frequency components. The low-frequency component retains the brightness and temperature distribution information, and the high-frequency component highlights the edge and texture details. Subsequently, the low-frequency component is weighted average fused, combining the geometric contour and thermal information of the target; the high-frequency component is fused using the maximum value method to retain significant edge and texture features. Finally, through layer-by-layer reconstruction, the fused low-frequency and high-frequency features are integrated into a complete preliminary fused image. This fused image contains not only the edge contours and detailed textures of the low-light night vision image, but also the temperature distribution and hot spot area information of the thermal imaging, providing comprehensive and accurate basic data for subsequent target identification and positioning.

[0027] In a possible implementation, based on the optical axis of the image acquisition device, the low-light night vision recognition image and the thermal imaging area information are spatially aligned, and step S210 further includes step S211, obtaining key points in the image. Specifically, obtaining key points in the image is a key step in image processing, which aims to extract significant feature points from the low-light night vision recognition image and the thermal imaging area information to support subsequent image registration and fusion. First, the input image is preprocessed, such as graying, denoising and contrast enhancement, to ensure the accuracy of feature point detection. Then, an appropriate feature point detection algorithm is selected, such as SIFT, ORB or Harris corner detection, and a multi-scale feature space is constructed according to the characteristics of the image. In the feature space, key points are detected by the criteria defined by the algorithm, such as locating key points using the extreme points of the Gaussian difference function or detecting corner points by grayscale gradient changes. Subsequently, the detected key points are screened to remove duplicate points, unstable points or noise points caused by boundary effects. Finally, feature description information is generated for each key point, such as direction, scale or gradient distribution, for subsequent image matching and fusion processing. Through the above steps, it is ensured that the extracted key points can accurately reflect the salient features in the image, laying the foundation for image alignment and fusion.

[0028] Step S212, based on the key points, the image matching point pairs of the low-light night vision recognition image and the thermal imaging area information are obtained by the nearest neighbor matching algorithm. Specifically, after obtaining the key points of the low-light night vision recognition image and the thermal imaging area information, the feature descriptors of the key points are matched by the nearest neighbor matching algorithm to establish the corresponding relationship between the two images. First, a feature descriptor is generated for each key point, such as a multidimensional vector extracted by the SIFT or ORB algorithm to describe the local features of the key point. Subsequently, the nearest matching point is found by calculating the Euclidean distance or Hamming distance between each key point descriptor in the low-light night vision image and all key point descriptors in the thermal imaging image, and the mismatched point pairs are eliminated by the ratio test. Next, a geometric constraint algorithm (such as RANSAC) is used to fit the transformation model, and abnormal point pairs that do not meet the geometric consistency are further eliminated, and finally accurate matching point pairs are retained. These matching point pairs provide a key basis for subsequent image alignment and fusion.

[0029] Step S213, using the image matching point pairs, fitting the transformation model by randomly sampling point pairs, identifying and eliminating erroneous matching points, and calculating the best homography matrix. Specifically, by extracting the image matching point pairs in the low-light night vision recognition image and the thermal imaging area information, the transformation model is fitted using the random sampling consensus algorithm (RANSAC) to complete the precise alignment of the images. The specific process is: first, a small number of point pairs are randomly selected from the matching point pairs to fit the initial homography transformation model, which is used to describe the perspective transformation relationship between the two images. Then, all point pairs are substituted into the model to calculate the error, and the point pairs below the preset threshold constitute the consistency point set. Through multiple random sampling and model fitting iterations, the model is gradually optimized to find the transformation relationship containing the maximum consistency point set, and the erroneous matching points are eliminated. On this basis, the consistency point set is refitted to finally generate an accurate best homography matrix to ensure that the thermal imaging area information is aligned with the perspective transformation of the low-light night vision recognition image, providing an accurate alignment basis for subsequent feature fusion and target recognition.

[0030] Step S214, using the optimal homography matrix to perform perspective transformation on the thermal imaging area information, and align it with the low-light-level night vision recognition image. Specifically, the thermal imaging area information is perspective transformed using the optimal homography matrix to ensure that it is completely aligned with the low-light-level night vision recognition image in spatial coordinates. First, through the optimal homography matrix generated in the early stage, the matrix accurately describes the geometric relationship between the thermal imaging information and the low-light-level night vision image, including parameters such as rotation, scaling, translation and perspective transformation. The matrix is ​​used to perform perspective transformation on the coordinates of each pixel point in the thermal imaging image, and it is mapped to the coordinate system of the low-light-level night vision image. At the same time, the interpolation algorithm is used to resample the data after perspective transformation to ensure that the transformed thermal imaging data is clear and retains the original temperature distribution and hot spot area information. Finally, the thermal imaging area information processed by perspective transformation is aligned with the low-light-level night vision recognition image at the pixel level, providing an accurate spatial correspondence for subsequent multimodal feature fusion and target detection and recognition, thereby effectively improving the recognition accuracy and robustness.

[0031] In a possible implementation, according to the spatial alignment relationship, the edge information and structural features of the object are feature fused with the temperature distribution and hot spot area information to obtain the preliminary fused image, and step S240 further includes step S241, extracting multi-scale low-frequency components from the edge information and structural features of the object, the temperature distribution and hot spot area information respectively through Gaussian pyramid. Specifically, extracting multi-scale low-frequency components from the edge information and structural features of the object in the low-light night vision image, and the temperature distribution and hot spot area information of the thermal imaging image through Gaussian pyramid is an efficient multi-scale analysis method. This process smoothes and downsamples the image layer by layer through Gaussian filtering to generate an image with gradually reduced resolution, thereby extracting low-frequency components, retaining the global features of the image, and removing high-frequency details and noise. In the low-light night vision image, the low-frequency component highlights the main contour and structural features of the object, while in the thermal imaging image, the low-frequency component strengthens the brightness gradient of the temperature distribution and hot spot area. The layer-by-layer extraction of these components can not only retain the global characteristics, but also separate the high-frequency detail layer through differential calculation, providing a reliable basis for subsequent image feature fusion. The multi-scale characteristics of the Gaussian pyramid enable it to simultaneously express the details and overall changes of the image at different scales, laying a solid technical foundation for multimodal information fusion.

[0032] Step S242, calculate the difference of the low-frequency components between adjacent pyramid layers to obtain a high-frequency detail layer. Specifically, by performing difference calculation on the low-frequency components of adjacent layers through the Gaussian pyramid, the high-frequency detail layer of the image can be extracted, thereby separating high-frequency information such as edges, textures, and local contrast in the image. Specifically, the original image is subjected to Gaussian filtering to generate a multi-layer pyramid, each layer of which contains a low-frequency component with a lower resolution, representing the overall brightness and structural information of the image, while removing details and noise. Subsequently, the pixel value difference is calculated for the low-frequency components of two adjacent layers of the pyramid to generate a high-frequency detail layer. The high-frequency detail layer retains details such as edge features, texture changes, and temperature gradient changes in the image, such as highlighting the edges of objects in low-light night vision images, and strengthening the detail features of hot spots and temperature distribution in thermal imaging images. As the pyramid levels progress, the resolution of the high-frequency detail layer gradually decreases, but its feature coverage increases, and it can effectively extract detail information from local to global, providing rich multi-scale detail support for subsequent feature fusion and depth analysis.

[0033] Step S243, repeat the hierarchical decomposition until the image size threshold is reached, and construct a feature hierarchical architecture, wherein the bottom layer of the feature hierarchical architecture is a low-frequency layer including brightness information, and each layer difference represents texture and edge details. Specifically, the image is hierarchically decomposed by a Gaussian pyramid, and the low-frequency and high-frequency features of the image are extracted layer by layer until the set image size threshold or the predetermined number of decomposition layers is reached. First, the original image is Gaussian filtered to generate a low-frequency component, which contains the brightness information and basic structural features of the entire image. Subsequently, the next layer of low-frequency components is further obtained by downsampling, and the difference between the two adjacent layers of low-frequency components is calculated to extract the high-frequency detail layer, which mainly represents the texture, edge features and hot spots in the image. The decomposition process continues until the minimum edge size of the image reaches the set threshold (such as 32×32 pixels) or reaches the specified number of decomposition layers (such as 3 to 5 layers). Finally, the constructed feature hierarchical architecture contains multiple levels: the bottom layer is a low-frequency component, which reflects the overall brightness information; the remaining layers are high-frequency detail layers, which record the texture, edge and local features of the image layer by layer. This architecture realizes multi-scale feature expression from global to local, providing a solid foundation for subsequent feature fusion and in-depth analysis.

[0034] Step S244, based on the feature hierarchical architecture, feature fusion is performed hierarchically to obtain the preliminary fused image. Specifically, based on the feature hierarchical architecture, the edge information, structural features, temperature distribution and hot spot area information of the object are fused hierarchically. First, the low-light night vision image and the thermal imaging image are decomposed into a multi-scale feature hierarchical architecture through Gaussian pyramid decomposition, wherein the bottom layer is a low-frequency layer, which contains global information such as overall brightness and temperature distribution, and the upper layer gradually contains more high-frequency details, such as edge, texture and hot spot features. In the fusion process, the low-frequency layer is fused by weighted average, and the brightness and temperature information are superimposed according to the weight distribution; for the high-frequency detail layer, the maximum value method is used for fusion layer by layer to highlight the significant edge and hot spot detail information. Finally, starting from the low-frequency fusion layer, it is reconstructed layer by layer, and the high-frequency details are gradually superimposed to form the final preliminary fused image. This image has both the edge and structural features of the low-light night vision image and the temperature distribution and hot spot features of the thermal imaging image, providing accurate multi-modal information support for subsequent target detection, recognition and positioning.

[0035] In a possible implementation, based on the feature hierarchy, feature fusion is performed hierarchically to obtain the preliminary fused image, and step S244 further includes step S2441, based on the feature hierarchy, weighted average fusion is performed on the low-frequency layer to obtain low-frequency fusion features. Specifically, based on the feature hierarchy, the process of weighted average fusion of the low-frequency layer is intended to integrate the low-frequency information of the low-light night vision image and the thermal imaging image to retain the temperature distribution characteristics of the thermal imaging and enhance the overall brightness of the low-light night vision. First, the low-frequency component F of the low-light night vision image is extracted from the feature hierarchy vis，low (x, y) and the low-frequency component F of the thermal imaging image ir，low (x, y). The low-frequency component of low-light night vision reflects the brightness distribution and contour characteristics of the scene, while the low-frequency component of thermal imaging contains the global characteristics of temperature distribution and hot spot areas. Then, the weighted average method is used to fuse the two, and the formula is: F low (x, y) = α·F vis，low (x, y)+(1-α)·F ir，low (x, y), where α is a weighting coefficient, usually in the range of 0.6 to 0.7, which controls the contribution ratio of low-light night vision and thermal imaging in low-frequency features. By adjusting α reasonably, the brightness of low-light night vision or the temperature characteristics of thermal imaging can be highlighted according to scene requirements. The fused low-frequency features not only enhance the global brightness consistency of the image, but also retain the integrity of the temperature distribution, providing a reliable foundation for subsequent high-frequency detail fusion and image reconstruction. This method effectively smoothes the noise in the original data while ensuring that the overall brightness and thermal distribution information of the fused image is clear and accurate.

[0036] Step S2442, based on the feature hierarchy, the high-frequency layer is fused layer by layer using the maximum value method to obtain the fusion features of the high-frequency layer. Specifically, based on the feature hierarchy, the high-frequency layer is fused layer by layer using the maximum value method, and the complementary fusion of the low-light-level night vision image and the thermal imaging image features is achieved by selecting the more significant high-frequency components in each layer. The specific process includes extracting the high-frequency components F of the low-light-level night vision image layer by layer. vis，high (x, y) and the high-frequency component F of the thermal imaging image ir，high (x, y), compare the pixel values ​​at the same position and apply the maximum fusion rule: F high (x, y) = max(F vis，high (x,y), F ir，high (x, y)) where F vis，high (x, y) represents the high-frequency feature value of the low-light night vision image, F ir，high (x, y) represents the high-frequency feature value of the thermal imaging image, F high (x, y) is the high-frequency feature value after fusion. By fusing high-frequency details layer by layer, the edge details in low-light night vision images and the hotspot information in thermal imaging images are retained to the greatest extent, so that the final fused high-frequency layer has both sharp edge features and prominent hotspot areas, providing refined textures and significant features for subsequent image reconstruction, and improving the effect of target detection and recognition.

[0037] Step S2443, starting from the bottom layer of low-frequency fusion features, reconstructing the fusion image layer by layer to obtain the preliminary fusion image. Specifically, starting from the bottom layer of low-frequency fusion features, superimposing high-frequency detail layers layer by layer to gradually complete the reconstruction of the fusion image. First, the low-frequency fusion feature F of the feature hierarchy is used. low (x, y) is used as the initial image, which contains the overall brightness and temperature distribution information of thermal imaging. Then, starting from the lowest high-frequency layer, the corresponding high-frequency fusion features are extracted And superimposed on the current image, through the formula Complete the fusion of each layer. This process is carried out layer by layer to ensure that the high-frequency details such as texture, edges and hot spots of each layer are completely superimposed on the low-frequency image. When the high-frequency details of the highest layer are also superimposed, the final fused image is generated. It not only includes the brightness and temperature distribution of thermal imaging, but also retains the texture and edge features of low-light night vision, presenting a smooth and detail-rich effect, providing a high-quality image foundation for subsequent detection and identification.

[0038] Step S300, mapping the preliminary spatial signal distribution to the position coordinates of the preliminary fused image, and establishing the spatial correspondence between the RF signal and the image data. Specifically, in order to achieve the spatial correspondence between the RF signal and the image data, it is first necessary to map the preliminary spatial distribution of the RF signal to the pixel coordinates of the image. Through geometric calibration technology, the three-dimensional spatial position of the RF signal is aligned with the two-dimensional plane coordinates of the image to unify the two coordinate systems. Specifically, using the optical parameters of the image acquisition device (such as focal length and optical axis position) and the spatial distribution information of the RF signal, a projection mapping model is established to associate the spatial characteristics of the RF signal (such as signal strength or frequency information) with the corresponding pixel points in the image. Through this mapping, the position of the RF signal on the image can be accurately calibrated, and the spatial association between the RF data and the image features can be achieved, providing an accurate data basis for subsequent feature fusion and target detection and recognition.

[0039] Step S400, according to the spatial correspondence, the extracted RF signal features are feature-fused with the preliminary fused image features to obtain joint features. Specifically, in order to achieve effective fusion of the RF signal features and the preliminary fused image features, the RF signal features are first accurately mapped to the corresponding positions in the preliminary fused image according to the established spatial correspondence to ensure the spatial consistency of the two different modal data. The RF signal features include information such as angle, intensity and frequency distribution, while the preliminary fused image features include features such as edge contours, brightness distribution and temperature hot spots. Subsequently, the two features are deeply fused through a multimodal fusion model. The model first extracts the significant features of the RF signal and the image respectively through a convolutional feature extraction module, and then dynamically adjusts the weights of the RF signal and image features in the fusion process using the cross-modal attention mechanism in the fusion module. For example, the RF signal feature weight is enhanced in a low-light environment, while the detailed features of the image are highlighted when the target edge is clear. Finally, a joint feature of the comprehensive RF signal and image features is generated through the output layer. The joint feature contains both the precise positioning information of the RF signal and the fusion of the visual details and temperature hot spots of the image, providing comprehensive data support for target detection and positioning.

[0040] In a possible implementation, according to the spatial correspondence, the extracted RF signal features are fused with the preliminary fused image features to obtain joint features, and step S400 further includes step S410, constructing a multimodal fusion model, including a convolution extraction module, a fusion module, and an output layer, and the fusion module performs deep fusion of RF features and image features through a cross-modal attention mechanism. Specifically, in order to achieve deep fusion of RF signal features and image features, the multimodal fusion model is designed to include three major parts: a convolution extraction module, a fusion module, and an output layer. First, the convolution extraction module extracts features from the RF signal and the preliminary fused image respectively, and the RF signal extracts spatial direction information and signal strength features through the convolution layer, while the image feature extraction focuses on the multi-scale feature representation of texture, edge, and brightness distribution. Subsequently, the fusion module uses a cross-modal attention mechanism to dynamically assign weights to the RF features and image features, adjusts the importance of the image features according to the significance of the RF signal, and captures the semantic association between the two modalities through feature alignment to generate a fused joint feature. Finally, the output layer further optimizes the joint features and extracts high-level semantics to ensure that the output joint features can fully represent the spatial position and visual details of the target, providing accurate support for subsequent detection, positioning and labeling tasks. Through this fusion method, the RF signal and image features can work together in complex scenes, improving the accuracy and efficiency of detection and positioning.

[0041] Step S420, input the RF signal features and the preliminary fused image features into the multimodal fusion model to obtain the joint features. Specifically, in the multimodal fusion model, the RF signal features and the preliminary fused image features are input into the model, and the deep fusion of the two modal features is achieved through the convolution extraction module, fusion module and output layer of the model to generate a comprehensive joint feature. The RF signal features contain the spatial distribution features such as the angle information, signal amplitude and intensity of the target, which can accurately locate the directionality and scattering characteristics of the target; the preliminary fused image features contain information such as the edge, structural details and temperature hot spot distribution of the object, which intuitively reflects the morphology and thermal distribution characteristics of the target. In the fusion process, the cross-modal attention mechanism dynamically allocates the weights of the RF signal and image features, focusing on the feature dimensions that have a significant contribution to target recognition, thereby achieving accurate alignment and fusion in the feature space. Finally, the joint features are further optimized through the output layer, integrating the information of spatial positioning and visual expression, and providing high-precision and high-robustness support for subsequent target detection, recognition and positioning.

[0042] Step S500, detecting, identifying, locating and marking the object according to the joint features. Specifically, the joint features are used to detect, identify, locate and mark the target, and the deep fusion results of the RF signal and image features are integrated to provide multi-dimensional support for high-precision target positioning. First, the target geometry, edge information, temperature distribution and RF signal strength and angle characteristics contained in the joint features are extracted. These information jointly reflect the spatial position and physical characteristics of the target. Subsequently, the joint features are classified and screened using the target recognition algorithm to distinguish the target from the background signal. The RF signal provides direction information, and the image features further accurately locate the shape and position of the target, and map the target location to the specific coordinates of the actual scene through the established spatial correspondence. Finally, combined with the positioning results, the target is marked, including contour display, key point marking or heat distribution area marking, and the target type, location coordinates and other attribute information are attached. This process realizes the accurate positioning and visual marking of the target, providing reliable data support for subsequent applications such as monitoring, early warning and tracking.

[0043] The embodiment of the present application obtains the angle information of the radio frequency signal through an array antenna to generate a preliminary spatial signal distribution of the target area; combines the low-light night vision image with the thermal imaging information to perform image alignment and fusion to generate a preliminary fused image; maps the spatial signal distribution to the fused image coordinates to establish a spatial correspondence between the radio frequency signal and the image; based on the correspondence, the radio frequency signal features and the image features are fused to generate joint features; and the joint features are used to realize target positioning and labeling, thereby achieving a technical effect of improving target recognition and positioning accuracy through more comprehensive information acquisition and data fusion.

[0044] In the above, refer to Figure 1 The radio frequency detection and identification method integrating low light and thermal imaging according to an embodiment of the present invention is described in detail. Figure 2 A radio frequency detection and identification device integrating low-light imaging and thermal imaging according to an embodiment of the present invention is described.

[0045] The radio frequency detection and identification device integrating low light and thermal imaging according to the embodiment of the present invention is used to solve the technical problem of low target recognition and positioning accuracy in complex environments in the prior art, and achieves the technical effect of improving target recognition and positioning accuracy through more comprehensive information acquisition and data fusion. The radio frequency detection and identification device integrating low light and thermal imaging includes: a preliminary spatial signal distribution construction module 10, an image data acquisition module 20, a spatial correspondence construction module 30, a joint feature acquisition module 40, and a detection and identification object positioning module 50.

[0046] The preliminary spatial signal distribution construction module 10 is used to obtain angle information of radio frequency signals using an array antenna to construct a preliminary spatial signal distribution of a target area.

[0047] The image data acquisition module 20 is used to acquire image data, which includes a low-light-level night vision recognition image and thermal imaging area information, and uses the low-light-level night vision recognition image and thermal imaging area information to perform image alignment and fusion to obtain a preliminary fused image.

[0048] The spatial correspondence building module 30 is used to map the preliminary spatial signal distribution to the position coordinates of the preliminary fused image, and to establish the spatial correspondence between the radio frequency signal and the image data.

[0049] The joint feature acquisition module 40 is used to perform feature fusion on the extracted radio frequency signal features and the preliminary fused image features according to the spatial correspondence to obtain joint features.

[0050] The detection identification object positioning module 50 is used to locate and mark the detection identification object according to the joint features.

[0051] The specific configuration of the preliminary spatial signal distribution construction module 10 will be described in detail below. As described above, the angle information of the RF signal is obtained by using the array antenna to construct the preliminary spatial signal distribution of the target area. The preliminary spatial signal distribution construction module 10 further includes: a RF signal information acquisition unit, which is used to obtain the RF signal amplitude and phase of each antenna unit in the array; a direction vector calculation unit, which is used to calculate the direction vector of the signal input angle based on the RF signal amplitude and phase of each antenna unit; a preliminary spatial signal distribution acquisition unit, which is used to map the signal to a three-dimensional space coordinate system according to the direction vector of the signal input angle to obtain the preliminary spatial signal distribution.

[0052] The specific configuration of the image data acquisition module 20 will be described in detail below. As described above, image data is acquired, and the image data includes a low-light night vision recognition image and thermal imaging area information. The low-light night vision recognition image and thermal imaging area information are used to perform image alignment and fusion to obtain a preliminary fused image. The image data acquisition module 20 further includes: a spatial alignment unit, which is used to spatially align the low-light night vision recognition image and the thermal imaging area information based on the optical axis of the image acquisition device; a high-frequency detail adding unit, which is used to add high-frequency details to the low-light night vision recognition image and extract object edge information and structural features; a low-frequency brightness feature extraction unit, which is used to extract low-frequency brightness features from the thermal imaging area information to obtain temperature distribution and hot spot area information; a preliminary fused image acquisition unit, which is used to perform feature fusion of the object edge information and structural features with the temperature distribution and hot spot area information according to the spatial alignment relationship to obtain the preliminary fused image.

[0053] Among them, based on the optical axis of the image acquisition device, the low-light level night vision recognition image and the thermal imaging area information are spatially aligned, and the spatial alignment unit further includes: a key point acquisition subunit, the key point acquisition subunit is used to acquire key points in the image; an image matching point pair acquisition subunit, the image matching point pair acquisition subunit is used to obtain image matching point pairs of the low-light level night vision recognition image and the thermal imaging area information based on the key points through the nearest neighbor matching algorithm; an optimal homography matrix calculation subunit, the optimal homography matrix calculation subunit is used to use the image matching point pairs, fit the transformation model through random sampling point pairs, identify and eliminate erroneous matching points, and calculate the optimal homography matrix; a perspective transformation subunit, the perspective transformation subunit is used to use the optimal homography matrix to perform perspective transformation on the thermal imaging area information to align it with the low-light level night vision recognition image.

[0054] Among them, according to the spatial alignment relationship, the edge information and structural features of the object are feature fused with the temperature distribution and hot spot area information to obtain the preliminary fused image, and the preliminary fused image acquisition unit further includes: a low-frequency component extraction subunit, the low-frequency component extraction subunit is used to extract multi-scale low-frequency components from the edge information and structural features of the object, the temperature distribution and hot spot area information through a Gaussian pyramid; a difference calculation subunit, the difference calculation subunit is used to calculate the difference of the low-frequency components between adjacent pyramid layers to obtain a high-frequency detail layer; a feature hierarchy architecture construction subunit, the feature hierarchy architecture construction subunit is used to repeatedly perform hierarchical decomposition until the image size threshold is reached, and a feature hierarchy architecture is constructed, wherein the bottom layer of the feature hierarchy architecture is a low-frequency layer including brightness information, and each layer difference represents texture and edge details; a preliminary fused image acquisition subunit, the preliminary fused image acquisition subunit is used to perform feature fusion hierarchically based on the feature hierarchy architecture to obtain the preliminary fused image.

[0055] Among them, based on the feature hierarchy architecture, feature fusion is performed hierarchically to obtain the preliminary fused image, and the preliminary fused image acquisition subunit further includes: a weighted average fusion micro-unit, the weighted average fusion micro-unit is used to perform weighted average fusion on the low-frequency layer based on the feature hierarchy architecture to obtain low-frequency fusion features; a maximum method fusion micro-unit, the maximum method fusion micro-unit is used to perform maximum method fusion on the high-frequency layer layer by layer based on the feature hierarchy architecture to obtain high-frequency layer fusion features; a reconstructed fused image micro-unit, the reconstructed fused image micro-unit is used to reconstruct the fused image layer by layer starting from the bottom layer of low-frequency fusion features to obtain the preliminary fused image.

[0056] The specific configuration of the joint feature acquisition module 40 will be described in detail below. As described above, according to the spatial correspondence, the extracted RF signal features are fused with the preliminary fused image features to obtain the joint features. The joint feature acquisition module 40 further includes: a multimodal fusion model construction unit, which is used to construct a multimodal fusion model, including a convolution extraction module, a fusion module, and an output layer. The fusion module performs deep fusion of RF features and image features through a cross-modal attention mechanism; a joint feature acquisition unit, which is used to input the RF signal features and the preliminary fused image features into the multimodal fusion model to obtain the joint features.

[0057] The radio frequency detection and identification device integrating low light level and thermal imaging provided by the embodiment of the present invention can execute the radio frequency detection and identification method integrating low light level and thermal imaging provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0058] Although the present application makes various references to certain modules in the device according to the embodiments of the present application, any number of different modules may be used and run on the user terminal and / or server, and the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0059] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A radio frequency detection and identification method integrating low light and thermal imaging, characterized in that: include: Use array antennas to obtain the angle information of radio frequency signals and construct preliminary spatial signal distribution in the target area; Acquire image data, wherein the image data includes a low-light-level night vision recognition image and thermal imaging area information, and use the low-light-level night vision recognition image and thermal imaging area information to perform image alignment and fusion to obtain a preliminary fused image; Mapping the preliminary spatial signal distribution to the position coordinates of the preliminary fused image to establish a spatial correspondence between the radio frequency signal and the image data; According to the spatial correspondence, the extracted radio frequency signal features are fused with the preliminary fused image features to obtain joint features; The detection identification object is located and marked according to the joint features.

2. The radio frequency detection and identification method integrating low light and thermal imaging as claimed in claim 1, characterized in that: The method of using an array antenna to obtain angle information of radio frequency signals and constructing a preliminary spatial signal distribution of a target area includes: Obtain the RF signal amplitude and phase of each antenna element in the array; Calculate the direction vector of the signal input angle based on the RF signal amplitude and phase of each antenna unit; According to the direction vector of the signal input angle, the signal is mapped into a three-dimensional space coordinate system to obtain the preliminary spatial signal distribution.

3. The radio frequency detection and identification method integrating low light and thermal imaging as claimed in claim 2, characterized in that: Using the low-light-level night vision recognition image and thermal imaging area information to perform image alignment and fusion to obtain a preliminary fused image, including: Based on the optical axis of the image acquisition device, spatially aligning the low-light-level night vision recognition image and the thermal imaging area information; Adding high-frequency details to the low-light-level night vision recognition image to extract edge information and structural features of the object; Extracting low-frequency brightness features from the thermal imaging area information to obtain temperature distribution and hot spot area information; According to the spatial alignment relationship, the edge information and structural features of the object are feature fused with the temperature distribution and hot spot area information to obtain the preliminary fused image.

4. The radio frequency detection and identification method integrating low light and thermal imaging as claimed in claim 3, characterized in that: The spatial alignment of the low-light-level night vision recognition image and the thermal imaging area information based on the optical axis of the image acquisition device includes: Get the key points in the image; Based on the key points, the image matching point pairs of low-light night vision recognition images and thermal imaging area information are obtained through the nearest neighbor matching algorithm; Using the image matching point pairs, fitting the transformation model by randomly sampling point pairs, identifying and eliminating wrong matching points, and calculating the best homography matrix; The thermal imaging area information is perspectively transformed using the optimal homography matrix to align it with the low-light-level night vision recognition image.

5. The radio frequency detection and identification method integrating low light and thermal imaging as claimed in claim 3, characterized in that: According to the spatial alignment relationship, the edge information and structural features of the object are feature-fused with the temperature distribution and hot spot area information to obtain the preliminary fused image, including: Extracting multi-scale low-frequency components from the edge information and structural features of the object, the temperature distribution and the hot spot area information through a Gaussian pyramid; Calculating the difference of the low-frequency components between adjacent pyramid layers to obtain a high-frequency detail layer; Repeat the hierarchical decomposition until the image size threshold is reached, and construct a feature hierarchical architecture, wherein the bottom layer of the feature hierarchical architecture is a low-frequency layer including brightness information, and each layer difference represents texture and edge details; Based on the feature hierarchical architecture, feature fusion is performed hierarchically to obtain the preliminary fused image.

6. The radio frequency detection and identification method integrating low light and thermal imaging as claimed in claim 5, characterized in that: Based on the feature hierarchical architecture, feature fusion is performed hierarchically to obtain the preliminary fused image, including: Based on the feature hierarchical architecture, weighted average fusion is performed on the low-frequency layer to obtain low-frequency fusion features; Based on the feature hierarchical architecture, the high-frequency layer is fused layer by layer using the maximum value method to obtain the high-frequency layer fusion features; Starting from the bottom-level low-frequency fusion features, the fusion image is reconstructed layer by layer upwards to obtain the preliminary fusion image.

7. The radio frequency detection and identification method integrating low light and thermal imaging as claimed in claim 1, characterized in that: The step of fusing the extracted radio frequency signal features with the preliminary fused image features to obtain joint features includes: Constructing a multimodal fusion model, including a convolutional extraction module, a fusion module, and an output layer, wherein the fusion module deeply fuses RF features with image features through a cross-modal attention mechanism; The radio frequency signal features and the preliminary fusion image features are input into the multimodal fusion model to obtain the joint features.

8. A radio frequency detection and identification device integrating low light and thermal imaging, characterized in that: The device is used to implement the radio frequency detection and identification method integrating low-light and thermal imaging as described in any one of claims 1 to 7, and the device comprises: A preliminary spatial signal distribution construction module, wherein the preliminary spatial signal distribution construction module is used to obtain angle information of radio frequency signals using an array antenna to construct a preliminary spatial signal distribution of a target area; An image data acquisition module, wherein the image data acquisition module is used to acquire image data, wherein the image data includes a low-light-level night vision recognition image and thermal imaging area information, and the low-light-level night vision recognition image and thermal imaging area information are used to perform image alignment and fusion to obtain a preliminary fused image; A spatial correspondence building module, the spatial correspondence building module is used to map the preliminary spatial signal distribution to the position coordinates of the preliminary fused image, and establish a spatial correspondence between the radio frequency signal and the image data; A joint feature acquisition module, the joint feature acquisition module is used to perform feature fusion on the extracted radio frequency signal features and the preliminary fusion image features according to the spatial correspondence to obtain a joint feature; A detection identification object positioning module is used to locate and mark the detection identification object according to the joint feature.

Images