Radar and optoelectronic feature fusion target recognition system and method based on deep network

Abstract

A radar and optoelectronic feature fusion target recognition system and method based on a deep network belongs to the field of data processing technology. In the system, the template matching module includes a circular sampling filter, a radial sampling filter and a template matching filter, wherein the circular sampling filter respectively projects the target image and the template image on a set of rings to detect the first-level candidate pixels; the radial sampling filter respectively projects the target image and the template image on a set of radial lines to upgrade some first-level candidate pixels to second-level candidate pixels; the template matching filter resizes the template image and rotates it to N angles and M scales, and calculates the image at scale S i and angle α j The contrast and brightness perception correlation between the template image at the pixel and the target image at the pixel is calculated. If the absolute value of the correlation is higher than the threshold, it is considered that the template is found at the second-level candidate pixel. The present invention greatly reduces the amount of calculation and improves the speed of target recognition.

Description

Technical Field

[0001] The present invention relates to a radar and optoelectronic feature fusion target recognition system and method based on a deep network, belonging to the technical field of data processing. Background Art

[0002] A Chinese invention patent application with publication number CN117079117A discloses a method for calculating a target recognition rate interval based on an electrical digital data processing technology. The method is applied to calculating the target recognition rate interval of three types of heterogeneous sensor systems: communication signals, SAR radars, and optoelectronics. The method includes: calculating the maximum target recognition rate interval of the three types of heterogeneous sensor systems, calculating the target recognition rate interval of two types of heterogeneous sensors: communication signal sensors and SAR radar sensors supported by a target association fusion recognition algorithm, calculating the target recognition rate interval of two types of heterogeneous sensors: communication signal sensors and SAR radar sensors supported by a target association fusion recognition algorithm, and calculating the target recognition rate interval of two types of heterogeneous sensors: optoelectronic sensors and SAR radar sensors supported by a target association fusion recognition algorithm.

[0003] However, the target recognition speed of the technical solution disclosed in the above patent application is slow. Summary of the invention

[0004] To achieve the above-mentioned purpose of the invention, the present invention provides a radar and optoelectronic feature fusion target recognition system and method based on a deep network, which has small calculation amount and fast target recognition speed.

[0005] To achieve the above-mentioned purpose, the present invention provides a radar and optoelectronic feature fusion target recognition system based on a deep network, which includes a radar, a camera and an RCNN module. The radar is configured to obtain a scanning area A. r The camera is configured to obtain the characteristic information of the target in the scanning area A. r The RCNN module is configured to scan the area A according to rThe invention is characterized in that it also includes a data fusion module and a template matching module, wherein the data fusion module is configured to fuse the target feature information obtained by the radar and the bounding box coordinates obtained by the RCNN module to obtain the target image; the template matching module includes a circular sampling filter, a radial sampling filter and a template matching filter, wherein the circular sampling filter uses the projection of the target image and the template image on a set of rings to divide the pixels of the target image into two categories: pixels that have no chance of matching the template image and first-level candidate pixels that have a certain chance of matching the template image, and determines a possible scaling factor for each first-level candidate pixel; the radial sampling filter upgrades some first-level candidate pixels to second-level candidate pixels according to the projection of the target image and the template image on a set of radial lines and the possible scaling factors provided by the circular sampling filter, and also assigns a possible rotation angle to each second-level candidate pixel, and discards the pixels that are not upgraded; the template matching filter resizes and rotates the template image to N angles and M scales, and calculates the rotation angle at the scale S i and angle α j The contrast and brightness perception correlation between the template image at the second-level candidate pixel and the target image at the second-level candidate pixel. If the absolute value of the correlation is higher than the threshold, it is considered that the template is found at the second-level candidate pixel, j=0,2…,N-1; i=0,2…,M-1, N and M are positive integers greater than or equal to 2.

[0006] To achieve the above-mentioned purpose, the present invention also provides a target recognition method based on deep network and radar and photoelectric feature fusion, which is characterized by comprising the following steps:

[0007] S01: Get scanning area A through radar r Feature information of internal targets;

[0008] S02: Step camera acquires scanning area A r images;

[0009] S03: Scan area A through RCNN module r Get the bounding box coordinates of the target from the image;

[0010] S04: fusing the feature information obtained by the radar and the bounding box coordinates obtained by the RCNN module to obtain a target image through a data fusion module;

[0011] S05: Obtain the type of the target image through the template matching module; the template matching module includes a circular sampling filter, a radial sampling filter and a template matching filter, wherein the circular sampling filter uses the projections of the target image and the template image on a set of rings to divide the pixels of the target image into two categories: pixels that have no chance of matching the template image and first-level candidate pixels that have a certain chance of matching the template image, and determines a possible scaling factor for each first-level candidate pixel; the radial sampling filter uses the projections of the target image and the template image on a set of radial lines and the possible scaling factors provided by the circular sampling filter to upgrade some first-level candidate pixels to second-level candidate pixels, and also assigns a possible rotation angle to each second-level candidate pixel, discarding pixels that are not upgraded; the template matching filter resizes and rotates the template image to N angles and M scales, and calculates the rotation angle at scale S i and angle α j The contrast and brightness perception correlation between the template image at the second-level candidate pixel and the target image at the second-level candidate pixel. If the absolute value of the correlation is higher than the threshold, it is considered that the template is found at the second-level candidate pixel (x, y), j=1,1,…,N-1; i=0,1,…,M-1, N and M are both positive integers greater than or equal to 2.

[0012] Compared with the prior art, the radar and optoelectronic feature fusion target recognition system and method based on deep network provided by the present invention has small calculation amount and fast target recognition speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a radar and optoelectronic feature fusion target recognition system based on deep network provided by the first embodiment of the present invention.

[0014] Figure 2 It is a flow chart of radar scanning speed optimization provided by the first embodiment of the present invention. DETAILED DESCRIPTION

[0015] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0016] First embodiment

[0017] Figure 1 The radar and optoelectronic feature fusion target recognition system based on deep network is provided in the first embodiment of the present invention. Figure 1 As shown, the radar and optoelectronic feature fusion target recognition system based on the deep network provided in the first embodiment includes a radar, a camera and a processor, the processor includes an RCNN module, and the radar is configured to obtain a scanning area Ar The camera is configured to obtain the characteristic information of the target in the scanning area A. r The RCNN module is configured to scan the area A according to r The processor further comprises a data fusion module and a template matching module, wherein the data fusion module is configured to fuse the characteristic information of the target acquired by the radar and the bounding box coordinates acquired by the RCNN module to acquire the target image; the template matching module comprises a circular sampling filter, a radial sampling filter and a template matching filter, wherein the circular sampling filter uses the projection of the target image and the template image on a set of rings to divide the pixels of the target image into two categories: pixels that have no chance of matching the template image and first-level candidate pixels that have a certain chance of matching the template image, and determines a possible scaling factor for each first-level candidate pixel; the radial sampling filter uses the projection of the target image and the template image on a set of radial lines and the possible scaling factors provided by the circular sampling filter to upgrade some first-level candidate pixels to second-level candidate pixels, and also assigns a possible rotation angle to each second-level candidate pixel, discarding pixels that have not been upgraded; the template matching filter resizes and rotates the template image to N angles and M scales, and calculates the image at scale S i and angle α j The contrast and brightness perception correlation between the template image at the second-level candidate pixel and the target image at the second-level candidate pixel. If the absolute value of the correlation is higher than the threshold, it is considered that the template is found at the second-level candidate pixel (x, y), j=0,1…,N-1; i=0,1…,M-1, N and M are both positive integers greater than or equal to 2.

[0018] In the first embodiment, if the sensor is a synthetic aperture radar (SAR), the synthetic aperture radar (SAR) can generate a two-dimensional or three-dimensional image of the target and provide richer target information. In this case, the camera can be omitted. The RCNN module obtains the boundary coordinates of the target based on the image information provided by the synthetic aperture radar (SAR). The template matching module performs target recognition based on the target image provided by the data fusion module.

[0019] In the first embodiment, the contrast and brightness perception correlation is:

[0020] ,

[0021] In the formula , yes The average value of is the vector obtained by copying the grayscale of the pixels of the template image; ; yes , q is a vector obtained by copying the grayscale of the pixels of the target image corresponding to the template image; q=βa+γI+ε, I is the brightness vector; β is the contrast correction factor, and γ is the brightness correction factor, ε is the vector of residual errors; t β is the contrast correction threshold; t γ is the brightness correction threshold.

[0022] In the first embodiment, the circular sampling filter uses the projections of the target image and the template image on a set of rings to divide the pixels of the target image into two categories: pixels that have no chance of matching the template image and first-level candidate pixels that have a certain chance of matching the template image, and determines a possible scaling factor for each first-level candidate pixel, specifically including:

[0023] S3-01: Resize the template image Q to each scale S i , obtain the resized templates Q0, Q1, ..., Q M-1；

[0024] S3-02: For each resized template Q i Circular sampling is performed with a predefined radius, resulting in a 2D matrix of multi-scale rotation-invariant features CQ with M rows and L columns:

[0025] C Q i,k = Cis Q i x 0 , y 0 , r k = 1 2 π r k ∑ j=0 N-1 Q i [ x- x 0 + r k cos α j , y- y 0 + r k sin α j ] ,

[0026] Where (x0), y0) is the center pixel coordinate of the template image Q; i = 1, 2…, M; k = 0, 2…, L-1;

[0027] S3-03: Given a target image A for analysis, construct a 3D image matrix CA(x, y, k):

[0028] ;

[0029] S3-04: Using Matrix C Q and C A and contrast and brightness thresholds T β and T γTo detect the circular sampling correlation CisCorr at the best matching scale for each pixel (x, y) in the target image A: ,

[0030] if , t1 is the threshold, the pixel (x, y) in the target image A is classified as a first-level candidate pixel, and the best matching scale is:

[0031] .

[0032] In the first embodiment, the radial sampling filter uses the projections of the target image and the template image on a set of radial lines and possible scaling factors provided by the circular sampling filter to upgrade some primary candidate pixels to secondary candidate pixels, and also assigns possible rotation angles to each secondary candidate pixel including:

[0033] S4-01: Given an image template Q and N angles of inclination (α0, α1, ..., α N-1 ), using λ=r L-1 Radial sampling is performed on the image template Q to generate a vector R with M features Q (j):

[0034] R Q j = Ras Q r L-1 x 0 , y 0 , α j = 1 L-1 ∑ k=0 L-1 Q[ x- x 0 + r k cos α j , y- y 0 + r k sin α j ] ,

[0035] S4-02: For each first-level candidate pixel (x, y) in the target image A, the target image A is calculated at its possible scales. Radial sampling produces a vector with N features :

[0036] ;

[0037] S4-03: At each first-level candidate pixel (x, y) in the target image A, the radial sampling filter uses the vector R A (x,y,j),R Q and contrast and brightness thresholds T β and T γ To detect the radial sampling correlation RasCorr at the best matching angle:

[0038] ,

[0039] In the formula, represents the j positions of the cyclic shifted independent variable vector;

[0040] S4-04:If , t2 is the threshold, then it is the second-level candidate pixel (x, y), otherwise it is discarded. The optimal rotation angle for the second-level candidate pixel (x, y) is:

[0041] .

[0042] The present invention can achieve the beneficial effects of small matching calculation amount and fast matching through the above technical solution.

[0043] In the first embodiment, the RCNN module includes a region generation module, a selective search module, a feature extraction module and a bounding box regression module, wherein the region generation module is configured to generate a forward propagation neural network network of candidate regions based on the image of the scanning area Ar, and output the targets, target types and bounding box coordinates contained in each candidate region; the selective search module uses an image segmentation method based on feature points to quickly generate a batch of candidate regions that may contain targets, and further filters these candidate regions using an intersection-over-union ratio threshold; the feature extraction module uses a pre-trained convolutional neural network to extract feature maps of the image; and the bounding box regression module calculates the bounding box coordinates of the target in the candidate region based on its actual position.

[0044] The deep network-based radar and optoelectronic feature fusion target recognition system provided in the first embodiment of the present invention also includes a storage device, which is used to store computer program codes for implementing the RCNN module, the data fusion module, and the template matching module, and can be called by the processor.

[0045] The storage device includes a storage array, which can provide persistent data storage for the processor. The storage array can include one or more storage array controllers. The storage array controller can be implemented as an automatic computer module including computer hardware, computer software, or a combination of computer hardware and software. The storage array controller can be configured to perform various storage tasks. The storage tasks can include writing data received from the processor to the storage array, erasing data from the storage array, retrieving data from the storage array and providing data to the processor, monitoring and reporting disk utilization and performance, performing redundancy operations, compressing data, encrypting data, etc.

[0046] The storage array controller may be implemented in various ways, including as a field programmable gate array ("FPGA"), a programmable logic chip ("PLC"), an application specific integrated circuit ("ASIC"), a system on a chip ("SOC"), or any computing device including discrete components such as a processing unit, a central processing unit, a computer memory, or various adapters. The persistent storage resources may include any number of storage drives and any number of non-volatile random access memory devices.

[0047] The radar and optoelectronic feature fusion target recognition system based on deep network provided in the first embodiment of the present invention also includes a communication module, which can be used to communicate with other devices.

[0048] The deep network-based radar and optoelectronic feature fusion target recognition system provided in the first embodiment of the present invention also includes a display module, which includes an interface for interacting with the user and is used to display the intermediate process and final result of the system.

[0049] Figure 2 is a flow chart of radar scanning speed optimization provided by the first embodiment of the present invention, such as Figure 2 As shown, the relative scanning speed v between the radar and the target r Determine by following these steps:

[0050] S2-01: Use the following formula to calculate the activation equation of the rth neuron in the Hopfield neural network :

[0051] ,,

[0052] In the formula, is the radar scanning time period t r The probability of finding the target; v r The scanning time period t between the radar and the target r The relative speed of W r is the radar scanning time period t r The scanning width of A r is the radar scanning time period t r The scanning area; R is the scanning time period t into which the total scanning time is divided r Total number of;

[0053] S2-02: Calculate the input variable of the rth neuron using the first-order Euler method :

[0054] + ;

[0055] S2-03: Calculate the output variable of the rth neuron according to the following formula :

[0056] ,,

[0057] Where U0 is the initial input value of the rth neuron;

[0058] S2-04: Assign a value to t and calculate the energy function according to the following formula:

[0059] ,

[0060] Where R is the total scan time divided into scan time periods t r Total number of;

[0061] S2-05: Determine whether the energy function E has reached the minimum. If the energy function E has not reached the minimum, return to step S2-01; if the energy function E has reached the minimum, the iteration ends and the output is , when the energy is the smallest r It is the optimal relative scanning speed between radar and target.

[0062] The present invention can achieve the beneficial effect of optimizing the relative scanning speed between the radar and the target through the above technical solution.

[0063] Second embodiment

[0064] The second embodiment of the present invention provides a target recognition method based on deep network and fusion of radar and photoelectric features, which includes the following steps: S01: acquiring the target position coordinates and running speed in the scanning area Ar by radar;

[0065] S02: The camera acquires an image of the scanning area Ar;

[0066] S03: Obtain the bounding box coordinates of the target based on the image of the scan area Ar through the RCNN module

[0067] S04: acquiring a target image by fusing the target position coordinates acquired by the radar and the bounding box coordinates acquired by the target recognition module through a data fusion module;

[0068] S05: Obtain the type of the target image through the template matching module; the template matching module includes a circular sampling filter, a radial sampling filter and a template matching filter, wherein the circular sampling filter uses the projections of the target image and the template image on a set of rings to divide the pixels of the target image into two categories: pixels that have no chance of matching the template image and first-level candidate pixels that have a certain chance of matching the template image, and determines a possible scaling factor for each first-level candidate pixel; the radial sampling filter uses the projections of the target image and the template image on a set of radial lines and the possible scaling factors provided by the circular sampling filter to upgrade some first-level candidate pixels to second-level candidate pixels, and also assigns a possible rotation angle to each second-level candidate pixel, discarding pixels that are not upgraded; the template matching filter resizes and rotates the template image to N angles and M scales, and calculates the rotation angle at scale S i and angle α j The contrast and brightness perception correlation between the template image at the second-level candidate pixel and the target image at the second-level candidate pixel. If the absolute value of the correlation is higher than the threshold, it is considered that the template is found at the second-level candidate pixel (x, y), j=1,2…,N-1; i=0,2…,M-1.

[0069] The beneficial effects of the method provided by the second embodiment are the same as those of the first embodiment, and will not be described again here.

[0070] Third embodiment

[0071] The third embodiment of the present invention further provides a program product, which includes a computer program code, and the computer program code can be called by a processor to execute the method described in the first and second embodiments.

[0072] The beneficial effects of the program product provided by the third embodiment are the same as those of the first and second embodiments, and will not be described again here.

[0073] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined. The meaning of "several" is one or more, unless otherwise clearly and specifically defined.

[0074] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A radar and optoelectronic feature fusion target recognition system based on a deep network, which includes a radar, a camera and an RCNN module. The radar is configured to obtain a scanning area A. r The camera is configured to obtain the characteristic information of the target in the scanning area A. r The RCNN module is configured to scan the area A according to r The bounding box coordinates of the target are obtained from the image, characterized in that It also includes a data fusion module and a template matching module. The data fusion module is configured to fuse the target feature information obtained by the radar and the bounding box coordinates obtained by the RCNN module to obtain the target image; the template matching module includes a circular sampling filter, a radial sampling filter and a template matching filter, wherein the circular sampling filter uses the projection of the target image and the template image on a set of rings to divide the pixels of the target image into two categories: pixels that do not match the template image and first-level candidate pixels that match the template image, and determines a scale factor for each first-level candidate pixel; the radial sampling filter upgrades some first-level candidate pixels to second-level candidate pixels according to the projection of the target image and the template image on a set of radial lines and the scale factor provided by the circular sampling filter, and also assigns a rotation angle to each second-level candidate pixel, discarding pixels that have not been upgraded; the template matching filter resizes and rotates the template image to N angles and M scales, and calculates the scale factor at the scale S i and angle α j The contrast and brightness perception correlation between the template image at the second-level candidate pixel and the target image at the second-level candidate pixel. If the absolute value of the correlation is higher than the threshold, it is considered that the template is found at the second-level candidate pixel. j=0,2…,N-1;i=0,2…,M-1, N and M are positive integers greater than or equal to 2. Among them, the relative scanning speed v between the radar and the target r Determine by following these steps: S2-01: Use the following formula to calculate the activation equation of the rth neuron in the Hopfield neural network : , In the formula, is the radar scanning time period t r The probability of finding the target; v r The scanning time period t between the radar and the target r The relative speed of W r is the radar scanning time period t r The scanning width of A r is the radar scanning time period t r The scanning area; S2-02: Calculate the input variable of the rth neuron using the first-order Euler method : + ; S2-03: Calculate the output variable of the rth neuron according to the following formula : , Where U0 is the initial input value of the rth neuron; S2-04: Assign a value to t and calculate the energy function according to the following formula: , Where R is the total scan time divided into scan time periods t r Total number of; S2-05: Determine whether the energy function E has reached the minimum. If the energy function E has not reached the minimum, return to step S2-01; if the energy function E has reached the minimum, the iteration ends and the output is .

2. The radar and optoelectronic feature fusion target recognition system based on deep network according to claim 1 is characterized in that: Contrast and brightness perception are related by: , where , yes The average value of is the vector obtained by copying the grayscale of the pixels of the template image; ; yes , q is a vector obtained by copying the grayscale of the pixels of the target image corresponding to the template image; q=βa+γI+ε, I is the brightness vector; β is the contrast correction factor, and γ is the brightness correction factor, ε is the vector of residual errors; t β is the contrast correction threshold; t γ is the brightness correction threshold.

3. The radar and optoelectronic feature fusion target recognition system based on deep network according to claim 1 is characterized in that: The RCNN module includes a region generation module, a selective search module, a feature extraction module and a bounding box regression module, wherein the region generation module is configured to generate a forward propagation neural network of candidate regions based on the image of the scanning area Ar, and output the targets, target types and bounding box coordinates contained in each candidate region; the selective search module uses an image segmentation method based on feature points to quickly generate a batch of candidate regions that may contain targets, and further filters these candidate regions using an intersection-over-union ratio threshold; the feature extraction module uses a pre-trained convolutional neural network to extract feature maps of the image; and the bounding box regression module calculates the bounding box coordinates of the target in the candidate region based on its actual position.

4. A target recognition method based on deep network by integrating radar and photoelectric features, characterized in that The steps include: S01: Get scanning area A through radar r Feature information of internal targets; S02: Step camera acquires scanning area A r images; S03: Through the RCNN module according to the scan area A r Get the bounding box coordinates of the target from the image; S04: fusing the feature information obtained by the radar and the bounding box coordinates obtained by the RCNN module to obtain a target image through a data fusion module; S05: Obtain the type of the target image through the template matching module; the template matching module includes a circular sampling filter, a radial sampling filter and a template matching filter, wherein the circular sampling filter uses the projections of the target image and the template image on a set of rings to divide the pixels of the target image into two categories: pixels that do not match the template image and first-level candidate pixels that match the template image, and determines a scaling factor for each first-level candidate pixel; the radial sampling filter uses the projections of the target image and the template image on a set of radial lines and the scaling factor provided by the circular sampling filter to upgrade some first-level candidate pixels to second-level candidate pixels, and also assigns a rotation angle to each second-level candidate pixel, discarding pixels that are not upgraded; the template matching filter resizes and rotates the template image to N angles and M scales, and calculates the resolution at scale S i and angle α j The contrast and brightness perception correlation between the template image at the second-level candidate pixel and the target image at the second-level candidate pixel. If the absolute value of the correlation is higher than the threshold, it is considered that the template is found at the second-level candidate pixel (x, y), j = 1, 1, ..., N-1; i = 0, 1, ..., M-1, N and M are both positive integers greater than or equal to 2, Among them, the relative scanning speed v between the radar and the target r Determine by following these steps: S2-01: Use the following formula to calculate the activation equation of the rth neuron in the Hopfield neural network : , In the formula, is the radar scanning time period t r The probability of finding the target; v r The scanning time period t between the radar and the target r The relative speed of W r is the radar scanning time period t r The scanning width of A r is the radar scanning time period t r The scanning area; S2-02: Calculate the input variable of the rth neuron using the first-order Euler method : + ; S2-03: Calculate the output variable of the rth neuron according to the following formula : , Where U0 is the initial input value of the rth neuron; S2-04: Assign a value to t and calculate the energy function according to the following formula: , Where R is the total scan time divided into scan time periods t r Total number of; S2-05: Determine whether the energy function E has reached the minimum. If the energy function E has not reached the minimum, return to step S2-01; if the energy function E has reached the minimum, the iteration ends and the output is .

5. The target recognition method based on deep network and fusion of radar and optoelectronic features according to claim 4 is characterized in that: Contrast and brightness perception are related by: , In the formula , yes The average value of is the vector obtained by copying the grayscale of the pixels of the template image; ; yes , q is a vector obtained by copying the grayscale of the pixels of the target image corresponding to the template image; q=βa+γI+ε, I is the brightness vector; β is the contrast correction factor, and γ is the brightness correction factor, ε is the vector of residual errors; t β is the contrast correction threshold; t γ is the brightness correction threshold.

Images