A full-space laser radar scattering cross section calculation method based on multi-modal data

Abstract

The application discloses a kind of full space laser radar scattering cross section calculation method based on multi-modal data, comprising: S1, the background in visible light image is masked and the laser radar scattering cross section data under the corresponding visual angle of each image is measured;S2, construct the deep learning inference model based on finite visible light image to full space laser radar scattering cross section;S3, visible light image and visual angle data are input to the deep learning inference model trained in step S2 and carry out calculation inference, output the laser radar scattering cross section distribution map of full space;S4, according to the sampling accuracy of full space laser radar scattering cross section, the laser radar scattering cross section distribution map calibrated is interpolated, and the final result is output.The application realizes the method for solving the laser radar scattering cross section of target full space by using visible light and laser multi-modal data and combining deep learning technology, and enhances the rapid calculation ability for the laser scattering characteristics of the target full space to be measured.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of laser radar scattering characteristics calculation, in particular to a full-space laser radar scattering cross section calculation method based on multi-modal data. BACKGROUND

[0002] The laser radar scattering cross section (LRCS) is widely used in the fields of national defense, aviation, meteorology and other fields with important application value, and is a comprehensive reflection of the target laser scattering characteristics. The size of the laser radar scattering cross section is affected by many factors such as detection angle, target geometric structure shape, target surface material and roughness. The existing calculation method of laser radar scattering cross section depends on the construction of sufficient geometric structure and surface material representation of the target. However, there are many non-cooperative targets in reality, and the accurate structure and material characteristics of the target cannot be obtained, resulting in large deviation of the calculation result of the LRCS obtained by the traditional method.

[0003] With the rapid development of big data and artificial intelligence technology, it is possible to realize the LRCS characteristic calculation of non-cooperative targets by combining multi-modal data and artificial intelligence. Multi-modal data can help to complement information from multiple dimensions and overcome the limitations of single source data. Deep learning technology can enhance the information mining of multi-modal data by using the implicit relationship between data, and realize the solution and reasoning of the full-space non-cooperative target LRCS. Therefore, it is of great significance to design a full-space laser radar scattering cross section calculation method based on multi-modal data. SUMMARY

[0004] The purpose of the present application is to provide a full-space laser radar scattering cross section calculation method based on multi-modal data, which combines visible light image data and laser radar scattering cross section data, and builds a method framework for quickly solving the target full-space laser radar scattering interface based on multi-modal data fusion and deep learning technology, to provide a feasible technical approach for obtaining target full-space laser observation characteristics under limited measurement conditions.

[0005] In order to achieve the above purpose, the present application is realized by the following technical scheme:

[0006] A full-space laser radar scattering cross section calculation method based on multi-modal data, characterized in that the method comprises the following steps:

[0007] S1, a large number of targets are collected under different viewing angle conditions to record the viewing angle information of each image, then the background in the visible light image is masked, and the laser radar scattering cross section data corresponding to each viewing angle is measured;

[0008] S2, constructing a deep learning inference model based on limited visible light images to full space laser radar scattering cross section, wherein the deep learning inference model takes a limited number of visible light images and observation angle vectors corresponding to the images as input data, and takes a distribution diagram of the full space laser radar scattering cross section around the target in the spherical coordinate system as output data;

[0009] S3, for the target to be measured, taking multiple visible light images and recording the corresponding observation angles, masking the background in the visible light images, inputting the visible light images and the angle data into the deep learning inference model trained in step S2 for calculation and inference, and outputting the laser radar scattering cross section distribution diagram in the full space;

[0010] S4, taking the laser radar scattering cross section data of the target to be measured at least one angle as a reference, using the measured laser radar scattering cross section data to calibrate the laser radar scattering cross section distribution diagram output in step S3, and then interpolating the calibrated laser radar scattering cross section distribution diagram according to the sampling accuracy of the full space laser radar scattering cross section to output the final result.

[0011] Further, the masking process in steps S1 and S3 refers to taking the observed target as the foreground and the rest of the image pixels as the background, and setting the background image pixel value to 0 through image foreground and background separation method or manual annotation method, and only keeping the foreground pixels in the image.

[0012] Further, the deep learning inference model in step S2 includes two sub-models:

[0013] S21, collecting a certain amount of visible light images of the target in the full space as a data set, and based on the neural radiation field technology, constructing a generative neural network taking the masked visible light images and the image observation vectors as input and the visible light images of the target in the observation direction as output;

[0014] S22, taking a certain amount of masked visible light images collected in step S1 and laser radar scattering cross section data under the corresponding angle as a data set, and based on the convolutional neural network technology, constructing a regression neural network taking the masked visible light images as input and the laser radar scattering cross section values under the same angle of the visible light images as output.

[0015] Further, the input to the trained deep learning inference model for calculation and inference in step S3 further includes the following steps:

[0016] S31, inputting a limited number of masked visible light images and their observation angles into the generative neural network based on the neural radiation field technology trained in step S21 to generate visible light images around the target at a fixed sampling interval in the spherical coordinate system.

[0017] S32, input the generated visible light image into the regression neural network based on the convolutional neural network technology trained in step S22 to calculate the laser radar scattering cross section corresponding to each image at each view angle;

[0018] S33, store the laser radar scattering cross section data at each angle calculated in step S32 in a matrix as output in the order of the zenith angle and the azimuth angle of the observed target in the spherical coordinate.

[0019] Further, the numerical calibration method in step S4 comprises the following steps:

[0020] S41, based on the reference laser radar scattering cross section of the observation view angle query the value corresponding to the view angle in the laser radar scattering cross section distribution map

[0021] S42, calculate the calibration ratio R of the reference laser radar scattering cross section and the query value in the distribution map.

[0022] S43, multiply all elements in the distribution map by the calibration ratio R.

[0023] Further, the visible light image as input in step S3 comprises at least 6 images with an observation angle difference of 10 degrees.

[0024] Compared with the prior art, the present application has the following advantages:

[0025] (1) The prior art usually needs to construct a digital model of the target in advance, determine the representation model and parameters of the surface material. In actual engineering applications, the geometric structure and material characteristics of many non-cooperative targets are difficult to obtain, resulting in large errors in the above method. The present application uses limited visible light images and single-view laser radar scattering cross section data to avoid direct construction of geometric models and modeling of surface materials, and widens the solvable target objects under limited detection conditions.

[0026] (2) The prior art usually uses analytical methods to solve simple ideal models, and uses graphic electromagnetic technology, OpenGL technology or ray tracing technology and rough surface laser scattering theory to solve complex target models, and the solving process can only obtain data at one view angle at a time. The present application uses deep learning technology to generate multi-view observation images of the target to be measured, and solves the laser radar scattering cross section data in the whole space at one time, thereby enhancing the calculation efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1A flowchart of a full-space lidar scattering cross section calculation method based on multi-modal data according to the present application;

[0028] Figure 2 A specific flowchart of a full-space lidar scattering cross section calculation method based on multi-modal data according to the present application. DETAILED DESCRIPTION

[0029] The present application will be further described in conjunction with the accompanying drawings and a preferred embodiment.

[0030] It should be noted that the accompanying drawings are in a very simplified form and all use non-precise proportions, only to facilitate, clear to assist in explaining the purpose of the embodiments of the present application, and not to limit the implementation of the present application defined conditions, so not have the technical substance of the meaning, any structure modification, the change of proportional relationship or size adjustment, without affecting the effect and can achieve the purpose of the present application can produce, should still fall within the scope of the technical content disclosed by the present application can cover.

[0031] It should be noted that in the present application, such as the relationship between the first and second terms are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the term "include", "contain" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes the explicitly listed elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or equipment.

[0032] As shown in Figure 1 , 2 A full-space lidar scattering cross section calculation method based on multi-modal data includes:

[0033] S1, collect visible light images of a large number of targets under different viewing angle conditions, record the viewing angle information of each image, then mask the background in the visible light image, and measure the laser radar scattering cross section (LRCS) data under the corresponding viewing angle of each image;

[0034] In this embodiment, 1000 targets with different geometric shapes are simulated, with an azimuth angle of 1 degree and a zenith angle of 1 degree in the spherical coordinate system as the sampling interval, the visible light images of the targets are collected and the LRCS data of the targets are solved, wherein the surface material of the target is set to Lambert material with reflectivity of 1.

[0035] Wherein, the mask processing refers to taking the observation target as the foreground and the rest of the image pixels as the background, and setting the background image pixel value to 0 through an image foreground and background separation method or a manual labeling method, so that only the pixels of the foreground are retained in the image.

[0036] S2, a deep learning inference model based on limited visible light images to full space LRCS is constructed, which takes a limited number of visible light images and image corresponding observation view vectors as input data, and a distribution diagram of the full space LRCS around the target in the spherical coordinate system as output data;

[0037] The deep learning inference model in step S2 is composed of two sub-models, and the model training stage further includes the following steps

[0038] S21, a large number of visible light images of the target full space are collected as a data set, and based on the neural radiation field (NeRF) technology, a generative neural network is constructed, which takes the visible light image processed by the mask and the image observation vector as the input, and only retains the visible light image of the target in the observed direction as the output.

[0039] In this embodiment, the visible light image as the data set refers to the visible light image collected in step S1, and the observation vector refers to the observation vector in the spherical coordinate system Wherein, θ is the zenith angle, is the azimuth angle.

[0040] S22, a large number of visible light images processed by the mask and the corresponding LRCS data under the observation angle in step S1 are taken as a data set, and based on the convolutional neural network technology, a regression neural network is constructed, which takes the visible light image processed by the mask as the input, and the LRCS value under the same observation angle of the visible light image as the output;

[0041] In this embodiment, the neural network is a 5-layer convolutional neural network, and the mean square error of the true value and the predicted value is taken as the loss function.

[0042] S3, for the target to be measured, a plurality of visible light images are photographed and the corresponding observation view is recorded, the background in the visible light image is masked, and the visible light image and the view data are input into the deep learning inference model trained in step S2 for calculation and inference, and the LRCS distribution diagram of the full space is output.

[0043] The input into the trained deep learning inference model for calculation and inference in step S3 further includes the following steps:

[0044] S31, input the limited number of visible light images and their observation view angles after mask processing into the generative neural network based on neural radiation field technology trained in step S21, and generate visible light images around the target at fixed sampling intervals in the spherical coordinate system;

[0045] S32, input the generated visible light images into the regression neural network based on convolutional neural network technology trained in step S22, and calculate the corresponding LRCS of each image at each view angle;

[0046] S33, store the LRCS data of each angle calculated in step S32 in the matrix as output in the order of the zenith angle and the azimuth angle of the observed target in the spherical coordinate system.

[0047] In the embodiment, step S31 uses 6 visible light images after mask processing, and the observation view angles of the images are the view angles corresponding to the six views of the target; in step S32, the zenith angle is sampled at an interval of 1 degree, and the azimuth angle is sampled at an interval of 1 degree, and a total of 64800 images are generated; in step S33, the LRCS data is stored in a two-dimensional matrix, wherein the x-axis of the matrix has 180 elements, and the y-axis has 360 elements.

[0048] S4, obtain the LRCS data of at least one view angle of the target to be measured as a reference, use the measured LRCS data to numerically calibrate the LRCS distribution map output in step S3, and then interpolate the calibrated LRCS distribution map according to the sampling accuracy of the full-space LRCS to output the final result.

[0049] The numerical calibration method in step S4 includes the following steps:

[0050] S41, based on the reference observation view angle query the value corresponding to the view angle in the LRCS distribution map

[0051] S42, calculate the calibration ratio R of the reference LRCS and the queried value in the distribution map;

[0052] S43, multiply all elements in the distribution map by the calibration ratio R.

[0053] In the embodiment, the LRCS data observed in the forward direction of the target, i.e., the observation direction with the zenith angle and the azimuth angle of 0, is used as the reference for the numerical calibration in step S4.

[0054] In summary, the application is a method framework for quickly solving the full-space laser radar scattering interface of a target by combining visible light image data and laser radar scattering cross-section data, based on multi-modal data fusion and deep learning technology, providing a feasible technical approach for obtaining the full-space laser observation characteristics of a target under limited measurement conditions.

[0055] Although the present application has been described in detail by the preferred embodiments above, it should be appreciated that the above description should not be considered as limiting the present application. Various modifications and alternatives to the present application will be apparent to those skilled in the art after reading the above description. Therefore, the scope of protection of the present application should be defined by the appended claims.

Claims

1. A method for full-space laser radar scattering cross section calculation based on multi-modal data, characterized in that, The method comprises the following steps: S1, collecting visible light images of a plurality of targets under different viewing angle conditions, recording the viewing angle information of each image, then performing mask processing on the background in the visible light image, and measuring the laser radar scattering cross section data under the viewing angle corresponding to each image; S2, constructing a deep learning inference model based on limited visible light images to full-space laser radar scattering cross section, wherein the deep learning inference model takes a limited number of visible light images and the observation viewing angle vector corresponding to the images as input data, and takes a distribution map of the full-space laser radar scattering cross section around the target in the spherical coordinate system as output data; S3, for a target to be measured, taking multiple visible light images and recording the corresponding observation viewing angle, performing mask processing on the background in the visible light image, inputting the visible light image and the viewing angle data into the deep learning inference model trained in step S2 for calculation and inference, and outputting a full-space laser radar scattering cross section distribution map; S4, obtaining laser radar scattering cross section data under at least one viewing angle of the target to be measured as a reference, numerically calibrating the laser radar scattering cross section distribution map output in step S3 using the measured laser radar scattering cross section data, and then interpolating the calibrated laser radar scattering cross section distribution map according to the sampling accuracy of the full-space laser radar scattering cross section to output the final result.

2. The method for full-space laser radar scattering cross section calculation based on multi-modal data of claim 1, wherein, The mask processing in steps S1 and S3 refers to taking the observed target as the foreground and the rest of the image pixels as the background, and setting the background image pixel value to 0 through image foreground and background separation methods or manual annotation methods, and only keeping the foreground pixels in the image.

3. The method of claim 1, wherein, The deep learning inference model in step S2 comprises two sub-models: S21, collecting a certain amount of visible light images of a target in full space as a data set, based on neural radiation field technology, constructing a generative neural network taking the mask-processed visible light image and the image observation vector as input, and only retaining the visible light image of the target in the observation direction as output; S22, taking a certain amount of mask-processed visible light images and laser radar scattering cross section data under the corresponding viewing angle in step S1 as a data set, based on convolutional neural network technology, constructing a regression neural network taking the mask-processed visible light image as input, and the laser radar scattering cross section value under the same viewing angle of the visible light image as output.

4. The method of claim 1, wherein, The calculation and inference of the trained deep learning inference model in step S3 further comprises the following steps: S31, inputting a limited number of mask-processed visible light images and their observation viewing angles into the generative neural network based on neural radiation field technology trained in step S21, generating visible light images around the target at a fixed sampling interval in the spherical coordinate system, and generating visible light images at each viewing angle; S32, inputting the generated visible light image into the regression neural network based on convolutional neural network technology trained in step S22, calculating the laser radar scattering cross section corresponding to each image at each viewing angle; S33, in the form of a matrix, according to the zenith angle and azimuth angle of the observed target under the spherical coordinate as the arrangement order, the laser radar scattering cross section data of each angle calculated in step S32 is stored in the matrix as the output.

5. The full-space laser radar scattering cross section calculation method based on multi-modal data according to claim 1, characterized in that, The numerical calibration method in step S4 comprises the following steps: S41, observing the reference laser radar scattering cross section at the observation angle querying the value corresponding to the observation angle in the laser radar scattering cross section distribution S42, calculating the calibration ratio R of the reference laser radar scattering cross section and the query value in the distribution map; S43, multiplying all elements in the distribution map by the calibration ratio R.

6. The method of full-space laser radar cross section calculation based on multi-modal data of claim 1, wherein, The visible light image as the input in step S3 comprises at least 6 images with an observation angle difference of 10 degrees.

Images