Automatic driving target detection method and device based on hyperspectrum, electronic equipment and storage medium

Abstract

The invention relates to the technical field of automatic driving, in particular to an automatic driving target detection method and device based on hyperspectrum, electronic equipment and a storage medium, which are used for solving the defects that the automatic driving target detection result is inaccurate and interference is generated on automatic driving under the traditional similar background interference condition. According to the method, the deep learning network is trained by using the RGB image to assist the hyperspectral data, the preprocessed hyperspectral data is synthesized into the pseudo-color image through dimension reduction, the pseudo-color image is input into the network, the final accurate target detection result is obtained through comparison and analysis of the spectral database and optimization of the output of the network, and the driving safety of the autonomous vehicle is improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and more specifically, to a hyperspectral-based autonomous driving target detection method, device, electronic device, and storage medium. Background Art

[0002] Autonomous driving has made significant progress and holds great potential for reducing traffic congestion, improving road safety, and reducing carbon emissions. Object detection in autonomous driving is a key technology used to identify and locate various objects in the vehicle's field of view, such as vehicles, pedestrians, and obstacles. Developing reliable autonomous driving remains a very challenging task because autonomous vehicles are intelligent agents that need to perceive, predict, make decisions, plan, and execute them in the real world, often in uncontrolled or complex environments. The perception systems of autonomous vehicles need to be accurate and real-time, especially in scenarios such as autonomous driving, where even a small error can lead to a fatal accident.

[0003] Traditional RGB camera-based detection methods based on color and texture features have difficulty distinguishing objects with similar appearance to the background. This results in low target detection accuracy in the presence of similar background interference, impacting vehicle driving safety. Hyperspectral imaging technology combines imaging and spectroscopy techniques to simultaneously detect both the two-dimensional geometric space and one-dimensional spectral information of a target.

[0004] Hyperspectral cameras can overcome the shortcomings of RGB cameras in detecting homochromatic objects. Compared to traditional RGB images, hyperspectral images can accurately identify and distinguish the materials of target objects, facilitating target detection even under similar background interference. Furthermore, with the development of related technologies, the cost of hyperspectral cameras has continued to decrease, and imaging speed has significantly increased. Currently, hyperspectral cameras based on array filters can achieve the imaging speed and resolution of conventional RGB cameras, meeting the real-time and safety requirements of autonomous driving systems. Although hyperspectral imaging provides rich spectral information, the scarcity of hyperspectral data limits its widespread application in certain scenarios. However, compared to hyperspectral data, RGB image datasets suitable for autonomous driving are readily available, and a large number of annotated datasets are already available for training deep learning models. Summary of the Invention

[0005] The present invention provides a hyperspectral-based autonomous driving target detection method to address the drawbacks of traditional autonomous driving target detection methods, such as inaccurate detection results under similar background interference conditions, which interferes with autonomous driving. Specifically, the method includes the following steps:

[0006] We collected photos of different target objects using RGB cameras and hyperspectral cameras, and created target detection datasets for each. We also recorded the spectra of different types of objects in the hyperspectral images and created a spectral database.

[0007] Correct and denoise the collected original hyperspectral image to obtain a high-resolution hyperspectral image;

[0008] Build a deep learning object detection network for RGB images and use the established RGB dataset for training;

[0009] Extract and calculate the spatial texture features of each band of the preprocessed hyperspectral image, arrange the texture indices of all bands in order from large to small, and remove the bands in the bottom 30%;

[0010] The removed hyperspectral data are reduced to three characteristic bands;

[0011] The three feature bands after dimensionality reduction are spliced ​​and reconstructed into a pseudo-color image, which contains information that cannot be detected by ordinary RGB images;

[0012] Input the pseudo-color image into the trained deep learning model to detect it;

[0013] Compare the detected objects with the data in the spectral library to optimize the network output;

[0014] According to the optimized network, the final target detection results are output.

[0015] Furthermore, the correction and denoising of the collected original hyperspectral image to obtain a high-resolution hyperspectral image is specifically performed as follows:

[0016] The black and white correction method is used to correct the image and obtain an image with high spectral reflectance. The formula is as follows:

[0017]

[0018] Where R is the final hyperspectral reflectance image, S is the original hyperspectral sample image directly acquired, D is the reference image in the presence of system dark current, and W is the reference image of the standard white board.

[0019] The minimum noise separation method is used for noise reduction preprocessing. The data X is separated by a high-pass filter template to obtain the noise X N With signal X S :

[0020] X=X N +X S

[0021] The transformation matrix V is obtained by maximizing the ratio of the signal covariance to the noise covariance.

[0022]

[0023] Where Y N and Y S are the noise and signal of the transformed data, respectively, C N and C S are the covariance of the noise and signal of the data respectively, and this optimization problem can be equivalent to:

[0024]

[0025] Where C represents the population covariance of the data, C = Cov(X), C = C N +C S According to the Lagrange multiplier method, the optimal solution is:

[0026] CV=λC N V

[0027] Take the eigenvectors corresponding to the first d largest eigenvalues ​​and construct the transformation matrix:

[0028] V=[V1,V2...,V d ]

[0029] Furthermore, the deep learning target detection network for RGB images is built and trained using the established RGB dataset as follows:

[0030] The constructed model is based on the YOLOV8 network, and optimizes the detection head by introducing the asymptotic feature pyramid network. It uses the strategy of progressive feature fusion to gradually integrate features of different levels into the target detection process, reducing the semantic differences between features at different levels, thereby improving the generalization of the model at different scales.

[0031] Furthermore, the spatial texture feature continuity extraction method is specifically as follows:

[0032] Use Sobel operator to extract the texture of each band image;

[0033] Use Matlab's bwlabel function to extract continuous segments of the texture image;

[0034] Calculate the continuity index of each continuous segment using the following formula:

[0035]

[0036] In the formula is the center point of each continuous segment in the band image.

[0037]

[0038] Where, is the distance from each point to the center point, The contribution of each point to the continuous segment it is in.

[0039] Normalize all continuous segments, the formula is as follows

[0040]

[0041] S is the final continuity index, and the formula is as follows:

[0042]

[0043] Where n1 is the number of continuous segments with only a single pixel in the band image, n i The total number of pixels in each continuous segment.

[0044] Furthermore, the dimensionality reduction method is PCA dimensionality reduction.

[0045] Another object of the present invention is to provide an autonomous driving target detection device based on a hyperspectral spectrum, comprising:

[0046] The data acquisition module includes an RGB camera and a hyperspectral camera. The RGB camera and the hyperspectral camera are used to collect photos of different target objects and generate target detection datasets. At the same time, the spectra of different types of objects in the hyperspectral images are recorded to generate a spectral database.

[0047] The preprocessing module is used to correct and denoise the collected original hyperspectral image to obtain a high-resolution hyperspectral image;

[0048] The training module uses a deep learning object detection network for RGB images and is trained using the established RGB dataset;

[0049] The spatial texture feature extraction module is used to extract and calculate the spatial texture features of each band of the pre-processed hyperspectral image, arrange the texture indicators of all bands in order from large to small, and eliminate the bands in the bottom 30%;

[0050] The image processing module is used to reduce the dimension of the hyperspectral data after elimination to three characteristic bands, and then splice the three characteristic bands after dimension reduction to reconstruct a pseudo-color image, which contains information that cannot be detected by ordinary RGB images;

[0051] Detection module, used to detect pseudo-color images in a trained deep learning model;

[0052] The data comparison module compares the detected objects with the data in the spectral library, optimizes the network output, and outputs the final target detection results based on the optimized network.

[0053] Another object of the present invention is to provide an electronic device comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the program, an autonomous driving target detection method as described in any one of the above is implemented.

[0054] Another object of the present invention is to provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the autonomous driving target detection method as described in any one of the above.

[0055] The autonomous driving target detection method, device, electronic device, and storage medium provided by this invention overcome the difficulty of RGB cameras in detecting objects under background interference. By recombining the bands of hyperspectral data to construct pseudo-RGB images, deep learning networks can be trained using labeled RGB image data, improving the model's generalization and training efficiency, reducing the impact of background on detected targets, and enhancing target detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0057] Figure 1 Schematic diagram of the process of the automatic driving target detection method according to an embodiment of the present invention.

[0058] Figure 2 This is the spectral database of an embodiment of the present invention.

[0059] Figure 3 This is a pseudo-color picture synthesized by an embodiment of the present invention. Figure 4 For composite pictures. DETAILED DESCRIPTION

[0060] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0061] like Figure 1 FIG2 is a flow chart of an autonomous driving target detection method according to an embodiment of the present invention. The autonomous driving target detection method according to an embodiment of the present invention includes:

[0062] S1. Collect photos of different target objects using RGB cameras and hyperspectral cameras, and create target detection datasets respectively. At the same time, record the spectrum of the object in the hyperspectral image and create a spectral database, such as Figure 2 As shown;

[0063] S2, correcting and denoising the original hyperspectral image collected in S1 to obtain a high-resolution hyperspectral image;

[0064] S201: Image correction uses a black and white correction method to obtain an image with high spectral reflectance. The formula is as follows:

[0065]

[0066] Where R represents the final hyperspectral reflectance image, S represents the original hyperspectral sample image directly acquired, D represents the reference image in the presence of system dark current, and W represents the reference image of the standard white board.

[0067] S202, the image denoising method uses the minimum noise separation method to perform noise reduction preprocessing, and separates the data X through the high-pass filter template to obtain the noise X N With signal X S , then:

[0068] X=X N +X S

[0069] The transformation matrix V is obtained by maximizing the ratio of the signal covariance to the noise covariance.

[0070]

[0071] Where Y N and Y S are the noise and signal of the transformed data, respectively, C N and C S are the covariance of the noise and signal of the data respectively, and this optimization problem can be equivalent to:

[0072]

[0073] Where C represents the population covariance of the data, C = Cov(X), C = C N +C S According to the Lagrange multiplier method, the optimal solution is:

[0074] CV=λC N V

[0075] Take the eigenvectors corresponding to the first d largest eigenvalues ​​and construct the transformation matrix:

[0076] V=[V1,V2...,V d ]

[0077] S3. Build a deep learning object detection network for RGB images and use the established RGB dataset for training;

[0078] The constructed model is based on the YOLOV8 network, and optimizes the detection head by introducing the asymptotic feature pyramid network. It uses the strategy of progressive feature fusion to gradually integrate features of different levels into the target detection process, reducing the semantic differences between features at different levels, thereby improving the generalization of the model at different scales.

[0079] S4. Extract and calculate the spatial texture features of each band of the preprocessed hyperspectral image, arrange the texture indices of all bands in descending order, and remove the bands in the bottom 30%;

[0080] S401, extracting the texture of each band image using the Sobel operator;

[0081] S402, using the bwlabel function of Matlab to extract continuous segments of the texture image;

[0082] S403. Calculate the continuity index of each continuous segment using the following formula:

[0083]

[0084] In the formula is the center point of each continuous segment in the band image.

[0085]

[0086] Where, is the distance from each point to the center point, The contribution of each point to the continuous segment it is in.

[0087] Normalize all continuous segments, the formula is as follows

[0088]

[0089] S is the final continuity index, and the formula is as follows:

[0090]

[0091] Where n1 is the number of continuous segments with only a single pixel in the band image, n i The total number of pixels in each continuous segment.

[0092] S5, reducing the dimension of the hyperspectral data filtered in S4 to three characteristic bands;

[0093] S6, splice the three feature bands after dimensionality reduction in S5 and reconstruct them into a pseudo-color image containing information that cannot be detected by ordinary RGB images. Figure 3 As shown in the figure, the dimensionality reduction method used is PCA dimensionality reduction.

[0094] S7, input the pseudo-color image in S6 into the trained deep learning model to detect it;

[0095] S8, compare the objects detected by S7 with the data in the spectral library to optimize the output of the network;

[0096] S9, based on the network optimized by S8, output the final target detection result as follows Figure 4 shown.

[0097] Based on the above embodiments, the autonomous driving target detection model provided by the embodiments of the present invention is used for target detection under similar background interference conditions, including:

[0098] The data acquisition module includes an RGB camera and a hyperspectral camera. The RGB camera and the hyperspectral camera are used to collect photos of different target objects and generate target detection datasets. At the same time, the spectra of different types of objects in the hyperspectral images are recorded to generate a spectral database.

[0099] The preprocessing module is used to correct and denoise the collected original hyperspectral image to obtain a high-resolution hyperspectral image;

[0100] The training module uses a deep learning object detection network for RGB images and is trained using the established RGB dataset;

[0101] The spatial texture feature extraction module is used to extract and calculate the spatial texture features of each band of the pre-processed hyperspectral image, arrange the texture indicators of all bands in order from large to small, and eliminate the bands in the bottom 30%;

[0102] The image processing module is used to reduce the dimension of the hyperspectral data after elimination to three characteristic bands, and then splice the three characteristic bands after dimension reduction to reconstruct a pseudo-color image, which contains information that cannot be detected by ordinary RGB images;

[0103] The detection module is used to detect pseudo-color images in a well-trained deep learning model; the data comparison module compares the detected objects with the data in the spectral library, optimizes the network output, and outputs the final target detection results based on the optimized network.

[0104] Based on the above embodiments, the present invention provides, on one hand, a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the program, an autonomous driving target detection method as described in any one of the above embodiments is implemented.

[0105] The logical instructions in the memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk, or an optical disk.

[0106] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the autonomous driving target detection method provided by the above-mentioned methods.

[0107] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0108] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A hyperspectral-based automatic driving target detection method, characterized in that: The steps include: Collect photos of different target objects through RGB camera and hyperspectral camera respectively, and make target detection data sets respectively. Meanwhile, record the spectra of different types of objects in hyperspectral images and make spectral databases. Correct and denoise the collected original hyperspectral image to obtain a high-resolution hyperspectral image; Build a deep learning object detection network for RGB images and use the established RGB object detection dataset for training; Extract and calculate the spatial texture features of each band of the preprocessed hyperspectral image, arrange the texture indexes of all bands in order from large to small, and remove the bands in the last 30%; The removed hyperspectral data is reduced in dimension to three characteristic bands, and then the three characteristic bands after dimension reduction are spliced ​​and reorganized into a pseudo-color image containing information that cannot be detected by ordinary RGB images; Input the pseudo-color image into the trained deep learning model to detect it; The detected objects are compared with the data in the spectral library, the output of the network is optimized, and the final target detection results are output based on the optimized network.

2. The hyperspectral-based automatic driving target detection method according to claim 1, characterized in that: The specific steps to obtain a high-resolution hyperspectral image are: The image correction formula is as follows: Where R is the final hyperspectral reflectance image, S is the original hyperspectral sample image directly acquired, D is the reference image in the presence of system dark current, and W is the reference image of the standard white board. The minimum noise separation method is used for noise reduction preprocessing. The data X is separated by a high-pass filter template to obtain the noise X N With signal X S : X=X N +X S By maximizing the ratio of signal covariance to noise covariance, we get the transformation matrix V: Where Y N and Y S are the noise and signal of the transformed data, respectively, C N and C S are the covariance of the noise and signal of the data respectively, and the optimization problem can be equivalent to: In the formula, C represents the population covariance of the data, C = Cov(X), C = C N +C S , according to the Lagrange multiplier method, the optimal solution is: CV=λC N V Take the eigenvectors corresponding to the first d largest eigenvalues ​​and construct the transformation matrix: V=[V1,V2...,V d ]。 3. The hyperspectral-based automatic driving target detection method according to claim 1, characterized in that: The established RGB target detection dataset is used for training based on the YOLOV8 network model, and the detection head is optimized by introducing the asymptotic feature pyramid network. The strategy of progressive feature fusion is used to gradually integrate features of different levels into the target detection process, reducing the semantic differences between features at different levels.

4. The hyperspectral-based automatic driving target detection method according to claim 1, characterized in that: The spatial texture feature continuity extraction method is specifically as follows: The Sobel operator is used to extract the texture of each band image, and the bwlabel function of MATLAB extracts the continuous segments of the texture image; the continuity index of each continuous segment is calculated, and the formula is as follows: In the formula is the center point of each continuous segment in the band image, In the formula, is the distance from each point to the center point, is the contribution of each point to the continuous segment it is in, Normalize all continuous segments, the formula is as follows S is the final continuity index, and the formula is as follows: Where n1 is the number of continuous segments with only a single pixel in the band image, n i The total number of pixels in each continuous segment.

5. The hyperspectral-based automatic driving target detection method according to claim 1, characterized in that: The dimensionality reduction method is PCA dimensionality reduction.

6. An automatic driving target detection device based on hyperspectral spectrum, characterized in that: include: The data acquisition module includes an RGB camera and a hyperspectral camera. The RGB camera and the hyperspectral camera are used to collect photos of different target objects and respectively generate target detection data sets. Meanwhile, the spectra of different types of objects are recorded for the hyperspectral images to generate a spectral database. The preprocessing module is used to correct and denoise the collected original hyperspectral image to obtain a high-resolution hyperspectral image; The training module uses a deep learning object detection network for RGB images and is trained using the established RGB dataset; The spatial texture feature extraction module is used to extract and calculate the spatial texture features of each band of the preprocessed hyperspectral image, arrange the texture indicators of all bands in order from large to small, and remove the bands in the last 30%; The image processing module is used to reduce the dimension of the removed hyperspectral data to three characteristic bands, and to splice the three characteristic bands after the dimension reduction to reorganize them into a pseudo-color image, which contains information that cannot be detected by ordinary RGB images; The detection module is used to detect the pseudo-color images in the trained deep learning model; The data comparison module compares the detected objects with the data in the spectral library, optimizes the network output, and outputs the final target detection results based on the optimized network.

7. An electronic device, characterized in that: It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, an autonomous driving target detection method as described in any one of the above is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the automatic driving target detection method as described in any one of the above is implemented.

Images