Method and system for constructing BEV space through fusion of visible light image and infrared image
By registering and fusing visible light images with infrared images, combining deep convolutional neural networks and dense convolutional networks, the BEV space is constructed using the timing self-attention mechanism, which solves the instability of pure visible light methods and the high cost of lidar Problem, achieving more accurate and robust BEV space construction, suitable for autonomous driving environments.
Patent Information
- Application Number
- CN202411062217.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-11-15
AI Technical Summary
In the prior art, pure visible light methods have instability in light, weather and false target information. Lidar lacks effective infrared images and visible light images to fusion to build BEV space due to high cost, sparse point clouds and poor real-time performance. method.
By registering, fusion, extracting feature information and weight generation of visible light images and infrared images, BEV space is constructed using deep convolutional neural network and dense convolutional network model, combining the timing self-attention mechanism and spatial attention mechanism to generate the Two fusion images are used to construct BEV space.
It improves the accuracy and robustness of BEV space, reduces interference from light and false information, retains original information and data integrity, enhances environmental consistency, and promotes autonomous driving decision planning.
Smart Images

Figure CN118967478A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image fusion, and in particular relates to a method and system for constructing a BEV space by fusing visible light images with infrared images. Background Art
[0002] At present, the methods for building BEV can be roughly divided into pure visible light and visible light lidar fusion methods. Among them, the pure visible light method is seriously affected by light, weather, false target information, etc., and lidar has the problems of high cost, sparse point cloud and poor real-time performance.
[0003] Generally speaking, the detection distance of ordinary visible light cameras decreases sharply under conditions of sudden light changes or low light or no light, and they are easily interfered by false target information, while infrared images can effectively resist interference from extreme conditions and false information. Therefore, the method of fusion of infrared images and visible light images can improve the accuracy and robustness of BEV space construction.
[0004] However, there is currently no method to fuse infrared images with visible light images to construct BEV. Summary of the invention
[0005] The purpose of the present invention is to overcome the defects of the prior art. The present invention provides a method and system for constructing a BEV space by fusing visible light images and infrared images. The present invention can improve the accuracy and robustness of BEV space construction.
[0006] In order to achieve the expected effect, the present invention adopts the following technical solutions:
[0007] The present invention discloses a method for constructing a BEV space by fusing a visible light image with an infrared image, comprising:
[0008] Register the original visible light image with the infrared image;
[0009] fusing the registered visible light image and the infrared image to generate a first fused image;
[0010] Extracting feature information of the registered visible light image and infrared image, and generating corresponding weights accordingly;
[0011] Input the weight, the first fused image, the registered visible light image and the infrared image into a preset loss function to calculate a second fused image;
[0012] Feature information of the second fused image is extracted to obtain a target feature map, and a BEV space is constructed based on the target feature map.
[0013] Furthermore, the registering of the original visible light image and the infrared image specifically includes: performing canny edge detection on the original visible light image and the infrared image to obtain a canny edge map, performing ORB detection and feature point pairing on the canny edge map, respectively, and performing RANSAC purification on the preliminary result of the feature point pairing to obtain an accurate transformation relationship between the visible light image and the infrared image, and finally completing the image registration through the transformation relationship.
[0014] Furthermore, the fusing of the registered visible light image and the infrared image to generate the first fused image specifically includes: inputting the registered visible light image and the infrared image into a single-channel fusion network for splicing, and then inputting them into a dense convolutional network model to generate the first fused image.
[0015] Furthermore, the extracting of feature information of the registered visible light image and infrared image and generating corresponding weights accordingly specifically includes: extracting the first several layers of features of the registered visible light image and infrared image through a deep convolutional neural network model to obtain a first feature map, and then inputting the first feature map into an information measurement module to calculate a feature gradient, and then inputting the feature gradient into an information protection module to generate corresponding weights.
[0016] Furthermore, the information metric module uses the following information metric formula to calculate the feature gradient:
[0017]
[0018] Among them, F is the F norm; k is the number of channels; D is the training set; C(I) is the feature map of different levels of the registered visible light image and infrared image; gI is the obtained gradient value; j is the number of maximum pooling layers; H and W represent the height and width of the feature map respectively.
[0019] Furthermore, the information protection degree module generates the corresponding weight using the following formula:
[0020]
[0021] Where c represents a positive hyperparameter; it is used to scale gI1 and gI2.
[0022] Furthermore, the preset loss function is:
[0023]
[0024] Among them, L sim (θ,D) represents the similarity loss between the fused image and the original image; λL ewc(θ, D) is used for continuous learning; L is the loss function; θ represents the parameters of DenseNet; λ represents the hyperparameters that control the weights.
[0025] Furthermore, extracting the feature information of the second fused image to obtain a target feature map and constructing a BEV space based on the feature information specifically includes:
[0026] Inputting the second fused image into a feature extraction network to extract feature information to obtain a target feature map;
[0027] The first temporal information is calculated through the temporal self-attention mechanism;
[0028] Performing a spatial attention mechanism calculation on the BEV space of the first time series information and the target feature map to obtain first spatial information;
[0029] The BEV space at the current moment is constructed using the first spatial information.
[0030] Furthermore, the calculating of the first time series information through the time series self-attention mechanism specifically includes: calculating the most relevant features of each feature of the current BEV space to be queried and the BEV space at the previous moment through the deformable attention mechanism, and calculating the first time series information.
[0031] The present invention also discloses a system for constructing a BEV space by fusing visible light images and infrared images, comprising:
[0032] An acquisition module, used for acquiring original visible light images and infrared images;
[0033] A construction module is used to construct a BEV space according to any of the methods described above.
[0034] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention provides a method and system for constructing a BEV space by fusing visible light images with infrared images. The present invention integrates infrared cameras to address the inevitable problems of visible light cameras in pure visible light BEVs, so that the perception level is no longer disturbed by light, environment, and false target information, thereby improving the accuracy and robustness of BEV construction. Compared with other fusion methods, front fusion has the ability to retain all original information, data integrity, and environmental consistency; it also has the ability to accurately identify and interpret the surrounding environment, which has a good promoting effect on subsequent autonomous driving decision-making and planning. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0036] Figure 1 It is a flow chart of a method for constructing a BEV space by fusing visible light images and infrared images provided by an embodiment of the present invention.
[0037] Figure 2 It is a flow chart of constructing a BEV space provided by an embodiment of the present invention.
[0038] Figure 3 This is a flowchart of image fusion provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0040] See also Figures 1 to 3 The present invention discloses a method for constructing a BEV space by fusing a visible light image with an infrared image. The specific steps are as follows:
[0041] S1. aligning the original visible light image with the infrared image;
[0042] In one embodiment, canny edge detection is performed on the original visible light image and the infrared image to obtain a canny edge map, and then ORB detection and feature point pairing are performed on the canny edge map respectively, and RANSAC purification is performed on the preliminary result of the feature point pairing to obtain an accurate transformation relationship between the visible light image and the infrared image, and finally image registration is completed through the transformation relationship.
[0043] Specifically, due to the different imaging principles of visible light images and infrared images, differences in camera structures, etc., there are problems such as different scaling ratios, relative translation, and rotation between visible light images and infrared images. Therefore, it is necessary to align the visible light image and the infrared image to ensure that the two images have the same size and viewing angle so that subsequent operations can be performed.
[0044] S2. Fusing the registered visible light image and the infrared image to generate a first fused image;
[0045] In one embodiment, the registered visible light image and infrared image are input into a single-channel fusion network for splicing, and then input into a dense convolutional network model to generate the first fused image.
[0046] Specifically, the registered visible light image and infrared image are input into u2fusion for splicing and then input into the dense convolutional network model DenseNet to generate the first fused image.
[0047] u2Fusion is a deep learning-based image fusion framework that can automatically evaluate the importance of source images and generate high-quality fused images through unsupervised learning. U2Fusion is designed to solve multiple image fusion problems such as multi-modality, multi-exposure and multi-focus.
[0048] It is worth noting that the present invention can take into account both visible light image and infrared image information during the front fusion to ensure the accuracy and robustness of BEV.
[0049] S3. Extract feature information of the registered visible light image and infrared image, and generate corresponding weights accordingly.
[0050] In one embodiment, the first several layers of features of the registered visible light image and infrared image are respectively extracted through a deep convolutional neural network model to obtain a first feature map, and then the first feature map is input into an information measurement module to calculate a feature gradient, and then the feature gradient is input into an information protection module to generate a corresponding weight.
[0051] Specifically, the registered visible light image and infrared image are subjected to feature extraction (Feature Extraction) through the deep convolutional neural network model VGG-16 network, and the first five layers of feature maps are extracted. Then, the first five layers of feature maps are input into the information measurement module to calculate the feature gradient (Information Measurement), and then the feature gradient is input into the information protection degree module to generate the corresponding weight (Information Preservation Degree).
[0052] Furthermore, the information metric module uses the following information metric formula to calculate the feature gradient:
[0053]
[0054] Among them, F is the F norm; k is the number of channels; D is the training set; C(I) is the feature map of different levels of the registered visible light image and infrared image; gI is the obtained gradient value; j is the number of maximum pooling layers; H and W represent the height and width of the feature map respectively.
[0055] Furthermore, the information protection degree module generates the corresponding weight using the following formula:
[0056]
[0057] Where c represents a positive hyperparameter; it is used to scale gI1 and gI2.
[0058] S4. The weights, the first fused image, the registered visible light image and the infrared image are input into a preset loss function to calculate a second fused image;
[0059] Furthermore, the above preset loss function is:
[0060]
[0061] Among them, L sim (θ,D) represents the similarity loss between the fused image and the original image; λL ewc (θ, D) is used for continuous learning; L is the loss function; θ represents the parameters of DenseNet; λ represents the hyperparameters that control the weights.
[0062] Densely connected network (DenseNet) is a deep convolutional neural network that enhances feature reuse and gradient flow by introducing dense connections in the network, thereby improving the performance and generalization ability of the model. In DenseNet, each layer takes the output of all previous layers as its input, forming a densely connected structure.
[0063] S5. Extract feature information of the second fused image to obtain a target feature map, and construct a BEV space based on the feature map.
[0064] In one embodiment, the second fused image is input into a feature extraction network to extract feature information to obtain a target feature map;
[0065] The first temporal information is calculated through the temporal self-attention mechanism;
[0066] Performing a spatial attention mechanism calculation on the BEV space of the first time series information and the target feature map to obtain first spatial information;
[0067] The BEV space at the current moment is constructed using the first spatial information.
[0068] In another embodiment, the calculating of the first time series information through the time series self-attention mechanism specifically includes: calculating the most relevant features of each feature of the current BEV space to be queried and the BEV space at the previous moment through the deformable attention mechanism, and calculating the first time series information.
[0069] In one embodiment, the steps include:
[0070] First, image features are extracted from the second fused image to obtain a feature map.
[0071] Then, an empty BEV space is constructed, and the temporal self-attention mechanism is calculated with the BEV space of the previous moment, specifically including: calculating each feature of the current BEV space to be queried and the most relevant features of the BEV space of the previous moment through the deformable attention mechanism, and performing calculations to obtain temporal information.
[0072] The BEV space with the initial time series information and the feature map of the fused image are then subjected to the second step of spatial attention mechanism calculation, which specifically includes: calculating the most relevant features of each feature of the current BEV space to be queried and the feature map through the deformable attention mechanism, and performing calculations to obtain spatial information, and finally constructing the BEV space at the current moment.
[0073] It is worth noting that the present invention uses BEVFormer as the basis to construct the BEV space. The BEVFomer of the present invention circumvents the problem of difficulty in obtaining depth information in visual sensors by using transformer, and at the same time uses the calculation of the temporal self-attention mechanism to extract temporal information from historical BEV features, which is beneficial to the speed estimation of moving targets and the detection of severely occluded targets.
[0074] In view of the inevitable problems of visible light cameras in pure visible light BEVs, the present invention integrates infrared cameras, so that the perception level is no longer disturbed by light, environment and false target information, thereby improving the accuracy and robustness of BEV construction. Compared with other fusion methods, front fusion has the ability to retain all original information, data integrity and environmental consistency; it also has the ability to accurately identify and interpret the surrounding environment, which has a good promoting effect on subsequent autonomous driving decision-making planning.
[0075] The present invention also discloses a system for constructing a BEV space by fusing visible light images and infrared images, comprising:
[0076] An acquisition module, used for acquiring original visible light images and infrared images;
[0077] A construction module is used to construct a BEV space according to any of the methods described above.
[0078] The system embodiments can be implemented one-to-one with the aforementioned method embodiments, and will not be described in detail here.
[0079] Based on the same inventive concept, the present invention also discloses an electronic device, which may include: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus. The processor may call the logic instructions in the memory to execute a method for constructing a BEV space by fusing visible light images with infrared images, including:
[0080] Register the original visible light image with the infrared image;
[0081] fusing the registered visible light image and the infrared image to generate a first fused image;
[0082] Extracting feature information of the registered visible light image and infrared image, and generating corresponding weights accordingly;
[0083] Input the weight, the first fused image, the registered visible light image and the infrared image into a preset loss function to calculate a second fused image;
[0084] Feature information of the second fused image is extracted to obtain a target feature map, and a BEV space is constructed based on the target feature map.
[0085] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0086] On the other hand, an embodiment of the present invention further provides a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer can execute a method for constructing a BEV space by fusing a visible light image with an infrared image provided by the above-mentioned method embodiments, including:
[0087] Register the original visible light image with the infrared image;
[0088] fusing the registered visible light image and the infrared image to generate a first fused image;
[0089] Extracting feature information of the registered visible light image and infrared image, and generating corresponding weights accordingly;
[0090] Input the weight, the first fused image, the registered visible light image and the infrared image into a preset loss function to calculate a second fused image;
[0091] Feature information of the second fused image is extracted to obtain a target feature map, and a BEV space is constructed based on the target feature map.
[0092] In another aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for constructing a BEV space by fusing a visible light image with an infrared image provided in the above embodiments is implemented, including:
[0093] Register the original visible light image with the infrared image;
[0094] fusing the registered visible light image and the infrared image to generate a first fused image;
[0095] Extracting feature information of the registered visible light image and infrared image, and generating corresponding weights accordingly;
[0096] Input the weight, the first fused image, the registered visible light image and the infrared image into a preset loss function to calculate a second fused image;
[0097] Feature information of the second fused image is extracted to obtain a target feature map, and a BEV space is constructed based on the target feature map.
[0098] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0099] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for constructing a BEV space by fusing visible light images and infrared images, characterized in that: include: Register the original visible light image with the infrared image; fusing the registered visible light image and the infrared image to generate a first fused image; Extracting feature information of the registered visible light image and infrared image, and generating corresponding weights accordingly; Input the weight, the first fused image, the registered visible light image and the infrared image into a preset loss function to calculate a second fused image; Feature information of the second fused image is extracted to obtain a target feature map, and a BEV space is constructed based on the target feature map.
2. The method according to claim 1, characterized in that: The registering of the original visible light image and the infrared image specifically includes: performing canny edge detection on the original visible light image and the infrared image to obtain a canny edge map, performing ORB detection and feature point pairing on the canny edge map respectively, and performing RANSAC purification on the preliminary result of the feature point pairing to obtain an accurate transformation relationship between the visible light image and the infrared image, and finally completing the image registration through the transformation relationship.
3. The method according to claim 1, characterized in that: The fusing of the registered visible light image and the infrared image to generate the first fused image specifically includes: inputting the registered visible light image and the infrared image into a single-channel fusion network for splicing, and then inputting them into a dense convolutional network model to generate the first fused image.
4. The method according to claim 1, characterized in that: The method of extracting feature information of the registered visible light image and infrared image and generating corresponding weights based on the feature information specifically includes: extracting the first several layers of features of the registered visible light image and infrared image through a deep convolutional neural network model to obtain a first feature map, and then inputting the first feature map into an information measurement module to calculate a feature gradient, and then inputting the feature gradient into an information protection module to generate corresponding weights.
5. The method according to claim 4, characterized in that: The information metric module uses the following information metric formula to calculate the feature gradient: Among them, F is the F norm; k is the number of channels; D is the training set; C(I) is the feature map of different levels of the registered visible light image and infrared image; gI is the obtained gradient value; j is the number of maximum pooling layers; H and W represent the height and width of the feature map respectively.
6. The method according to claim 5, characterized in that The information protection module uses the following formula to generate the corresponding weight: Where c represents a positive hyperparameter; it is used to scale gI1 and gI2.
7. The method according to claim 1, characterized in that: The preset loss function is: Among them, L sim (θ,D) represents the similarity loss between the fused image and the original image; λL ewc (θ, D) is used for continuous learning; L is the loss function; θ represents the parameters of DenseNet; λ represents the hyperparameters that control the weights.
8. The method according to claim 1, characterized in that: The step of extracting the feature information of the second fused image to obtain a target feature map and constructing a BEV space based on the feature information specifically includes: Inputting the second fused image into a feature extraction network to extract feature information to obtain a target feature map; The first temporal information is calculated through the temporal self-attention mechanism; Performing a spatial attention mechanism calculation on the BEV space of the first time series information and the target feature map to obtain first spatial information; The BEV space at the current moment is constructed using the first spatial information.
9. The method according to claim 8, characterized in that The method of calculating the first time series information through the time series self-attention mechanism specifically includes: calculating the most relevant features of each feature of the current BEV space to be queried and the BEV space at the previous moment through the deformable attention mechanism, and calculating the first time series information.
10. A system for constructing a BEV space by fusing visible light images and infrared images, characterized in that: include: An acquisition module, used for acquiring original visible light images and infrared images; A construction module for constructing a BEV space according to the method according to any one of claims 1-9.
