Multi-modal image fusion method and device, computer equipment and storage medium

Abstract

The invention provides a multi-modal image fusion method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining a three-dimensional image and a two-dimensional image to be subjected to image fusion, carrying out the region division according to the same preset shape, and obtaining a three-dimensional image region corresponding to the three-dimensional image and a two-dimensional image region corresponding to the two-dimensional image; determining three-dimensional feature points located at the same depth distance in the three-dimensional image area; determining two-dimensional feature points, corresponding to the three-dimensional feature points, in the two-dimensional image region; obtaining the similarity between the three-dimensional feature points and the two-dimensional feature points; in the same depth distance of the three-dimensional image area, determining the number of corresponding three-dimensional feature points with the similarity degree smaller than a preset similarity degree threshold value; if the number is greater than a preset number threshold value, removing the three-dimensional feature points of which the similarity degree is less than a preset similarity degree threshold value; and obtaining a fused image based on the three-dimensional feature points which are not removed and the corresponding two-dimensional feature points. By adopting the method, the image fusion quality can be improved.

Claims

1. A multimodal image fusion method, characterized in that: The method comprises: Acquire a three-dimensional image and a two-dimensional image to be fused, and divide the regions according to the same preset shape to obtain a three-dimensional image region corresponding to the three-dimensional image and a two-dimensional image region corresponding to the two-dimensional image; Determine three-dimensional feature points located at the same depth distance in the three-dimensional image area; and determine that the three-dimensional feature points correspond to two-dimensional feature points in the two-dimensional image area; Obtaining a degree of similarity between the three-dimensional feature point and the two-dimensional feature point; In the same depth distance of the three-dimensional image area, determine the number of three-dimensional feature points corresponding to a similarity less than a preset similarity threshold; if the number is greater than the preset number threshold, remove the three-dimensional feature points with a similarity less than the preset similarity threshold; Based on the uneliminated three-dimensional feature points and the corresponding two-dimensional feature points, image fusion is performed to obtain a fused image.

2. The method according to claim 1, characterized in that The obtaining the similarity between the three-dimensional feature point and the two-dimensional feature point includes: Obtaining the projection pixel coordinates corresponding to the three-dimensional feature points, and determining the two-dimensional pixel coordinates of the two-dimensional feature points; The similarity degree is obtained based on the projected pixel coordinates and the two-dimensional pixel coordinates.

3. The method according to claim 2, characterized in that The obtaining of the projection pixel coordinates corresponding to the three-dimensional feature points includes: Based on the three-dimensional feature points of the current depth and the two-dimensional feature points corresponding to the three-dimensional feature points, obtain the external parameter calibration parameters corresponding to the current depth; wherein the current depth is any depth; The three-dimensional feature points at the current depth are projected using the extrinsic calibration parameters to obtain the projected pixel coordinates of the three-dimensional feature points.

4. The method according to claim 3, characterized in that The method of projecting the three-dimensional feature point at the current depth by using the external parameter calibration parameter to obtain the projected pixel coordinates of the three-dimensional feature point includes: Obtaining camera internal parameters corresponding to the camera device of the two-dimensional image; The three-dimensional feature points at the current depth are projected using the external calibration parameters and the camera internal parameters to obtain the projected pixel coordinates of the three-dimensional feature points.

5. The method according to claim 2, characterized in that: The obtaining the similarity based on the projection pixel coordinates and the two-dimensional pixel coordinates includes: Determining the Euclidean distance between the projected pixel coordinates and the two-dimensional pixel coordinates; Based on the Euclidean distance, the similarity degree is obtained.

6. The method according to claim 1, characterized in that The three-dimensional image and the two-dimensional image are both divided into a central area and an outer ring area; the outer ring area of ​​the three-dimensional image includes a plurality of the three-dimensional image areas; the outer ring area of ​​the two-dimensional image includes a plurality of the two-dimensional image areas; The step of fusing the images based on the unremoved three-dimensional feature points and the corresponding two-dimensional feature points to obtain a fused image includes: Determine the three-dimensional feature points and the two-dimensional feature points corresponding to the central area, and determine the extrinsic calibration parameters of the central area based on the three-dimensional feature points and the two-dimensional feature points corresponding to the central area; Determine the extrinsic calibration parameters of the outer ring area based on the three-dimensional feature points that are not eliminated in the outer ring area and the two-dimensional feature points corresponding to the three-dimensional feature points that are not eliminated; According to the extrinsic calibration parameters of the central area and the extrinsic calibration parameters of the outer ring area, the images of the central area and the outer ring area are fused respectively to obtain the fused image.

7. The method according to any one of claims 1 to 6, characterized in that: The method is applied to electric robots; The three-dimensional image is a point cloud image; The step of acquiring the three-dimensional image and the two-dimensional image to be fused includes: Acquiring the point cloud image by using a laser radar installed on the electric robot; The two-dimensional image is acquired by a camera device installed on the electric robot.

8. A multimodal image fusion device, characterized in that: The device comprises: An image region determination module is used to obtain a 3D image and a 2D image to be fused, and divide the regions according to the same preset shape to obtain a 3D image region corresponding to the 3D image and a 2D image region corresponding to the 2D image; A feature point determination module, used to determine three-dimensional feature points located at the same depth distance in the three-dimensional image area; and determine that the three-dimensional feature points correspond to two-dimensional feature points in the two-dimensional image area; A similarity determination module, used to obtain the similarity between the three-dimensional feature point and the two-dimensional feature point; a quantity determination module, configured to determine the number of three-dimensional feature points corresponding to a similarity less than a preset similarity threshold value at the same depth distance of the three-dimensional image area; if the number is greater than the preset number threshold value, then eliminate the three-dimensional feature points whose similarity is less than the preset similarity threshold value; The image fusion module is used to perform image fusion based on the unremoved three-dimensional feature points and the corresponding two-dimensional feature points to obtain a fused image.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images