An intelligent fusion method and system based on image information mechanism
By calculating image weight values and selecting appropriate algorithms, the problems of edge noise and resolution inconsistency in multi-source image fusion were solved, achieving efficient fusion of depth images and RGB images to generate high-quality 3D point clouds.
Patent Information
- Application Number
- CN202310501442.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-06
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2043-05-06
AI Technical Summary
Existing technologies for multi-source image fusion suffer from problems such as edge noise interference and inconsistent resolutions of images from different sources, which make fusion difficult.
By acquiring image information from depth images and RGB images respectively, calculating weight values, selecting suitable clustering and point-filling algorithms, and processing the images using a preset time-domain filter, the standard deviation is judged to determine whether to reprocess or directly fuse, thus achieving the fusion of images of different resolutions.
The image fusion effect has been optimized, enabling images of different resolutions to be effectively fused to generate 3D point clouds containing depth and color information.
Smart Images

Figure CN116912136B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image processing, and in particular relates to an intelligent fusion method and system based on image information mechanism. Background Art
[0002] Image fusion is the process of combining images or image sequence information of a specific scene from two or more sensors at the same time or different times to generate new useful information for interpreting the scene. This information cannot be obtained from a single image source. Image fusion can obtain more reliable, richer and more accurate useful information.
[0003] However, it is difficult to design an image fusion system that can adapt to various types of image sources. Therefore, in practical applications, different image fusion systems are usually developed for different application environments.
[0004] At present, multi-source image fusion has the following difficulties: 1. When regional shielding is used as the index for image fusion, there will be noise interference at the edge of the object; 2. The CMOS image sensor cannot receive the position of reflected infrared light, and "holes" will appear, affecting the quality of image fusion; 3. When fusing images, the resolutions of images from different sources need to be the same, otherwise fusion cannot be performed. Summary of the invention
[0005] The embodiments of the present invention provide an intelligent fusion method and system based on an image information mechanism to solve the problems in the prior art of edge noise in multi-source image fusion and the need for the same resolution of images from different sources.
[0006] In order to solve the above technical problems, the embodiments of the present invention disclose the following technical solutions:
[0007] One aspect of the present invention provides an intelligent fusion method based on an image information mechanism, which is applied to fuse a depth image and an RGB image, comprising:
[0008] Acquire the depth image and RGB image of the same scene taken at the same time respectively;
[0009] Obtain image information of depth image and RGB image respectively;
[0010] Calculate the weight values of the two images according to the image information of the depth image and the RGB image respectively;
[0011] Based on the weight values of the depth image and the RGB image, select the clustering algorithm and the point filling algorithm;
[0012] The two images are processed respectively using the clustering algorithm and the point filling algorithm;
[0013] The preset time domain filter is used to obtain the standard deviation of the processed depth image and RGB image respectively;
[0014] Determine whether the standard deviation of the two images is greater than the preset threshold.
[0015] If yes, the two images are reprocessed using a preset clustering algorithm and a point filling algorithm; the two reprocessed images are fused to obtain a 3D point cloud;
[0016] If not, the two images processed by the preset time domain filter are fused to obtain a 3D point cloud.
[0017] Optionally, the acquiring the image information of the depth image and the RGB image respectively includes:
[0018] According to the pixel data values of the depth image and the RGB image, the image information of the two images is calculated respectively, and the image information at least includes gradient value change, color complexity and three-dimensional information.
[0019] Optionally, the calculating weight values of the depth image and the RGB image respectively according to the image information of the two images includes:
[0020] For each image information of the depth image and the RGB image, the weight values of the depth image and the RGB image are calculated as follows:
[0021] Get the current weight values of the depth image and RGB image respectively;
[0022] comparing the image information of the depth image with the image information of the RGB image;
[0023] Determine an image with a larger image information value, and add a preset value to the current weight value of the image;
[0024] An image having a smaller image information value is determined, and a preset value is reduced on a current weight value of the image.
[0025] Optionally, the selecting of a clustering algorithm and a point filling algorithm based on the weights of the depth image and the RGB image includes:
[0026] Compare the current weight value of the depth image with the current weight value of the RGB image;
[0027] The preset clustering algorithm and point filling algorithm corresponding to the image with a larger current weight value are used as the selected algorithms.
[0028] Optionally, the using a preset time domain filter to obtain the standard deviation of the processed depth image and the RGB image respectively includes:
[0029] Obtain depth images and RGB images of the same scene taken at adjacent times as comparison images;
[0030] Based on the preset time domain filter, the standard deviation between the processed depth image and RGB image and the corresponding comparison image is calculated respectively.
[0031] Optionally, acquiring a depth image and an RGB image of the same scene captured at adjacent moments as comparison images includes:
[0032] Obtain a depth image and an RGB image of the same scene taken at the previous moment as comparison images;
[0033] or,
[0034] The depth image and RGB image of the same scene taken at a preset number of time moments are obtained as comparison images.
[0035] Optionally, the method further comprises:
[0036] Based on the preset time domain filter, the average values between the processed depth image and RGB image and the corresponding contrast image are calculated respectively;
[0037] The processed depth image and RGB image are fused using the average value.
[0038] Another aspect of the present invention provides an intelligent fusion system based on an image information mechanism, which is applied to fuse a depth image and an RGB image, comprising:
[0039] An image acquisition module is used to respectively acquire a depth image and an RGB image of the same scene shot at the same time;
[0040] An image information acquisition module, used to respectively acquire image information of a depth image and an RGB image;
[0041] A weight value calculation module is used to calculate the weight values of the depth image and the RGB image respectively according to the image information of the two images;
[0042] The algorithm selection module is used to select the clustering algorithm and the point filling algorithm based on the weight values of the depth image and the RGB image;
[0043] An image processing module, used to process the two images respectively using the clustering algorithm and the point filling algorithm;
[0044] A standard deviation acquisition module is used to obtain the standard deviations of the processed depth image and RGB image respectively using a preset time domain filter;
[0045] The judgment module is used to judge whether the standard deviations of the two images are both greater than a preset threshold.
[0046] A fusion module is used to reprocess the two images using a preset clustering algorithm and a point filling algorithm when the standard deviations of the two images are both greater than a preset threshold; the two reprocessed images are fused to obtain a 3D point cloud;
[0047] Furthermore, when the standard deviations of the two images are not both greater than a preset threshold, the two images processed by a preset time domain filter are fused to obtain a 3D point cloud.
[0048] The intelligent fusion method and system based on the image information mechanism provided by the embodiment of the present invention, firstly, based on the algorithm selection mechanism of the image information, by selecting the most suitable algorithm, greatly optimizes the effect of image fusion. Secondly, the image is processed with points so that images of different resolutions can also be fused. Thirdly, the feedback mechanism of the time domain filter is introduced to iterate and optimize the algorithm. Finally, the depth image and the RGB image are fused so that the fused image has both the depth information of the depth image and the color information of the RGB image. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 A schematic diagram of a flow chart of an intelligent fusion method based on an image information mechanism provided by an embodiment of the present invention;
[0050] Figure 2 A schematic diagram of capturing a depth image and an RGB image provided by an embodiment of the present invention;
[0051] Figure 3 An implementation provided for an embodiment of the present invention Figure 1 Schematic diagram of the process of step S103;
[0052] Figure 4 An implementation provided for an embodiment of the present invention Figure 1 Schematic diagram of the process of step S104;
[0053] Figure 5 An implementation provided for an embodiment of the present invention Figure 1 Schematic diagram of the process of step S106;
[0054] Figure 6 A schematic diagram of the structure of an intelligent fusion system based on image information mechanism provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0055] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0056] Figure 1It is a flowchart of an intelligent fusion method based on image information mechanism, which is applied to fuse depth image and RGB image. Figure 1 As shown, the method comprises the following steps:
[0057] Step S101: respectively obtaining a depth image and an RGB image of the same scene captured at the same time.
[0058] In the embodiments disclosed in the present invention, the following can be used: Figure 2 The device in the embodiment simultaneously captures a depth image and an RGB image of the same scene, wherein the depth camera is used to capture a depth image of the scene under infrared light, and the visible light camera is used to capture an RGB image of the scene under visible light.
[0059] Step S102: Obtain image information of the depth image and the RGB image respectively.
[0060] In the embodiments disclosed in the present invention, the image information of the depth image and the RGB image can be calculated using existing methods according to the pixel data values of the depth image and the RGB image, and the specific calculation process is not repeated here. The image information includes gradient value changes, color complexity, and three-dimensional information.
[0061] Step S103: Calculate the weight values of the depth image and the RGB image respectively according to the image information of the two images.
[0062] In one embodiment disclosed in the present invention, for each image information of the depth image and the RGB image, such as Figure 3 As shown, the weight values of the depth image and the RGB image are calculated in the following way:
[0063] Step S1031: respectively obtain the current weight values of the depth image and the RGB image.
[0064] In a specific embodiment disclosed in the present invention, the initial weight values of the depth image and the RGB image are both 0.5.
[0065] Step S1032: Compare the image information of the depth image and the image information of the RGB image.
[0066] Compare the values of the same image information in the two images one by one to determine the image with higher scene complexity. For example, compare the values of color complexity in the two images. If the color complexity value of the depth image is larger, it is considered that the scene complexity of the depth image in terms of color complexity is higher than that of the RGB image. Similarly, compare the different image information of the two images one by one to determine the image with higher scene complexity under each image information.
[0067] Step S1033: Determine an image with a larger image information value, and add a preset value to the current weight value of the image.
[0068] According to the comparison result in step S1032, an image with a larger value of each type of image information is obtained respectively, and a preset value is added to the current weight value of the image.
[0069] Step S1034: Determine an image with a smaller image information value, and reduce the preset value on the current weight value of the image.
[0070] According to the comparison result in step S1032, an image with a smaller value of each image information is obtained respectively, and a preset value is reduced from the current weight value of the image.
[0071] Step S1033 and step S1034 can be understood through the following examples:
[0072] For example, the current weight values of the depth image and the RGB image are both 0.5. The default value is 0.1. If the color complexity of the depth image is greater than that of the RGB image, the current weight value of the depth image is increased by 0.1, so that the current weight value of the depth image is 0.6. At the same time, the current weight value of the RGB image is reduced by 0.1, so that the current weight value of the RGB image is 0.4.
[0073] If the gradient value change of the depth image is less than that of the RGB image, then add 0.1 to the current weight value of the RGB image, so that the current weight value of the RGB image is 0.5. At the same time, reduce 0.1 from the current weight value of the depth image, so that the current weight value of the depth image is 0.5.
[0074] If the three-dimensional information value of the depth image is greater than the three-dimensional information value of the RGB image, then the current weight value of the depth image is increased by 0.1, so that the current weight value of the depth image is 0.6. At the same time, the current weight value of the RGB image is reduced by 0.1, so that the current weight value of the RGB image is 0.4.
[0075] By analogy, the weight values of the two images are calculated respectively according to the above method.
[0076] Step S104: based on the weight values of the depth image and the RGB image, a clustering algorithm and a point filling algorithm are selected.
[0077] In the embodiments disclosed in the present invention, Figure 4 As shown, the following sub-steps are used to select the clustering algorithm and the point filling algorithm:
[0078] Step S1041: Compare the current weight value of the depth image with the current weight value of the RGB image.
[0079] After obtaining the current weight values of the depth image and the RGB image, the sizes of the two are compared.
[0080] Step S1042: The preset clustering algorithm and point filling algorithm corresponding to the image with a larger current weight value are selected as the algorithm.
[0081] In the embodiments disclosed in the present invention, the most suitable clustering algorithm and point filling algorithm are pre-set for the two types of images respectively. For example, the clustering algorithm preset for the depth image is the Markov random field algorithm, and the point filling algorithm is the bilateral filter algorithm; the clustering algorithm preset for the RGB image is the superpixel algorithm, and the point filling algorithm is the linear interpolation algorithm.
[0082] For example, if the current weight value of the RGB image is larger, the preset clustering algorithm and point filling algorithm corresponding to the RGB image are used as the selected clustering algorithm and point filling algorithm, that is, the clustering algorithm is the superpixel algorithm, and the point filling algorithm is the linear interpolation algorithm.
[0083] Step S105: using a clustering algorithm and a point filling algorithm to process the two images respectively.
[0084] The clustering algorithm and the point filling algorithm determined in step S104 are used to process the two images respectively. After the images with different resolutions are filled with points, the subsequent image fusion can be completed. Therefore, the depth image and the RGB image in the embodiment disclosed by the present invention do not require the resolution to be consistent.
[0085] Step S106: using a preset time domain filter to obtain the standard deviations of the processed depth image and RGB image respectively.
[0086] In one embodiment disclosed in the present invention, Figure 5 As shown, the standard deviation of the processed depth image and RGB image can be obtained in the following way:
[0087] Step S1061: Acquire a depth image and an RGB image of the same scene captured at adjacent moments as comparison images.
[0088] In one embodiment disclosed in the present invention, the contrast image is obtained by the following method:
[0089] The depth image and RGB image of the same scene taken at the previous moment are obtained as comparison images. For example, the moment when the depth image and RGB image that need to be fused are taken is N, and the depth image and RGB image taken at the moment N-1, which is one time period before the moment N, are obtained, and the two images taken at the moment N-1 are used as comparison images for comparison with the corresponding type of images taken at the moment N.
[0090] or,
[0091] The depth image and RGB image of the same scene taken at a preset number of time moments are obtained as comparison images.
[0092] Obtain depth images and RGB images of the same scene taken at the next few moments as comparison images. For example, the moment when the depth image and RGB image that need to be fused are taken is N, and the depth image and RGB image are obtained at intervals after moment N. For example, obtain the depth image and RGB image taken at moments N+1 to N+n, and use the two images taken at these moments as comparison images.
[0093] Step S1062: Based on a preset time domain filter, the standard deviations between the processed depth image and the RGB image and the corresponding contrast image are calculated respectively.
[0094] Using the preset time domain filter, the standard deviation between the depth image processed by the clustering algorithm and the point filling algorithm and the depth image in the comparison image is calculated, and the standard deviation between the RGB image processed by the clustering algorithm and the point filling algorithm and the RGB image in the comparison image is calculated. The specific calculation method is the same as the existing method and will not be repeated here.
[0095] In one embodiment disclosed in the present invention, the following processing is also included:
[0096] (1) Based on the preset time domain filter, the average values between the processed depth image and RGB image and the corresponding contrast image are calculated respectively by using the existing method.
[0097] (2) Use the average value to smooth the processed depth image and RGB image.
[0098] Step S107: determining whether the standard deviations of the two images are both greater than a preset threshold.
[0099] If the standard deviations of the two images are both greater than the preset threshold, step S108 is executed.
[0100] Step S108: Reprocess the two images using a preset clustering algorithm and a point filling algorithm; fuse the two reprocessed images to obtain a 3D point cloud.
[0101] The decision tree algorithm is used as the preset clustering algorithm, and the interpolation search algorithm is used as the preset point filling algorithm to reprocess the depth image and RGB image to be fused. And the two images after fusion are obtained to obtain a 3D point cloud.
[0102] If the standard deviations of the two images are not both greater than the preset threshold, step S109 is executed.
[0103] Step S109: Fusing the two images processed by the preset time domain filter to obtain a 3D point cloud.
[0104] Directly fuse the two images processed by the preset time domain filter to obtain a 3D point cloud.
[0105] The embodiment of the present invention introduces a time domain filter to compare the image to be fused with the images taken at the previous and next moments, and can provide real-time and iterative feedback on whether the currently selected clustering algorithm and point filling algorithm are suitable, and make changes when the currently selected algorithm is not suitable.
[0106] Existing image fusion methods usually only use a certain fixed algorithm, and when applied to multi-source images, they often cannot achieve a good fusion effect. However, the embodiments disclosed in the present invention can dynamically and in real time select the most suitable clustering algorithm and point filling algorithm according to the actual situation of the image, greatly optimizing the effect of multi-source image fusion.
[0107] Figure 6 It is a schematic diagram of the structure of an intelligent fusion system based on image information mechanism, which is applied to fuse depth image and RGB image. Figure 6 As shown, the system includes the following modules:
[0108] The image acquisition module 11 is configured to respectively acquire a depth image and an RGB image of the same scene captured at the same time;
[0109] An image information acquisition module 12 is configured to acquire image information of a depth image and an RGB image respectively;
[0110] The weight value calculation module 13 is configured to calculate the weight values of the depth image and the RGB image respectively according to the image information of the two images;
[0111] The algorithm selection module 14 is configured to select a clustering algorithm and a point filling algorithm based on the weight values of the depth image and the RGB image;
[0112] An image processing module 15 is configured to process the two images respectively using the clustering algorithm and the point filling algorithm;
[0113] A standard deviation acquisition module 16 is configured to respectively obtain the standard deviations of the processed depth image and the RGB image using a preset time domain filter;
[0114] The judging module 17 is configured to judge whether the standard deviations of the two images are both greater than a preset threshold.
[0115] The fusion module 18 is configured to reprocess the two images using a preset clustering algorithm and a point filling algorithm when the standard deviations of the two images are both greater than a preset threshold; and fuse the two reprocessed images to obtain a 3D point cloud;
[0116] Furthermore, when the standard deviations of the two images are not both greater than a preset threshold, the two images processed by a preset time domain filter are fused to obtain a 3D point cloud.
[0117] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present invention, but the present invention is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.
Claims
1. An intelligent fusion method based on image information mechanism, applied to fuse depth image and RGB image, characterized in that: include: Acquire the depth image and RGB image of the same scene taken at the same time respectively; Acquiring image information of the depth image and the RGB image respectively, including: calculating image information of the two images respectively according to pixel data values of the depth image and the RGB image, wherein the image information includes at least gradient value change, color complexity and three-dimensional information; According to the image information of the depth image and the RGB image, the weight values of the two images are calculated respectively, including: For each image information of the depth image and the RGB image, the weight values of the depth image and the RGB image are calculated as follows: Get the current weight values of the depth image and RGB image respectively; comparing the image information of the depth image with the image information of the RGB image; Determine an image with a larger image information value, and add a preset value to the current weight value of the image; Determine an image with a smaller image information value, and reduce a preset value on a current weight value of the image; Based on the weight values of the depth image and the RGB image, the clustering algorithm and the point filling algorithm are selected, including: Compare the current weight value of the depth image with the current weight value of the RGB image; The preset clustering algorithm and point filling algorithm corresponding to the image with a larger current weight value are used as the selected algorithms, wherein the preset clustering algorithm for the depth image is the Markov random field algorithm, and the point filling algorithm is the bilateral filter algorithm; the preset clustering algorithm for the RGB image is the superpixel algorithm, and the point filling algorithm is the linear interpolation algorithm; The two images are processed respectively using the clustering algorithm and the point filling algorithm; The preset time domain filter is used to obtain the standard deviation of the processed depth image and RGB image respectively; Determine whether the standard deviation of the two images is greater than the preset threshold. If yes, use the decision tree algorithm as the clustering algorithm and the interpolation search algorithm as the point filling algorithm to reprocess the two images; fuse the two reprocessed images to obtain a 3D point cloud; If not, the two images processed by the preset time domain filter are fused to obtain a 3D point cloud.
2. The method according to claim 1, characterized in that The method of using a preset time domain filter to obtain the standard deviation of the processed depth image and the RGB image respectively includes: Obtain depth images and RGB images of the same scene taken at adjacent times as comparison images; Based on the preset time domain filter, the standard deviation between the processed depth image and RGB image and the corresponding comparison image is calculated respectively.
3. The method according to claim 2, characterized in that The step of acquiring a depth image and an RGB image of the same scene captured at adjacent moments as comparison images includes: Obtain a depth image and an RGB image of the same scene taken at the previous moment as comparison images; or, The depth image and RGB image of the same scene taken at a preset number of time moments are obtained as comparison images.
4. The method according to claim 2, characterized in that: The method further comprises: Based on the preset time domain filter, the average values between the processed depth image and RGB image and the corresponding contrast image are calculated respectively; The processed depth image and RGB image are smoothed using the average value.
5. An intelligent fusion system based on image information mechanism, applied to fuse depth image and RGB image, characterized in that: include: An image acquisition module is used to respectively acquire a depth image and an RGB image of the same scene shot at the same time; An image information acquisition module, used to acquire image information of a depth image and an RGB image respectively, including: calculating image information of the two images respectively according to pixel data values of the depth image and the RGB image, wherein the image information includes at least gradient value change, color complexity and three-dimensional information; The weight value calculation module is used to calculate the weight values of the two images according to the image information of the depth image and the RGB image, including: For each image information of the depth image and the RGB image, the weight values of the depth image and the RGB image are calculated as follows: Get the current weight values of the depth image and RGB image respectively; comparing the image information of the depth image with the image information of the RGB image; Determine an image with a larger image information value, and add a preset value to the current weight value of the image; Determine an image with a smaller image information value, and reduce a preset value on a current weight value of the image; The algorithm selection module is used to select the clustering algorithm and point filling algorithm based on the weight values of the depth image and RGB image, including: Compare the current weight value of the depth image with the current weight value of the RGB image; The preset clustering algorithm and point filling algorithm corresponding to the image with a larger current weight value are used as the selected algorithms, wherein the preset clustering algorithm for the depth image is the Markov random field algorithm, and the point filling algorithm is the bilateral filter algorithm; the preset clustering algorithm for the RGB image is the superpixel algorithm, and the point filling algorithm is the linear interpolation algorithm; An image processing module, used to process the two images respectively using the clustering algorithm and the point filling algorithm; A standard deviation acquisition module is used to obtain the standard deviations of the processed depth image and RGB image respectively using a preset time domain filter; The judgment module is used to judge whether the standard deviations of the two images are both greater than a preset threshold. A fusion module is used to use a decision tree algorithm as a clustering algorithm and an interpolation search algorithm as a point filling algorithm to reprocess the two images when the standard deviations of the two images are both greater than a preset threshold; the two reprocessed images are fused to obtain a 3D point cloud; Furthermore, when the standard deviations of the two images are not both greater than a preset threshold, the two images processed by a preset time domain filter are fused to obtain a 3D point cloud.
Citation Information
Patent Citations
Real-time dense reconstruction method and system based on RGB-D data
CN110349249A
Data fusion method, electronic equipment and storage medium
CN111428759A
Depth super-resolution recovery system of camera
CN115147282A
