RGBD image fusion method and storage medium applied to mobile carrier

Abstract

The present invention relates to an RGBD image fusion method and storage medium applied to a mobile carrier. The fusion method comprises: obtaining a process image model by training a multi-layer perceptron; acquiring position data, RGB images and depth images in real time; updating a training set to train the process image model; determining a current depth image, a current RGB image and a matching period, extracting position data within the matching period to form a sequence X. 12 ; Sequence X 12 Input process image model to get sequence F 12 ; From sequence F 12 The RGB image that best matches the current depth image is found and fused with the current depth image to obtain an RGBD image. In the present invention, more intermediate RGB images can be generated between two adjacent RGB image frames based on the movement path of the mobile carrier, thereby obtaining an RGB image that better matches the depth image and substantially eliminating the displacement that occurs when the RGB image and the depth image are superimposed.

Description

Technical Field

[0001] The present invention belongs to the technical field of RGBD image fusion, and relates to an RGBD image fusion method and a storage medium applied to a mobile carrier. Background Art

[0002] RGBD images are obtained by aligning the RGB three-channel color image (hereinafter referred to as RGB image) generated by a regular RGB camera and the depth image (i.e., depth image) generated by a depth camera, so that the pixels of the RGB image correspond one-to-one with the pixels of the depth image, and then fusing them. However, there will be errors when reading images between the RGB camera and the depth camera. If the FPS (i.e., the number of frames transmitted per second) of the two cameras is different, the RGB image and the depth image may not be able to find the same moment in the picture, and the RGBD image will be superimposed and shifted. The mobile carrier may vibrate during the shooting process (for example, drones often vibrate when hovering). At this time, the RGB image and the depth image will also be offset. In the point cloud (point cloud data) picture, the RGB image and the depth image will usually be shifted after superimposing, affecting the clarity of the fused RGBD image. Summary of the Invention

[0003] In view of the above-mentioned deficiencies in the prior art, the technical problem to be solved by the present invention is to provide an RGBD image fusion method and storage medium for mobile carriers that can greatly reduce the displacement after the RGB image and the depth image are superimposed.

[0004] In order to achieve the above object, the present invention provides the following technical solutions:

[0005] A method for RGBD image fusion applied to a mobile carrier, wherein the mobile carrier is provided with an inertial measurement unit, an RGB camera, and a depth camera; comprising the following steps:

[0006] S1. Training a multilayer perceptron using position data and RGB image information to obtain a process image model; the process image model is configured to output an RGB image corresponding to each position data based on a change in each position data relative to the previous position data and the RGB image corresponding to the previous position data;

[0007] S2, acquiring position data of the inertial measurement unit and an RGB image generated by the RGB camera and a depth image generated by the depth camera in real time during the movement of the mobile carrier;

[0008] S3, updating the training set based on the real-time acquired position data and RGB images, and training the process image model;

[0009] S4. When a depth image is generated, the depth image is used as the current depth image, and the most recently generated RGB image is used as the current RGB image; a period that matches the generation time of the current RGB image is used as the matching period, and the position data within the matching period is extracted to form a sequence X. 12 ;

[0010] S5, sequence X 12 Input process image model, the RGB image output by the process image model forms a sequence F 12 ;

[0011] S6, from sequence F 12 Find the RGB image that has the highest matching degree with the current depth image;

[0012] S7. Fuse the current depth image with the RGB image with the highest matching degree to obtain an RGBD image.

[0013] Furthermore, the step S1 includes the following steps:

[0014] S11, acquiring position data generated by an inertial measurement unit and an RGB image generated by shooting during the movement of the mobile carrier;

[0015] S12, forming a position data sequence with each set of position data as an element; forming an RGB image sequence with each frame of RGB image as an element;

[0016] S13, performing time synchronization on the position data sequence and the RGB image sequence;

[0017] S14. Input the position data sequence and the RGB image sequence into a multilayer perceptron, and obtain a process image model through training.

[0018] Furthermore, the step S2 includes the following steps:

[0019] S21, acquiring position data of the inertial measurement unit in real time during the movement of the mobile carrier, and forming a sequence X with a set of position data as an element;

[0020] S22, acquiring RGB images captured in real time during the movement of the mobile carrier, and forming a sequence F with one frame of RGB images as an element;

[0021] S23, acquiring depth images generated by shooting in real time during the movement of the mobile carrier, and forming a sequence D with each depth image as an element;

[0022] S24: Time synchronization is performed on sequence X, sequence F, and sequence D.

[0023] Furthermore, the maximum number of elements of sequence X, sequence F, and sequence D is predetermined;

[0024] When executing step S2, when the number of elements in sequence X, sequence F or sequence D reaches the maximum number of elements in the sequence, each time an element is added to the sequence, the earliest element in the sequence is removed first.

[0025] Furthermore, in the step S3, the start time of the matching period is the generation time of the current RGB image, and the end time of the matching period is the generation time of the next frame of RGB image.

[0026] Furthermore, in the step S3, the start time of the matching period is a time of a predetermined length before the current RGB image is generated, and the end time of the matching period is a time of a predetermined length after the current RGB image is generated.

[0027] Furthermore, the mobile carrier is a drone, and the position data of the inertial measurement unit includes the navigation angle of the drone and the horizontal coordinate and vertical coordinate of the drone projected on the plane.

[0028] Furthermore, the step S6 includes the following steps:

[0029] S601, projecting the current depth image onto a two-dimensional plane by scanning to obtain a two-dimensional image;

[0030] S602, two-dimensional image and sequence F 12 The RGB images in are normalized and edge detected in turn;

[0031] S603, find sequence F 12 The RGB image with the smallest edge error with the two-dimensional image is taken as the RGB image with the highest matching degree with the current depth image.

[0032] Furthermore, in steps S601 and S602, the Sobel operator is used to perform edge detection.

[0033] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements an RGBD image fusion method applied to a mobile carrier.

[0034] In the present invention, a deep learning method is adopted to generate RGB images based on the position data of the inertial measurement unit. Since the rate at which the inertial measurement unit generates position data is much higher than the rate at which the RGB camera and the depth camera generate images, more intermediate RGB images can be generated between two adjacent frames of RGB images generated by the RGB camera according to the movement route of the mobile carrier, thereby obtaining an RGB image that better matches the depth image and basically eliminating the displacement that occurs after the RGB image and the depth image are superimposed. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0036] Figure 1 The figure is a flow chart of an embodiment of the RGBD image fusion method applied to a mobile carrier according to the present invention.

[0037] Figure 2 Flowchart of the process image model obtained for training.

[0038] Figure 3 Flowchart for finding the RGB image that best matches the depth image. DETAILED DESCRIPTION

[0039] The following describes the implementation of the present invention through specific examples. The illustrations provided in the following embodiments are only used to schematically illustrate the basic concept of the present invention. The following embodiments and features in the embodiments may be combined with each other unless there is any conflict.

[0040] The present invention discloses an RGBD image fusion method applied to a mobile carrier; the mobile carrier is provided with an inertial measurement unit, an RGB camera and a depth camera; in this embodiment, the mobile carrier is a drone, of course, the mobile carrier can also be other movable devices. Figure 1 A preferred embodiment of the RGBD image fusion method for a mobile carrier includes the following steps:

[0041] S1. A process image model is obtained by training a multilayer perceptron using position data and RGB image information. The process image model is capable of outputting the RGB image corresponding to each position data based on the change of each position data relative to the previous position data and the RGB image corresponding to the previous position data. Figure 2 , this step may include the following sub-steps:

[0042] S11. Acquire position data (x, y, theta) measured by the inertial measurement unit and the captured RGB image during the movement of the mobile carrier. Here, x and y represent the horizontal and vertical coordinates of the drone projected on the plane, respectively, and theta represents the navigation angle of the drone.

[0043] S12. Use each set of position data (including the time when the position data was generated) as an element to form a position data sequence X'; use each frame of RGB image (including the time when the RGB image was generated) as an element to form an RGB image sequence F'. Use the position data sequence X' and the RGB image sequence F' as the training set.

[0044] S13 , performing time synchronization on the position data sequence X′ and the RGB image sequence F′, so that the time information of the RGB images in the RGB image sequence F′ and the time information of the position data in the position data sequence X′ are synchronized.

[0045] S14. NeRF (Neural Radiance Fields, 3D reconstruction technology) is used to input the position data sequence and the RGB image sequence into a multilayer perceptron, and a process image model is obtained through training.

[0046] S2. Acquire the position data of the inertial measurement unit, the RGB image generated by the RGB camera, and the depth image generated by the depth camera in real time during the movement of the drone. This step may include the following sub-steps:

[0047] S21. Acquire the position data of the inertial measurement unit in real time during the movement of the UAV, and form a position data sequence X with a set of position data (including the time when the position data is generated) as an element.

[0048] S22. Acquire the RGB images generated by shooting in real time during the movement of the UAV, and form an RGB image sequence F with one frame of RGB image (including the time when the RGB image is generated) as an element.

[0049] S23 . Acquire depth images generated by shooting in real time during the movement of the drone, and form a depth image sequence D using one depth image (including the time when the depth image is generated) as an element.

[0050] S24 , time-synchronize the position data sequence X, the RGB image sequence F, and the depth image sequence D, so as to extract the position data generated in the corresponding period in sequence X according to the generation time of the RGB image in sequence F or the depth image in sequence D.

[0051] S3. Update sequences X, F, and D based on the real-time acquired position data, RGB images, and depth images. After the updates, use sequences X and F as training sets to train the process image model. Training is typically performed once each time sequence F is updated. After the updates, the parameters of the process image model can be fine-tuned based on the most recently generated RGB image, adapting the process image model to the changing patterns of RGB images. To prevent excessive training time due to an excessive number of elements in sequences X and F, which could affect the coherence of the generated RGBD images, a maximum number of elements in sequences X, F, and D can be pre-determined. When the number of elements in sequence X, F, or D reaches the maximum, the oldest element in the sequence is removed for each additional element. This prevents excessive elements in sequences X, F, and D.

[0052] S4. When acquiring a depth image, use that depth image as the current depth image, and the most recently acquired RGB image as the current RGB image. For example, if the depth image and RGB image are generated synchronously, the RGB image generated simultaneously with the current depth image is the current RGB image. If the FPS of the depth image and RGB image are not synchronized, the RGB image generated most recently before the current depth image is used as the current RGB image (RGB images after the current depth image have not yet been generated and are therefore not included in the comparison range).

[0053] A period that matches the generation time of the current RGB image is used as the matching period, and the position data within the matching period is extracted to form a sequence X 12 . The generation time of the current RGB image can be used as the starting time of the matching period, and the generation time of the next frame of RGB image can be used as the end time of the matching period (the matching period includes the generation time of the current RGB image, but does not include the generation time of the next frame of RGB image). For example, when the depth image and the RGB image are generated synchronously, the generation time of the depth image is the starting time of the matching period, and the matching period must include the generation time of the depth image. When the depth is the same and the FPS of the RGB camera is different, since the current RGB image is the last frame of RGB image generated before the current depth image, the generation time of the current depth image must be between the generation time of the current RGB image and the time when the next RGB image is generated, that is, the matching period must also include the generation time of the depth image.

[0054] Of course, the first predetermined time before the current RGB image is generated can also be used as the start time of the matching period, and the second predetermined time after the current RGB image is generated can be used as the end time of the matching period. For example, when the depth image and the RGB image are generated synchronously, the first predetermined time and the second predetermined time can be much smaller than the generation time interval between two adjacent frames of the RGB camera, that is, the matching period can only include a short period before and after the current RGB image is generated, thereby reducing the length of the matching period and the sequence X. 12 To reduce the amount of subsequent calculations.

[0055] S5, sequence X 12 Input process image model, the RGB image output by the process image model forms a sequence F 12 Since the sequence F 12 RGB image and sequence X 12 Therefore, the process image model can add multiple RGB images between the current RGB image and the next frame RGB image, which is equivalent to greatly improving the FPS of the RGB camera. In this way, an RGB image that better matches the current depth image can be obtained in the case of drone vibration.

[0056] S6, from sequence F 12 Find the RGB image that best matches the current depth image. Figure 3 , this step may specifically include the following sub-steps:

[0057] S601, projecting the current depth image onto a two-dimensional plane by scanning (e.g., using the z coordinate depth in the depth image) to obtain a two-dimensional image;

[0058] S602, two-dimensional image and sequence F 12 The RGB images in the image are normalized and edge detected in turn. The normalization function can be used for normalization, and the soble operator can be used for edge detection.

[0059] S603, find sequence F 12 The RGB image with the smallest edge error with the current depth image is taken as the RGB image with the highest matching degree with the current depth image. The sequence F can be translated by shifting the pixels in the row and column directions. 12 The elements in F are shifted in sequence, so that 12 The RGB images in are compared with the two-dimensional images respectively, and the pixel value error is used to calculate the edge error between the RGB image and the current depth image.

[0060] S7, compare the current depth image with the sequence F 12 The RGB images with the highest matching degree are superimposed together, and the pixels of the RGB image are made to correspond one to one with the pixels of the depth image, and then fused to obtain an RGBD image. The method of fusing the depth image with the RGB image is a prior art and will not be described in detail here. After that, steps S2 to S7 are repeated to generate continuous RGBD images and form an RGBD video image. Since the time required for the algorithm in this embodiment is extremely short, the time required to fuse and generate the RGBD image can be made less than the time interval between the depth camera generating two adjacent frames of depth images, thereby achieving quasi-real-time RGBD video image generation without affecting the continuity of the RGBD video image.

[0061] In this embodiment, a deep learning method is used to generate an RGB image based on the position data of the inertial measurement unit. Since the rate at which the inertial measurement unit generates position data is much higher than the rate at which the RGB camera and the depth camera generate images, more intermediate RGB images can be generated between two adjacent frames of RGB images generated by the RGB camera according to the movement route of the drone (including the vibration direction, vibration amplitude, and vibration frequency when vibration occurs), thereby obtaining an RGB image that better matches the depth image. This can basically eliminate the displacement that occurs after the RGB image and the depth image are superimposed due to reasons such as the different FPS of the two cameras and the vibration of the drone.

[0062] The present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a computer, part or all of the actual operations of the above-mentioned RGBD image fusion method applied to a mobile carrier are performed, thereby realizing the functions of any one of the above-mentioned embodiments.

[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A RGBD image fusion method applied to a mobile carrier, wherein the mobile carrier is provided with an inertial measurement unit, an RGB camera, and a depth camera; characterized in that: The following steps are involved: S1. Training a multilayer perceptron using position data and RGB image information to obtain a process image model; the process image model is configured to output an RGB image corresponding to each position data based on a change in each position data relative to the previous position data and the RGB image corresponding to the previous position data; S2, acquiring position data of the inertial measurement unit and an RGB image generated by the RGB camera and a depth image generated by the depth camera in real time during the movement of the mobile carrier; S3, updating the training set based on the real-time acquired position data and RGB images, and training the process image model; S4. When a depth image is generated, the depth image is used as the current depth image, and the most recently generated RGB image is used as the current RGB image; A period that matches the generation time of the current RGB image is used as the matching period, and the position data within the matching period is extracted to form a sequence X 12 ; S5, sequence X 12 Input process image model, the RGB image output by the process image model forms a sequence F 12 ; S6, from sequence F 12 Find the RGB image that has the highest matching degree with the current depth image; S7. Fuse the current depth image with the RGB image with the highest matching degree to obtain an RGBD image.

2. The RGBD image fusion method applied to a mobile carrier according to claim 1, characterized in that: The S1 step includes the following steps: S11, acquiring position data generated by an inertial measurement unit and an RGB image generated by shooting during the movement of the mobile carrier; S12, forming a position data sequence with each set of position data as an element; forming an RGB image sequence with each frame of RGB image as an element; S13, performing time synchronization on the position data sequence and the RGB image sequence; S14. Input the position data sequence and the RGB image sequence into a multilayer perceptron, and obtain a process image model through training.

3. The RGBD image fusion method applied to a mobile carrier according to claim 2, characterized in that: The S2 step includes the following steps: S21, acquiring position data of the inertial measurement unit in real time during the movement of the mobile carrier, and forming a sequence X with a set of position data as an element; S22, acquiring RGB images captured in real time during the movement of the mobile carrier, and forming a sequence F with one frame of RGB images as an element; S23, acquiring depth images generated by shooting in real time during the movement of the mobile carrier, and forming a sequence D with each depth image as an element; S24: Time synchronization is performed on sequence X, sequence F, and sequence D.

4. The RGBD image fusion method applied to a mobile carrier according to claim 3, characterized in that: The maximum number of elements of sequence X, sequence F and sequence D is determined in advance; When executing step S2, when the number of elements in sequence X, sequence F or sequence D reaches the maximum number of elements in the sequence, each time an element is added to the sequence, the earliest element in the sequence is removed first.

5. The RGBD image fusion method applied to a mobile carrier according to claim 1, characterized in that: In the step S3, the start time of the matching period is the generation time of the current RGB image, and the end time of the matching period is the generation time of the next frame of RGB image.

6. The RGBD image fusion method applied to a mobile carrier according to claim 1, characterized in that: In the step S3, the start time of the matching period is a time of a predetermined length before the current RGB image generation time, and the end time of the matching period is a time of a predetermined length after the current RGB image generation time.

7. The RGBD image fusion method applied to a mobile carrier according to claim 1, characterized in that: The mobile carrier is a drone, and the position data of the inertial measurement unit includes the navigation angle of the drone and the horizontal coordinate and vertical coordinate of the drone projected on the plane.

8. The RGBD image fusion method applied to a mobile carrier according to any one of claims 1 to 7, characterized in that: The S6 step includes the following steps: S601, projecting the current depth image onto a two-dimensional plane by scanning to obtain a two-dimensional image; S602, two-dimensional image and sequence F 12 The RGB images in are normalized and edge detected in turn; S603, find sequence F 12 The RGB image with the smallest edge error with the two-dimensional image is taken as the RGB image with the highest matching degree with the current depth image.

9. The RGBD image fusion method applied to a mobile carrier according to claim 8, characterized in that: In steps S601 and S602, the Sobel operator is used to perform edge detection.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the RGBD image fusion method applied to a mobile carrier according to any one of claims 1 to 9 is implemented.

Images