Monobinocular 3D reconstruction method, system and equipment for lunar surface flag image
Through the 3D reconstruction method of monodibular images of moon flag images, stereo matching and depth estimation are used for binocular or monodomain cameras, combined with VR device parameters, dynamic 3D reconstruction of flags and panoramic image generation are realized, solving the lack of existing technology The issues of dynamic capture and 3D visual effects provide an immersive moon-surface experience.
Patent Information
- Application Number
- CN202411144075.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-11-15
AI Technical Summary
The existing lunar exploration mission can only upload the static image data of the flag. It lacks technological innovation and visual effects, and it cannot achieve accurate capture and 3D reconstruction of the dynamic form of the flag.
The 3D reconstruction method of the lunar flag image is used to perform stereo matching and depth estimation through the binocular camera, or the keyframe extraction and depth estimation of the monocular camera, and three-dimensional reconstruction and spherical projection are combined with VR device parameters to generate a panoramic image.
It realizes the accurate 3D reconstruction of the dynamic form of the banner and has good visual effects, providing users with a first-person realistic experience, and providing immersive lunar experience scenes for popular science displays.
Smart Images

Figure CN118967938A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of aerospace engineering technology, and in particular to a method, system and device for monocular and binocular 3D reconstruction of lunar flag images. Background Art
[0002] The ground demonstration system is a key component of aerospace engineering missions. It can not only be used to simulate and reproduce on-orbit missions, but also receive data transmitted from the satellite for post-production. However, most of the current lunar exploration missions at home and abroad only transmit image data of the flag, which can only be displayed statically, lacking technical innovation and visual effects. Summary of the invention
[0003] In order to solve the deficiencies mentioned in the above background technology, the purpose of the present invention is to provide a method, system and equipment for monocular and binocular 3D reconstruction of lunar flag images, which can accurately capture and reconstruct the dynamic shape of the flag through advanced stereo vision technology and image processing technology to achieve good 3D visual effects.
[0004] In a first aspect, the purpose of the present invention can be achieved by the following technical solution: a method for monocular and binocular 3D reconstruction of lunar flag images, the method comprising the following steps:
[0005] receiving lunar flag dynamic display video data, determining a camera used for the lunar flag dynamic display video data, and processing the lunar flag dynamic display video data according to the determination result, wherein the camera used includes a binocular camera and a monocular camera;
[0006] If it is a binocular camera, stereo matching and binocular depth estimation are performed on the video data of the dynamic display of the lunar flag to obtain a binocular depth map, and the binocular depth map is 3D reconstructed to obtain 3D point cloud data; if it is a monocular camera, key frame extraction and monocular depth estimation are performed on the video data of the dynamic display of the lunar flag to obtain a monocular depth map, and the monocular depth map is 3D reconstructed to obtain 3D point cloud data;
[0007] The VR device parameters are obtained, the three-dimensional point cloud data is combined with the VR device parameters, and the three-dimensional scene is rendered to obtain a processed image, and the processed image is spherically projected to obtain the final panoramic image.
[0008] In combination with the first aspect, in some implementations of the first aspect, the method further includes: under the binocular camera, preprocessing the lunar flag dynamic display video data by selecting key frames to obtain a set of binocular image sequences, wherein the left image is represented by I left , the right figure indicates I right .
[0009] In combination with the first aspect, in some implementations of the first aspect, the method further includes: a process of stereo matching the lunar flag dynamic display video data:
[0010] For each pixel I in the left image left (x, y), calculate the matching cost with the corresponding point in the right image, use Census transform matching, define a rectangular window W in the image area, use W to traverse the entire image, select the central pixel (x, y) as the reference pixel, compare the gray value of each pixel in the rectangular window with the gray value of the reference pixel, and mark the pixel with a gray value less than or equal to the reference value as 0, and the pixel with a gray value greater than the reference value as 1. Then connect them bit by bit to get the transformed result. The transformed result is a binary code stream composed of 0 and 1, and the formula is expressed as:
[0011] Where f is a comparison function defined as:
[0012]
[0013] here Represents a bitwise XOR operation, W is a rectangular window centered at (x, y), (i, j) is the offset of the window W relative to the center (x, y), and each pixel in the window is represented by I(x+i, y+i);
[0014] Apply Census transformation to each pixel of the left and right images respectively to obtain two binary code streams and Where d represents the disparity. For each pixel (x, y) in the left image, all disparity values d will be considered in the right image. The value range of d is [0, width], and width represents the width of the image.
[0015] Get the two Census transformation results in the left image and the right image, and then use the Hamming distance to calculate the cost C between the left and right images. Census , expressed as:
[0016]
[0017] Among them, H represents the Hamming distance, and the Census distance is part of the matching cost;
[0018] Calculate the cost C based on the intensity difference AD , expressed as:
[0019] C AD =|I left (x,y)-I right (xd,y)|,
[0020] The total cost is expressed as:
[0021] C(x,y,d)=ρ(C Census (x,y,d),λ census )+ρ(C AD (x,y,d),λ AD ),
[0022] in , C is the cost value, λ is the control term;
[0023] The matching cost of each pixel point (x, y) and disparity d is aggregated, and the cost aggregation P(x, y, d) adopts the average or weighted aggregation of the cost within the pixel neighborhood:
[0024] P(x,y,d)=aggregate(x+m,y+n,d), m, n represent the position offset within the neighborhood;
[0025] Using the integrated matching cost, the optimal disparity map D(x,y) is calculated by the stereo matching algorithm SGM, which is expressed as:
[0026] D(x,y)=argmin d {C(x,y,d)+μ·S(x,y,d)}, S(x,y,d) is the parallax smoothing term, and μ is a parameter that controls the degree of smoothing.
[0027] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the binocular depth estimation process:
[0028] The depth Z(x,y) of each pixel is calculated using the optimal disparity map obtained and the geometric parameters of the camera obtained by pre-calibration.
[0029] The formula is as follows:
[0030] b represents the camera baseline, f represents the focal length, and D(x,y) is the optimal disparity map.
[0031] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the three-dimensional reconstruction process is as follows:
[0032] After obtaining the depth map, the depth map is converted into three-dimensional point cloud data in combination with the camera parameters. The formula is expressed as:
[0033]
[0034] (X t ,Y t ,Z t ) represents three-dimensional coordinates.
[0035] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the monocular depth estimation process is as follows:
[0036] The monocular depth estimation network is an encoder-decoder structure, where the encoder consists of four downsampling layers and the decoder consists of four upsampling layers;
[0037] Input image I, first pass through the preprocessing layer to get I'=conv1(I), conv represents the ordinary convolution operation, expands the three channels of the image into multiple channels, and then downsamples through average pooling and the basic network. There are four identical downsampling layers. The basic network consists of two convolutional layers, each of which is followed by batch normalization and relu activation functions. The first convolutional layer is composed of three 1×1 convolutions in parallel, and the input comes from the output of the previous layer. The second convolutional layer is composed of two 5×5, one 3×3, and one 1×1 convolutions in parallel, where the input of the 1×1 convolution comes from the output of the previous network layer, and the input of the remaining convolutions comes from the output of the previous convolution. The outputs of these four convolutions are connected to obtain the output feature map of the basic network;
[0038] The output of the last downsampling stage is used as the input of the upsampling stage. There are four identical upsampling stages in total. Each stage consists of a basic network and an upsampling function. The upsampling function is bilinear interpolation. The output of the last downsampling stage is predicted by the prediction layer, and the output is a monocular depth map.
[0039] In combination with the first aspect, in some implementations of the first aspect, the method further includes: an upsampling stage and a downsampling stage of the monocular depth estimation network correspond one to one, and the formula is as follows:
[0040] E 1 ,E 2 ,E 3 ,E 4 =Encoder(I')
[0041] Where Encoder represents the encoder structure, E 1 ,E 2 ,E 3 ,E 4 Represents feature maps of different sizes output in the encoder structure;
[0042] E 1 ,E 2 ,E 3 The three feature maps of different sizes will be obtained through the basic network respectively. 1 ,B 2 ,B 3 , the decoder structure is expressed as:
[0043] D 1 =Upsample(E 4 )+B 3 ,
[0044] D 2 = Upsample(D 1 )+B 2 ,
[0045] D 3 = Upsample(D 2 )+B 1 ,
[0046] D 4 = Upsample(D 3 )
[0047] Finally, the prediction layer predicts the output depth map Z = conv(D 4 ).
[0048] In a second aspect, in order to achieve the above-mentioned purpose, the present invention discloses a monocular and binocular 3D reconstruction system for a lunar flag image, comprising:
[0049] An image determination module is used to receive the lunar flag dynamic display video data, determine the camera used for the lunar flag dynamic display video data, and process the lunar flag dynamic display video data according to the determination result, wherein the camera used includes a binocular camera and a monocular camera;
[0050] The 3D reconstruction module is used to perform stereo matching and binocular depth estimation on the video data of the lunar flag dynamic display to obtain a binocular depth map if the camera is a binocular camera, and perform 3D reconstruction on the binocular depth map to obtain 3D point cloud data; if the camera is a monocular camera, perform key frame extraction and monocular depth estimation on the video data of the lunar flag dynamic display to obtain a monocular depth map, and perform 3D reconstruction on the monocular depth map to obtain 3D point cloud data;
[0051] The rendering and projection module is used to obtain VR device parameters, combine the three-dimensional point cloud data with the VR device parameters, and perform three-dimensional scene rendering to obtain a processed image, and spherically project the processed image to obtain the final panoramic image.
[0052] In conjunction with the second aspect, in some implementations of the second aspect, the system further includes: under the binocular camera in the image determination module, the moon flag dynamic display video data is preprocessed by selecting key frames to obtain a set of binocular image sequences, wherein the left image is represented as I left , the right figure indicates I right ;
[0053] The process of stereo matching of the lunar flag dynamic display video data in the 3D reconstruction module:
[0054] For each pixel I in the left image left (x, y), calculate the matching cost with the corresponding point in the right image, use Census transform matching, define a rectangular window W in the image area, use W to traverse the entire image, select the central pixel (x, y) as the reference pixel, compare the gray value of each pixel in the rectangular window with the gray value of the reference pixel, and mark the pixel with a gray value less than or equal to the reference value as 0, and the pixel with a gray value greater than the reference value as 1. Then connect them bit by bit to get the transformed result. The transformed result is a binary code stream composed of 0 and 1, and the formula is expressed as:
[0055] Where f is a comparison function defined as:
[0056]
[0057] here Represents a bitwise XOR operation, W is a rectangular window centered at (x, y), (i, j) is the offset of the window W relative to the center (x, y), and each pixel in the window is represented by I(x+i, y+i);
[0058] Apply Census transformation to each pixel of the left and right images respectively to obtain two binary code streams and Where d represents disparity. For each pixel (x, y) in the left image, all disparity values d will be considered in the right image. The value range of d is [0, width], and width represents the width of the image.
[0059] Get the two Census transformation results in the left image and the right image, and then use the Hamming distance to calculate the cost C between the left and right images. Census , expressed as:
[0060]
[0061] Among them, H represents the Hamming distance, and the Census distance is part of the matching cost;
[0062] Calculate the cost C based on the intensity difference AD , expressed as:
[0063] C AD =|I left (x,y)-I right (xd,y)|,
[0064] The total cost is expressed as:
[0065] C(x,y,d)=ρ(CCensus (x,y,d),λ census )+ρ(C AD (x,y,d),λ AD ),
[0066] in C is the cost value, λ is the control term;
[0067] The matching cost of each pixel point (x, y) and disparity d is aggregated, and the cost aggregation P(x, y, d) adopts the average or weighted aggregation of the cost within the pixel neighborhood:
[0068] P(x,y,d)=aggregate(x+m,y+n,d), m, n represent the position offset within the neighborhood;
[0069] Using the integrated matching cost, the optimal disparity map D(x,y) is calculated by the stereo matching algorithm SGM, which is expressed as:
[0070] D(x,y)=argmin d {C(x,y,d)+μ·S(x,y,d)}, S(x,y,d) is the parallax smoothing term, and μ is a parameter that controls the degree of smoothing;
[0071] Stereo depth estimation process in the 3D reconstruction module:
[0072] The depth Z(x,y) of each pixel is calculated using the optimal disparity map obtained and the geometric parameters of the camera obtained by pre-calibration.
[0073] The formula is as follows:
[0074] b represents the camera baseline, f represents the focal length, and D(x,y) is the optimal disparity map;
[0075] The process of 3D reconstruction in the 3D reconstruction module is as follows:
[0076] After obtaining the depth map, the depth map is converted into three-dimensional point cloud data in combination with the camera parameters. The formula is expressed as:
[0077]
[0078] (X t ,Y t ,Z t ) represents three-dimensional coordinates;
[0079] The monocular depth estimation process is as follows:
[0080] The monocular depth estimation network is an encoder-decoder structure, where the encoder consists of four downsampling layers and the decoder consists of four upsampling layers;
[0081] Input image I, first pass through the preprocessing layer to get I'=conv1(I), conv represents the ordinary convolution operation, expands the three channels of the image into multiple channels, and then downsamples through average pooling and the basic network. There are four identical downsampling layers. The basic network consists of two convolutional layers, each of which is followed by batch normalization and relu activation functions. The first convolutional layer is composed of three 1×1 convolutions in parallel, and the input comes from the output of the previous layer. The second convolutional layer is composed of two 5×5, one 3×3, and one 1×1 convolutions in parallel, where the input of the 1×1 convolution comes from the output of the previous network layer, and the input of the remaining convolutions comes from the output of the previous convolution. The outputs of these four convolutions are connected to obtain the output feature map of the basic network;
[0082] The output of the last downsampling stage is used as the input of the upsampling stage. There are four identical upsampling stages. Each stage consists of a basic network and an upsampling function. The upsampling function is bilinear interpolation. The output of the last downsampling stage is predicted by the prediction layer, and the output is a monocular depth map.
[0083] The upsampling stage and downsampling stage of the monocular depth estimation network in the 3D reconstruction module correspond one to one, and the formula is as follows:
[0084] E 1 ,E 2 ,E 3 ,E 4 =Encoder(I')
[0085] Where Encoder represents the encoder structure, E 1 ,E 2 ,E 3 ,E 4 Represents feature maps of different sizes output in the encoder structure;
[0086] E 1 ,E 2 ,E 3 The three feature maps of different sizes will be obtained through the basic network respectively. 1 ,B 2 ,B 3 , the decoder structure is expressed as:
[0087] D 1 =Upsample(E 4 )+B 3 ,
[0088] D 2 = Upsample(D 1 )+B 2 ,
[0089] D 3 = Upsample(D 2 )+B 1 ,
[0090] D 4 = Upsample(D 3 )
[0091] Finally, the prediction layer predicts the output depth map Z = conv(D 4 ).
[0092] In another aspect of the present invention, in order to achieve the above-mentioned purpose, a terminal device is disclosed, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the memory stores a computer program capable of running on the processor, and when the processor loads and executes the computer program, the monocular and binocular 3D reconstruction method of a lunar flag image as described above is adopted.
[0093] Beneficial effects of the present invention:
[0094] The present invention comprehensively considers two different shooting methods of monocular cameras and binocular cameras in aerospace engineering missions, and designs an exclusive 3D reconstruction algorithm for the captured video. The algorithm is novel and highly innovative. By constructing a three-dimensional scene in the virtual world and combining it with VR equipment display, users can have a first-person realistic experience. Due to the use of spherical projection, users can view the lunar surface scene in 360 degrees in the virtual world. At the same time, combined with the lunar surface materials, sound effects, real objects, etc. already available on the ground, an immersive lunar surface experience scene is constructed for users, which has important scientific and popularization significance. BRIEF DESCRIPTION OF THE DRAWINGS
[0095] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0096] Figure 1 It is a schematic flow chart of the method of the present invention;
[0097] Figure 2 It is a schematic diagram of the basic principle of binocular stereo vision measurement of the present invention;
[0098] Figure 3 It is a schematic diagram of the monocular and binocular 3D reconstruction and demonstration process of the present invention;
[0099] Figure 4 It is a schematic diagram of a monocular depth estimation network of the present invention;
[0100] Figure 5 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION
[0101] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0102] Embodiment 1:
[0103] The following is an introduction to the relevant terms involved in the embodiments of the present application:
[0104] Monocular and binocular cameras: Monocular cameras have only one lens, which captures the scene through this lens, forming a single visual angle. On the other hand, binocular cameras have two lenses, which capture the scene from different angles, forming a three-dimensional vision. These two visual methods have their own advantages in machine vision applications.
[0105] like Figure 1 As shown, a method for monocular and binocular 3D reconstruction of lunar flag images comprises the following steps:
[0106] S101: receiving lunar flag dynamic display video data, determining a camera used for the lunar flag dynamic display video data, and processing the lunar flag dynamic display video data according to the determination result, wherein the camera used includes a binocular camera and a monocular camera;
[0107] The lunar surface binocular camera data refers to the dual-view images and videos taken by two parallel fixed cameras. After receiving the video data, the key frames of the video are selected for preprocessing to obtain a set of binocular image sequences. The left figure can be represented as I left , the right figure indicates I right .
[0108] S102: If it is a binocular camera, stereo matching and binocular depth estimation are performed on the video data of the dynamic display of the moon flag to obtain a binocular depth map, and the binocular depth map is 3D reconstructed to obtain 3D point cloud data; if it is a monocular camera, key frame extraction and monocular depth estimation are performed on the video data of the dynamic display of the moon flag to obtain a monocular depth map, and the monocular depth map is 3D reconstructed to obtain 3D point cloud data;
[0109] The process of stereo matching of the lunar flag dynamic display video data:
[0110] First, for each pixel I in the left image left(x, y), calculate the matching cost with the corresponding point in the right image. Here, Census transformation matching is used. A rectangular window W is defined in the image area, and the entire image is traversed with this rectangular window. The central pixel (x, y) is selected as the reference pixel, and the grayscale value of each pixel in the rectangular window is compared with the grayscale value of the reference pixel. Pixels with grayscale values less than or equal to the reference value are marked as 0, and pixels greater than the reference value are marked as 1. Finally, they are connected bit by bit to obtain the transformed result. The transformed result is a binary code stream composed of 0 and 1. The whole process is expressed by the formula: Where f is a comparison function defined as:
[0111]
[0112] here Represents a bitwise exclusive OR (XOR) operation, which is used to concatenate the comparison results to form a binary code stream. W is a rectangular window centered at (x, y), (i, j) is the offset of the window W relative to the center (x, y), and each pixel in the window can be represented as I(x+i, y+i).
[0113] Apply Census transformation to each pixel of the left and right images respectively to obtain two binary code streams and Where d represents disparity. For each pixel (x, y) in the left image, all possible disparity values d, that is, the number of pixels moved horizontally, will be considered in the right image. The value range of d is [0, width], and width represents the width of the image.
[0114] Get the two Census transformation results in the left image and the right image, and then use the Hamming distance to calculate the cost C between the left and right images. Census , which can be expressed as:
[0115]
[0116] Where H represents the Hamming distance, that is, the number of different corresponding bits in two binary strings. The Census distance is part of the matching cost. It is also necessary to calculate the cost C based on the intensity difference AD , which can be expressed as: C AD =|I left (x,y)-I right (xd,y)|, so the total cost can be expressed as:
[0117] C(x,y,d)=ρ(C Census (x,y,d),λ census )+ρ(C AD (x,y,d),λ AD),in
[0118] C is the cost value, λ is the control term. When C and λ are both positive, the value of the function is in the range of [0,1]. This function can normalize the two matching cost values, that is, the final result range is [0,2].
[0119] The matching costs of each pixel point (x, y) and disparity d are aggregated to reduce the impact of noise and improve the stability of matching. The cost aggregation P(x, y, d) adopts the average cost or weighted aggregation within the pixel neighborhood: P(x, y, d) = aggregate(x+m, y+n, d), where m and n represent the position offset within the neighborhood.
[0120] Using the integrated matching cost, the optimal disparity D(x,y) is calculated by the stereo matching algorithm SGM, which is expressed as: D(x,y) = argmin d {C(x,y,d)+μ·S(x,y,d)}, S(x,y,d) is the parallax smoothing term, and μ is a parameter that controls the degree of smoothing.
[0121] The binocular depth estimation process:
[0122] Use the obtained disparity map and the geometric parameters of the camera obtained in advance to estimate the depth Z(x,y) of each pixel. The formula is as follows: b represents the camera baseline (the line connecting the optical centers of two cameras is called the baseline), and f represents the focal length.
[0123] The process of 3D reconstruction is as follows:
[0124] After obtaining the depth map, the depth map data is converted into three-dimensional point cloud data in combination with the camera parameters. The formula is expressed as:
[0125]
[0126] (X t ,Y t ,Z t ) represents the three-dimensional coordinates. After obtaining the three-dimensional point cloud data, denoising, filtering and other operations are performed to improve the data quality. Then a continuous surface representation, such as a polygonal mesh or surface, is generated from the point cloud data. The color or texture information is mapped from the original image to the reconstructed three-dimensional model. At this point, the three-dimensional reconstruction of the flag image is completed.
[0127] The lunar surface monocular camera data refers to the single-view images and videos taken by a single camera. After receiving the video data, the key frames of the video are selected for preprocessing to obtain a set of image sequences. The image can be represented as I.
[0128] Monocular depth estimation process: The entire monocular depth estimation network is an encoder-decoder structure. The encoder consists of four downsampling layers and the decoder consists of four upsampling layers.
[0129] The input image I first passes through the preprocessing layer to obtain I'=conv1(I), where conv represents a normal convolution operation, which expands the three channels of the image into multiple channels, and then downsamples through average pooling and the basic network. There are four identical downsampling layers. The basic network consists of two convolutional layers, each of which is followed by batch normalization and relu activation functions. The first convolutional layer is composed of three 1×1 convolutions in parallel, and the input comes from the output of the previous layer. The second convolutional layer is composed of two 5×5, one 3×3, and one 1×1 convolutions in parallel, where the input of the 1×1 convolution comes from the output of the previous network layer, and the input of the remaining convolutions comes from the output of the previous convolution layer. The outputs of these four convolutions are connected to obtain the output feature map of the basic network.
[0130] The output of the last downsampling stage is used as the input of the upsampling stage. There are four identical upsampling stages, each of which consists of a base network and an upsampling function, where the upsampling function is bilinear interpolation. The output of the last downsampling stage is predicted by the prediction layer and the depth map is output.
[0131] The downsampling stage and upsampling stage of the monocular estimation network correspond one to one. The output of each downsampling stage will also pass through the basic network, and then the output feature map will be added to the feature map output of the corresponding upsampling stage, so that the generated depth map is smoother. The entire network is similar to a variant of the Unet network. The above process can be expressed as:
[0132] E 1 ,E 2 ,E 3 ,E 4 =Encoder(I'), where Encoder represents the encoder structure, E 1 ,E 2 ,E 3 ,E 4 Represents the feature maps of different sizes output in the encoder structure. 1 ,E 2 ,E 3 The three feature maps of different sizes will be obtained through the basic network respectively. 1 ,B 2 ,B 3 . The decoder structure can be expressed as:
[0133] D 1 =Upsample(E 4 )+B3 ,
[0134] D 2 = Upsample(D 1 )+B 2 ,
[0135] D 3 = Upsample(D 2 )+B 1 ,
[0136] D 4 = Upsample(D 3 )
[0137] Finally, the prediction layer predicts the output depth map Z = conv(D 4 ), with the image depth information, the three-dimensional reconstruction process in the above-mentioned binocular camera 3D reconstruction module can be carried out.
[0138] S103: Obtain VR device parameters, combine the three-dimensional point cloud data with the VR device parameters, and perform three-dimensional scene rendering to obtain a processed image, and perform spherical projection on the processed image to obtain a final panoramic image.
[0139] Specifically, the scheme of the present invention is further described below through an embodiment: After the basic construction of the three-dimensional scene is completed through the algorithm, preparations are made to project the scene and build an immersive experience. First, the parameters of the VR device are obtained, the world coordinate system of the scene is constructed, and the initial position parameters of the VR device are defined. Combined with the obtained device parameters, a virtual 3D immersive scene is constructed. The coordinate system of the reconstructed three-dimensional point cloud scene is transformed, and the generated scene is mapped and rendered. The image is first projected onto the spherical surface, and finally the seams of adjacent images are calculated and the fusion of the overlapping areas is completed to obtain the final panoramic image.
[0140] Embodiment 2: In the second aspect, as Figure 5 As shown, in order to achieve the above-mentioned purpose, the present invention discloses a single-eye and binocular 3D reconstruction system for lunar flag images, comprising:
[0141] The image determination module 11 is used to receive the lunar flag dynamic display video data, determine the camera used for the lunar flag dynamic display video data, and process the lunar flag dynamic display video data according to the determination result, wherein the camera used includes a binocular camera and a monocular camera;
[0142] The 3D reconstruction module 12 is used for, if it is a binocular camera, performing stereo matching and binocular depth estimation on the lunar flag dynamic display video data to obtain a binocular depth map, and performing 3D reconstruction on the binocular depth map to obtain 3D point cloud data; if it is a monocular camera, performing key frame extraction and monocular depth estimation on the lunar flag dynamic display video data to obtain a monocular depth map, and performing 3D reconstruction on the monocular depth map to obtain 3D point cloud data;
[0143] The rendering and projection module 13 is used to obtain VR device parameters, combine the three-dimensional point cloud data with the VR device parameters, and perform three-dimensional scene rendering to obtain a processed image, and spherically project the processed image to obtain a final panoramic image.
[0144] In conjunction with the second aspect, in some implementations of the second aspect, the system further includes: under the binocular camera in the image determination module 11, the dynamic display video data of the moon flag is preprocessed by selecting key frames to obtain a set of binocular image sequences, wherein the left image is represented as I left , the right figure indicates I right ;
[0145] The process of stereo matching of the lunar flag dynamic display video data in the three-dimensional reconstruction module 12:
[0146] For each pixel I in the left image left (x, y), calculate the matching cost with the corresponding point in the right image, use Census transform matching, define a rectangular window W in the image area, use W to traverse the entire image, select the central pixel (x, y) as the reference pixel, compare the gray value of each pixel in the rectangular window with the gray value of the reference pixel, and mark the pixel with a gray value less than or equal to the reference value as 0, and the pixel with a gray value greater than the reference value as 1. Then connect them bit by bit to get the transformed result. The transformed result is a binary code stream composed of 0 and 1, and the formula is expressed as:
[0147] Where f is a comparison function defined as:
[0148]
[0149] here Represents a bitwise XOR operation, W is a rectangular window centered at (x, y), (i, j) is the offset of the window W relative to the center (x, y), and each pixel in the window is represented by I(x+i, y+i);
[0150] Apply Census transformation to each pixel of the left and right images respectively to obtain two binary code streams and Where d represents the disparity. For each pixel (x, y) in the left image, all disparity values d will be considered in the right image. The value range of d is [0, width], and width represents the width of the image.
[0151] Get the two Census transformation results in the left image and the right image, and then use the Hamming distance to calculate the cost C between the left and right images. Census , expressed as:
[0152]
[0153] Among them, H represents the Hamming distance, and the Census distance is part of the matching cost;
[0154] Calculate the cost C based on the intensity difference AD , expressed as:
[0155] C AD =|I left (x,y)-I right (xd,y)|,
[0156] The total cost is expressed as:
[0157] C(x,y,d)=ρ(C Census (x,y,d),λ census )+ρ(C AD (x,y,d),λ AD ),
[0158] in C is the cost value, λ is the control term;
[0159] The matching cost of each pixel point (x, y) and disparity d is aggregated, and the cost aggregation P(x, y, d) adopts the average or weighted aggregation of the cost within the pixel neighborhood:
[0160] P(x,y,d)=aggregate(x+m,y+n,d), m, n represent the position offset within the neighborhood;
[0161] Using the integrated matching cost, the optimal disparity map D(x,y) is calculated by the stereo matching algorithm SGM, which is expressed as:
[0162] D(x,y)=argmin d {C(x,y,d)+μ·S(x,y,d)}, S(x,y,d) is the parallax smoothing term, and μ is a parameter that controls the degree of smoothing;
[0163] Stereo depth estimation process in 3D reconstruction module 12:
[0164] The depth Z(x,y) of each pixel is calculated using the optimal disparity map obtained and the geometric parameters of the camera obtained by pre-calibration.
[0165] The formula is as follows:
[0166] b represents the camera baseline, f represents the focal length, and D(x,y) is the optimal disparity map;
[0167] The process of 3D reconstruction in the 3D reconstruction module 12 is as follows:
[0168] After obtaining the depth map, the depth map is converted into three-dimensional point cloud data in combination with the camera parameters. The formula is expressed as:
[0169]
[0170] (X t ,Y t ,Z t ) represents three-dimensional coordinates;
[0171] The monocular depth estimation process is as follows:
[0172] The monocular depth estimation network is an encoder-decoder structure, where the encoder consists of four downsampling layers and the decoder consists of four upsampling layers;
[0173] Input image I, first pass through the preprocessing layer to get I'=conv1(I), conv represents the ordinary convolution operation, expands the three channels of the image into multiple channels, and then downsamples through average pooling and the basic network. There are four identical downsampling layers. The basic network consists of two convolutional layers, each of which is followed by batch normalization and relu activation functions. The first convolutional layer is composed of three 1×1 convolutions in parallel, and the input comes from the output of the previous layer. The second convolutional layer is composed of two 5×5, one 3×3, and one 1×1 convolutions in parallel, where the input of the 1×1 convolution comes from the output of the previous network layer, and the input of the remaining convolutions comes from the output of the previous convolution. The outputs of these four convolutions are connected to obtain the output feature map of the basic network;
[0174] The output of the last downsampling stage is used as the input of the upsampling stage. There are four identical upsampling stages. Each stage consists of a basic network and an upsampling function. The upsampling function is bilinear interpolation. The output of the last downsampling stage is predicted by the prediction layer, and the output is a monocular depth map.
[0175] The upsampling stage and downsampling stage of the monocular depth estimation network in the 3D reconstruction module 12 correspond one to one, and the formula is as follows:
[0176] E 1 ,E2 ,E 3 ,E 4 =Encoder(I')
[0177] Where Encoder represents the encoder structure, E 1 ,E 2 ,E 3 ,E 4 Represents feature maps of different sizes output in the encoder structure;
[0178] E 1 ,E 2 ,E 3 The three feature maps of different sizes will be obtained through the basic network respectively. 1 ,B 2 ,B 3 , the decoder structure is expressed as:
[0179] D 1 =Upsample(E 4 )+B 3 ,
[0180] D 2 = Upsample(D 1 )+B 2 ,
[0181] D 3 = Upsample(D 2 )+B 1 ,
[0182] D 4 = Upsample(D 3 )
[0183] Finally, the prediction layer predicts the output depth map Z = conv(D 4 ).
[0184] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors, and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.
[0185] It needs to be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium, on which a computer program is stored, and the computer program is executed by a processor to execute the above method. The storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.
[0186] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.
[0187] The above shows and describes the basic principles, main features and advantages of the present disclosure. Those skilled in the art should understand that the present disclosure is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, the present disclosure may have various changes and improvements, and these changes and improvements fall within the scope of the present disclosure to be protected.
Claims
1. A method for reconstructing lunar flag images by single or binocular 3D imaging, characterized in that: The method comprises the following steps: receiving lunar flag dynamic display video data, determining a camera used for the lunar flag dynamic display video data, and processing the lunar flag dynamic display video data according to the determination result, wherein the camera used includes a binocular camera and a monocular camera; If it is a binocular camera, stereo matching and binocular depth estimation are performed on the video data of the dynamic display of the lunar flag to obtain a binocular depth map, and the binocular depth map is 3D reconstructed to obtain 3D point cloud data; if it is a monocular camera, key frame extraction and monocular depth estimation are performed on the video data of the dynamic display of the lunar flag to obtain a monocular depth map, and the monocular depth map is 3D reconstructed to obtain 3D point cloud data; The VR device parameters are obtained, the three-dimensional point cloud data is combined with the VR device parameters, and the three-dimensional scene is rendered to obtain a processed image, and the processed image is spherically projected to obtain the final panoramic image.
2. The method for monocular and binocular 3D reconstruction of lunar flag images according to claim 1, characterized in that: Under the binocular camera, the video data of the dynamic display of the moon flag is preprocessed by selecting key frames to obtain a set of binocular image sequences, where the left image is represented by I left , the right figure indicates I right .
3. The method for monocular and binocular 3D reconstruction of lunar flag images according to claim 1, characterized in that: The process of stereo matching the video data of the dynamic display of the moon flag: For each pixel I in the left image left (x, y), calculate the matching cost with the corresponding point in the right image, use Census transform matching, define a rectangular window W in the image area, use W to traverse the entire image, select the central pixel (x, y) as the reference pixel, compare the gray value of each pixel in the rectangular window with the gray value of the reference pixel, and mark the pixel with a gray value less than or equal to the reference value as 0, and the pixel with a gray value greater than the reference value as 1. Then connect them bit by bit to get the transformed result. The transformed result is a binary code stream composed of 0 and 1, and the formula is expressed as: Where f is a comparison function defined as: here Represents a bitwise exclusive OR operation, W is a rectangular window centered at (x, y), (i, j) is the offset of the window W relative to the center (x, y), and each pixel in the window is represented as I(x+i, y+i); Apply Census transformation to each pixel of the left and right images respectively to obtain two binary code streams and Where d represents the disparity. For each pixel (x, y) in the left image, all disparity values d will be considered in the right image. The value range of d is [0, width], and width represents the width of the image. Get the two Census transformation results in the left image and the right image, and then use the Hamming distance to calculate the cost C between the left and right images. Cen sus, expressed as: Among them, H represents the Hamming distance, and the Census distance is part of the matching cost; Calculate the cost C based on the intensity difference AD , expressed as: C AD =|I left (x,y)-I right (x-d,y)|, The total cost is expressed as: C(x,y,d)=ρ(C Census (x, y, d), λ census )+ρ(C AD (x, y, d), λ AD ), in C is the cost value, λ is the control term; The matching cost of each pixel point (x, y) and disparity d is aggregated, and the cost aggregation P(x, y, d) adopts the average cost or weighted aggregation within the pixel neighborhood: P(x, y, d) = aggregate(x+m, y+n, d), where m and n represent the position offset within the neighborhood; Using the integrated matching cost, the optimal disparity map D(x, y) is calculated by the stereo matching algorithm SGM, which is expressed as: D(x,y)=argmin d {C(x, y, d)+μ·S(x, y, d)}, S(x, y, d) is a parallax smoothing term, and μ is a parameter that controls the degree of smoothing.
4. The method for monocular and binocular 3D reconstruction of lunar flag images according to claim 1, characterized in that: The binocular depth estimation process: The depth Z(x, y) of each pixel is calculated using the obtained optimal disparity map and the geometric parameters of the camera obtained by pre-calibration. The formula is as follows: b represents the camera baseline, f represents the focal length, and D(x, y) is the optimal disparity map.
5. The method for monocular and binocular 3D reconstruction of lunar flag images according to claim 1, characterized in that: The process of the three-dimensional reconstruction is as follows: After obtaining the depth map, the depth map is converted into three-dimensional point cloud data in combination with the camera parameters. The formula is expressed as: (X t , Yt, Z t ) represents three-dimensional coordinates.
6. The method for monocular and binocular 3D reconstruction of lunar flag images according to claim 1, characterized in that: The monocular depth estimation process is as follows: The monocular depth estimation network is an encoder-decoder structure, where the encoder consists of four downsampling layers and the decoder consists of four upsampling layers; Input image I, first pass through the preprocessing layer to get I′=convl(I), conv represents the ordinary convolution operation, expands the three channels of the image into multiple channels, and then downsamples through average pooling and the basic network. There are four identical downsampling layers. The basic network consists of two convolutional layers, each of which is followed by batch normalization and relu activation function. The first convolutional layer is composed of three 1×1 convolutions in parallel, and the input comes from the output of the previous layer. The second convolutional layer is composed of two 5×5, one 3×3, and one 1×1 convolutions in parallel, where the input of the 1×1 convolution comes from the output of the previous layer of the network, and the input of the remaining convolutions comes from the output of the previous convolution. The outputs of these four convolutions are connected to obtain the output feature map of the basic network; The output of the last downsampling stage is used as the input of the upsampling stage. There are four identical upsampling stages in total. Each stage consists of a basic network and an upsampling function. The upsampling function is bilinear interpolation. The output of the last downsampling stage is predicted by the prediction layer, and the output is a monocular depth map.
7. The method for monocular and binocular 3D reconstruction of lunar flag images according to claim 6, characterized in that: The upsampling stage and downsampling stage of the monocular depth estimation network correspond one to one, and the formula is as follows: E1, E2, E3, E4 = Encoder (I') Where Encoder represents the encoder structure, E1, E2, E3, and E4 represent feature maps of different sizes output in the encoder structure; The three feature maps of different sizes E1, E2, and E3 will be obtained through the basic network respectively B1, B2, and B3. The decoder structure is expressed as: D1=Upsample(E4)+B3, D2=Upsample(D1)+B2, D3=Upsample(D2)+B1, D4=Upsample(D3) Finally, the prediction layer predicts the output depth map Z=conv(D4).
8. A single-eye and binocular 3D reconstruction system for lunar flag images, characterized in that: include: An image determination module is used to receive the lunar flag dynamic display video data, determine the camera used for the lunar flag dynamic display video data, and process the lunar flag dynamic display video data according to the determination result, wherein the camera used includes a binocular camera and a monocular camera; The 3D reconstruction module is used to perform stereo matching and binocular depth estimation on the video data of the lunar flag dynamic display to obtain a binocular depth map if the camera is a binocular camera, and perform 3D reconstruction on the binocular depth map to obtain 3D point cloud data; if the camera is a monocular camera, perform key frame extraction and monocular depth estimation on the video data of the lunar flag dynamic display to obtain a monocular depth map, and perform 3D reconstruction on the monocular depth map to obtain 3D point cloud data; The rendering and projection module is used to obtain VR device parameters, combine the three-dimensional point cloud data with the VR device parameters, and perform three-dimensional scene rendering to obtain a processed image, and spherically project the processed image to obtain the final panoramic image.
9. The system for reconstructing moon flag images by single or binocular 3D imaging according to claim 8, characterized in that: Under the binocular camera in the image determination module, the dynamic display video data of the moon flag is preprocessed by selecting key frames to obtain a set of binocular image sequences, where the left image is represented by I left , the right figure indicates I right ; The process of stereo matching of the lunar flag dynamic display video data in the 3D reconstruction module: For each pixel I in the left image left (x, y), calculate the matching cost with the corresponding point in the right image, use Census transform matching, define a rectangular window W in the image area, use W to traverse the entire image, select the central pixel (x, y) as the reference pixel, compare the gray value of each pixel in the rectangular window with the gray value of the reference pixel, and mark the pixels with gray values less than or equal to the reference value as 0, and the pixels with gray values greater than the reference value as 1. Then connect them bit by bit to get the transformed result. The transformed result is a binary code stream composed of 0 and 1, and the formula is expressed as: Where f is a comparison function defined as: here Represents a bitwise exclusive OR operation, W is a rectangular window centered at (x, y), (i, j) is the offset of the window W relative to the center (x, y), and each pixel in the window is represented as I(x+i, y+i); Apply Census transformation to each pixel of the left and right images respectively to obtain two binary code streams and Where d represents the disparity. For each pixel (x, y) in the left image, all disparity values d will be considered in the right image. The value range of d is [0, width], and width represents the width of the image. Get the two Census transformation results in the left image and the right image, and then use the Hamming distance to calculate the cost C between the left and right images. Census , expressed as: Among them, H represents the Hamming distance, and the Census distance is part of the matching cost; Calculate the cost C based on the intensity difference AD , expressed as: C AD =|I left (x,y)-I right (x-d,y)|, The total cost is expressed as: C(x,y,d)=ρ(C Census (x, y, d), λ census )+ρ(C AD (x, y, d), λ AD ), in C is the cost value, λ is the control term; The matching cost of each pixel point (x, y) and disparity d is aggregated, and the cost aggregation P(x, y, d) adopts the average cost or weighted aggregation within the pixel neighborhood: P(x, y, d) = aggregate(x+m, y+n, d), where m and n represent the position offset within the neighborhood; Using the integrated matching cost, the optimal disparity map D(x, y) is calculated by the stereo matching algorithm SGM, which is expressed as: D(x,y)=argmin d {C(x, y, d) + μ·S(x, y, d)}, S(x, y, d) is the parallax smoothing term, and μ is a parameter that controls the degree of smoothing; Stereo depth estimation process in the 3D reconstruction module: The depth Z(x,y) of each pixel is calculated using the optimal disparity map obtained and the geometric parameters of the camera obtained by pre-calibration. The formula is as follows: b represents the camera baseline, f represents the focal length, and D(x,y) is the optimal disparity map; The process of 3D reconstruction in the 3D reconstruction module is as follows: After obtaining the depth map, the depth map is converted into three-dimensional point cloud data in combination with the camera parameters. The formula is expressed as: (X t ,Y t ,Z t ) represents three-dimensional coordinates; The monocular depth estimation process is as follows: The monocular depth estimation network is an encoder-decoder structure, where the encoder consists of four downsampling layers and the decoder consists of four upsampling layers; Input image I, first pass through the preprocessing layer to get I'=conv1(I), conv represents the ordinary convolution operation, expands the three channels of the image into multiple channels, and then downsamples through average pooling and the basic network. There are four identical downsampling layers. The basic network consists of two convolutional layers, each of which is followed by batch normalization and relu activation functions. The first convolutional layer is composed of three 1×1 convolutions in parallel, and the input comes from the output of the previous layer. The second convolutional layer is composed of two 5×5, one 3×3, and one 1×1 convolutions in parallel, where the input of the 1×1 convolution comes from the output of the previous network layer, and the input of the remaining convolutions comes from the output of the previous convolution. The outputs of these four convolutions are connected to obtain the output feature map of the basic network; The output of the last downsampling stage is used as the input of the upsampling stage. There are four identical upsampling stages. Each stage consists of a basic network and an upsampling function. The upsampling function is bilinear interpolation. The output of the last downsampling stage is predicted by the prediction layer, and the output is a monocular depth map. The upsampling stage and downsampling stage of the monocular depth estimation network in the 3D reconstruction module correspond one to one, and the formula is as follows: E1, E2, E3, E4 = Encoder (I') Where Encoder represents the encoder structure, E1, E2, E3, and E4 represent feature maps of different sizes output in the encoder structure; The three feature maps of different sizes E1, E2, and E3 will be obtained through the basic network respectively B1, B2, and B3. The decoder structure is expressed as: D1=Upsample(E4)+B3, D2=Upsample(D1)+B2, D3=Upsample(D2)+B1, D4=Upsample(D3) Finally, the prediction layer predicts the output depth map Z=conv(D4).
10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: The memory stores a computer program that can be run on the processor. When the processor loads and executes the computer program, a method for monocular and binocular 3D reconstruction of a lunar flag image according to any one of claims 1 to 7 is adopted.
