An image generation method, apparatus and electronic device

Abstract

This invention discloses an image generation method, apparatus, and electronic device. The method involves acquiring a first three-channel color RGB image and a first depth map; obtaining a first mask corresponding to the first RGB image through semantic segmentation; sampling the first depth map in a target region based on the first mask to obtain a first sparse depth map; and inputting the first RGB image and the first sparse depth map into a depth filling model to obtain a first dense depth map corresponding to a first object. The technical solution of this invention replaces depth denoising with depth completion, avoiding quality problems such as depth loss and abrupt changes caused by camera physical performance limitations in depth denoising techniques, thereby improving the quality and accuracy of the completed depth map.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to an image generation method, apparatus, and electronic device. Background Technology

[0002] In downstream applications such as autonomous driving, robotics, and augmented reality, the physical performance of depth cameras is insufficient, such as lens optical distortion, sensor resolution limitations, lighting environment interference, and hardware noise. These factors can cause distortion, jumps, and noise in depth maps.

[0003] Currently, depth denoising is commonly used to process depth maps in order to provide relatively clean input data for subsequent visual tasks. Depth denoising is a post-processing technique for depth images, and its core goal is to effectively suppress various noises and artifacts while preserving edge details, structural information, and true depth values ​​in the depth map. However, existing depth denoising techniques can only smooth existing noisy depth data and cannot compensate for fundamental quality problems such as depth loss and abrupt changes caused by camera physical limitations. Furthermore, the processed data still falls short of the requirements for scenarios with stringent error control, such as object distance recognition, object grasping, and high-precision scene reconstruction. Summary of the Invention

[0004] This invention provides an image generation method, apparatus, and electronic device to solve quality problems such as depth loss and abrupt changes caused by the physical performance limitations of cameras in depth denoising technology.

[0005] According to one aspect of the present invention, an image generation method is provided, comprising: Acquire a first three-channel color RGB image and a first depth map, wherein the first RGB image includes an RGB image obtained after photographing the first object, and the first depth map includes a depth image obtained after photographing the first object; By performing semantic segmentation on the first RGB image, a first mask corresponding to the first RGB image is obtained, and the first mask includes the mask of the location of the first object. Based on the first mask, the first depth map is sampled in the target area to obtain a first sparse depth map; The first RGB image and the first sparse depth map are input into the depth filling model to obtain the first dense depth map corresponding to the first object. The first dense depth map includes the depth map obtained after depth filling the first sparse depth map.

[0006] According to another aspect of the present invention, an image generation apparatus is provided, comprising: The acquisition module is used to acquire a first three-channel color RGB image and a first depth map. The first RGB image includes an RGB image obtained after photographing the first object, and the first depth map includes a depth image obtained after photographing the first object. The semantic segmentation module is used to obtain a first mask corresponding to the first RGB image by performing semantic segmentation on the first RGB image, wherein the first mask includes a mask of the location of the first object. The sampling module is used to sample the first depth map in the target area based on the first mask to obtain a first sparse depth map; The depth filling model input module is used to input the first RGB image and the first sparse depth map into the depth filling model to obtain the first dense depth map corresponding to the first object. The first dense depth map includes the depth map obtained after depth filling the first sparse depth map.

[0007] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the method described in any embodiment of the present invention.

[0008] The technical solution of this invention involves acquiring a first three-channel color RGB image and a first depth map. The first RGB image includes an RGB image obtained after photographing a first object, and the first depth map includes a depth image obtained after photographing the first object. Semantic segmentation is performed on the first RGB image to obtain a first mask corresponding to the first RGB image, the first mask including a mask of the location of the first object. Based on the first mask, the first depth map is sampled in the target region to obtain a first sparse depth map. The first RGB image and the first sparse depth map are input into a depth filling model to obtain a first dense depth map corresponding to the first object, the first dense depth map including a depth map obtained after depth filling the first sparse depth map. The technical solution of this invention fills the first sparse depth map with a depth filling model based on the first RGB image and the first sparse depth map, and completes the first sparse depth map with a depth completion method to obtain a first dense depth map, replacing the depth map processing method of depth denoising. Based on the texture semantic information of a single first RGB image and the effective data of the first sparse depth map, the depth is accurately completed to correct distortion and jump problems, thereby improving the quality and accuracy of the first dense depth map obtained after the first RGB image and the first sparse depth map are processed by the depth filling model.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart of an image generation method provided in Embodiment 1 of the present invention; Figure 2 This is a flowchart of an image generation method provided in an embodiment of the present invention; Figure 3 This is a flowchart of an image generation method provided in Embodiment 2 of the present invention; Figure 4 This is a flowchart of a training method for an initial model provided in an embodiment of the present invention; Figure 5 This is a flowchart of a loss function determination method provided in an embodiment of the present invention; Figure 6 This is a flowchart of a method for determining a second sparse depth map provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of an image generation device provided in Embodiment 3 of the present invention; Figure 8 A schematic diagram of an electronic device that can be used to implement embodiments of the present invention is shown. Detailed Implementation

[0012] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0013] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0014] Example 1 Figure 1 This is a flowchart of an image generation method provided in Embodiment 1 of the present invention. This embodiment is applicable to depth image processing. The method can be executed by an image generation device, which can be implemented in hardware and / or software and can be configured in an electronic device. For example, the electronic device can be a computer, workstation, or server, etc. Figure 1 As shown, the method includes: S110. Acquire a first three-channel color RGB image and a first depth map, wherein the first RGB image includes an RGB image obtained after photographing the first object, and the first depth map includes a depth image obtained after photographing the first object.

[0015] In this embodiment, the first three-channel color (Red, Green, Blue, RGB) image can be an RGB image captured by a color camera. The first RGB image can describe the geometry and surface texture of the photographed object, and each pixel in the first RGB image contains the intensity values ​​of the red, green, and blue channels. The first depth map can be a depth map captured by a depth camera. Each pixel in the first depth map can represent the straight-line distance from the corresponding point on the surface of the photographed object to the depth camera, and the unit can be millimeters or meters.

[0016] The pixel dimensions of the first RGB image and the first depth map can be the same. The first object can be the object being photographed. The first object can be the target of depth completion.

[0017] Specifically, a color camera is used to photograph the first object in the scene, recording the light reflected from the surface of the first object to obtain a first RGB image. A depth camera is then used to photograph the first object, and a first depth map is generated by calculating the straight-line distance from each point in the scene to the depth camera.

[0018] Color and depth cameras can be integrated into one camera, such as a color-depth (Red Green Blue-Depth, RGB-D) camera.

[0019] S120. By performing semantic segmentation on the first RGB image, a first mask corresponding to the first RGB image is obtained, wherein the first mask includes a mask of the location of the first object.

[0020] In this embodiment, the first mask can be a binary mask obtained by semantic segmentation of the first RGB image. The first mask may include a mask of the location of the first object, and the pixel size of the first mask may be the same as that of the first RGB image.

[0021] Specifically, the first RGB image is processed using a neural network to analyze the image's color, texture, and contextual information pixel by pixel, predicting the semantic category of each pixel to complete semantic segmentation of the first RGB image. Based on the semantic segmentation results, all pixels whose semantic category is the first object are extracted, and the mask corresponding to that pixel is marked as 1, while the masks corresponding to the remaining pixels are marked as 0, thereby generating a binary mask of the same size as the first RGB image, i.e., the first mask.

[0022] S130. Based on the first mask, sample the first depth map in the target area to obtain a first sparse depth map.

[0023] In this embodiment, the target region can be the area sampled from the first depth map. The target region can be determined based on the performance parameters of the depth camera or the requirements of subsequent downstream application scenarios. For example, the region within the effective measurement range where the depth camera measurement error is less than the error threshold can be used as the target region. Alternatively, if the downstream application scenario is robot grasping, the target region can be predefined as the projection area of ​​the robotic arm's workspace in the image.

[0024] The first sparse depth map can be a depth map obtained by sampling a first depth map. The first sparse depth map can include the depth values ​​corresponding to the pixels in the region where the first object is located, as identified by the first mask. The pixel size of the first sparse depth map can be the same as the first depth map. Figure 1 To.

[0025] Specifically, the first mask is aligned pixel-by-pixel with the first depth map. Within the target area, the positions of pixels with a mask value of 1 are sampled. The sampling rules could be, for example: dividing the 3D scene space where the first object is located into several spatial boxes, assigning corresponding weights to each spatial box based on depth camera parameters (such as measurement accuracy distribution) or downstream application scenarios, determining the ratio of the weight of each spatial box to the total weight, allocating the number of pixels to be sampled in the projection area of ​​each spatial box in the first depth map according to the ratio, and sampling the pixels according to the number of pixels to be sampled to obtain the first sparse depth map.

[0026] For example, a target region is determined, such as the location where the measurement accuracy distribution error of the depth camera parameters is less than a preset error threshold. Then, a logical AND operation is performed between the region with a mask value of 1 in the first mask and the target region to determine the intersection region. The positions of all pixels within this intersection region are determined as the positions of pixels that can be sampled. Subsequently, the sampling ratio is determined, and the 3D scene space where the first object is located is divided into several spatial boxes. Based on the downstream application scenario, the weight of each spatial box is determined, and then the ratio of the weight of each spatial box to the total weight is determined. This ratio is multiplied by the total number of sampling points to determine the number of pixels to be sampled for each spatial box. The spatial boxes are mapped onto the first depth map, and sampling is performed according to the number of pixels to be sampled to obtain the first sparse depth map. For pixel regions outside the target region in the first depth map, the original depth value of the pixel region in the first depth map can be retained and filled into the corresponding position of the pixel region in the first sparse depth map, according to the needs of the downstream application scenario.

[0027] S140. Input the first RGB image and the first sparse depth map into the depth filling model to obtain the first dense depth map corresponding to the first object. The first dense depth map includes the depth map obtained after depth filling the first sparse depth map.

[0028] In this embodiment, the depth-filling model can be a model used for depth filling. The depth-filling model can be a neural network model. The depth-filling model can include an encoder and a decoder. The first dense depth map can be a depth map that has been depth-filled. The first dense depth map can be generated by the depth-filling model. The pixel size of the first dense depth map can be the same as that of the first sparse depth map.

[0029] Specifically, the first RGB image and the first sparse depth map are input into the depth filling model. The encoder in the depth filling model extracts the first RGB image as a strong semantic and edge prior, clearly establishes the alignment relationship between color edges and depth discontinuities, and ensures that the outline and surface boundary of the first object remain continuous after completion. The decoder performs spatial propagation and optimization on the first sparse depth map, and finally obtains the first dense depth map.

[0030] For example, Figure 2 This is a flowchart of an image generation method provided by an embodiment of the present invention. First, a first depth map is processed. Based on a first mask obtained by semantic segmentation of a first RGB image, the first depth map is sampled in the target region to obtain a first sparse depth map. The first sparse depth map and the first RGB image are then input into a depth filling model to obtain a first dense depth map.

[0031] The technical solution of this invention involves acquiring a first three-channel color RGB image and a first depth map. The first RGB image includes an RGB image obtained after photographing a first object, and the first depth map includes a depth image obtained after photographing the first object. Semantic segmentation is performed on the first RGB image to obtain a first mask corresponding to the first RGB image, the first mask including a mask of the location of the first object. Based on the first mask, the first depth map is sampled in the target region to obtain a first sparse depth map. The first RGB image and the first sparse depth map are input into a depth filling model to obtain a first dense depth map corresponding to the first object, the first dense depth map including a depth map obtained after depth filling the first sparse depth map. The technical solution of this invention fills the first sparse depth map with a depth filling model based on the first RGB image and the first sparse depth map, and completes the first sparse depth map with a depth completion method to obtain a first dense depth map, replacing the depth map processing method of depth denoising. Based on the texture semantic information of a single first RGB image and the effective data of the first sparse depth map, the depth is accurately completed to correct distortion and jump problems, thereby improving the quality and accuracy of the first dense depth map obtained after the first RGB image and the first sparse depth map are processed by the depth filling model.

[0032] Example 2 Figure 3This is a flowchart of an image generation method provided in Embodiment 2 of the present invention. This embodiment is an optimization based on any of the above embodiments, and mainly includes a detailed description of the training process of the depth filling model. It should be noted that technical details not described in detail in this embodiment can be found in any of the above embodiments. Figure 3 As shown, the method includes: S210. Obtain the second RGB image corresponding to the second object.

[0033] In this embodiment, the second object can be understood as an object used as a training sample during the model training phase. The second object can be distinct from or be the first object, and the second object can be an object belonging to the same category as the first object. The second RGB image can be an RGB image of the second object captured by a color camera. The second RGB image can represent the color, outline, and texture of the surface of the second object.

[0034] Specifically, a color camera is used to photograph the second object, and a second RGB image is generated based on the light reflected from the surface of the second object.

[0035] S220. Determine a second sparse depth map of the second object. The second sparse depth map includes a depth map of the location of the second object in the second depth map. The second depth map includes a depth map obtained after taking a picture of the second object.

[0036] In this embodiment, the second sparse depth map can be a depth map obtained by sampling a second depth map. The second sparse depth map can include the depth value of the pixel location of the second object in the second depth map. The second depth map can be a depth map of the second object captured by a depth camera. The second depth map can represent the straight-line distance from the corresponding point on the surface of the second object to the depth camera.

[0037] Specifically, the second RGB image is semantically segmented to obtain the corresponding mask. The distance between the second object measured by the depth camera and the depth camera is found in the mask, and the error between the distance between the second object and the depth camera and the actual distance is less than a predefined error. The mask is then combined with the mask in this area to obtain the second sparse depth map.

[0038] S230. Input the second RGB image and the second sparse depth map into the initial model to obtain the second dense depth map.

[0039] In this embodiment, the initial model can be an untrained depth-filling model. The initial model can be a neural network model. The initial model can include an encoder and a decoder. The second dense depth map can be understood as the depth map output by the initial model.

[0040] Specifically, the second RGB image and the second sparse depth map are input into the initial model. The encoder and decoder in the initial model learn the alignment relationship between the color edges in the second RGB image and the depth discontinuities in the second sparse depth map. The second sparse depth map is then spatially propagated and optimized to ensure that the outline and surface boundary of the second object remain continuous after depth filling, and finally the second dense depth map is obtained.

[0041] S240. Adjust the initial model based on the second dense depth map and the ground truth depth map of the second object.

[0042] In this embodiment, the ground truth depth map value can be the actual distance value from a point on the surface of the second object to the depth camera. The ground truth depth map value can be obtained by scanning the second object, or it can be generated based on the three-dimensional model of the second object by rendering and calculating the distance from each point on the surface of the three-dimensional model to the optical center of the depth camera.

[0043] Specifically, the second dense depth map is compared pixel-by-pixel with the ground truth depth map of the second object. Based on a preset loss function (such as L1 loss and / or L2 loss), the error between the second dense depth map and the ground truth depth map is quantified as the loss value. Then, using a backpropagation algorithm, the loss value is propagated back from the output layer to the input layer layer by layer, and the gradient of the loss value relative to each learnable parameter in the initial model is calculated. Based on the gradient, an optimization algorithm is used to iteratively update the model parameters.

[0044] For example, in each training round, the second RGB image and the second sparse depth map are input into the initial model. The initial model compares the second dense depth map with the ground truth depth map pixel by pixel, calculates the loss value, performs backpropagation based on the loss value, and calculates the gradient. During training, multiple predefined node thresholds are set (e.g., the node threshold can be 50%, 75%, and / or 90% of the total number of training epochs), denoted as milestone nodes. Whenever the training process reaches a milestone node, the learning rate is adjusted according to the gradient. After each training epoch, inference evaluation is performed on the validation dataset, and a performance matrix composed of multiple loss values ​​is calculated and recorded. After all training epochs are completed, the performance matrices of the initial model on the validation set in each epoch are compared, and the initial model corresponding to the best-performing performance matrix is ​​selected as the depth-filling model.

[0045] S250. Acquire a first three-channel color RGB image and a first depth map, wherein the first RGB image includes an RGB image obtained after photographing the first object, and the first depth map includes a depth image obtained after photographing the first object.

[0046] S260. By performing semantic segmentation on the first RGB image, a first mask corresponding to the first RGB image is obtained, wherein the first mask includes a mask of the location of the first object.

[0047] S270. Based on the first mask, sample the first depth map in the target area to obtain a first sparse depth map.

[0048] S280. Input the first RGB image and the first sparse depth map into the depth filling model to obtain the first dense depth map corresponding to the first object. The first dense depth map includes the depth map obtained after depth filling the first sparse depth map.

[0049] The technical solution of this invention involves: acquiring a second RGB image corresponding to a second object; determining a second sparse depth map of the second object; inputting the second RGB image and the second sparse depth map into an initial model to obtain a second dense depth map; adjusting the initial model based on the second dense depth map and the ground truth depth map of the second object; acquiring a first three-channel color RGB image and a first depth map, wherein the first RGB image includes an RGB image obtained after photographing the first object, and the first depth map includes a depth image obtained after photographing the first object; obtaining a first mask corresponding to the first RGB image by performing semantic segmentation on the first RGB image; sampling the first depth map in the target region based on the first mask to obtain a first sparse depth map; and inputting the first RGB image and the first sparse depth map into a depth filling model to obtain a first dense depth map corresponding to the first object. By processing the second depth map with the second mask, the initial model focuses only on the second object during training, reducing learning and prediction costs and solving the problem of high global denoising costs. By using the second sparse depth map and the second RGB image as input to the initial model, and the second RGB image as a strong semantic and edge prior, the initial model explicitly establishes the alignment relationship between color edges and depth discontinuities, ensuring that the contours and surface boundaries of objects remain continuous after completion, thereby improving the depth filling capability of the depth filling model and thus improving the image quality of the filled depth map output by the depth filling model.

[0050] In one embodiment, adjusting the initial model based on the second dense depth map and the ground truth depth map of the second object includes: S241. Determine the geometric loss value based on the true value of the depth map and the parameters associated with geometric topological features in the second dense depth map.

[0051] In this embodiment, the geometric topological features can be the geometric topological features of the second object represented by the ground truth depth map and the second dense depth map. Geometric topological features can describe, for example, the local shape and global structure of the second object's surface. Geometric topological features can be, for example, the position of the second object, the orientation angle of each surface, and / or the curvature of each surface. The geometric loss value can be the loss value of the geometry between the second object represented by the ground truth depth map and the second object predicted in the second dense depth map. The geometric loss value can be determined based on, for example, surface normals, curvature, and / or the global point cloud distribution. The geometric loss value can include displacement loss values ​​and / or loss values ​​for the orientation of each surface of the object.

[0052] Specifically, by mapping the ground truth depth map and the second dense depth map to 3D point clouds respectively, the parameters associated with the two sets of geometric topological features are determined, and the deviation between the two sets of parameters is compared. This deviation is then defined as the geometric loss value. The geometric loss value quantifies the difference between the predicted and the true values; the smaller the value, the higher the geometric topological consistency between the second object represented by the second dense depth map and the true second object.

[0053] S242. Based on the true value of the depth map and the parameters associated with the depth value in the second dense depth map, determine the image domain loss value.

[0054] In this embodiment, the image domain loss value can be understood as the deviation between the second dense depth map and the ground truth depth map in terms of the depth value of the corresponding pixel. The image domain loss value can be obtained by comparing pixel by pixel or by comparing the rate of change of pixel depth values. The image domain loss value can be expressed, for example, as the loss of pixel depth values ​​and / or the loss of gradient values, and the corresponding statistics.

[0055] Specifically, the ground truth depth map and the second dense depth map are aligned pixel by pixel. The predicted depth value of each pixel in the second dense depth map is compared with the depth value of the corresponding pixel in the ground truth depth map to determine the deviation between the two. The statistical measure of this deviation is then used as the image domain loss value.

[0056] For example, the predicted depth value of each pixel in the second dense depth map is compared one by one with the depth value of the corresponding pixel position in the ground truth depth map to determine the pixel-level loss, and the absolute error or mean square error of the pixel-level loss is calculated as the image domain loss value. Furthermore, the first-order partial derivatives (i.e., gradients) of the second dense depth map and the ground truth depth map in the horizontal and / or vertical directions can be calculated, and the difference between their gradients is used as the image domain loss value.

[0057] S243. The geometric loss value and the image domain loss value are weighted and summed to obtain the comprehensive loss value.

[0058] In this embodiment, the combined loss value can be a weighted sum of the geometric loss value and the image domain loss value. The combined loss value can be used as the final loss function of the initial model during the training phase.

[0059] Specifically, preset weights are assigned to the geometric loss value and the image domain loss value. The geometric loss value and the image domain loss value are multiplied by the preset weights respectively, and then the two weighted results are added together. The sum is the comprehensive loss value.

[0060] S244. Adjust the initial model according to the comprehensive loss value.

[0061] Specifically, the gradient of the overall loss value relative to all learnable parameters in the initial model is calculated using the backpropagation algorithm. Then, the learnable parameters of the initial model are iteratively updated using algorithms such as stochastic gradient descent, based on factors such as the gradient descent direction and the learning rate, so that the overall loss value generated by the model in subsequent forward propagation tends to decrease.

[0062] In one embodiment, determining the geometric loss value based on the ground truth of the depth map and parameters associated with geometric topological features in the second dense depth map includes: A1. Determine the first surface normal vector of each surface in the second object based on the true value of the depth map.

[0063] In this embodiment, the parameters associated with the geometric topological features include the surface normal vectors of each face in the second object, and the surface normal vectors include a first surface normal vector and a second surface normal vector.

[0064] In this embodiment, the first surface normal vector can be a normal vector determined based on the ground truth of the depth map. The first surface normal vector can characterize the geometric orientation of each surface of the second object in reality. The second surface normal vector can be a normal vector determined based on the second dense depth map. The second surface normal vector can characterize the geometric orientation of each surface of the second object predicted by the second dense depth map.

[0065] Specifically, the ground truth depth map is mapped into 3D space to obtain a 3D point cloud on the surface of the second object. Then, surface maps of each surface of the second object are constructed based on the 3D point cloud, and the normal vectors of each surface map are calculated to ensure that the coordinate system of each normal vector is consistent. The result obtained is the normal vector of the first surface.

[0066] A2. Determine the second surface normal vector of each surface in the second object based on the second dense depth map.

[0067] Specifically, by combining the intrinsic parameters of the depth camera, the depth values ​​of each pixel in the second dense depth map are mapped to three-dimensional space to obtain a three-dimensional point cloud on the surface of the second object. Based on the three-dimensional point cloud, surface maps of each surface of the second object are constructed, and the normal vectors of each surface map are determined. The result is the normal vector of the second surface.

[0068] A3. Determine the surface normal vector loss based on the deviation between the first surface normal vector and the second surface normal vector corresponding to the same surface.

[0069] In this embodiment, the surface normal vector loss can be understood as the loss value between the first surface normal vector and the corresponding second surface normal vector. The surface normal vector loss characterizes the directional differences between the surfaces of the second object predicted by the second dense depth map and the surfaces of the actual second object. The smaller the value of the surface normal vector loss, the higher the geometric consistency between the predicted second object and the actual second object.

[0070] Specifically, the first surface normal vector and the second surface normal vector are matched one by one. Then, for the matching results, the directional difference between the first surface normal vector and the corresponding second surface normal vector is calculated. The statistical value of all directional differences is calculated and the statistical value is used as the surface normal vector loss.

[0071] For example, the cosine distance between the angle between the first surface normal vector and the corresponding second surface normal vector is calculated, and the statistical value (such as the mean or variance) of all cosine distances is calculated. The statistical value is used as the surface normal vector loss.

[0072] For example, Figure 4 This is a flowchart of a training method for an initial model provided by an embodiment of the present invention. First, a second RGB image and a second sparse depth map are input into the initial model. The encoder and decoder in the initial model learn the features of the second RGB image and the second sparse depth map, and finally output a second dense depth map. Based on the ground truth of the depth map and the first surface normal vector, the loss function of this training round is calculated, and the initial model corresponding to the loss function with the best performance is selected as the depth filling model.

[0073] In one embodiment, determining the geometric loss value based on the ground truth of the depth map and parameters associated with geometric topological features in the second dense depth map includes: The displacement loss is determined based on the positional deviation of the second object in three-dimensional space from the second dense depth map and the true value of the depth map.

[0074] In this embodiment, the parameters associated with the geometric topological features include the position of the second object in three-dimensional space.

[0075] In this embodiment, displacement loss can be understood as the positional deviation between the predicted second object and the actual second object. Displacement loss can be measured by means such as chamfer distance.

[0076] Specifically, the second dense depth map and the ground truth depth map are mapped into three-dimensional space to obtain the three-dimensional point cloud of the second dense depth map and the ground truth depth map. The three-dimensional point cloud includes the three-dimensional coordinates of each point on the surface of the second object. The three-dimensional point cloud of the second dense depth map is matched with the three-dimensional point cloud of the ground truth depth map, and the distance between the matched point pairs is calculated. The statistical value of the distance, such as the average, is used as the displacement loss.

[0077] For example, the 3D point cloud of the second dense depth map is denoted as the predicted point cloud, and the ground truth of the depth map is denoted as the real point cloud. First, starting from the predicted point cloud, for each point in the predicted point cloud, the nearest corresponding point in the real point cloud is found, and the squared distance between all predicted points and their corresponding points is calculated and averaged to obtain the average distance. Second, starting from the real point cloud, for each point in the real point cloud, the nearest corresponding point in the predicted point cloud is matched, and the squared distance between the point in the real point cloud and its corresponding point is calculated and averaged to obtain another average distance. Finally, the two average distances are weighted and summed to obtain the displacement loss.

[0078] For example, Figure 5 This is a flowchart of a loss function determination method provided by an embodiment of the present invention. Based on the ground truth depth map and the second dense depth map, an image domain loss value is determined. The image domain loss value may include pixel-level loss of mean absolute error loss (L1) or mean squared error loss (L2) (i.e., L1 / L2 pixel-level loss), and sequence gradient consistency loss (i.e., gradient loss). Based on the ground truth depth map and the second dense depth map, a geometric loss value is determined. The geometric loss value may include surface normal consistency loss (i.e., surface normal vector loss) and chamfer distance loss (i.e., displacement loss). The process of determining the surface normal consistency loss includes: determining a first surface normal vector and a second surface normal vector based on the ground truth depth map and the second dense depth map, respectively, and using the deviation between the first surface normal vector and the second surface normal vector as the surface normal consistency loss; the process of determining the chamfer distance loss includes: mapping the ground truth depth map and the second dense depth map to three-dimensional space, determining the corresponding three-dimensional point clouds for each, and calculating the distance deviation between the corresponding three-dimensional point clouds as the chamfer distance loss. The loss values ​​of each image domain and each geometric loss are weighted and summed to obtain the comprehensive loss value.

[0079] In one embodiment, determining a second sparse depth map of the second object includes: S221. Obtain the second depth map, mesh map, and second RGB image of the second object.

[0080] In this embodiment, the mesh map can be a depth map generated by rendering a 3D mesh of the second object. The mesh map can have the same depth camera viewpoint and parameters as the second depth map. The depth value of each pixel in the mesh map is the true value of the depth map. The pixel size of the mesh map can be the same as that of the second depth map. Figure 1 To.

[0081] Specifically, a depth camera is used to photograph the second object, resulting in a second depth map. The 3D mesh of the second object is placed in the same coordinate system as the depth camera used for the second depth map. The mesh is then rendered to obtain a 3D model of the second object. The vertical distance from each point on the model's surface to the imaging plane of the depth camera is calculated, generating a depth map with the exact same pixel size and resolution as the second depth map—this is the mesh map. A color camera is then used to photograph the second object, resulting in a second RGB image.

[0082] S222. Perform semantic segmentation on the second RGB image to obtain a second mask corresponding to the second RGB image. The second mask includes a mask of the location of the second object.

[0083] In this embodiment, the second mask can be a mask obtained by semantic segmentation of the second RGB image. The second mask can be a binary mask. The pixel size of the second mask can be the same as the pixel size of the second RGB image.

[0084] Specifically, semantic segmentation is performed on the second RGB image, analyzing the color, texture, and contextual information of the image pixel by pixel, predicting the semantic category of each pixel, determining the mask corresponding to each pixel position based on the semantic category, and generating the second mask.

[0085] For example, if the semantic category corresponding to a pixel is the second object, then a mask of 1 is assigned to that pixel, and a mask of 0 is assigned to the pixels of other semantic categories, thereby obtaining the second mask.

[0086] S223. Compare the second depth map with the mesh map to determine the sampling area in the second depth map.

[0087] In this embodiment, the sampling area can be a sampling area in the second depth map. The sampling area can be a region where the depth value error between the second depth map and the mesh map is less than a preset threshold.

[0088] Specifically, the second depth map is aligned with the mesh map, and the error of the depth values ​​at the corresponding pixel positions of the two is calculated pixel by pixel. The region with an error less than a preset threshold is determined and used as the sampling region.

[0089] S224. Sampling is performed in the sampling area according to the second mask to obtain a second sparse depth map; wherein the true value of the depth map is determined based on the mesh map.

[0090] Specifically, the second depth map is aligned pixel-by-pixel with the second mask. Within the sampling area, pixels with a mask value of 1 are sampled. The number of sampling points at different locations can be determined, for example, by the ratio of the weight of that location to the total weight. The sampled result is used as the second sparse depth map.

[0091] In one embodiment, comparing the second depth map with the mesh map to determine the sampling area in the second depth map includes: The second depth map is compared with the mesh map to determine the region where the pixel depth value is less than the depth value threshold, and the region is used as the sampling region.

[0092] In this embodiment, the depth value threshold can be a threshold for the error of the depth value. The depth value threshold represents the maximum deviation between the second depth map and the mesh map.

[0093] Specifically, the second depth map is aligned with the mesh map, and the depth values ​​of the second depth map and the mesh map are compared pixel by pixel. The difference between the two is calculated. If the difference is less than the depth value threshold, the corresponding pixel area is determined as the sampling area.

[0094] For example, Figure 6 This is a flowchart of a method for determining a second sparse depth map provided by an embodiment of the present invention. First, a second depth map and a second RGB image are acquired. The second RGB image is semantically segmented to obtain a second mask. The second mask is used to perform calculations on the second depth map to obtain the depth observation values ​​of the second object (i.e., the depth values ​​of each pixel in the second depth map). The second object's 3D mesh is rendered to obtain a mesh map, which includes the ground truth depth map values ​​of the second object. A sampling region is determined based on the mesh map and the second depth map. Within the sampling region, the second depth map is sampled using the second mask to obtain the second sparse depth map.

[0095] In one embodiment, the step of sampling within the sampling area according to the second mask to obtain a second sparse depth map includes: B1. Within the sampling area, the second depth map is sampled according to the object instance ratio based on the second mask to obtain a sampled depth map. The object instance ratio includes the ratio of the weight of the spatial position occupied by the second object to the total weight.

[0096] In this embodiment, the object instance proportion can be understood as the ratio of the weight of the space occupied by the second object to the total weight. The weight of the space occupied by the second object can be allocated empirically or determined during the initial model training. The sampled depth map can be a depth map that has been sampled. The sampled depth map can be composed of all pixels in the second depth map that simultaneously satisfy the following two conditions: the pixel is located within the specified sampling area, and its corresponding value in the second mask is 1.

[0097] Specifically, a logical AND operation is performed between the second mask and the sampling area to obtain a set of pixels of the second object in the sampling area. Within this set, sampling is performed according to the proportion of object instances to obtain a sampled depth map. For example, the sampling process can be based on the ratio of the weight of the second object in the current scene space to the total weight to determine the number of sampling points, and then randomly select a corresponding number of pixels from the set.

[0098] B2. Fill the sampled depth map with the region other than the sampling region in the second depth map to obtain the second sparse depth map, which includes a depth map with the same size as the second depth map.

[0099] Specifically, the sampled depth map is filled in the area outside the sampling region to obtain a second sparse depth map. The filling depth value can be determined according to the downstream application scenario. For example, if the downstream application scenario does not pay much attention to the scene in which the second object is located, then 0 is used to fill the area outside the sampling region. If the downstream application scenario also needs to pay attention to the details of the scene in which the second object is located, then the depth value of the corresponding pixel position in the second depth map is used to fill the area outside the sampling region. The pixel size of the final second sparse depth map is the same as the size of the second depth map.

[0100] The present invention will be described by way of example below: In downstream applications such as autonomous driving, robotics, and augmented reality, depth maps often suffer from distortion, abrupt changes, and noise due to limitations in the physical performance of depth cameras, including lens optical distortion, sensor resolution limitations, lighting interference, and hardware noise. These issues severely reduce the reliability of depth data and interfere with the accuracy of core tasks such as object detection, scene reconstruction, and path planning. Currently, depth denoising is commonly used to process depth maps to provide relatively clean input data for subsequent vision tasks.

[0101] Depth denoising is a post-processing technique for depth images. Its core objective is to effectively suppress various types of noise and artifacts while preserving edge details, structural information, and true depth values ​​in the depth map. This technique optimizes noisy depth data through algorithmic models. It can rely on traditional filtering methods (such as Gaussian filtering, bilateral filtering, and joint bilateral filtering) to smooth noise using similarities in the spatial or value domains; or it can leverage deep learning models (such as denoising networks based on convolutional neural networks) to learn the true features of the depth map from a large amount of labeled data, achieving more accurate noise removal and detail restoration.

[0102] However, deep denoising methods suffer from drawbacks such as insufficient accuracy, high cost of global denoising, and lack of geometric consistency in prediction results.

[0103] Due to the physical limitations of depth camera accuracy, depth denoising can only reduce noise on the basis of the original depth map's accuracy. It cannot improve the accuracy of the original depth map, and therefore cannot provide depth information that meets industrial requirements in scenarios such as "object distance recognition and object grasping" that require high precision.

[0104] Denoising an entire image requires a significant amount of time and computational resources. In some industrial scenarios where real-time depth map generation is necessary, the time and computational costs of global denoising are too high, making it impossible to complete the response within the system's required timeframe.

[0105] Existing denoising methods mostly focus on eliminating noise without taking into account the semantic and geometric topological consistency of object structure, making it difficult to fundamentally solve the impact of deep data quality defects on subsequent completion tasks.

[0106] Therefore, this invention uses depth filling instead of depth denoising to solve the depth map quality problem more efficiently.

[0107] In one example, the present invention addresses the problem of obtaining high-precision depth maps through depth denoising, including: Existing depth denoising techniques can only smooth existing noisy depth data, failing to address fundamental quality issues such as depth loss and abrupt changes caused by camera physical limitations. Furthermore, the processed data still falls short of the stringent error control requirements of scenarios like object distance recognition and high-precision scene reconstruction. This invention replaces simple denoising with depth completion. Based on the texture semantic information of a single RGB image and the effective data from a sparse depth map, it accurately completes the depth to correct distortion and abrupt changes.

[0108] In one example, the present invention addresses the problems of high global denoising computational cost and time cost, making it difficult to respond to real-time scenarios, including: Existing global denoising techniques require processing the entire depth map, incurring the same computational and time costs regardless of whether a region is of interest to downstream tasks. This makes it difficult to meet response requirements in real-time scenarios. This invention uses semantic segmentation technology to divide the scene into critical and non-critical regions (i.e., determining the target region and sampling region). Targeted sparse depth sampling and fine-tuning are performed on critical regions, while processing of non-critical regions is simplified, thus eliminating wasted computational resources. This improves the depth accuracy of core local regions while reducing overall processing time and computational cost.

[0109] In one example, the present invention addresses the problem of unpredictable geometric consistency, including: Existing denoising methods often focus on noise removal, neglecting the structural semantics of objects and the geometric topology of the scene. This leads to problems such as local structural distortion and discontinuities in object surface normals in the processed depth map, compromising geometric consistency. This invention uses high-resolution RGB images (i.e., the first and second RGB images) as strong semantic and edge priors. An encoder-decoder network (i.e., the encoder and decoder included in the depth filling model and the initial model) explicitly establishes the alignment relationship between color edges and depth discontinuities, ensuring that the object's contour and surface boundaries remain continuous after completion. Simultaneously, surface normal consistency constraints (i.e., surface normal loss) and displacement geometric consistency constraints (i.e., displacement loss) are fused into the training objective, forcing the network to learn to conform to the physical 3D scene's geometric topology, thereby effectively suppressing structural distortion and recovering a geometrically consistent depth map.

[0110] In one example, sparse depth values ​​(i.e., a second sparse depth map) are sampled from regions with small target depth errors (i.e., sampling regions) through semantic segmentation, including: To focus more on the depth value of the target object (i.e., the second object) during depth completion training and prediction, before sampling the sparse depth map, a mask (i.e., the second mask) is found in the three-channel color image (i.e., the second RGB image) using semantic segmentation to locate the target object (i.e., the second object). The depth map rendered using the object mesh (i.e., the 3D mesh) is compared with the depth map obtained by the depth camera (i.e., the second depth map). The region with the smallest error (approximately 5 mm) in the mask is identified (i.e., the depth value threshold is 5 mm, and areas with pixel depth values ​​less than the depth value threshold are defined as sampling regions). From this region with the smallest error in the depth map obtained by the depth camera (i.e., the sampling region), a sparse depth map (i.e., the second sparse depth map) is obtained.

[0111] In one example, a method for determining the loss function is provided, including: By constructing a face map to represent the surface regions of objects (i.e., the second object) in the scene, and calculating the surface normals (i.e., the first and second surface normals) of each region to capture geometric and topological features, a face normal loss is introduced into the model training process to constrain the consistency between the predicted depth map and the real scene in terms of surface normals and topological structure. This ensures the geometric rationality of the depth map from an algorithmic perspective, thereby reducing the negative impact of depth data quality defects on subsequent completion and downstream tasks. The face normals of the object mesh (i.e., the ground truth depth map value) and the depth-filled predicted point cloud (i.e., the 3D point cloud generated based on the second dense depth map) are calculated. By calculating the L1 / L2 loss of the face normals between the ground truth (i.e., the ground truth depth map value) and the predicted value (i.e., the second dense depth map), combined with the pixel-level gradient loss of the initial model, gradient descent is performed to train the initial model.

[0112] In one example, Chamfer Distance is used to compensate for the loss of the face map in the displacement direction, including: Because face normal loss is calculated based on the normal vector of each face, and the normal vector of a face can only represent the orientation of the face, it cannot reflect whether there is a deviation between the face and the position of the point. Therefore, by adding Chamfer Distance, the loss between the predicted point cloud (i.e., the 3D point cloud generated from the second dense depth map) and the ground truth point cloud (i.e., the 3D point cloud generated from the ground truth depth map) is calculated, further constraining the geometric consistency of the predicted depth values.

[0113] The beneficial effects of this invention are as follows: because it uses depth completion instead of depth denoising and uses semantic segmentation to generate the mask, the model only focuses on the target object during the training and learning process, which reduces the learning and prediction costs and solves the problem of high global denoising costs.

[0114] By generating ground truth values ​​through mesh rendering of the target object and comparing them with observations from a depth camera, a subset of the more accurate depth camera observations is used as the source for generating a sparse depth map. Simultaneously, face normalization and Chamfer Distance losses are introduced to improve the geometric consistency of the predictions. This is because these two losses allow the model to learn how to make the depth map geometrically conform to the normal distribution and distances of the real surface, reducing the phenomenon of depth value jumps during 3D reconstruction. Through these two approaches, while maintaining the generalization advantages of the depth filling model, geometric consistency and physical plausibility are improved, solving the problems of "insufficient depth denoising accuracy" and "lack of geometric consistency in prediction results." This approach is suitable for high-precision applications such as robot grasping, industrial inspection, and box recognition.

[0115] In one example, an image generation method is provided, including: Obtain the following input data: ground truth depth map of the target object (i.e., mesh map), ground truth surface normal vector of the target object (i.e., first surface normal vector), camera RGB image (i.e., second RGB image), and sparse depth map of the target region (i.e., second sparse depth map).

[0116] For example, the ground truth of the target object depth map is obtained by modeling the target object mesh through pre-scanning, and the ground truth of where the second object should be in the scene is obtained through rendering.

[0117] For example, the true value of the surface normal vector of the target object is obtained by calculating the normal vector value of each face after obtaining the true value of the depth map of the target object.

[0118] For example, the RGB image from the camera is obtained directly from a color camera.

[0119] For example, the sparse depth map of the target region is first obtained by finding the mask of the region where the target object is located through the RGB image obtained by the semantic segmentation camera. The ground truth depth map of the target region is compared with the observation value of the camera depth map in the depth map. The region with the more accurate camera observation value is selected (when the target is a box (i.e., a spatial box), the region with an error range of less than 5 mm is usually selected). In the region with the more accurate camera observation value, the depth is sampled according to the area weight ratio (i.e., the proportion of object instances) to obtain the sparse depth map of the target region.

[0120] The model's loss is primarily calculated using two types of data: depth values ​​and object surface normal vectors. It is divided into two parts: the initial model loss, based on pixel depth and gradient, and the geometric constraint loss, based on surface normal consistency loss and Chamfer Distance loss.

[0121] In the basic loss calculation section, the difference between the depth pixel value and the gradient value, L1 / L2 loss, is calculated between the predicted value and the true value.

[0122] In the geometric constraint loss calculation section, based on the point cloud (i.e., 3D point cloud) generated by the depth-filled prediction, the normal vector of each face is calculated and compared with the true value of the normal vector of the target object surface to calculate the loss. At the same time, ChamferDistance is used to calculate whether there is a geometric loss in displacement between the point cloud generated by the depth-filled prediction and the true value.

[0123] The depth pixel-level loss, gradient loss, surface normal loss, and Chamfer Distance displacement loss are combined through a weighted loss function to generate a total loss value (i.e., a combined loss value) for the gradient descent process during model training.

[0124] After processing the data and labels (i.e., the second sparse depth map), the RGB image (i.e., the second RGB image) and the sparse depth map (i.e., the second sparse depth map) can be fed into the initial model. The predicted depth values ​​are compared with the ground truth depth map values ​​and the ground truth surface normal vector values ​​of the target object to calculate the combined loss value. Backpropagation and gradient descent algorithms are then performed based on the loss. After multiple rounds of training, several milestone nodes (i.e., predefined node thresholds) are set to control the model's learning rate. The model's performance on the training and validation datasets is compared, and the model with the optimal performance matrix is ​​selected as the final model (i.e., the depth-filling model).

[0125] The depth-filling model requires two types of data input during inference: the camera's RGB image (i.e., the first RGB image) and a sparse depth map (i.e., the first sparse depth map). The camera's RGB image can be used directly. The sparse depth map requires first obtaining a mask (i.e., the first mask) using semantic segmentation. Then, sampling is performed on tested regions with small depth errors (e.g., 5 mm for a box) within the mask (i.e., the target region) to obtain the sparse depth map.

[0126] After inputting the RGB image and sparse depth map from the camera into the pre-trained depth filling model, the depth filling model will output the inference depth map (i.e., the first dense depth map) based on the model's parameters.

[0127] This invention replaces traditional denoising with depth filling, solving the problem of inaccurate denoising. Addressing the issues of depth map distortion, jumps, and noise caused by the insufficient physical performance of depth cameras, traditional depth denoising preprocessing methods struggle to provide high-precision results and handle geometric object boundaries, failing to provide high-quality input for downstream tasks. This invention uses depth filling to replace depth denoising, improving filling accuracy through target region sampling and geometric constraints, thereby eliminating noise and correcting distortion.

[0128] This invention is based on proportional sampling of object instances. It uses target region sampling instead of random sampling across the entire image, and weights sampling by selecting reliable regions from depth camera observations. Sampling is allocated according to the proportion of object instances, reducing outlier data in the training data affected by noise and distortion. This also reduces the area the model needs to focus on, thereby reducing model learning time and computational costs. The algorithm is optimized in terms of sampling rationality.

[0129] This invention uses face map normal vector constraints to enhance geometric consistency. In the initial model, by calculating the loss of the face normal vectors, the initial model learns the geometric structure of the target object during depth filling, thereby reducing the problem of insufficient modeling of the object's surface structure in the initial model, and reducing problems such as surface discontinuities, shape distortion, and normal vector deviation.

[0130] This invention adds geometric constraints through a loss function. It also uses surface normal vectors to calculate the loss, enhancing the model's understanding of the geometric shape. Furthermore, to avoid the problem that normal vectors only represent the object's orientation and cannot calculate whether there is displacement compared to the true depth, Chamfer Distance is added to the loss function to constrain the displacement in depth prediction.

[0131] This invention adapts to industrial-grade target scenarios by combining face map recognition, target area proportional sampling, and geometric loss constraints to improve the geometric shape of targets with clear surface structures, such as boxes and regular geometric shapes, thereby specifically enhancing the adaptability to structured targets in scenarios such as industrial inspection and robot grasping.

[0132] Example 3 Figure 7 This is a schematic diagram of the structure of an image generation device provided in Embodiment 3 of the present invention. Figure 7 As shown, the device includes: The acquisition module 310 is used to acquire a first three-channel color RGB image and a first depth map. The first RGB image includes an RGB image obtained after photographing the first object, and the first depth map includes a depth image obtained after photographing the first object. The semantic segmentation module 320 is used to obtain a first mask corresponding to the first RGB image by performing semantic segmentation on the first RGB image, wherein the first mask includes a mask of the location of the first object. The sampling module 330 is used to sample the first depth map in the target area based on the first mask to obtain a first sparse depth map; The depth filling model input module 340 is used to input the first RGB image and the first sparse depth map into the depth filling model to obtain the first dense depth map corresponding to the first object. The first dense depth map includes the depth map obtained after depth filling the first sparse depth map.

[0133] The technical solution of this invention involves acquiring a first three-channel color RGB image and a first depth map via an acquisition module. The first RGB image includes an RGB image obtained after photographing a first object, and the first depth map includes a depth image obtained after photographing the first object. A semantic segmentation module performs semantic segmentation on the first RGB image to obtain a first mask corresponding to the first RGB image. The first mask includes a mask of the location of the first object. A sampling module samples the first depth map in the target area based on the first mask to obtain a first sparse depth map. A depth filling model input module inputs the first RGB image and the first sparse depth map into a depth filling model to obtain a first dense depth map corresponding to the first object. The first dense depth map includes a depth map obtained by depth filling the first sparse depth map. The technical solution of this invention fills the first sparse depth map with a depth filling model based on the first RGB image and the first sparse depth map, and completes the first sparse depth map with a depth completion method to obtain a first dense depth map, replacing the depth map processing method of depth denoising. Based on the texture semantic information of a single first RGB image and the effective data of the first sparse depth map, the depth is accurately completed to correct distortion and jump problems, thereby improving the quality and accuracy of the first dense depth map obtained after the first RGB image and the first sparse depth map are processed by the depth filling model.

[0134] In one embodiment, the training method of the depth filling model includes: The second RGB image acquisition module is used to acquire the second RGB image corresponding to the second object; The second sparse depth map determination module is used to determine the second sparse depth map of the second object. The second sparse depth map includes the depth map of the location of the second object in the second depth map. The second depth map includes the depth map obtained after taking a picture of the second object. The input module is used to input the second RGB image and the second sparse depth map into the initial model to obtain the second dense depth map; An initial model adjustment module is used to adjust the initial model based on the second dense depth map and the ground truth depth map of the second object.

[0135] In one embodiment, the initial model adjustment module further includes: A geometric loss value determination unit is used to determine a geometric loss value based on the true value of the depth map and parameters associated with geometric topological features in the second dense depth map; The image domain loss value determination unit is used to determine the image domain loss value based on the true value of the depth map and the parameters associated with the depth value in the second dense depth map; The comprehensive loss value acquisition unit is used to weightedly sum the geometric loss value and the image domain loss value to obtain the comprehensive loss value; An initial model adjustment unit is used to adjust the initial model based on the comprehensive loss value.

[0136] In one embodiment, the parameters associated with the geometric topological features include surface normal vectors of each face in the second object, the surface normal vectors including a first surface normal vector and a second surface normal vector; the geometric loss value determination unit is specifically used for: Determine the first surface normal vector of each surface in the second object based on the true value of the depth map; The second surface normal vector of each surface in the second object is determined based on the second dense depth map; The surface normal vector loss is determined based on the deviation between the first surface normal vector and the second surface normal vector corresponding to the same surface.

[0137] In one embodiment, the parameters associated with the geometric topological features include the position of the second object in three-dimensional space, and the geometric loss value determination unit is specifically used for: The displacement loss is determined based on the positional deviation of the second object in three-dimensional space from the second dense depth map and the true value of the depth map.

[0138] In one embodiment, the second sparse depth map determination module further includes: The acquisition unit is used to acquire the second depth map, the mesh map, and the second RGB image of the second object; The semantic segmentation unit is used to perform semantic segmentation on the second RGB image to obtain a second mask corresponding to the second RGB image, wherein the second mask includes a mask of the location of the second object; The comparison unit is used to compare the second depth map with the mesh map to determine the sampling area in the second depth map; A sampling unit is used to sample within the sampling area according to the second mask to obtain a second sparse depth map; wherein the true value of the depth map is a true value determined based on the mesh map.

[0139] In one embodiment, the comparison unit is specifically used for: The second depth map is compared with the mesh map to determine the region where the pixel depth value is less than the depth value threshold, and the region is used as the sampling region.

[0140] In one embodiment, the sampling unit is specifically used for: Within the sampling area, the second depth map is sampled according to the object instance ratio based on the second mask to obtain the sampled depth map. The object instance ratio includes the ratio of the weight of the spatial position occupied by the second object to the total weight. The sampled depth map is filled with the region outside the sampling region in the second depth map to obtain the second sparse depth map, which includes a depth map with the same size as the second depth map.

[0141] The image generation apparatus provided in this embodiment of the invention can execute an image generation method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0142] Example 4 Figure 8 A schematic diagram of an electronic device that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0143] like Figure 8 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0144] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0145] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the methods proposed in this invention.

[0146] In some embodiments, the method proposed in this invention can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the method proposed in this invention by any other suitable means (e.g., by means of firmware).

[0147] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0148] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0149] In the context of this invention, a computer-readable storage medium stores computer instructions that are used to cause a processor to execute and implement the method provided by this invention.

[0150] The present invention also provides a computer program product comprising a computer program that, when executed by a processor, implements the method provided according to embodiments of the present invention.

[0151] Computer-readable storage media can be tangible media that may contain or store computer programs for use by or in conjunction with an instruction execution system, apparatus, or device. Computer-readable storage media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0152] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device for displaying information to the user, such as a cathode ray tube (CRT) or a liquid crystal display (LCD); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0153] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0154] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system. It addresses the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0155] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0156] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An image generation method, characterized in that, include: Acquire a first three-channel color RGB image and a first depth map, wherein the first RGB image includes an RGB image obtained after photographing the first object, and the first depth map includes a depth image obtained after photographing the first object; By performing semantic segmentation on the first RGB image, a first mask corresponding to the first RGB image is obtained, and the first mask includes the mask of the location of the first object. Based on the first mask, the first depth map is sampled in the target area to obtain a first sparse depth map; The first RGB image and the first sparse depth map are input into the depth filling model to obtain the first dense depth map corresponding to the first object. The first dense depth map includes the depth map obtained after depth filling the first sparse depth map.

2. The method according to claim 1, characterized in that, The training method for the depth filling model includes: Obtain the second RGB image corresponding to the second object; A second sparse depth map of the second object is determined. The second sparse depth map includes a depth map of the location of the second object in the second depth map. The second depth map includes a depth map obtained after photographing the second object. The second RGB image and the second sparse depth map are input into the initial model to obtain the second dense depth map; The initial model is adjusted based on the second dense depth map and the ground truth depth map of the second object.

3. The method according to claim 2, characterized in that, The step of adjusting the initial model based on the second dense depth map and the ground truth depth map of the second object includes: Based on the ground truth of the depth map and the parameters associated with geometric topological features in the second dense depth map, the geometric loss value is determined; Based on the true value of the depth map and the parameters associated with the depth value in the second dense depth map, the image domain loss value is determined; The geometric loss value and the image domain loss value are weighted and summed to obtain the comprehensive loss value; The initial model is adjusted based on the comprehensive loss value.

4. The method according to claim 3, characterized in that, The parameters associated with the geometric topological features include the surface normal vectors of each face in the second object, and the surface normal vectors include a first surface normal vector and a second surface normal vector; The determination of the geometric loss value based on the ground truth of the depth map and parameters associated with geometric topological features in the second dense depth map includes: Determine the first surface normal vector of each surface in the second object based on the true value of the depth map; The second surface normal vector of each surface in the second object is determined based on the second dense depth map; The surface normal vector loss is determined based on the deviation between the first surface normal vector and the second surface normal vector corresponding to the same surface.

5. The method according to claim 3, characterized in that, The parameters associated with the geometric topological features include the position of the second object in three-dimensional space. The determination of the geometric loss value based on the ground truth depth map and the parameters associated with the geometric topological features in the second dense depth map includes: The displacement loss is determined based on the positional deviation of the second object in three-dimensional space from the second dense depth map and the true value of the depth map.

6. The method according to claim 2, characterized in that, Determining the second sparse depth map of the second object includes: Acquire the second depth map, mesh map, and second RGB image of the second object; Semantic segmentation is performed on the second RGB image to obtain a second mask corresponding to the second RGB image. The second mask includes a mask of the location of the second object. The second depth map is compared with the mesh map to determine the sampling area in the second depth map; Sampling is performed within the sampling area according to the second mask to obtain a second sparse depth map; The true value of the depth map is determined based on the true value of the mesh map.

7. The method according to claim 6, characterized in that, The step of comparing the second depth map with the mesh map to determine the sampling area in the second depth map includes: The second depth map is compared with the mesh map to determine the region where the pixel depth value is less than the depth value threshold, and the region is used as the sampling region.

8. The method according to claim 6, characterized in that, The step of sampling within the sampling area according to the second mask to obtain a second sparse depth map includes: Within the sampling area, the second depth map is sampled according to the object instance ratio based on the second mask to obtain the sampled depth map. The object instance ratio includes the ratio of the weight of the spatial position occupied by the second object to the total weight. The sampled depth map is filled with the region outside the sampling region in the second depth map to obtain the second sparse depth map, which includes a depth map with the same size as the second depth map.

9. An image generation apparatus, characterized in that, include: The acquisition module is used to acquire a first three-channel color RGB image and a first depth map. The first RGB image includes an RGB image obtained after photographing the first object, and the first depth map includes a depth image obtained after photographing the first object. The semantic segmentation module is used to obtain a first mask corresponding to the first RGB image by performing semantic segmentation on the first RGB image, wherein the first mask includes a mask of the location of the first object. The sampling module is used to sample the first depth map in the target area based on the first mask to obtain a first sparse depth map; The depth filling model input module is used to input the first RGB image and the first sparse depth map into the depth filling model to obtain the first dense depth map corresponding to the first object. The first dense depth map includes the depth map obtained after depth filling the first sparse depth map.

10. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

Images