A three-dimensional imaging method, apparatus, device, and storage medium

By using image processing models to segment and optimize the depth of transparent objects in 3D imaging technology, the problem of poor imaging of transparent objects in existing technologies is solved, and accurate 3D reconstruction and improved imaging effect of transparent objects are achieved.

CN115578516BActive Publication Date: 2026-11-17JINGDONG TECH HLDG CO LTD +1
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202211281815.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-10-19
Publication Date
2026-11-17
Estimated Expiration
2042-10-19

AI Technical Summary

Technical Problem

Existing 3D imaging technologies suffer from insufficient imaging accuracy and poor performance when dealing with transparent objects. In particular, passive binocular 3D imaging has high requirements for texture features and cannot measure scenes with indistinct textures. 3D imaging based on the time-of-flight principle has poor resolution and accuracy in close-range scenes, and structured light projection 3D imaging cannot effectively image transparent objects.

Method used

By acquiring color images and original depth maps of the target shooting scene, a preset image processing model is used to segment transparent objects, predict boundaries and normal vectors, remove the depth information of transparent objects, perform global optimization of the depth information, and construct an optimized depth map to fill in the missing depth areas of transparent objects.

Benefits of technology

It improves the 3D imaging effect of transparent objects, realizes accurate 3D reconstruction of transparent objects, and enhances the imaging accuracy and effect of the imaging system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115578516B_ABST
    Figure CN115578516B_ABST
Patent Text Reader

Abstract

This invention discloses a three-dimensional imaging method, apparatus, device, and storage medium. The method includes: acquiring a target color image and an original depth map corresponding to a target shooting scene, wherein the target shooting scene contains at least one transparent object; inputting the target color image into a preset image processing model, and obtaining a transparent object segmentation result, a boundary prediction result, and a normal vector prediction result based on the output of the preset image processing model; performing a cropping process on the original depth map based on the transparent object segmentation result to obtain a first depth map with the transparent object's depth information removed; performing global optimization of the depth map based on the first depth map, the transparent object segmentation result, the boundary prediction result, and the normal vector prediction result to determine an optimized second depth map; and determining a target three-dimensional image corresponding to the target shooting scene based on the second depth map. The technical solution of this invention can effectively improve the three-dimensional imaging effect of transparent objects.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present invention relate to computer technology, and more particularly to a three-dimensional imaging method, apparatus, device, and storage medium. Background Technology

[0002] With the rapid development of computer technology, 3D cameras can be used to acquire three-dimensional images of scenes of interest in an optical manner.

[0003] Currently, 3D imaging methods can include passive binocular 3D imaging, time-of-flight-based 3D imaging, and structured light projection 3D imaging. Passive binocular 3D imaging indirectly calculates 3D information using triangulation through image feature matching. Time-of-flight-based 3D imaging directly measures the time of flight of light. Structured light projection 3D imaging actively projects a known coded pattern, improving feature matching accuracy.

[0004] However, in the process of realizing this invention, the inventors discovered at least the following problems in the prior art:

[0005] Passive binocular 3D imaging requires high detail in the texture of the object being measured, making it unsuitable for scenes with indistinct textures and thus unsuitable as the "eyes" of industrial composite robots. The measurement accuracy of time-of-flight-based 3D imaging depends on the precision of the beam detection time; its imaging resolution and accuracy are poor in close-range scenes, thus it is more often used in long-range scenarios, such as autonomous driving and long-distance search and detection. In structured light projection 3D imaging, the projected light is prone to projection and reflection on transparent objects, making it unsuitable for effective 3D imaging of transparent objects. Summary of the Invention

[0006] This invention provides a three-dimensional imaging method, apparatus, device, and storage medium to effectively improve the three-dimensional imaging effect of transparent objects in a shooting scene.

[0007] In a first aspect, embodiments of the present invention provide a three-dimensional imaging method, comprising:

[0008] Acquire a target color image and original depth map corresponding to the target shooting scene, wherein the target shooting scene contains at least one transparent object;

[0009] The target color image is input into a preset image processing model for image processing, and based on the output of the preset image processing model, the transparent object segmentation result, boundary prediction result, and normal vector prediction result in the target shooting scene are obtained.

[0010] Based on the transparent object segmentation results, the original depth map is cropped to obtain a first depth map with the transparent object depth information removed.

[0011] Based on the first depth map, the transparent object segmentation result, the boundary prediction result, and the normal vector prediction result, global optimization of depth information is performed to determine the optimized second depth map;

[0012] Based on the second depth map, the target 3D image corresponding to the target shooting scene is determined.

[0013] Secondly, embodiments of the present invention also provide a three-dimensional imaging device, comprising:

[0014] An image acquisition module is used to acquire a target color image and an original depth map corresponding to a target shooting scene, wherein the target shooting scene contains at least one transparent object;

[0015] The target color image input module is used to input the target color image into a preset image processing model for image processing, and obtain the transparent object segmentation result, boundary prediction result and normal vector prediction result in the target shooting scene based on the output of the preset image processing model;

[0016] The shearing module is used to shear the original depth map based on the transparent object segmentation result to obtain a first depth map with the transparent object depth information removed.

[0017] The global optimization module is used to perform global optimization of depth information based on the first depth map, the transparent object segmentation result, the boundary prediction result and the normal vector prediction result, and to determine the optimized second depth map.

[0018] The target 3D image determination module is used to determine the target 3D image corresponding to the target shooting scene based on the second depth map.

[0019] Thirdly, embodiments of the present invention also provide an electronic device, the electronic device comprising:

[0020] One or more processors;

[0021] Memory, used to store one or more programs;

[0022] When the one or more programs are executed by the one or more processors, the one or more processors implement the three-dimensional imaging method as provided in any embodiment of the present invention.

[0023] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the three-dimensional imaging method as provided in any embodiment of the present invention.

[0024] One embodiment of the above invention has the following advantages or beneficial effects:

[0025] By inputting the target color image corresponding to the target shooting scene into a preset image processing model for image processing, the transparent object segmentation result, boundary prediction result, and normal vector prediction result in the target shooting scene are obtained. Based on the transparent object segmentation result, the original depth map corresponding to the target shooting scene is cropped to obtain a first depth map with the depth information of the transparent object removed. Based on the first depth map, the transparent object segmentation result, the boundary prediction result, and the normal vector prediction result, the depth information is globally optimized to fill in the depth missing areas caused by the object's transparency. Based on the complete second depth map after completion and optimization, the target 3D image corresponding to the target shooting scene containing the transparent object can be constructed more accurately, thereby effectively improving the 3D imaging effect of the transparent object. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart of a three-dimensional imaging method provided in one embodiment of the present invention;

[0028] Figure 2 This is an example diagram of a three-dimensional imaging process according to an embodiment of the present invention;

[0029] Figure 3 This is a flowchart of another three-dimensional imaging method provided in one embodiment of the present invention;

[0030] Figure 4 This is a schematic diagram of the structure of a three-dimensional imaging device provided in one embodiment of the present invention;

[0031] Figure 5 This is a schematic diagram of the structure of an electronic device provided in one embodiment of the present invention. Detailed Implementation

[0032] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0033] Figure 1This is a flowchart illustrating a three-dimensional imaging method according to an embodiment of the present invention. This embodiment is applicable to three-dimensional imaging of scenes containing transparent objects. The method can be executed by a three-dimensional imaging device, which can be implemented in software and / or hardware and integrated into electronic devices, such as 3D cameras or visual sensors of composite robots in industrial settings, to assist robots in performing tasks such as scene recognition and target detection under complex working conditions. Figure 1 As shown, the method specifically includes the following steps:

[0034] S110. Obtain the target color image and original depth map corresponding to the target shooting scene, wherein the target shooting scene contains at least one transparent object.

[0035] Transparent objects can include fully transparent objects that allow complete light transmission and semi-transparent objects that allow partial light transmission, such as glass cups and plastic bottles. The target shooting scene can refer to the area of ​​the scene currently being shot. One or more transparent objects can exist in the target shooting scene. The target shooting scene can contain only transparent objects, or it can contain other non-transparent objects in addition to transparent objects. The target color image can be an RGB image synthesized using red, blue, and green colors. The original depth map can be a depth map that includes depth information of the transparent objects.

[0036] Specifically, a 2D camera can be used to photograph the target scene to obtain a color image of the target scene. A 3D camera can be used to photograph the target scene to obtain a raw depth map of the target scene.

[0037] S120. Input the target color image into the preset image processing model for image processing, and obtain the transparent object segmentation result, boundary prediction result and normal vector prediction result in the target shooting scene based on the output of the preset image processing model.

[0038] The preset image processing model can be a neural network model used for segmenting and extracting transparent objects from a color image, predicting the boundaries between objects in the image, and predicting the normal vectors of each element position in the image. The transparent object segmentation result can be represented using a grayscale image; for example, the area where the transparent object is located is a white area with a grayscale value of 255, and other areas besides the transparent object are black areas with a grayscale value of 0. The boundary prediction result can be a boundary prediction image of the same size as the input target color image. The boundary prediction image includes the boundaries between the transparent object and the background, and the boundaries between the transparent object and non-transparent objects, represented by lines. The normal vector prediction result can be a normal vector prediction image of the same size as the input target color image. Different colors can be used to represent the normal vector corresponding to each element position in the image. The normal vector corresponding to each element position can be the normal vector of the plane formed by the element position and its adjacent element positions. It should be noted that the preset image processing model is obtained by pre-training the model based on sample data. The sample data includes sample color images containing at least one transparent object, and transparent object segmentation labels, boundary labels, and normal vector labels obtained by calibrating the sample color images.

[0039] Specifically, Figure 2 An example diagram of a three-dimensional imaging process is given. For example... Figure 2 As shown, the target color image can be input into a pre-trained preset image processing model. The preset image processing model can perform image processing on the input target color image, and simultaneously determine and output the transparent object segmentation results, boundary prediction results, and normal vector prediction results in the target shooting scene. Thus, the preset image processing model can be used to quickly obtain the transparent object segmentation results, boundary prediction results, and normal vector prediction results.

[0040] For example, the preset image processing model may include: an encoding sub-model, a first decoding branch sub-model, a second decoding branch sub-model, and a third decoding branch sub-model. Correspondingly, S120 may include: inputting the target color image into the encoding sub-model for feature extraction to obtain extracted target image feature information; inputting the target image feature information into the first decoding branch sub-model for transparent object position prediction to determine the transparent object segmentation result; inputting the target image feature information into the second decoding branch sub-model for transparent and non-transparent object boundary prediction to determine the boundary prediction result; and inputting the target image feature information into the third decoding branch sub-model for predicting the normal vector corresponding to the pixel position to determine the normal vector prediction result.

[0041] Specifically, the encoding sub-model can refer to a network model that encodes a color image and extracts its features. For example, the first two inference stages of the original Swin Transformer network structure can be used as the encoding sub-model, while the third and fourth inference stages can be removed, thus reducing the overall time cost of the model. The single-branch decoding network in the Swin Transformer network structure can be modified into a three-branch decoding network model, namely, the first decoding branch sub-model, the second decoding branch sub-model, and the third decoding branch sub-model. The first, second, and third decoding branch sub-models are three parallel decoding network models used to predict different information. Thus, the first, second, and third decoding branch sub-models can simultaneously predict the transparent object segmentation result, boundary prediction result, and normal vector prediction result in the input image, avoiding repeated inference in the encoding network, improving information prediction efficiency, and further improving 3D imaging efficiency.

[0042] S130. Based on the transparent object segmentation results, the original depth map is cropped to obtain a first depth map with the transparent object depth information removed.

[0043] The first depth map can refer to a depth map that does not contain depth information of transparent objects. When acquiring the original depth map, the depth information of the transparent objects acquired is inaccurate due to the transmission and reflection of projected light on the transparent objects. Therefore, it is necessary to cut off the depth information of the transparent objects whose positions are incorrectly predicted in the original depth map.

[0044] Specifically, such as Figure 2 As shown, based on the transparent object position information in the transparent object segmentation result, the depth information of the transparent object in the original depth map is determined, and the transparent object depth information in the original depth map is clipped and removed, thereby obtaining the first depth map after removing the incorrectly predicted transparent object depth information.

[0045] S140. Based on the first depth map, the transparent object segmentation result, the boundary prediction result, and the normal vector prediction result, perform global optimization of the depth information to determine the optimized second depth map.

[0046] The second depth map can be the optimal depth map obtained by supplementing the first depth map with depth information, that is, the depth map containing the most accurate depth information of transparent objects.

[0047] Specifically, such as Figure 2As shown, the first depth map, the transparent object segmentation result, the boundary prediction result, and the normal vector prediction result can be input into a global optimizer. The global optimizer solves the optimal solution for the depth information based on the input information and outputs the obtained optimal depth map, i.e., the second depth map, thereby completing the depth information of the depth-deficient region caused by the transparency of the object.

[0048] For example, S140 may include: taking the target depth map as the optimization object, constructing a target optimization function corresponding to the target depth map based on the first depth map, the transparent object segmentation result, the boundary prediction result, and the normal vector prediction result; minimizing the target optimization function, and determining the target depth map corresponding to the minimum solution as the optimized second depth map.

[0049] The target depth map can be obtained by completing the depth information of the first depth map. In other words, the target depth map is a complete depth map with the same size as the original depth map. The target depth map can refer to the optimization object in the target optimization function. The depth information of each pixel in the target depth map can be adjusted and optimized to determine the optimal depth information for each pixel. Specifically, during global optimization of the depth information, the position of each pixel in the target depth map can be modeled based on the first depth map, the transparent object segmentation results, the boundary prediction results, and the normal vector prediction results. This constructs a target optimization function with the target depth map as the optimization object, thus transforming the optimization of the entire depth map into a mathematical process of solving the minimum solution of an n-variable linear polynomial. The sparse square root Cholesky decomposition method can be used to minimize the target optimization function. Each solution corresponds to a target depth map containing specific depth values. The minimum solution is the optimal solution, and the target depth map corresponding to the minimum solution is taken as the optimal depth map, i.e., the second depth map, thereby completing the depth missing regions caused by the transparency of objects.

[0050] S150. Based on the second depth map, determine the target 3D image corresponding to the target shooting scene.

[0051] Specifically, such as Figure 2 As shown, the complete second depth map, after depth completion optimization, can be transformed according to pre-calibrated camera parameters to obtain the final 3D point cloud data of the target shooting scene. Based on the 3D point cloud data, a more accurate 3D image of the target shooting scene containing transparent objects can be constructed. By using a preset image processing model and global optimization method based on the target color image and the original depth map, a post-processing optimization workflow is implemented, which can effectively improve the 3D imaging effect of the imaging system on transparent objects with limited computing power.

[0052] The technical solution of this embodiment involves inputting the target color image corresponding to the target shooting scene into a preset image processing model for image processing to obtain the transparent object segmentation result, boundary prediction result, and normal vector prediction result in the target shooting scene. Based on the transparent object segmentation result, the original depth map corresponding to the target shooting scene is cropped to obtain a first depth map with the transparent object depth information removed. Based on the first depth map, the transparent object segmentation result, the boundary prediction result, and the normal vector prediction result, the depth information is globally optimized to fill in the depth missing areas caused by the object's transparency. Based on the complete second depth map after completion and optimization, the target 3D image corresponding to the target shooting scene containing the transparent object can be constructed more accurately, thereby effectively improving the 3D imaging effect of the transparent object.

[0053] Figure 3 This is a flowchart illustrating another three-dimensional imaging method provided by an embodiment of the present invention. Based on the above embodiments, this embodiment describes in detail the process of constructing the target optimization function corresponding to the target depth map. Explanations of terms that are the same as or corresponding to those in the above embodiments are not repeated here.

[0054] See Figure 3 The other three-dimensional imaging method provided in this embodiment specifically includes the following steps:

[0055] S310. Obtain the target color image and original depth map corresponding to the target shooting scene, wherein the target shooting scene contains at least one transparent object.

[0056] S320. Input the target color image into the preset image processing model for image processing, and obtain the transparent object segmentation result, boundary prediction result and normal vector prediction result in the target shooting scene based on the output of the preset image processing model.

[0057] S330. Based on the transparent object segmentation results, the original depth map is cropped to obtain a first depth map with the transparent object depth information removed.

[0058] S340. Using the target depth map as the optimization object, construct the depth deviation sub-function corresponding to the target depth map based on the first depth map.

[0059] Specifically, a pixel-level depth deviation loss, i.e., a depth deviation sub-function, can be constructed based on the depth values ​​of each pixel in the first depth map and the target depth map. It can also be required that the deviation between the predicted depth value of the current pixel in the target depth map and the original depth value of the pixel in the first depth map be as small as possible.

[0060] For example, S340 may include: obtaining a first depth value corresponding to each first pixel in the first depth map; determining the depth deviation between the first depth value and the optimized depth value corresponding to the same first pixel in the first depth map and the target depth map; and constructing a depth deviation sub-function corresponding to the target depth map based on each depth deviation.

[0061] In this context, the first pixel refers to all pixels in the first depth map except for those of transparent objects. The first depth map is created by cropping and removing the depth information of transparent objects; therefore, the transparent object pixels in the first depth map are invalid pixels without a first depth value. In this case, each valid pixel in the first depth map that has a first depth value can be used as the first pixel, i.e., p1∈T. obs .

[0062] Specifically, for each first pixel, in the first depth map and the target depth map, the depth deviation between the first depth value D0(p1) and the optimized depth value D(p1) corresponding to the same first pixel p1 is determined. The squares of the depth deviations corresponding to each first pixel are added together, and the sum is used as the constructed depth deviation sub-function E. D For example, the constructed depth deviation sub-function is:

[0063] It should be noted that the depth deviation sub-function does not calculate invalid pixels in the first depth map. Therefore, for the transparent object region that has been cut off, the deviation of its optimized depth from the original depth is not required to be as small as possible. Only the deviation of the optimized depth of the valid pixels from the original depth is required to be as small as possible, so as to ensure the accuracy of the depth optimization of the valid pixels.

[0064] S350. Based on the boundary prediction results and normal vector prediction results, construct the normal vector deviation sub-function corresponding to the target depth map.

[0065] Specifically, a pixel-level normal vector bias loss, i.e., a normal vector bias sub-function, can be constructed based on the normal vectors calculated for each pixel and its neighboring pixels in the target depth map and the predicted normal vectors corresponding to each pixel in the normal vector prediction results. Furthermore, it is required that the angle between the normal vectors calculated for each pixel and its neighboring pixels in the target depth map (excluding boundary pixels) and the predicted normal vector of that pixel in the normal vector prediction results be as small as possible.

[0066] For example, S350 may include: obtaining boundary pixels in the boundary prediction result; determining a first normal vector corresponding to each second pixel in the target depth map; obtaining a second normal vector corresponding to each second pixel in the normal vector prediction result; determining the vector angle corresponding to each second pixel based on the first and second normal vectors corresponding to each second pixel; and constructing a normal vector deviation sub-function corresponding to the target depth map based on the vector angles.

[0067] Here, the second pixel refers to all pixels in the target depth map other than the boundary pixels. Specifically, all pixels on each boundary in the boundary prediction result can be considered as boundary pixels, and all pixels in the target depth map other than all boundary pixels can be defined as second pixels, i.e., p2∈N. For each second pixel p2, the first normal vector v(p2,q) corresponding to the second pixel can be determined in the plane formed by the second pixel and its four adjacent pixels q in the four directions of up, down, left, and right in the target depth map. The second normal vector N(p2) of the second pixel in the normal vector prediction result is obtained. Based on the first and second normal vectors corresponding to the second pixel, the vector angle <v(p2,q),N(p2)> corresponding to the second pixel is determined. The squares of the vector angles corresponding to each second pixel can be added together, and the sum is used as the constructed normal vector deviation sub-function.

[0068] For example, the constructed normal vector deviation sub-function is:

[0069] It should be noted that the normal vector deviation sub-function does not calculate the pixels predicted as boundary points in the target depth map, thus allowing the normal vectors at the boundary positions in the target shooting scene to change drastically, thereby effectively ensuring the accuracy of depth optimization.

[0070] For example, determining the first normal vector corresponding to each second pixel in the target depth map may include: determining the normal vector corresponding to each second pixel based on the pixel position of each second pixel in the target depth map and the pixel positions of two adjacent pixels in each of two adjacent directions, wherein the adjacent directions include the four directions of up, down, left, and right; averaging the normal vectors to determine the first normal vector corresponding to each second pixel.

[0071] Specifically, for each second pixel, the pixel position can be connected to the positions of its two adjacent pixels in each of the four directions (up, down, left, and right) to construct a plane corresponding to each pair of adjacent pixels. The normal vector of each plane is then determined, and the average normal vector for the second pixel is calculated and used as the first normal vector for that second pixel. By averaging the normal vectors for each pair of adjacent pixels, the normal vector deviation can be more accurately represented, further improving the global optimization effect of depth information.

[0072] S360. Based on the boundary prediction results, construct the depth smoothing sub-function corresponding to the target depth map.

[0073] Specifically, a pixel-level depth smoothing loss, i.e., a depth smoothing sub-function, can be constructed based on the depth values ​​of each pixel and its neighboring pixels in the target depth map. It can also be required that the change in depth value between each pixel and its neighboring pixels in the target depth map, except for the boundary pixels, be as small as possible.

[0074] For example, S360 may include: obtaining boundary pixels in the boundary prediction result; determining the change depth corresponding to each second pixel based on the optimized depth value corresponding to each second pixel in the target depth map and the adjacent depth values ​​corresponding to adjacent pixels in each adjacent direction; and constructing a depth smoothing sub-function corresponding to the target depth map based on the change depth corresponding to each second pixel in each adjacent direction.

[0075] Here, the second pixel is any pixel in the target depth map other than the boundary pixels. Adjacent directions include four directions: up, down, left, and right. Specifically, all pixels on each boundary in the boundary prediction result can be considered as boundary pixels, and all pixels in the target depth map other than all boundary pixels can be determined as second pixels, i.e., p2∈N. For each second pixel p2, the optimized depth value D(p2) corresponding to the second pixel in the target depth map can be subtracted from the adjacent depth value D(q) corresponding to the adjacent pixels in each adjacent direction to obtain the change in depth between the second pixel and the adjacent pixels in each adjacent direction. For each adjacent direction, the squares of the change in depth corresponding to each second pixel in that adjacent direction can be added together, and the sum is used as the constructed depth smoothing sub-function for that adjacent direction. For example, the constructed normal vector deviation sub-function is: The solution can be obtained by solving the simultaneous solution of the depth smoothing sub-functions corresponding to each adjacent direction, or the average function can be obtained by averaging the depth smoothing sub-functions corresponding to each adjacent direction and using the average function as the normal vector deviation sub-function.

[0076] It should be noted that the depth smoothing sub-function does not consider the consistency of depth changes of pixels predicted as boundary points in the target depth map, thus allowing depth jumps at boundary positions in the target shooting scene, thereby effectively ensuring the accuracy of depth optimization.

[0077] S370. Based on the depth deviation sub-function, normal vector deviation sub-function, and depth smoothing sub-function, construct the target optimization function corresponding to the target depth map.

[0078] Specifically, based on the depth deviation weight λ D Normal vector deviation weight λ N and depth smoothing weight λ S The depth deviation subfunction, normal vector deviation subfunction, and depth smoothing subfunction are weighted and summed. The sum is used as the constructed objective optimization function E, i.e., E = λ. D E D +λ N E N +λ S E S Among them, the depth deviation weight λ D It can be greater than the normal vector deviation weight λ N and depth smoothing weight λ S To ensure the accuracy of depth optimization, for example, the depth deviation weight λ D The value can be 1000, and the normal vector deviation weight λ N and depth smoothing weight λ S All are 1.

[0079] S380. Minimize the objective optimization function and determine the target depth map corresponding to the minimum solution as the optimized second depth map.

[0080] Specifically, the sparse square root Cholesky decomposition method can be used to minimize the objective optimization function. Each solution corresponds to a target depth map. The minimum solution is the optimal solution. The target depth map corresponding to the minimum solution is taken as the optimal depth map, i.e., the second depth map, thereby completing the depth missing area caused by the transparency of the object.

[0081] S390. Based on the second depth map, determine the target 3D image corresponding to the target shooting scene.

[0082] The technical solution of this embodiment constructs a depth deviation sub-function, a normal vector deviation sub-function, and a depth smoothing sub-function corresponding to the target depth map, respectively. This allows for global optimization of the depth deviation loss, normal vector deviation loss, and depth smoothing loss, thereby enabling more accurate depth completion, ensuring the global optimization effect of depth information, and further improving the 3D imaging effect of transparent objects.

[0083] The following are embodiments of the three-dimensional imaging device provided in this invention. This device and the three-dimensional imaging methods in the above embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the three-dimensional imaging device, please refer to the embodiments of the above three-dimensional imaging methods.

[0084] Figure 4 This is a schematic diagram of a three-dimensional imaging device provided in an embodiment of the present invention. This embodiment is applicable to situations where three-dimensional imaging is performed on a scene containing transparent objects. Figure 4 As shown, the device specifically includes: an image acquisition module 410, a target color image input module 420, a cropping processing module 430, a global optimization module 440, and a target three-dimensional image determination module 450.

[0085] The system includes: an image acquisition module 410 for acquiring a target color image and an original depth map corresponding to a target shooting scene, wherein the target shooting scene contains at least one transparent object; a target color image input module 420 for inputting the target color image into a preset image processing model for image processing, and obtaining, based on the output of the preset image processing model, the transparent object segmentation result, boundary prediction result, and normal vector prediction result in the target shooting scene; a cropping module 430 for cropping the original depth map based on the transparent object segmentation result to obtain a first depth map with the transparent object depth information removed; a global optimization module 440 for globally optimizing the depth information based on the first depth map, the transparent object segmentation result, the boundary prediction result, and the normal vector prediction result to determine an optimized second depth map; and a target 3D image determination module 450 for determining the target 3D image corresponding to the target shooting scene based on the second depth map.

[0086] The technical solution of this embodiment involves inputting the target color image corresponding to the target shooting scene into a preset image processing model for image processing to obtain the transparent object segmentation result, boundary prediction result, and normal vector prediction result in the target shooting scene. Based on the transparent object segmentation result, the original depth map corresponding to the target shooting scene is cropped to obtain a first depth map with the transparent object depth information removed. Based on the first depth map, the transparent object segmentation result, the boundary prediction result, and the normal vector prediction result, the depth information is globally optimized to fill in the depth missing areas caused by the object's transparency. Based on the complete second depth map after completion and optimization, the target 3D image corresponding to the target shooting scene containing the transparent object can be constructed more accurately, thereby effectively improving the 3D imaging effect of the transparent object.

[0087] Optionally, the global optimization module 440 includes:

[0088] The target optimization function construction unit is used to construct a target optimization function corresponding to the target depth map based on the first depth map, the transparent object segmentation result, the boundary prediction result, and the normal vector prediction result, with the target depth map as the optimization object. The target depth map is a depth map obtained after completing the depth information of the first depth map.

[0089] The objective optimization function solving unit is used to minimize the objective optimization function and determine the target depth map corresponding to the minimum solution as the optimized second depth map.

[0090] Optionally, the objective optimization function building unit includes:

[0091] The depth deviation sub-function construction sub-unit is used to construct the depth deviation sub-function corresponding to the target depth map based on the first depth map, with the target depth map as the optimization object;

[0092] The normal vector deviation sub-function construction unit is used to construct the normal vector deviation sub-function corresponding to the target depth map based on the boundary prediction result and the normal vector prediction result;

[0093] A depth smoothing sub-function construction unit is used to construct the depth smoothing sub-function corresponding to the target depth map based on the boundary prediction results;

[0094] The objective optimization function construction sub-unit is used to construct the objective optimization function corresponding to the objective depth map based on the depth deviation sub-function, the normal vector deviation sub-function, and the depth smoothing sub-function.

[0095] Optionally, the depth deviation sub-function constructs sub-units, specifically for:

[0096] Obtain the first depth value corresponding to each first pixel in the first depth map, wherein the first pixel is any pixel in the first depth map other than the transparent object pixel; determine the depth deviation between the first depth value and the optimized depth value corresponding to the same first pixel in the first depth map and the target depth map; construct the depth deviation sub-function corresponding to the target depth map based on each of the depth deviations.

[0097] Optionally, the normal vector deviation sub-function constructs sub-units, specifically used for:

[0098] Obtain the boundary pixels in the boundary prediction result; determine the first normal vector corresponding to each second pixel in the target depth map, wherein the second pixel is any pixel in the target depth map other than the boundary pixels; obtain the second normal vector corresponding to each second pixel in the normal vector prediction result; determine the vector angle corresponding to each second pixel based on the first normal vector and the second normal vector corresponding to each second pixel; construct the normal vector deviation sub-function corresponding to the target depth map according to each vector angle.

[0099] Optionally, the normal vector deviation sub-function constructs sub-units and is also specifically used for:

[0100] Based on the pixel position of each second pixel in the target depth map and the pixel positions of two adjacent pixels in each of two adjacent directions, the normal vector corresponding to each second pixel is determined, wherein the adjacent directions include four directions: up, down, left, and right; the normal vectors are averaged to determine the first normal vector corresponding to each second pixel.

[0101] Optionally, the depth smoothing sub-function constructs sub-units, specifically for:

[0102] Obtain the boundary pixels in the boundary prediction result; determine the change depth corresponding to each second pixel based on the optimized depth value corresponding to each second pixel in the target depth map and the adjacent depth values ​​corresponding to adjacent pixels in each adjacent direction; wherein, the second pixel is other pixels in the target depth map besides the boundary pixels; construct the depth smoothing sub-function corresponding to the target depth map based on the change depth corresponding to each second pixel in each adjacent direction.

[0103] Optionally, the preset image processing model includes: an encoding sub-model, a first decoding branch sub-model, a second decoding branch sub-model, and a third decoding branch sub-model;

[0104] The target color image input module 420 is specifically used for: inputting the target color image into the encoding sub-model for feature extraction to obtain extracted target image feature information; inputting the target image feature information into the first decoding branch sub-model for position prediction of transparent objects to determine the transparent object segmentation result; inputting the target image feature information into the second decoding branch sub-model for boundary prediction of transparent and non-transparent objects to determine the boundary prediction result; and inputting the target image feature information into the third decoding branch sub-model for prediction of the normal vector corresponding to the pixel position to determine the normal vector prediction result.

[0105] The three-dimensional imaging device provided in the embodiments of the present invention can execute the three-dimensional imaging method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the three-dimensional imaging method.

[0106] It is worth noting that in the embodiments of the above-mentioned three-dimensional imaging device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy distinction between each other and are not used to limit the scope of protection of the present invention.

[0107] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Figure 5 A block diagram is shown of an exemplary electronic device 12 suitable for implementing embodiments of the present invention. Figure 5 The electronic device 12 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0108] like Figure 5 As shown, the electronic device 12 is represented in the form of a general-purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0109] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0110] Electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0111] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (… Figure 5 Not shown; usually referred to as a "hard drive"). Although Figure 5Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0112] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of the present invention.

[0113] Electronic device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with electronic device 12, and / or with any device that enables electronic device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 22. Furthermore, electronic device 12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20. As shown, network adapter 20 communicates with other modules of electronic device 12 via bus 18. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0114] Processing unit 16 executes various functional applications and data processing by running programs stored in system memory 28, such as implementing the steps of a three-dimensional imaging method provided in this embodiment, the method including:

[0115] Acquire a target color image and original depth map corresponding to the target shooting scene, wherein the target shooting scene contains at least one transparent object;

[0116] The target color image is input into a preset image processing model for image processing, and based on the output of the preset image processing model, the transparent object segmentation result, boundary prediction result, and normal vector prediction result in the target shooting scene are obtained.

[0117] Based on the transparent object segmentation results, the original depth map is cropped to obtain a first depth map with the transparent object depth information removed.

[0118] Based on the first depth map, the transparent object segmentation result, the boundary prediction result, and the normal vector prediction result, global optimization of depth information is performed to determine the optimized second depth map;

[0119] Based on the second depth map, the target 3D image corresponding to the target shooting scene is determined.

[0120] Of course, those skilled in the art will understand that the processor can also implement the technical solutions of the three-dimensional imaging method provided in any embodiment of the present invention.

[0121] This embodiment provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the program implements the steps of the three-dimensional imaging method provided in any embodiment of the present invention. The method includes:

[0122] Acquire a target color image and original depth map corresponding to the target shooting scene, wherein the target shooting scene contains at least one transparent object;

[0123] The target color image is input into a preset image processing model for image processing, and based on the output of the preset image processing model, the transparent object segmentation result, boundary prediction result, and normal vector prediction result in the target shooting scene are obtained.

[0124] Based on the transparent object segmentation results, the original depth map is cropped to obtain a first depth map with the transparent object depth information removed.

[0125] Based on the first depth map, the transparent object segmentation result, the boundary prediction result, and the normal vector prediction result, global optimization of depth information is performed to determine the optimized second depth map;

[0126] Based on the second depth map, the target 3D image corresponding to the target shooting scene is determined.

[0127] The computer storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. For example, a computer-readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0128] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0129] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0130] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0131] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computing device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0132] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A method of three-dimensional imaging, characterized by, include: Acquire a target color image and original depth map corresponding to the target shooting scene, wherein the target shooting scene contains at least one transparent object; The target color image is input into a preset image processing model for image processing, and based on the output of the preset image processing model, the transparent object segmentation result, boundary prediction result, and normal vector prediction result in the target shooting scene are obtained. Based on the transparent object segmentation results, the original depth map is cropped to obtain a first depth map with the transparent object depth information removed. Using the target depth map as the optimization object, a depth deviation sub-function corresponding to the target depth map is constructed based on the first depth map. The target depth map is the depth map obtained after the first depth map is filled with depth information. The depth deviation sub-function is used to characterize the depth deviation loss of the first pixel. The first pixel is the other pixel in the first depth map except for the transparent object pixel. Based on the boundary prediction result and the normal vector prediction result, a normal vector deviation sub-function corresponding to the target depth map is constructed. The normal vector deviation sub-function is used to characterize the normal vector deviation loss of the second pixel point, which is the other pixel point in the target depth map besides the boundary pixel point. Based on the boundary prediction results, a depth smoothing sub-function corresponding to the target depth map is constructed. The depth smoothing sub-function is used to characterize the depth smoothing loss of the second pixel. Based on the depth deviation sub-function, the normal vector deviation sub-function, and the depth smoothing sub-function, a target optimization function corresponding to the target depth map is constructed. The objective optimization function is minimized, and the target depth map corresponding to the minimum solution is determined as the optimized second depth map. Based on the second depth map, the target 3D image corresponding to the target shooting scene is determined.

2. The method of claim 1, wherein, The step of constructing the depth deviation sub-function corresponding to the target depth map based on the first depth map includes: Obtain the first depth value corresponding to each first pixel in the first depth map, wherein the first pixel is any pixel in the first depth map other than the transparent object pixel; In the first depth map and the target depth map, the depth deviation between the first depth value and the optimized depth value corresponding to the same first pixel is determined; Based on each of the aforementioned depth deviations, a depth deviation sub-function corresponding to the target depth map is constructed.

3. The method of claim 1, wherein, The step of constructing the normal vector deviation sub-function corresponding to the target depth map based on the boundary prediction result and the normal vector prediction result includes: Obtain the boundary pixels in the boundary prediction result; Determine the first normal vector corresponding to each second pixel in the target depth map, wherein the second pixel is any pixel in the target depth map other than the boundary pixel; Obtain the second normal vector corresponding to each second pixel in the normal vector prediction result; Based on the first normal vector and the second normal vector corresponding to each second pixel, determine the vector angle corresponding to each second pixel; Based on the included angles of each vector, a normal vector deviation sub-function corresponding to the target depth map is constructed.

4. The method of claim 3, wherein, Determining the first normal vector corresponding to each second pixel in the target depth map includes: Based on the pixel position of each second pixel in the target depth map and the pixel positions of the two adjacent pixels in each of the two adjacent directions, the normal vector corresponding to each second pixel is determined, wherein the adjacent directions include the four directions of up, down, left, and right. The first normal vector corresponding to each second pixel is determined by averaging the values ​​of each normal vector.

5. The method of claim 1, wherein, The step of constructing a depth smoothing sub-function corresponding to the target depth map based on the boundary prediction results includes: Obtain the boundary pixels in the boundary prediction result; Based on the optimized depth value corresponding to each second pixel in the target depth map and the adjacent depth values ​​corresponding to adjacent pixels in each adjacent direction, the changing depth corresponding to each second pixel is determined; wherein, the second pixel is any pixel in the target depth map other than the boundary pixel; Based on the changing depth corresponding to each of the second pixels in each adjacent direction, a depth smoothing sub-function corresponding to the target depth map is constructed.

6. The method according to any one of claims 1 to 5, characterized in that, The preset image processing model includes: an encoding sub-model, a first decoding branch sub-model, a second decoding branch sub-model, and a third decoding branch sub-model; The step of inputting the target color image into a preset image processing model for image processing includes: The target color image is input into the coding sub-model for feature extraction to obtain the extracted target image feature information; The target image feature information is input into the first decoding branch sub-model to predict the position of the transparent object and determine the transparent object segmentation result. The target image feature information is input into the second decoding branch sub-model to predict the boundaries of transparent and non-transparent objects, and the boundary prediction result is determined. The target image feature information is input into the third decoding branch sub-model to predict the normal vector corresponding to the pixel position, and the normal vector prediction result is determined.

7. A three-dimensional imaging device, characterized in that, include: An image acquisition module is used to acquire a target color image and an original depth map corresponding to a target shooting scene, wherein the target shooting scene contains at least one transparent object; The target color image input module is used to input the target color image into a preset image processing model for image processing, and obtain the transparent object segmentation result, boundary prediction result and normal vector prediction result in the target shooting scene based on the output of the preset image processing model; The shearing module is used to shear the original depth map based on the transparent object segmentation result to obtain a first depth map with the transparent object depth information removed. A global optimization module is used to optimize a target depth map. Based on the first depth map, it constructs a depth deviation sub-function corresponding to the target depth map, where the target depth map is obtained by completing the depth information of the first depth map. The depth deviation sub-function characterizes the depth deviation loss of a first pixel, which is any pixel in the first depth map except for transparent object pixels. Based on the boundary prediction result and the normal vector prediction result, it constructs a normal vector deviation sub-function corresponding to the target depth map, which characterizes the normal vector deviation loss of a second pixel, which is any pixel in the target depth map except for boundary pixels. Based on the boundary prediction result, it constructs a depth smoothing sub-function corresponding to the target depth map, which characterizes the depth smoothing loss of the second pixel. Based on the depth deviation sub-function, the normal vector deviation sub-function, and the depth smoothing sub-function, it constructs a target optimization function corresponding to the target depth map. It then minimizes the target optimization function and determines the target depth map corresponding to the minimum solution as the optimized second depth map. The target 3D image determination module is used to determine the target 3D image corresponding to the target shooting scene based on the second depth map.

8. An electronic device, characterized in that, The electronic device includes: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the three-dimensional imaging method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the three-dimensional imaging method as described in any one of claims 1-6.

Citation Information

Patent Citations

  • Transparent object grabbing control method and device, terminal and storage medium

    CN113160313A