Method, apparatus and electronic device for updating 3D model

Abstract

The present disclosure relates to a method and device for updating a 3D model and an electronic device, and particularly relates to the technical field of image processing. The method comprises: obtaining a first multi-view image of a target frame; determining a target estimated depth image of the target frame according to the first multi-view image, a second multi-view image of a previous frame of the target frame, and a target 3D model; and updating the target 3D model according to the target estimated depth image. The target 3D model is a 3D model established or updated based on at least two continuous multi-view images obtained before the first multi-view image, and the second multi-view image is included in the at least two continuous multi-view images. The embodiments of the present disclosure can update the 3D model without increasing the cost and power consumption of the device.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more particularly to a method, apparatus, and electronic device for updating 3D models. Background Technology

[0002] In some scenarios, electronic devices need to be able to recreate 3D models of the real world in real time. Currently, 3D models can be created and updated by using depth sensors. However, for head-mounted displays, mobile platforms such as smartphones, using depth sensors would be too costly and consume too much power. Therefore, there is an urgent need for a method to update 3D models without using depth sensors. Summary of the Invention

[0003] To solve the above-mentioned technical problems, or at least partially solve them, this disclosure provides a method, apparatus, and electronic device for updating 3D models, which can update 3D models without increasing equipment costs and power consumption.

[0004] To achieve the above objectives, the technical solutions provided by the embodiments of this disclosure are as follows:

[0005] Firstly, a method for updating 3D models is provided, including:

[0006] Acquire the first multiview image of the target frame;

[0007] Based on the first multi-view image, the second multi-view image of the previous frame of the target frame, and the target 3D model, determine the target estimated depth image of the target frame;

[0008] Update the target 3D model based on the target's estimated depth image;

[0009] The target 3D model is a 3D model established or updated based on at least two consecutive frames of multi-view images acquired before the first multi-view image, and the at least two frames of multi-view images include the second multi-view image.

[0010] As an optional implementation of this disclosure, acquiring the first multi-view image of the target frame includes:

[0011] The first multi-view image is the original multi-view image acquired by a multi-view image sensor;

[0012] or,

[0013] The first multiview image is a downsampled image of the original multiview image.

[0014] As an optional implementation of this disclosure, determining the target estimated depth image of the target frame based on the first multi-view image, the second multi-view image of the previous frame of the target frame, and the target 3D model includes:

[0015] Determine the first depth image of the target frame based on the first multi-view image;

[0016] The second multi-view image is projected onto the target 3D model to determine the first estimated multi-view image and the first estimated depth image of the target frame;

[0017] The depth value of the first pixel is removed from the first estimated depth image to obtain the target estimated depth image. The first pixel includes: pixels in the first estimated depth image and the first depth image whose depth difference is greater than a first depth difference threshold, and / or pixels in the first estimated multi-view image and the first multi-view image whose pixel difference is greater than a first pixel difference threshold.

[0018] As an optional implementation of this disclosure, the step of removing the depth value of the first pixel from the first estimated depth image to obtain the target estimated depth image includes:

[0019] The depth value of the first pixel is removed from the first estimated depth image to obtain the second estimated depth image;

[0020] Depth completion is performed on the second estimated depth image to obtain the target estimated depth image.

[0021] As an optional implementation of this disclosure, updating the target 3D model based on the target estimated depth image includes:

[0022] The target 3D model is projected onto the target frame to obtain a mask image. The mask image includes a covered area and an uncovered area. The covered area is the region where the pixel coordinates corresponding to the existing 3D points in the target 3D model are located. The uncovered area is the region in the mask image other than the covered area.

[0023] Obtain depth information corresponding to the covered area from the target estimated depth image; update existing 3D points in the target 3D model based on the depth information corresponding to the covered area; and / or obtain depth information corresponding to the uncovered area from the target estimated depth image; supplement and establish new 3D points based on the depth information corresponding to the uncovered area and the first multi-view image.

[0024] As an optional implementation of this disclosure, the step of supplementing the target 3D model with new 3D points based on the depth information corresponding to the non-covered area and the first multi-view image includes:

[0025] Determine the region to be processed in the first multi-view image that corresponds to the non-covered region;

[0026] A second pixel is determined from the region to be processed. The second pixel includes: a pixel in the region to be processed whose pixel difference with its neighboring pixel is greater than a second pixel difference threshold, and / or a pixel in the region to be processed that corresponds to a first target pixel in the target estimated depth image. The first target pixel is a pixel whose depth difference with its neighboring pixel is greater than a second depth difference threshold.

[0027] Based on the second pixel, the area to be processed is triangulated to obtain several first triangular meshes;

[0028] Based on the target 3D model, spatial mapping is performed on the several first triangular meshes based on the depth information corresponding to the non-covered areas, so as to supplement and establish new 3D points on the basis of the target 3D model.

[0029] As an optional implementation of this disclosure, before determining the target estimated depth image of the target frame based on the first multi-view image, the second multi-view image of the previous frame of the target frame, and the target 3D model, the method further includes:

[0030] A third estimated depth image is determined based on the at least two frames of multi-view images;

[0031] Based on the third pixel, the last multi-view image in the at least two multi-view images is triangulated to obtain several second triangular meshes. Based on the third estimated depth image, spatial mapping is performed on the several second triangular meshes to obtain the target 3D model.

[0032] The third pixel includes: a pixel in the last frame of the multi-view image whose pixel difference with its neighboring pixel is greater than a second pixel difference threshold, and / or a pixel in the last frame of the multi-view image that corresponds to a second target pixel in the third estimated depth image, wherein the second target pixel is a pixel in the third estimated depth image whose depth difference with its neighboring pixel is greater than a second depth difference threshold.

[0033] As an optional implementation of this disclosure, determining the third estimated depth image based on the at least two multi-view images includes:

[0034] Based on the at least two multi-view images, a fourth estimated depth image is determined; the overlapping region between the fourth estimated depth image and the earliest acquired depth image has a size ratio of less than or equal to a preset ratio relative to the size of the fourth estimated depth image; the earliest acquired depth image is the depth image corresponding to the earliest acquired multi-view image among the at least two multi-view images.

[0035] The depth values ​​corresponding to the repeated regions in the fourth estimated depth image are projected onto the latest frame to obtain the fifth estimated depth image.

[0036] The depth value of the fourth pixel is removed from the fifth estimated depth image, and depth completion is performed to obtain the third estimated depth image; the fourth pixel includes: the pixel whose depth difference between the fifth estimated depth image and the latest frame depth image is greater than the third depth difference threshold, and the latest frame depth image is the depth map corresponding to the multi-view image at the end of the time series among the at least two multi-view images.

[0037] As an optional implementation of this disclosure, projecting the depth value corresponding to the repeating region in the fourth estimated depth image onto the latest frame includes:

[0038] The depth values ​​corresponding to the repeated regions in the fourth estimated depth image are projected onto the depth map in the middle of the time series, and depth completion is performed to obtain the intermediate depth image; wherein, the depth map in the middle of the time series is the depth map corresponding to the multi-view image in the middle of the time series among the at least two frames of multi-view images;

[0039] The intermediate depth map is projected onto the latest frame to obtain the fifth estimated depth image.

[0040] As an optional implementation of this disclosure, determining the fourth estimated depth image based on the at least two multi-view images includes:

[0041] Step 1: Obtain the multi-view image of the current frame, and calculate the second depth image of the multi-view image of the current frame based on the multi-view image of the current frame;

[0042] Step 2: Obtain the previous frame multiview image and the third depth image of the previous frame multiview image of the current frame;

[0043] Step 3: Based on the third depth map, project the previous frame multi-view image onto the current frame to obtain the estimated binocular image of the current frame and the sixth estimated depth image of the current frame;

[0044] Step 4: Remove the depth value of the third pixel from the sixth estimated depth image of the current frame to obtain the seventh estimated depth image of the current frame; wherein, the third pixel includes: pixels in the fourth estimated depth image and the third depth image whose depth difference is greater than the second depth difference threshold, and / or pixels in the estimated multi-view image and the multi-view image of the current frame whose pixel difference is greater than the second pixel difference threshold;

[0045] Step 5: Determine the overlapping region between the seventh estimated depth image and the earliest acquired depth image;

[0046] Repeat steps 1 to 5 until the size ratio of the repeated region relative to the seventh estimated depth image is less than or equal to a preset ratio, then the seventh estimated image is determined as the fourth estimated depth image.

[0047] Secondly, a device for updating a 3D model is provided, comprising:

[0048] The acquisition module is used to acquire the first multi-view image of the target frame;

[0049] The depth estimation module is used to determine the target estimated depth image of the target frame based on the first multi-view image, the second multi-view image of the previous frame of the target frame, and the target 3D model;

[0050] An update module is used to update the target 3D model based on the target's estimated depth image.

[0051] As an optional implementation of this disclosure, the first multi-view image is an original multi-view image acquired by a multi-view image sensor.

[0052] As an optional implementation of this disclosure, the first multi-view image is a downsampled image of the original multi-view image.

[0053] As an optional implementation of this disclosure, the depth estimation module is specifically used for:

[0054] Determine the first depth image of the target frame based on the first multi-view image;

[0055] The second multi-view image is projected onto the target 3D model to determine the first estimated multi-view image and the first estimated depth image of the target frame;

[0056] The depth value of the first pixel is removed from the first estimated depth image to obtain the target estimated depth image. The first pixel includes: pixels in the first estimated depth image and the first depth image whose depth difference is greater than a first depth difference threshold, and / or pixels in the first estimated multi-view image and the first multi-view image whose pixel difference is greater than a first pixel difference threshold.

[0057] As an optional implementation of this disclosure, the depth estimation module is specifically used for:

[0058] The depth value of the first pixel is removed from the first estimated depth image to obtain the second estimated depth image;

[0059] Depth completion is performed on the second estimated depth image to obtain the target estimated depth image.

[0060] As an optional implementation of this disclosure, the updating module is specifically used for:

[0061] The target 3D model is projected onto the target frame to obtain a mask image. The mask image includes a covered area and an uncovered area. The covered area is the region where the pixel coordinates corresponding to the existing 3D points in the target 3D model are located. The uncovered area is the region in the mask image other than the covered area.

[0062] Obtain depth information corresponding to the covered area from the target estimated depth image; update existing 3D points in the target 3D model based on the depth information corresponding to the covered area; and / or obtain depth information corresponding to the uncovered area from the target estimated depth image; supplement and establish new 3D points based on the depth information corresponding to the uncovered area and the first multi-view image.

[0063] As an optional implementation of this disclosure, the updating module is specifically used for:

[0064] Determine the region to be processed in the first multi-view image that corresponds to the non-covered region;

[0065] A second pixel is determined from the region to be processed. The second pixel includes: a pixel in the region to be processed whose pixel difference with its neighboring pixel is greater than a second pixel difference threshold, and / or a pixel in the region to be processed that corresponds to a first target pixel in the target estimated depth image. The first target pixel is a pixel whose depth difference with its neighboring pixel is greater than a second depth difference threshold.

[0066] Based on the second pixel, the area to be processed is triangulated to obtain several first triangular meshes;

[0067] Based on the target 3D model, spatial mapping is performed on the several first triangular meshes based on the depth information corresponding to the non-covered areas, so as to supplement and establish new 3D points on the basis of the target 3D model.

[0068] As an optional implementation of this disclosure, the apparatus further includes: a creation module, configured to:

[0069] Before determining the target estimated depth image of the target frame based on the first multi-view image, the second multi-view image of the previous frame of the target frame, and the target 3D model, a third estimated depth image is determined based on the at least two multi-view images;

[0070] Based on the third pixel, the last multi-view image in the at least two multi-view images is triangulated to obtain several second triangular meshes. Based on the third estimated depth image, spatial mapping is performed on the several second triangular meshes to obtain the target 3D model.

[0071] The third pixel includes: a pixel in the last frame of the multi-view image whose pixel difference with its neighboring pixel is greater than a second pixel difference threshold, and / or a pixel in the last frame of the multi-view image that corresponds to a second target pixel in the third estimated depth image, wherein the second target pixel is a pixel in the third estimated depth image whose depth difference with its neighboring pixel is greater than a second depth difference threshold.

[0072] As an optional implementation of this disclosure, the creation module is specifically used for:

[0073] Based on the at least two multi-view images, a fourth estimated depth image is determined; the overlapping region between the fourth estimated depth image and the earliest acquired depth image has a size ratio of less than or equal to a preset ratio relative to the size of the fourth estimated depth image; the earliest acquired depth image is the depth image corresponding to the earliest acquired multi-view image among the at least two multi-view images.

[0074] The depth values ​​corresponding to the repeated regions in the fourth estimated depth image are projected onto the latest frame to obtain the fifth estimated depth image.

[0075] The depth value of the fourth pixel is removed from the fifth estimated depth image, and depth completion is performed to obtain the third estimated depth image; the fourth pixel includes: the pixel whose depth difference between the fifth estimated depth image and the latest frame depth image is greater than the third depth difference threshold, and the latest frame depth image is the depth map corresponding to the multi-view image at the end of the time series among the at least two multi-view images.

[0076] As an optional implementation of this disclosure, the creation module is specifically used for:

[0077] The depth values ​​corresponding to the repeated regions in the fourth estimated depth image are projected onto the depth map in the middle of the time series, and depth completion is performed to obtain the intermediate depth image; wherein, the depth map in the middle of the time series is the depth map corresponding to the multi-view image in the middle of the time series among the at least two frames of multi-view images;

[0078] The intermediate depth map is projected onto the latest frame to obtain the fifth estimated depth image.

[0079] As an optional implementation of this disclosure, the creation module is specifically used for:

[0080] Step 1: Obtain the multi-view image of the current frame, and calculate the second depth image of the multi-view image of the current frame based on the multi-view image of the current frame;

[0081] Step 2: Obtain the previous frame multiview image and the third depth image of the previous frame multiview image of the current frame;

[0082] Step 3: Based on the third depth map, project the previous frame multi-view image onto the current frame to obtain the estimated binocular image of the current frame and the sixth estimated depth image of the current frame;

[0083] Step 4: Remove the depth value of the third pixel from the sixth estimated depth image of the current frame to obtain the seventh estimated depth image of the current frame; wherein, the third pixel includes: pixels in the fourth estimated depth image and the third depth image whose depth difference is greater than the second depth difference threshold, and / or pixels in the estimated multi-view image and the multi-view image of the current frame whose pixel difference is greater than the second pixel difference threshold;

[0084] Step 5: Determine the overlapping region between the seventh estimated depth image and the earliest acquired depth image;

[0085] Repeat steps 1 to 5 until the size ratio of the repeated region relative to the seventh estimated depth image is less than or equal to a preset ratio, then the seventh estimated image is determined as the fourth estimated depth image.

[0086] Thirdly, an electronic device is provided, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the method for updating a 3D model as described in the first aspect or any alternative embodiment thereof.

[0087] Fourthly, a computer-readable storage medium is provided, comprising: storing a computer program on the computer-readable storage medium, wherein the computer program, when executed by a processor, implements the method for updating a 3D model as described in the first aspect or any alternative embodiment thereof.

[0088] Fifthly, a computer program product is provided, characterized in that it includes: when the computer program product is run on a computer, causing the computer to implement the method for updating a 3D model as described in the first aspect or any alternative embodiment thereof.

[0089] In this embodiment, a first multi-view image of the target frame can be acquired. Based on the first multi-view image, a second multi-view image of the previous frame of the target frame, and the target 3D model, a target estimated depth image of the target frame is determined. The target 3D model is then updated based on the target estimated depth image. The target 3D model is a 3D model established or updated based on at least two consecutive multi-view images acquired before the first multi-view image, including the second multi-view image. This scheme allows for depth estimation during target 3D model updates. Depth estimation is performed using the second multi-view image of the previous frame of the target frame, the target 3D model, and the first multi-view image of the target frame to obtain the target estimated depth image corresponding to the target frame. The target 3D model is then updated based on this estimated depth image. This method of updating the 3D model does not require a depth sensor and can achieve 3D model updates without increasing equipment cost or power consumption. Attached Figure Description

[0090] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0091] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0092] Figure 1 A flowchart illustrating a method for updating a 3D model provided in this disclosure embodiment. Figure 1 ;

[0093] Figure 2 A flowchart illustrating a method for updating a 3D model provided in this disclosure embodiment. Figure 2 ;

[0094] Figure 3 A structural block diagram of an apparatus for updating a 3D model provided in an embodiment of this disclosure;

[0095] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0096] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0097] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0098] Currently, some electronic devices require the ability to recreate 3D models of the real world in real time. While depth sensors can be used to create and update these models, this presents challenges for mobile platforms such as head-mounted displays and smartphones, leading to excessive costs and power consumption. Therefore, a method that eliminates the need for depth sensors is urgently needed for updating 3D models.

[0099] To address the aforementioned issues, this disclosure provides a method, apparatus, and electronic device for updating 3D models, which can update 3D models without increasing equipment costs or power consumption.

[0100] In this embodiment of the disclosure, the method for updating a 3D model can be implemented by a device or electronic device for updating the 3D model. The device for updating the 3D model can be a functional module or functional entity within an electronic device used to implement the method for updating the 3D model. The aforementioned electronic devices include: computers, tablet computers, mobile phones, laptops, handheld computers, vehicle terminals, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), personal computers (PCs), etc., and this embodiment of the disclosure does not specifically limit the types of devices used.

[0101] This disclosure involves the process of creating and updating a 3D model, which will be described separately below.

[0102] like Figure 1 The illustration shows a method for updating a 3D model according to an embodiment of this disclosure. Figure 1 The method includes the following steps:

[0103] S101. Obtain the first multi-view image of the target frame.

[0104] In this embodiment of the disclosure, the target frame can refer to any frame.

[0105] In some embodiments, the first multi-view image of the target frame may refer to an image acquired by a multi-view camera at the time indicated by the target frame. For example, the target frame is the current frame in real time, and the first multi-view image may be a first multi-view image acquired in real time by a device for updating the 3D model.

[0106] The device for updating the 3D model is equipped with multiple cameras (i.e., multi-view cameras), or the device for updating the 3D model is connected to multiple cameras, and the multiple cameras acquire multi-view images.

[0107] The first multi-view image may include images acquired by at least two cameras respectively. In a scenario where images are acquired by a binocular camera, the first multi-view image is a binocular image, including images acquired by the two cameras in the binocular camera.

[0108] In some embodiments, the first multiview image is a raw multiview image acquired by a multiview image sensor (such as the multiple cameras described above).

[0109] In some embodiments, the first multiview image is a downsampled image of the original multiview image.

[0110] In some embodiments, when acquiring the first multi-view image of a target frame, the original multi-view image of the target frame can be acquired first, and then downsampled to obtain the first multi-view image. That is, the original multi-view image is an image acquired by a multi-view camera, and the first multi-view image is obtained by downsampling the image acquired by the multi-view camera.

[0111] Image downsampling refers to reducing the resolution of an image. For example, it can be to reduce the resolution of a 1024*1024 image to 128*128 before processing.

[0112] In the above embodiments, the image is downsampled before processing, which can reduce the amount of computation in the image processing process and improve computational efficiency.

[0113] S102. Based on the first multi-view image, the second multi-view image of the previous frame of the target frame, and the target 3D model, determine the target estimated depth image of the target frame.

[0114] The target 3D model is a 3D model established or updated based on at least two consecutive frames of multi-view images acquired before the first multi-view image. The 3D model is a triangular mesh model, and the at least two frames of multi-view images include a second multi-view image.

[0115] For example, in the case where at least two multiview images refer to two multiview images, the target 3D model can be a 3D model built or updated based on the second multiview image and the previous multiview image of the second multiview image.

[0116] In some embodiments, in the process of determining the target estimated depth image of a target frame based on a first multi-view image, a second multi-view image of the previous frame of the target frame, and a target 3D model, firstly, a first depth image of the target frame can be determined based on the first multi-view image; then, the second multi-view image is projected onto the target 3D model to determine the first estimated multi-view image and the first estimated depth image of the target frame; subsequently, a first pixel can be determined based on the first estimated multi-view image and the first estimated depth image; finally, the depth value of the first pixel is removed from the first estimated depth image to obtain the target estimated depth image.

[0117] The first pixel includes: a pixel in the first estimated depth image whose depth difference with the first depth image is greater than a first depth difference threshold, and / or a pixel in the first estimated multi-view image whose pixel difference with the first multi-view image is greater than a first pixel difference threshold.

[0118] It can be seen that the first pixel point mentioned above is the first estimated depth map of the estimated target frame. Some points that are significantly different from the first multi-view image and / or the first depth image may have certain estimation errors.

[0119] In the above embodiments, after obtaining the first estimated depth map of the target frame, the first pixel can be determined based on the first estimated multi-view image and / or the first estimated depth image. After removing the first pixel from the first estimated depth map, the final target estimated depth map used to update the target 3D model is obtained. This can remove some points with large differences from the first multi-view image and / or the first depth image corresponding to the target frame in the estimated first estimated depth map of the target frame, avoiding these points from causing errors in the subsequent update of the target 3D model and ensuring the accuracy of the subsequent update of the target 3D model.

[0120] In some embodiments, the method of determining the first depth image of the target frame based on the first multi-view image may be to calculate the first depth image of the target frame based on the first multi-view image and a multi-view stereo vision algorithm.

[0121] Since the first multi-view image includes images of the same scene taken by a multi-view camera from different perspectives, a multi-view stereo vision algorithm can be used to calculate the disparity map of the images from different perspectives in the first multi-view image, and then the first depth image can be obtained based on the disparity map.

[0122] Among them, the aforementioned multi-view stereo vision algorithm may include, but is not limited to, at least one of the following algorithms: the sum of absolute differences (SAD) algorithm for corresponding pixel differences in the image sequence, the sum of squared differences (SSD) algorithm for corresponding pixel differences, and the semi-global matching algorithm (SGM), etc.

[0123] In some embodiments, the method of projecting the second multi-view image onto the target 3D model to determine the first estimated multi-view image and the first estimated depth image of the target frame may include, but is not limited to: first obtaining the camera pose corresponding to the previous frame of the target frame and the camera pose of the target frame, then combining the camera pose corresponding to the previous frame of the target frame and the camera pose of the target frame to determine pose change information, and determining the first estimated depth image and the first estimated multi-view image corresponding to the camera pose of the target frame based on the pose change information, the target 3D model, and the second multi-view image.

[0124] In some embodiments, the method of removing the depth value of the first pixel from the first estimated depth image to obtain the target estimated depth image may include, but is not limited to: removing the depth value of the first pixel from the first estimated depth image to obtain the second estimated depth image; and performing depth completion on the second estimated depth image to obtain the target estimated depth image.

[0125] Furthermore, after removing the depth value of the first pixel from the first depth image to obtain the second estimated depth image, depth completion is performed on the second estimated depth image. A depth completion algorithm is used to determine the depth value of the first pixel. This way, a target estimated depth image containing complete depth information can be obtained. In subsequent new target 3D models, there will be no omissions, and a more complete target 3D model can be updated.

[0126] S103. Update the target 3D model based on the target estimated depth image.

[0127] In some embodiments, a target 3D model is projected onto a target frame to obtain a mask image, which includes covered and uncovered areas.

[0128] The covered area is the region where the pixel coordinates of the existing 3D points in the target 3D model are located, and the uncovered area is the region in the mask image other than the covered area.

[0129] In one embodiment, after the covered and uncovered areas, depth information corresponding to the covered area can be determined from the target estimated depth image; based on the depth information corresponding to the covered area, existing 3D points in the target 3D model are updated.

[0130] In this process, updating the existing 3D points in the target 3D model based on the depth information corresponding to the coverage area can be achieved by adjusting the spatial position of the 3D points in the target 3D model corresponding to each pixel in the coverage area based on the depth value of each pixel in the coverage area.

[0131] In another embodiment, after the covered and uncovered areas, depth information corresponding to the uncovered area can be obtained from the target estimated depth image; based on the depth information corresponding to the uncovered area and the first multi-view image, new 3D points are supplemented and established on the target 3D model.

[0132] In another embodiment, after determining the covered and uncovered areas, depth information corresponding to the covered area can be obtained from the target estimated depth image; based on the depth information corresponding to the covered area, existing 3D points in the target 3D model are updated, and depth information corresponding to the uncovered area can be obtained from the target estimated depth image; based on the depth information corresponding to the uncovered area and the first multi-view image, new 3D points are supplemented and established on the basis of the target 3D model.

[0133] In some embodiments, the method of supplementing and establishing new 3D points on the basis of the target 3D model based on the depth information corresponding to the non-covered area and the first multi-view image may include, but is not limited to: determining the region to be processed in the first multi-view image corresponding to the non-covered area; determining a second pixel point from the region to be processed; triangulating the region to be processed based on the second pixel point to obtain a plurality of first triangular meshes; and performing spatial mapping on the plurality of first triangular meshes based on the depth information corresponding to the non-covered area on the basis of the target 3D model to supplement and establish new 3D points on the basis of the target 3D model.

[0134] The aforementioned depth information may include depth values, and the second pixel may include: a pixel in the region to be processed whose pixel difference with its neighboring pixel is greater than a second pixel difference threshold, and / or a pixel in the region to be processed that corresponds to a first target pixel in the target estimated depth image, wherein the first target pixel is a pixel whose depth difference with its neighboring pixel is greater than a second depth difference threshold.

[0135] The above-mentioned method of triangulating the region to be processed based on the second pixel to obtain several first triangular meshes may include, but is not limited to: determining the second pixel in the region to be processed, which may be multiple pixels, and connecting these multiple pixels to divide the region to be processed into multiple triangular images, namely, the above-mentioned several first triangular meshes.

[0136] Furthermore, the method of spatially mapping several first triangular meshes based on the depth information corresponding to the non-covered area to supplement and establish new 3D points on the basis of the target 3D model may include, but is not limited to: obtaining the depth information corresponding to the non-covered area, obtaining the depth value corresponding to the aforementioned second pixel point from the non-covered area, thus obtaining the depth value of the vertices of the aforementioned several first triangular meshes, and then spatially mapping several first triangular meshes based on the depth values ​​of the vertices of the several first triangular meshes to supplement and establish new 3D points on the basis of the target 3D model.

[0137] In this embodiment, a first multi-view image of the target frame can be acquired. Based on the first multi-view image, a second multi-view image of the previous frame of the target frame, and the target 3D model, a target estimated depth image of the target frame is determined. The target 3D model is then updated based on the target estimated depth image. The target 3D model is a 3D model established or updated based on at least two consecutive multi-view images acquired before the first multi-view image, including the second multi-view image. This scheme allows for depth estimation during target 3D model updates. Depth estimation is performed using the second multi-view image of the previous frame of the target frame, the target 3D model, and the first multi-view image of the target frame to obtain the target estimated depth image corresponding to the target frame. The target 3D model is then updated based on this estimated depth image. This method of updating the 3D model does not require a depth sensor and can achieve 3D model updates without increasing equipment cost or power consumption.

[0138] The method for updating a 3D model provided in this disclosure can first establish a target 3D model, and then, after establishing the target 3D model, use the method described above. Figure 1 The method shown updates the target 3D model in real time.

[0139] For example, such as Figure 2 The diagram illustrates a method for updating a 3D model according to an embodiment of this disclosure. The method includes a model creation stage and a model updating stage, implemented through the following steps S201 to S206:

[0140] The model building phase can be achieved through the following steps S201 to S203:

[0141] S201. Determine a third estimated depth image based on at least two multi-view images.

[0142] The at least two multi-view images are either two consecutive multi-view images or multiple consecutive multi-view images.

[0143] In some embodiments, the process of determining a third estimated depth image based on at least two multiview images includes the following steps 201a to 201c:

[0144] S201a First, a fourth estimated depth image needs to be determined based on at least two multi-view images.

[0145] Wherein, the overlapping region between the fourth estimated depth image and the earliest acquired depth image has a size ratio of less than or equal to a preset ratio relative to the fourth estimated depth image; the earliest acquired depth image is the depth image corresponding to the earliest acquired multi-view image among at least two multi-view images.

[0146] In the process of determining the fourth estimated depth image based on at least two multi-view images, it is necessary to first determine at least two corresponding depth images based on at least two multi-view images, and then determine the fourth estimated depth image from the at least two depth images whose overlapping region with the earliest acquired depth image has a size ratio of less than or equal to a preset ratio relative to the depth image size.

[0147] In some embodiments, the process of determining the fourth estimated depth image based on at least two frames of multi-view images can be achieved by repeatedly executing steps 1 to 5 below until the fourth estimated depth image is determined. The specific implementation process can be as follows:

[0148] Step 1: Obtain the multi-view image of the current frame, and calculate the second depth image of the multi-view image of the current frame based on the multi-view image of the current frame.

[0149] The above-mentioned calculation of the second depth image of the multi-view image of the current frame may include, but is not limited to, calculating the second depth image of the multi-view image of the current frame based on the multi-view image of the current frame using a multi-view stereo vision algorithm.

[0150] Step 2: Obtain the previous frame multiview image and the third depth image of the previous frame multiview image of the current frame.

[0151] The acquisition of the previous frame multiview image and the third depth image of the previous frame multiview image of the current frame may include, but is not limited to: first acquiring the previous frame multiview image of the current frame multiview image, and then calculating the third depth image of the previous frame multiview image based on the previous frame multiview image. Alternatively, the second depth image of the current frame multiview image may be calculated based on the previous frame multiview image using a multiview stereo vision algorithm. In some embodiments, the aforementioned current frame multiview image and previous frame multiview image may be multiview images obtained after downsampling.

[0152] Step 3: Based on the third depth map, project the multi-view image from the previous frame onto the current frame to obtain the estimated binocular image and the sixth estimated depth image of the current frame. The method described above, which projects the multi-view image from the previous frame onto the current frame based on the third depth map to obtain the estimated binocular image and the sixth estimated depth image of the current frame, includes: obtaining the camera pose corresponding to the multi-view image from the previous frame and the camera pose corresponding to the current frame; determining pose change information based on the two camera poses; and then estimating the estimated binocular image and the sixth estimated depth image of the current frame based on the pose change information, the third depth map, and the multi-view image from the previous frame.

[0153] Step 4: Remove the depth value of the third pixel from the sixth estimated depth image of the current frame to obtain the seventh estimated depth image of the current frame.

[0154] The third pixel includes: pixels in the fourth estimated depth image and the third depth image whose depth difference is greater than the second depth difference threshold, and / or pixels in the estimated multiview image and the current frame multiview image whose pixel difference is greater than the second pixel difference threshold.

[0155] Step 5: Identify the overlapping regions between the seventh estimated depth image and the earliest acquired depth image.

[0156] Repeat steps 1 to 5 until the size ratio of the repeated region relative to the seventh estimated depth image is less than or equal to a preset ratio, then the seventh estimated image is determined as the fourth estimated depth image.

[0157] In the above embodiments, during the process of determining the fourth estimated depth image based on at least two multi-view images, at least two corresponding depth images are first determined based on at least two multi-view images. Then, the fourth estimated depth image is determined from the at least two depth images where the size ratio of the overlapping region between the first and second obtained depth images is less than or equal to a preset ratio. This allows for the determination of the fourth estimated depth image, which differs significantly from the first obtained depth image due to large changes in camera pose. Since the depth values ​​of the overlapping regions in the fourth estimated depth image are relatively stable when the camera pose changes significantly, the 3D model established in this way requires fewer 3D points to be updated, which can reduce the computational load during subsequent 3D model updates and improve model update efficiency.

[0158] S201b: Project the depth values ​​corresponding to the repeated regions in the fourth estimated depth image onto the latest frame to obtain the fifth estimated depth image.

[0159] The method described above, which projects the depth values ​​corresponding to the repeated regions in the fourth estimated depth image onto the latest frame to obtain the fifth estimated depth image, first obtains the camera pose corresponding to the fourth estimated depth image and the camera pose corresponding to the latest frame. Then, based on the camera poses corresponding to the fourth estimated depth image and the latest frame, the pose change information is determined. Subsequently, based on the pose change information, the depth values ​​corresponding to the repeated regions in the fourth estimated depth image are updated to project the depth values ​​corresponding to the repeated regions in the fourth estimated depth image onto the latest frame to obtain the fifth estimated depth image.

[0160] S201c: Remove the depth value of the fourth pixel from the fifth estimated depth image and perform depth completion to obtain the third estimated depth image.

[0161] After obtaining the fifth estimated depth image, the depth map corresponding to the last multi-view image in the time series of at least two multi-view images is first determined, that is, the latest frame depth image. Then, the fourth pixel is determined based on the fifth estimated depth image and the latest frame depth image. Finally, the depth value of the fourth pixel is removed from the fifth estimated depth image, and depth completion is performed to obtain the third estimated depth image.

[0162] The fourth pixel may include: a pixel in the fifth estimated depth image whose depth difference with the latest frame depth image is greater than the third depth difference threshold.

[0163] In the above embodiments, pixels with large depth differences between the fifth estimated depth image and the latest frame depth image are removed. This avoids significant errors when using the depth values ​​of these pixels with large depth differences to build the 3D model. Furthermore, after removing the depth value of the fourth pixel from the fifth estimated depth image, depth completion is performed to obtain the third estimated depth image. Depth completion avoids the situation where the fifth estimated depth image contains missing depth values ​​after removing the depth value of the fourth pixel. Depth completion provides a complete third estimated depth image, thus providing complete depth information for subsequent 3D model building.

[0164] In some embodiments, during the process of projecting the depth values ​​corresponding to the repeated regions in the fourth estimated depth image to the latest frame, the depth values ​​corresponding to the repeated regions in the fourth estimated depth image can first be projected onto the depth map in the middle of the time series to obtain the intermediate depth image, and then the intermediate depth map image can be projected onto the latest frame to obtain the fifth estimated depth image.

[0165] In some embodiments, the process of projecting the depth values ​​corresponding to the repeated regions in the fourth estimated depth image onto the latest frame may include, but is not limited to: determining the depth map corresponding to the multi-view image in the middle of the time series in at least two multi-view images, i.e. the depth map in the middle of the time series, then projecting the depth values ​​corresponding to the repeated regions in the fourth estimated depth image onto the depth map in the middle of the time series, and performing depth completion on the projected depth map in the middle of the time series to obtain the intermediate depth image.

[0166] In the above embodiments, when the time series corresponds to a relatively long duration, the depth values ​​corresponding to the repeated regions in the fourth estimated depth image are first projected onto the depth map in the middle of the time series to obtain the intermediate depth image. Then, the intermediate depth map image is projected onto the latest frame to obtain the fifth estimated depth image. This process avoids the inaccuracy of the result obtained when directly projecting the fourth estimated depth image onto the latest frame due to the large changes in camera pose over a long time series. Therefore, the accuracy of the final third estimated depth image can be improved through the progressive processing of two projections.

[0167] It should be noted that the fourth estimated depth image can also be projected multiple times onto multiple depth maps in the time series before being projected onto the last frame to obtain the third estimated depth image. Further details are omitted in this embodiment.

[0168] S202. Based on the third pixel, perform triangulation on the last frame of the multi-view image in at least two frames of multi-view images to obtain several second triangular meshes.

[0169] The third pixel includes: a pixel in the last frame of the multi-view image whose pixel difference with its neighboring pixel is greater than the second pixel difference threshold, and / or a pixel in the last frame of the multi-view image that corresponds to the second target pixel in the third estimated depth image, wherein the second target pixel is a pixel in the third estimated depth image whose depth difference with its neighboring pixel is greater than the second depth difference threshold.

[0170] The above-mentioned implementation of triangulation of the last frame of at least two multi-view images based on the third pixel to obtain several second triangular meshes includes, but is not limited to: first obtaining the last frame of at least two multi-view images, then determining the third pixel in the last frame of the multi-view image, which can be multiple pixels, and connecting these multiple pixels to divide the last frame of the multi-view image into multiple triangular images, namely the aforementioned several second triangular meshes.

[0171] S203. Based on the third estimated depth image, spatial mapping is performed on several second triangular meshes to obtain the target 3D model.

[0172] The above-mentioned method of obtaining the target 3D model by spatially mapping several second triangular meshes based on the third estimated depth image includes, but is not limited to: obtaining the depth value corresponding to the third pixel point from the third estimated depth image, where these third pixels are the vertices of several second triangular meshes, thus obtaining the depth values ​​of the vertices of the several second triangular meshes; and then spatially mapping several second triangular meshes based on the depth values ​​of the vertices of the several second triangular meshes, thus establishing the initial 3D model, i.e., the target 3D model.

[0173] It should be noted that the above embodiments are illustrated using the target 3D model as the initial 3D model. In some embodiments, the target 3D model can also be understood as a model updated once or multiple times based on the initial 3D model. In this embodiment, when establishing the target 3D model, a depth map can be calculated by combining at least two frames of multi-view images with a multi-view stereo vision algorithm. A third estimated depth image is obtained through depth estimation. Based on the third pixel, the last frame of the at least two multi-view images is triangulated to obtain several second triangular meshes. Finally, spatial mapping is performed on the several second triangular meshes to obtain the target 3D model. This process of establishing the target 3D model does not require a depth sensor and can achieve 3D model updates without increasing equipment costs or power consumption.

[0174] The model update phase can be achieved through the following steps 204 to 206:

[0175] S204. Obtain the first multi-view image of the target frame.

[0176] The first multi-view image of the target frame is the multi-view image obtained after the above-mentioned at least two multi-view images.

[0177] S205. Based on the first multi-view image, the second multi-view image of the previous frame of the target frame, and the target 3D model, determine the target estimated depth image of the target frame.

[0178] S206. Update the target 3D model based on the target estimated depth image.

[0179] The descriptions of S204 to S206 in the above model update phase can be found in the descriptions of S101 to S103 above, and will not be repeated here.

[0180] With this scheme, after establishing the target 3D model, depth estimation can be performed using the second multi-view image of the previous frame of the target frame, the target 3D model, and the first multi-view image of the target frame to obtain the target estimated depth image corresponding to the target frame. Based on the target estimated depth image obtained by this estimation, the target 3D model is updated. This method of updating the 3D model does not require a depth sensor and can achieve the updating of the 3D model without increasing equipment cost and power consumption.

[0181] In this embodiment, no depth sensor is required during the creation and updating of the 3D model, thus enabling updates without increasing equipment costs or power consumption. Furthermore, compared to traditional methods of creating or updating 3D models using consumer-grade cameras, the solution in this embodiment does not incur significant memory consumption. Moreover, compared to traditional offline 3D reconstruction methods on servers, the solution in this embodiment reduces runtime and improves the efficiency of 3D model creation and updating.

[0182] like Figure 3 As shown in the diagram, this disclosure provides a structural block diagram of an apparatus for updating a 3D model, the apparatus comprising:

[0183] The acquisition module 301 is used to acquire the first multi-view image of the target frame;

[0184] The depth estimation module 302 is used to determine the target estimated depth image of the target frame based on the first multi-view image, the second multi-view image of the previous frame of the target frame, and the target 3D model;

[0185] The update module 303 is used to update the target 3D model based on the target estimated depth image.

[0186] As an optional implementation of this disclosure, the first multi-view image is an original multi-view image acquired by a multi-view image sensor;

[0187] As an optional implementation of this disclosure, the first multi-view image is a downsampled image of the original multi-view image.

[0188] As an optional implementation of this disclosure, the depth estimation module 302 is specifically used for:

[0189] Determine the first depth image of the target frame based on the first multi-view image;

[0190] The second multi-view image is projected onto the target 3D model to determine the first estimated multi-view image and the first estimated depth image of the target frame;

[0191] The depth value of the first pixel is removed from the first estimated depth image to obtain the target estimated depth image. The first pixel includes: pixels in the first estimated depth image and the first depth image whose depth difference is greater than a first depth difference threshold, and / or pixels in the first estimated multi-view image and the first multi-view image whose pixel difference is greater than a first pixel difference threshold.

[0192] As an optional implementation of this disclosure, the depth estimation module 302 is specifically used for:

[0193] The depth value of the first pixel is removed from the first estimated depth image to obtain the second estimated depth image;

[0194] Depth completion is performed on the second estimated depth image to obtain the target estimated depth image.

[0195] As an optional implementation of this disclosure, the updating module 302 is specifically used for:

[0196] The target 3D model is projected onto the target frame to obtain a mask image. The mask image includes a covered area and an uncovered area. The covered area is the region where the pixel coordinates corresponding to the existing 3D points in the target 3D model are located. The uncovered area is the region in the mask image other than the covered area.

[0197] Obtain depth information corresponding to the covered area from the target estimated depth image; update existing 3D points in the target 3D model based on the depth information corresponding to the covered area; and / or obtain depth information corresponding to the uncovered area from the target estimated depth image; supplement and establish new 3D points based on the depth information corresponding to the uncovered area and the first multi-view image.

[0198] As an optional implementation of this disclosure, the updating module 302 is specifically used for:

[0199] Determine the region to be processed in the first multi-view image that corresponds to the non-covered region;

[0200] A second pixel is determined from the region to be processed. The second pixel includes: a pixel in the region to be processed whose pixel difference with its neighboring pixel is greater than a second pixel difference threshold, and / or a pixel in the region to be processed that corresponds to a first target pixel in the target estimated depth image. The first target pixel is a pixel whose depth difference with its neighboring pixel is greater than a second depth difference threshold.

[0201] Based on the second pixel, the area to be processed is triangulated to obtain several first triangular meshes;

[0202] Based on the target 3D model, spatial mapping is performed on the several first triangular meshes based on the depth information corresponding to the non-covered areas, so as to supplement and establish new 3D points on the basis of the target 3D model.

[0203] As an optional implementation of this disclosure, the apparatus further includes:

[0204] A creation module 304 is configured to determine a third estimated depth image based on the at least two multi-view images before determining the target estimated depth image of the target frame based on the first multi-view image, the second multi-view image of the previous frame of the target frame, and the target 3D model.

[0205] Based on the third pixel, the last multi-view image in the at least two frames of multi-view images is triangulated to obtain several second triangular meshes.

[0206] Based on the third estimated depth image, spatial mapping is performed on the plurality of second triangular meshes to obtain the target 3D model;

[0207] The third pixel includes: a pixel in the last frame of the multi-view image whose pixel difference with its neighboring pixel is greater than a second pixel difference threshold, and / or a pixel in the last frame of the multi-view image that corresponds to a second target pixel in the third estimated depth image, wherein the second target pixel is a pixel in the third estimated depth image whose depth difference with its neighboring pixel is greater than a second depth difference threshold.

[0208] As an optional implementation of this disclosure, module 304 is specifically used for:

[0209] Based on the at least two multi-view images, a fourth estimated depth image is determined; the overlapping region between the fourth estimated depth image and the earliest acquired depth image has a size ratio of less than or equal to a preset ratio relative to the size of the fourth estimated depth image; the earliest acquired depth image is the depth image corresponding to the earliest acquired multi-view image among the at least two multi-view images.

[0210] The depth values ​​corresponding to the repeated regions in the fourth estimated depth image are projected onto the latest frame to obtain the fifth estimated depth image.

[0211] The depth value of the fourth pixel is removed from the fifth estimated depth image, and depth completion is performed to obtain the third estimated depth image; the fourth pixel includes: the pixel whose depth difference between the fifth estimated depth image and the latest frame depth image is greater than the third depth difference threshold, and the latest frame depth image is the depth map corresponding to the multi-view image at the end of the time series among the at least two multi-view images.

[0212] As an optional implementation of this disclosure, the creation module 304 is specifically used for:

[0213] The depth values ​​corresponding to the repeated regions in the fourth estimated depth image are projected onto the depth map in the middle of the time series, and depth completion is performed to obtain the intermediate depth image; wherein, the depth map in the middle of the time series is the depth map corresponding to the multi-view image in the middle of the time series among the at least two frames of multi-view images;

[0214] The intermediate depth map is projected onto the latest frame to obtain the fifth estimated depth image.

[0215] As an optional implementation of this disclosure, the creation module 304 is specifically used for:

[0216] Step 1: Obtain the multi-view image of the current frame, and calculate the second depth image of the multi-view image of the current frame based on the multi-view image of the current frame;

[0217] Step 2: Obtain the previous frame multiview image and the third depth image of the previous frame multiview image of the current frame;

[0218] Step 3: Based on the third depth map, project the previous frame multi-view image onto the current frame to obtain the estimated binocular image of the current frame and the sixth estimated depth image of the current frame;

[0219] Step 4: Remove the depth value of the third pixel from the sixth estimated depth image of the current frame to obtain the seventh estimated depth image of the current frame; wherein, the third pixel includes: pixels in the fourth estimated depth image and the third depth image whose depth difference is greater than the second depth difference threshold, and / or pixels in the estimated multi-view image and the multi-view image of the current frame whose pixel difference is greater than the second pixel difference threshold;

[0220] Step 5: Determine the overlapping region between the seventh estimated depth image and the earliest acquired depth image;

[0221] Repeat steps 1 to 5 until the size ratio of the repeated region relative to the seventh estimated depth image is less than or equal to a preset ratio, then the seventh estimated image is determined as the fourth estimated depth image.

[0222] like Figure 4 As shown in the diagram, this disclosure provides a schematic diagram of an electronic device. The electronic device includes a processor 401, a memory 402, and a computer program stored in the memory 402 and executable on the processor 401. When executed by the processor 401, the computer program implements the various processes of the method for updating the 3D model in the above-described method embodiments. Furthermore, it achieves the same technical effects, and to avoid repetition, it will not be described again here.

[0223] This disclosure provides a computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, it implements the various processes of the method for updating the 3D model in the above-described method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0224] The computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0225] This disclosure provides a computer program product that stores a computer program. When the computer program is executed by a processor, it implements the various processes of the method for updating the 3D model in the above-described method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0226] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media containing computer-usable program code.

[0227] In this disclosure, the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0228] In this disclosure, memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0229] In this disclosure, computer-readable media includes both permanent and non-permanent, removable and non-removable storage media. Storage media can store information using any method or technology; the information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient media, such as modulated data signals and carrier waves.

[0230] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0231] The above are merely specific embodiments of this disclosure, enabling those skilled in the art to understand or implement this disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to these embodiments, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for updating a 3D model, characterized in that, The method includes: Acquire a first multi-view image of the target frame; the first multi-view image includes images acquired by at least two cameras respectively; Based on the first multi-view image, the second multi-view image of the previous frame of the target frame, and the target 3D model, determine the target estimated depth image of the target frame; Update the target 3D model based on the target's estimated depth image; The target 3D model is a 3D model built or updated based on at least two consecutive frames of multi-view images acquired before the first multi-view image, and the at least two frames of multi-view images include the second multi-view image; The step of determining the target estimated depth image of the target frame based on the first multi-view image, the second multi-view image of the previous frame of the target frame, and the target 3D model includes: Determine the first depth image of the target frame based on the first multi-view image; The second multi-view image is projected onto the target 3D model to determine the first estimated multi-view image and the first estimated depth image of the target frame; The depth value of the first pixel is removed from the first estimated depth image to obtain the target estimated depth image. The first pixel includes at least one of the following: a pixel in the first estimated depth image whose depth difference with the first depth image is greater than a first depth difference threshold, and a pixel in the first estimated multi-view image whose pixel difference with the first multi-view image is greater than a first pixel difference threshold.

2. The method according to claim 1, characterized in that, The first multi-view image is the original multi-view image acquired by a multi-view image sensor; or, The first multiview image is a downsampled image of the original multiview image.

3. The method according to claim 1, characterized in that, The step of removing the depth value of the first pixel from the first estimated depth image to obtain the target estimated depth image includes: The depth value of the first pixel is removed from the first estimated depth image to obtain the second estimated depth image; Depth completion is performed on the second estimated depth image to obtain the target estimated depth image.

4. The method according to claim 1, characterized in that, Updating the target 3D model based on the estimated depth image of the target includes: The target 3D model is projected onto the target frame to obtain a mask image. The mask image includes a covered area and an uncovered area. The covered area is the region where the pixel coordinates corresponding to the existing 3D points in the target 3D model are located. The uncovered area is the region in the mask image other than the covered area. Obtain depth information corresponding to the covered area from the target estimated depth image; update existing 3D points in the target 3D model based on the depth information corresponding to the covered area; and / or obtain depth information corresponding to the uncovered area from the target estimated depth image; supplement and establish new 3D points based on the depth information corresponding to the uncovered area and the first multi-view image.

5. The method according to claim 4, characterized in that, The step of supplementing the target 3D model with new 3D points based on the depth information corresponding to the non-covered area and the first multi-view image includes: Determine the region to be processed in the first multi-view image that corresponds to the non-covered region; A second pixel is determined from the region to be processed. The second pixel includes: a pixel in the region to be processed whose pixel difference with its neighboring pixel is greater than a second pixel difference threshold, and / or a pixel in the region to be processed that corresponds to a first target pixel in the target estimated depth image. The first target pixel is a pixel whose depth difference with its neighboring pixel is greater than a second depth difference threshold. Based on the second pixel, the area to be processed is triangulated to obtain several first triangular meshes; Based on the target 3D model, spatial mapping is performed on the several first triangular meshes based on the depth information corresponding to the non-covered areas, so as to supplement and establish new 3D points on the basis of the target 3D model.

6. The method according to claim 1, characterized in that, Before determining the target estimated depth image of the target frame based on the first multi-view image, the second multi-view image of the previous frame of the target frame, and the target 3D model, the method further includes: A third estimated depth image is determined based on the at least two frames of multi-view images; Based on the third pixel, the last multi-view image in the at least two multi-view images is triangulated to obtain several second triangular meshes. Based on the third estimated depth image, spatial mapping is performed on the several second triangular meshes to obtain the target 3D model. The third pixel includes: a pixel in the last frame of the multi-view image whose pixel difference with its neighboring pixel is greater than a second pixel difference threshold, and / or a pixel in the last frame of the multi-view image that corresponds to a second target pixel in the third estimated depth image, wherein the second target pixel is a pixel in the third estimated depth image whose depth difference with its neighboring pixel is greater than a second depth difference threshold.

7. The method according to claim 6, characterized in that, The step of determining the third estimated depth image based on the at least two multi-view images includes: Based on the at least two multi-view images, a fourth estimated depth image is determined; the overlapping region between the fourth estimated depth image and the earliest acquired depth image has a size ratio of less than or equal to a preset ratio relative to the size of the fourth estimated depth image; the earliest acquired depth image is the depth image corresponding to the earliest acquired multi-view image among the at least two multi-view images. The depth values ​​corresponding to the repeated regions in the fourth estimated depth image are projected onto the latest frame to obtain the fifth estimated depth image. The depth value of the fourth pixel is removed from the fifth estimated depth image, and depth completion is performed to obtain the third estimated depth image; the fourth pixel includes: the pixel whose depth difference between the fifth estimated depth image and the latest frame depth image is greater than the third depth difference threshold, and the latest frame depth image is the depth map corresponding to the multi-view image at the end of the time series among the at least two multi-view images.

8. The method according to claim 7, characterized in that, The step of projecting the depth value corresponding to the repeated region in the fourth estimated depth image onto the latest frame includes: The depth values ​​corresponding to the repeated regions in the fourth estimated depth image are projected onto the depth map in the middle of the time series, and depth completion is performed to obtain the intermediate depth image; wherein, the depth map in the middle of the time series is the depth map corresponding to the multi-view image in the middle of the time series among the at least two frames of multi-view images; The intermediate depth map is projected onto the latest frame to obtain the fifth estimated depth image.

9. The method according to claim 7, characterized in that, The step of determining the fourth estimated depth image based on the at least two frames of multi-view images includes: Step 1: Obtain the multi-view image of the current frame, and calculate the second depth image of the multi-view image of the current frame based on the multi-view image of the current frame; Step 2: Obtain the previous frame multiview image and the third depth image of the previous frame multiview image of the current frame; Step 3: Based on the third depth map, project the previous frame multi-view image onto the current frame to obtain the estimated binocular image of the current frame and the sixth estimated depth image of the current frame; Step 4: Remove the depth value of the third pixel from the sixth estimated depth image of the current frame to obtain the seventh estimated depth image of the current frame; wherein, the third pixel includes: pixels in the fourth estimated depth image and the third depth image whose depth difference is greater than the second depth difference threshold, and / or pixels in the estimated multi-view image and the multi-view image of the current frame whose pixel difference is greater than the second pixel difference threshold; Step 5: Determine the overlapping region between the seventh estimated depth image and the earliest acquired depth image; Steps 1 to 5 are executed repeatedly until the size ratio of the repeated region relative to the seventh estimated depth image is less than or equal to a preset ratio, then the seventh estimated image is determined as the fourth estimated depth image.

10. An apparatus for updating a 3D model, characterized in that, include: The acquisition module is used to acquire the first multi-view image of the target frame; The first multi-view image includes images acquired by at least two cameras respectively; The depth estimation module is used to determine the target estimated depth image of the target frame based on the first multi-view image, the second multi-view image of the previous frame of the target frame, and the target 3D model; An update module is used to update the target 3D model based on the target's estimated depth image; The depth estimation module is specifically used for: Determine the first depth image of the target frame based on the first multi-view image; The second multi-view image is projected onto the target 3D model to determine the first estimated multi-view image and the first estimated depth image of the target frame; The depth value of the first pixel is removed from the first estimated depth image to obtain the target estimated depth image. The first pixel includes at least one of the following: a pixel in the first estimated depth image whose depth difference with the first depth image is greater than a first depth difference threshold, and a pixel in the first estimated multi-view image whose pixel difference with the first multi-view image is greater than a first pixel difference threshold.

11. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the method for updating a 3D model as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, include: A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the method for updating a 3D model as described in any one of claims 1 to 9.

Images