Method, apparatus, device, storage medium and computer program for three-dimensional reconstruction

Abstract

The application discloses a three-dimensional reconstruction method, device, equipment, storage medium and computer program, and belongs to the technical field of computer vision. The method comprises the following steps: determining a plurality of candidate foreground regions based on a plurality of shooting images, determining three-dimensional surface coordinates and / or depths of pixel points in the plurality of candidate foreground regions, determining at least one target foreground region from the plurality of candidate foreground regions based on the three-dimensional surface coordinates and / or depths of the pixel points in the plurality of candidate foreground regions, and performing three-dimensional reconstruction on a target object based on pixel points in the plurality of shooting images except the at least one target foreground region. The application can perform three-dimensional reconstruction on the target object based on the pixel points in the plurality of shooting images except the at least one target foreground region, thereby avoiding the interference of a foreground object on the three-dimensional reconstruction of the target object and effectively improving the three-dimensional reconstruction quality of the target object.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a three-dimensional reconstruction method, device, equipment, storage medium and computer program. BACKGROUND

[0002] Three-dimensional reconstruction is a technology of converting two-dimensional images or videos into three-dimensional models, which has important application value in the fields of autonomous driving, virtual reality, medical image processing, etc.

[0003] In related technologies, images of a target object under different viewing angles are usually first collected, and then a neural radiance field method is used to perform three-dimensional reconstruction on the target object based on the images under different viewing angles. That is, a neural radiance field model is trained based on the images of the target object under different viewing angles, and then the position coordinates and viewing angle of any point in the three-dimensional space where the target object is located are input into the trained neural radiance field model to obtain the volume density of the point and the color of the point under the viewing angle output by the trained neural radiance field model. Finally, the volume density of each point in the three-dimensional space and the color of each point under different viewing angles can be obtained, and then an image of the target object under any viewing angle is rendered by a voxel rendering method, so as to realize three-dimensional reconstruction of the target object.

[0004] However, in actual applications, when collecting images of the target object under different viewing angles, it is usually difficult to ensure that only the target object is included in the collected images. There may also be foreground objects such as pedestrians and vehicles in the collected images, which will interfere with the three-dimensional reconstruction of the target object, thereby reducing the quality of the three-dimensional reconstruction of the target object. SUMMARY

[0005] The present application provides a three-dimensional reconstruction method, device, equipment, storage medium and computer program, which can solve the problem of poor three-dimensional reconstruction quality in related technologies. The technical solution is as follows:

[0006] In a first aspect, a three-dimensional reconstruction method is provided, which includes: determining a plurality of candidate foreground regions based on a plurality of captured images, the plurality of captured images being images of a target object under different viewing angles, the candidate foreground region being an image region in which a foreground object exists in the captured image, determining three-dimensional surface coordinates and / or depths of pixel points in the plurality of candidate foreground regions, determining at least one target foreground region from the plurality of candidate foreground regions based on the three-dimensional surface coordinates and / or depths of the pixel points in the plurality of candidate foreground regions, the target foreground region not being a plane in form in a three-dimensional space, and performing three-dimensional reconstruction on the target object based on pixel points in the plurality of captured images except for the at least one target foreground region.

[0007] The application can perform three-dimensional reconstruction on the target object based on the pixel points in the multiple photographed images except the at least one target foreground region, that is, the pixel points in the foreground region are excluded when light sampling to avoid the interference of the foreground object on the three-dimensional reconstruction of the target object, thereby effectively improving the three-dimensional reconstruction quality of the target object. Moreover, since the morphology of the target foreground region in the three-dimensional space is not a plane, the application can also avoid the problems of missing or blurring of billboards and target objects with reflection ability after three-dimensional reconstruction, thereby ensuring the three-dimensional reconstruction quality of the target object.

[0008] The first photographed image is input into the foreground prediction model to obtain a region where a foreground object in the first photographed image output by the foreground prediction model is located and a confidence of the foreground object, the first photographed image being any one of the multiple photographed images, and then a region where a foreground object with a confidence greater than a second confidence threshold in the first photographed image is determined as a candidate foreground region. Each photographed image in the multiple photographed images is processed in the same way to obtain multiple candidate foreground regions.

[0009] The above-mentioned foreground object refers to an object in the photographed image that will block the target object, and the object category of the foreground object can include at least one.

[0010] In the case where the foreground prediction model is a semantic segmentation model, the region where the foreground object in the first photographed image is located refers to the pixel points where the foreground object in the first photographed image is located, and the confidence of the foreground object refers to the confidence of the foreground object corresponding to the corresponding pixel points. At this time, the pixel points with a confidence greater than the second confidence threshold can be determined as the candidate foreground region to obtain at least one candidate foreground region, and any one pixel point in each candidate foreground region has at least one pixel point adjacent to the position of the pixel point.

[0011] The implementation process of determining the pixel points with a confidence greater than the second confidence threshold as the candidate foreground region includes: determining the pixel points with a confidence greater than the second confidence threshold as candidate foreground pixel points to obtain multiple candidate foreground pixel points. At least one group of candidate foreground pixel points is determined from the multiple candidate foreground pixel points, any one pixel point in the candidate foreground pixel points in the same group has at least one pixel point adjacent to the position of the pixel point, and the region composed of the at least one group of candidate foreground pixel points is taken as at least one candidate foreground region of the first photographed image.

[0012] The above-mentioned any one pixel point in the candidate foreground pixel points in the same group has at least one pixel point adjacent to the position of the pixel point can be understood as that the positions of the candidate foreground pixel points in the same group are continuous.

[0013] Before determining multiple candidate foreground regions based on multiple captured images, a second confidence threshold can be configured. That is, the electronic device can display a third parameter configuration interface that instructs the user to configure the second confidence threshold. In response to the user's completion of the configuration of the second confidence threshold, the second confidence threshold is obtained from the third parameter configuration interface.

[0014] Since foreground objects are those that occlude the target object in the captured image, their object category varies depending on the circumstances. Therefore, before determining multiple candidate foreground regions based on multiple captured images, the object category of the foreground object can be determined.

[0015] There are several ways to determine the object category of a foreground object; two of these methods will be introduced below.

[0016] In the first implementation, the electronic device displays a foreground object configuration interface, which instructs the user to configure the object category of the foreground object. In response to the user's completion of the configuration of the object category of the foreground object, the object category of the foreground object is obtained from the foreground object configuration interface.

[0017] The second implementation method involves determining the target category, which is the object category of the target object, and determining the object category of the foreground object as the object category of the object that is not the target category.

[0018] Optionally, a target category configuration interface is displayed, which instructs the user to configure the object category of the target object. In response to the user's completion of the target category configuration, the category of the target object is obtained from the target category configuration interface.

[0019] Optionally, the target category is determined based on multiple captured images. That is, the multiple captured images are input into the object category detection model to obtain the object category in each captured image output by the object category detection model. The object categories in each captured image are counted to obtain the occurrence frequency of each object category, and the object category with the highest occurrence frequency is taken as the target category.

[0020] There are several ways to determine the 3D surface coordinates of pixels in multiple candidate foreground regions. Two of these methods will be introduced below.

[0021] The first implementation method involves determining a first neural radiation field model based on multiple captured images. Using this model, the volume density of multiple physical points corresponding to the target pixel in the first candidate foreground region in three-dimensional space is determined. The first candidate foreground region is any one of these multiple candidate foreground regions, the target pixel is any pixel within that region, and the multiple physical points are physical points along the imaging optical path of the target pixel. Based on the volume density of these physical points, the three-dimensional surface coordinates of the target pixel are determined. Processing each pixel in the first candidate foreground region in the same manner allows the determination of its three-dimensional surface coordinates. Similarly, processing each candidate foreground region in the same manner allows the determination of the three-dimensional surface coordinates of pixels within the multiple candidate foreground regions.

[0022] It should be noted that the above imaging optical path refers to the optical path through which the sampling light corresponding to the target pixel passes. This sampling light refers to the light emitted from the camera's optical center and passing through the target pixel.

[0023] Optionally, the process of determining the three-dimensional surface coordinates of the target pixel based on the volume density of the multiple physical points includes: taking physical points with a volume density greater than a volume density threshold as candidate physical points, and taking the three-dimensional coordinates of the candidate physical point that is closest to the target pixel as the three-dimensional surface coordinates of the target pixel.

[0024] Since volume density indicates the transparency of physical points, a higher volume density corresponds to lower transparency. Therefore, physical points with a volume density greater than a volume density threshold are considered opaque. Opaque physical points can be imaged in an image, and thus, they can be used as candidate physical points. Furthermore, since candidate physical points near the target pixel will occlude distant candidate physical points, thus affecting their image formation, the candidate physical point closest to the target pixel can be considered the physical point imaged on the target pixel in 3D space. Based on this, the 3D coordinates of the candidate physical point closest to the target pixel can be used as the 3D surface coordinates of the target pixel. This ensures the accuracy of the target pixel's 3D surface coordinates, resulting in a more accurate determination of the target foreground region and effectively improving the quality of the 3D reconstruction of the target object.

[0025] In the second implementation, the captured image is a depth image. In this case, each pixel in the captured image has depth. Therefore, the three-dimensional surface coordinates of the target pixel can be determined based on the depth of the target pixel in the first candidate foreground region. Processing each pixel in the first candidate foreground region in the same way allows the determination of the three-dimensional surface coordinates of each pixel in the first candidate foreground region. Processing each candidate foreground region in the same way allows the determination of the three-dimensional surface coordinates of pixels in multiple candidate foreground regions.

[0026] Optionally, the captured image is a depth image. In this case, each pixel in the captured image also has depth, and the depth of each pixel in the first candidate foreground region can be directly obtained. By processing each candidate foreground region in the same way, the depth of pixels in multiple candidate foreground regions can be determined.

[0027] Based on the 3D surface coordinates and / or depth of each pixel in the first candidate foreground region, it is determined whether the shape of the first candidate foreground region in 3D space is planar. The first candidate foreground region can be any one of the multiple candidate foreground regions. If the shape of the first candidate foreground region in 3D space is not planar, then the first candidate foreground region is determined as the target foreground region. If the shape of the first candidate foreground region in 3D space is planar, then the first candidate foreground region is not determined as the target foreground region. By processing each candidate foreground region in the same way, at least one target foreground region can be determined.

[0028] There are several ways to determine whether the shape of the first candidate foreground region is planar in three-dimensional space based on the three-dimensional surface coordinates of each pixel in the first candidate foreground region. Two of these methods will be introduced below.

[0029] In the first implementation, the three-dimensional surface coordinates of each pixel in the first candidate foreground region are input into the foreground discriminator to obtain the first shape output by the foreground discriminator and the confidence level of the first shape. If the first shape is planar and the confidence level of the first shape is greater than the first confidence level threshold, then the shape of the first candidate foreground region in three-dimensional space is determined to be planar. Otherwise, the shape of the first candidate foreground region in three-dimensional space is determined to be non-planar.

[0030] Optionally, before determining whether the shape of the first candidate foreground region in three-dimensional space is planar based on the three-dimensional surface coordinates of each pixel in the first candidate foreground region, a first confidence threshold can also be configured. That is, the electronic device displays a first parameter configuration interface, which instructs the user to configure the first confidence threshold. In response to the user's completion of the configuration of the first confidence threshold, the first confidence threshold is obtained from the first parameter configuration interface.

[0031] The second implementation method involves determining the 3D plane corresponding to the first candidate foreground region based on the 3D surface coordinates of each pixel in the first candidate foreground region. Then, based on the distance between the 3D surface coordinates of each pixel and this 3D plane, a morphological evaluation index for the first candidate foreground region is determined. This morphological evaluation index indicates the difference between the first candidate foreground region and its corresponding 3D plane. If the morphological evaluation index is less than a morphological evaluation index threshold, the first candidate foreground region is determined to be planar in 3D space. If the morphological evaluation index is not less than the morphological evaluation index threshold, the first candidate foreground region is determined to be non-planar in 3D space.

[0032] For ease of description, the distance between the three-dimensional surface coordinates of any pixel in the first candidate foreground region and the three-dimensional plane will be referred to as the first distance in the following text.

[0033] Optionally, the variance of multiple first distances is used as the morphological evaluation index of the first candidate foreground region, and the multiple first distances correspond one-to-one with multiple pixels in the first candidate foreground region.

[0034] Optionally, the average of multiple first distances can be used as the morphological evaluation index for the first candidate foreground region. Of course, in practical applications, in addition to using the variance or average as the morphological evaluation index for the first candidate foreground region, the maximum or minimum value among multiple first distances can also be used as the morphological evaluation index.

[0035] Before determining whether the shape of the first candidate foreground region is planar in three-dimensional space based on the three-dimensional surface coordinates of each pixel in the first candidate foreground region, the shape evaluation index threshold can also be configured. That is, the electronic device displays a second parameter configuration interface, which instructs the user to configure the shape evaluation index threshold. In response to the user's completion of the configuration of the shape evaluation index threshold, the shape evaluation index threshold is obtained from the second parameter configuration interface.

[0036] It should be noted that the first parameter configuration interface, the second parameter configuration interface, and the third parameter configuration interface mentioned above can be the same interface, or they can be different interfaces, or any two of the first parameter configuration interface, the second parameter configuration interface, and the third parameter configuration interface can be the same interface.

[0037] The process of determining whether the shape of the first candidate foreground region in three-dimensional space is planar based on the depth of each pixel in the first candidate foreground region includes: if the depth difference between any two pixels in the first candidate foreground region is less than the difference threshold, then the shape of the first candidate foreground region in three-dimensional space is determined to be planar; otherwise, the shape of the first candidate foreground region in three-dimensional space is determined not to be planar.

[0038] Optionally, the method of determining whether the shape of the first candidate foreground region in three-dimensional space is planar based on the three-dimensional surface coordinates and depth of each pixel in the first candidate foreground region includes: determining whether the shape of the first candidate foreground region in three-dimensional space is planar based on the three-dimensional surface coordinates of each pixel in the first candidate foreground region to obtain a first judgment result; determining whether the shape of the first candidate foreground region in three-dimensional space is planar based on the depth of each pixel in the first candidate foreground region to obtain a second judgment result; if at least one of the first judgment result and the second judgment result determines that the shape of the first candidate foreground region in three-dimensional space is planar, then the shape of the first candidate foreground region in three-dimensional space is finally determined to be planar; otherwise, the shape of the first candidate foreground region in three-dimensional space is determined not to be planar.

[0039] There are several ways to reconstruct a target object in three dimensions based on pixels in multiple captured images, excluding at least one target foreground region. Two of these methods will be introduced below.

[0040] The first implementation method is to determine a second neural radiation field model based on the pixels in multiple captured images, excluding at least one target foreground region. The second neural radiation field model is then used to render the image of the target object from any viewpoint to achieve three-dimensional reconstruction of the target object.

[0041] The second implementation involves determining a second neural radiation field model based on pixels in multiple captured images, excluding at least one target foreground region. This second neural radiation field model is then used to render the image of the target object from a target viewpoint, resulting in a rendered image from any of the multiple captured images. If the rendered image contains a problem area, a target image region (either a blurred or blank area) is identified from the target captured image at the same location as the problem area. Based on this target image region, the second neural radiation field model is further trained to obtain a third neural radiation field model. This third neural radiation field model is then used to determine the 3D reconstruction result of the target object. If the rendered image does not contain a problem area, the second neural radiation field model is used to render the image of the target object from any viewpoint to achieve 3D reconstruction of the target object.

[0042] The third neural radiation field model determines that there are multiple ways to achieve the three-dimensional reconstruction result of the target object. Two of these methods will be introduced below.

[0043] The first method involves rendering images of the target object from any viewpoint using a third neural radiation field model to achieve 3D reconstruction of the target object.

[0044] The second method uses a third neural radiation field model to render the image of the target object from the target's perspective. If the rendered image has problematic areas, a prompt interface is displayed, indicating that the image quality from the current target perspective is poor and a new image needs to be uploaded. If the rendered image does not have problematic areas, the third neural radiation field model is used to render the image of the target object from any perspective to achieve a 3D reconstruction of the target object.

[0045] Optionally, the prompt interface also includes an "End" button and a "Re-upload" button. The "End" button instructs the user not to re-upload the image taken from the target viewpoint, while the "Re-upload" button instructs the user to re-upload the image taken from the target viewpoint. In response to the user's triggering of the "End" button, the electronic device renders the image of the target object from any viewpoint using the third neural radiation field model to achieve 3D reconstruction of the target object. In response to the user's triggering of the "Re-upload" button, the electronic device displays an image re-upload interface, instructing the user to re-upload the image taken from the target viewpoint. In response to the user's triggered image upload completion operation, the electronic device retrieves the re-uploaded image taken from the target viewpoint from the image re-upload interface. The electronic device replaces the image taken from the target viewpoint among the multiple captured images with the re-uploaded image taken from the target viewpoint and returns to the step of determining multiple candidate foreground regions based on the multiple captured images.

[0046] Secondly, a three-dimensional reconstruction apparatus is provided, which has the function of implementing the three-dimensional reconstruction method described in the first aspect. The three-dimensional reconstruction apparatus includes at least one module for implementing the three-dimensional reconstruction method provided in the first aspect.

[0047] Thirdly, an electronic device is provided, comprising a processor and a memory, the memory being used to store a computer program for performing the three-dimensional reconstruction method provided in the first aspect. The processor is configured to execute the computer program stored in the memory to implement the three-dimensional reconstruction method described in the first aspect.

[0048] Optionally, the electronic device may further include a communication bus for establishing a connection between the processor and the memory.

[0049] Fourthly, a computer-readable storage medium is provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to perform the steps of the three-dimensional reconstruction method described in the first aspect.

[0050] Fifthly, a computer program product comprising instructions is provided, which, when executed on a computer, causes the computer to perform the steps of the three-dimensional reconstruction method described in the first aspect. Alternatively, a computer program is provided that, when executed on a computer, causes the computer to perform the steps of the three-dimensional reconstruction method described in the first aspect.

[0051] The technical effects achieved by the second, third, fourth, and fifth aspects mentioned above are similar to those achieved by the corresponding technical means in the first aspect, and will not be repeated here. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0053] Figure 2 This is a flowchart of a three-dimensional reconstruction method provided in an embodiment of this application;

[0054] Figure 3 This is a schematic diagram of an image captured according to an embodiment of this application;

[0055] Figure 4 This is a flowchart of another three-dimensional reconstruction method provided in the embodiments of this application;

[0056] Figure 5 This is a flowchart of another three-dimensional reconstruction method provided in the embodiments of this application;

[0057] Figure 6 This is a schematic diagram of the structure of a three-dimensional reconstruction device provided in an embodiment of this application. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0059] To facilitate understanding, before providing a detailed explanation of the three-dimensional reconstruction method provided in the embodiments of this application, the application scenarios involved in the embodiments of this application will be introduced first.

[0060] 3D reconstruction is a fundamental research area in computer vision, and thanks to the rapid development of computer technology, related research is receiving increasing attention. 3D reconstruction is a technique that transforms 2D images or videos into 3D models, and it has significant application value in fields such as autonomous driving, virtual reality, augmented reality, medical image processing, and map scene reconstruction. For example, in the field of autonomous driving, 3D reconstruction can reconstruct near-realistic street scenes, thus providing more diverse training scenarios for autonomous driving algorithms. In the field of map scene reconstruction, 3D reconstruction can provide users with map scenes that offer a realistic viewing experience. In the fields of virtual reality and augmented reality, 3D reconstruction technology can rapidly generate 3D scenes while maintaining low cost.

[0061] Currently, the common approach is to acquire images of the target object from different viewpoints (also known as captured images), and then use the neural radiation field method to reconstruct the target object in 3D based on these captured images from different viewpoints. Specifically, based on the captured images of the target object from different viewpoints, light is sampled at each pixel to determine the corresponding sampled light ray. This sampled light ray refers to the light ray emitted from the camera's optical center and passing through the corresponding pixel. Based on the color of each pixel in the captured image, the sampled light ray, and the viewpoint, the neural radiation field model is trained. Then, the position coordinates and viewpoint of any point in the 3D space containing the target object are input into the trained neural radiation field model. The trained neural radiation field model outputs the volume density and color of that point at that viewpoint. Ultimately, the volume density and color of each point in the 3D space at different viewpoints can be obtained. Finally, voxel rendering is used to render the image of the target object from any viewpoint, thus achieving 3D reconstruction of the target object.

[0062] However, in practical applications, when acquiring images of a target object from different perspectives, it is often difficult to guarantee that the acquired images contain only the target object. The acquired images may also contain foreground objects such as pedestrians and vehicles, which can interfere with the 3D reconstruction of the target object and thus reduce the quality of the 3D reconstruction.

[0063] It should be noted that the use of pedestrians and vehicles as foreground objects is merely an example and does not constitute a limitation on the embodiments of this application. In practical applications, the foreground object refers to an object in the captured image that will obstruct the target object, and specific analysis is required based on the actual situation.

[0064] To avoid interference from foreground objects, related technologies can identify the image region (also known as the foreground region) where the foreground object is located in the captured image. Based on the pixels in the captured image other than those in the foreground region, the neural radiation field model is trained to achieve three-dimensional reconstruction of the target object.

[0065] However, in practical applications, target objects may include reflective objects and billboards. Reflective objects such as glass and mirrors can reflect people, vehicles, and trees, while billboards contain images of people and vehicles. When using the aforementioned techniques to reconstruct the target object in 3D, these mirror-reflected objects, as well as the images of people and vehicles within the billboard, will be identified as foreground objects. This can lead to problems such as missing or blurred images of the rendered billboard and reflective target objects, thus reducing the quality of the 3D reconstruction.

[0066] Based on this, embodiments of this application provide a 3D reconstruction method capable of determining multiple candidate foreground regions, that is, determining image regions containing foreground objects in multiple captured images, and then determining at least one target foreground region from the multiple candidate foreground regions based on the 3D surface coordinates and / or depth of pixels in the multiple captured images, and performing 3D reconstruction of the target object based on pixels in the multiple captured images other than the at least one target foreground region. In other words, embodiments of this application can exclude pixels in the foreground region during light sampling, avoiding interference from foreground objects in the 3D reconstruction of the target object, thereby effectively improving the 3D reconstruction quality of the target object. Furthermore, since the target foreground region is not planar in 3D space, embodiments of this application can also avoid problems such as missing or blurred images of billboards and reflective target objects after 3D reconstruction, thereby ensuring the 3D reconstruction quality of the target object.

[0067] Please refer to Figure 1 , Figure 1 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. The electronic device includes at least one processor 101, a communication bus 102, a memory 103, and at least one communication interface 104.

[0068] Processor 101 can be a general-purpose central processing unit (CPU), a network processor (NP), a microprocessor, or one or more integrated circuits for implementing the solutions of this application, such as application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0069] The communication bus 102 is used to transmit information between the aforementioned components. The communication bus 102 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, it is represented by only one thick line in the figure, but this does not mean that there is only one bus or one type of bus.

[0070] The memory 103 may be a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable read-only memory (EEPROM), an optical disc (including a compact disc read-only memory (CD-ROM), a compressed optical disc, a laser disc, a digital versatile optical disc, a Blu-ray disc, etc.), a magnetic disk storage medium, or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but not limited thereto. The memory 103 may exist independently and be connected to the processor 101 via the communication bus 102. The memory 103 may also be integrated with the processor 101.

[0071] Communication interface 104 uses any transceiver-like device for communicating with other devices or communication networks. Communication interface 104 includes a wired communication interface and may also include a wireless communication interface. The wired communication interface may be, for example, an Ethernet interface. The Ethernet interface may be an optical interface, an electrical interface, or a combination thereof. The wireless communication interface may be a wireless local area network (WLAN) interface, a cellular network communication interface, or a combination thereof.

[0072] In a specific implementation, as one embodiment, the processor 101 may include one or more CPUs, such as Figure 1 CPU0 and CPU1 are shown in the diagram.

[0073] In a specific implementation, as one example, an electronic device may include multiple processors, such as... Figure 1 The processors 101 and 105 shown are illustrated. Each of these processors may be a single-core processor or a multi-core processor. Here, "processor" may refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0074] In a specific implementation, as one embodiment, the electronic device may further include output devices and input devices. The output device communicates with the processor 101 and can display information in various ways. For example, the output device may be a liquid crystal display (LCD), a light-emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. The input device communicates with the processor 101 and can receive user input in various ways. For example, the input device may be a mouse, keyboard, touchscreen device, or sensing device, etc.

[0075] In some embodiments, memory 103 stores program code 110 for executing the scheme of this application, and processor 101 can execute the program code 110 stored in memory 103. The program code 110 may include one or more software modules, and the electronic device can implement the following by using processor 101 and the program code 110 in memory 103. Figure 2 The three-dimensional reconstruction method provided in the embodiments.

[0076] Figure 2 This is a flowchart illustrating a three-dimensional reconstruction method provided in an embodiment of this application. This method is applied to electronic devices; please refer to [the documentation / reference]. Figure 2 The method includes the following steps.

[0077] Step 201: Based on multiple captured images, determine multiple candidate foreground regions. The multiple captured images are images of the target object from different perspectives. The candidate foreground region refers to the image region in the captured images where the foreground object exists.

[0078] In some embodiments, a first captured image is input into a foreground prediction model to obtain the region containing the foreground object in the first captured image and the confidence level of the foreground object output by the foreground prediction model. The first captured image can be any one of multiple captured images. Then, the region containing the foreground object whose confidence level in the first captured image is greater than a second confidence threshold is determined as a candidate foreground region. Processing each of the multiple captured images in the same way can yield multiple candidate foreground regions.

[0079] The second confidence threshold is preset and can be adjusted according to needs under different circumstances. For example, the second confidence threshold can be 0.8.

[0080] The foreground object mentioned above refers to an object in the captured image that will obstruct the target object. The object category of the foreground object may include at least one, but the embodiments of this application do not limit the object category of the foreground object.

[0081] It should be noted that when the foreground prediction model is a semantic segmentation model, the region where the foreground object is located in the first captured image refers to the pixel where the foreground object is located in the first captured image, and the confidence of the foreground object refers to the confidence of the foreground object corresponding to the pixel. In this case, pixels with a foreground object confidence greater than a second confidence threshold can be identified as candidate foreground regions to obtain at least one candidate foreground region. In each candidate foreground region, any pixel has at least one pixel adjacent to its position.

[0082] The process of identifying pixels with a confidence level greater than a second confidence threshold as candidate foreground regions includes: identifying pixels with a confidence level greater than the second confidence threshold as candidate foreground pixels, thus obtaining multiple candidate foreground pixels. From these multiple candidate foreground pixels, at least one group of candidate foreground pixels is selected, wherein any pixel in the same group of candidate foreground pixels has at least one pixel adjacent to it. The region formed by this at least one group of candidate foreground pixels is used as at least one candidate foreground region of the first captured image.

[0083] The fact that any pixel in the same group of candidate foreground pixels has at least one pixel adjacent to it can be understood as the candidate foreground pixels in the same group being in continuous positions.

[0084] Specifically, at least one set of candidate foreground pixels can be determined from a plurality of candidate foreground pixels based on their coordinates. For detailed implementation process, please refer to relevant technologies; it will not be elaborated here. Of course, at least one set of candidate foreground pixels can also be determined in other ways, and this application embodiment does not limit this approach.

[0085] For example, the first captured image includes 8 pixels, namely pixel 1, pixel 2, pixel 3, pixel 4, pixel 5, pixel 6, pixel 7, and pixel 8. Inputting the first captured image into the foreground prediction model, the model outputs pixels 1, 2, 4, 5, 6, and 8 as pixels containing a foreground object, and the confidence level of this foreground object is greater than a second confidence threshold. Therefore, pixels 1, 2, 4, 5, 6, and 8 can be identified as candidate foreground pixels. Since the coordinates of pixel 1 in the first captured image are (1,1), the coordinates of pixel 2 in the first captured image are (1,2), the coordinates of pixel 4 in the first captured image are (1,4), the coordinates of pixel 5 in the first captured image are (2,1), the coordinates of pixel 6 in the first captured image are (2,2), and the coordinates of pixel 8 in the first captured image are (2,4), therefore, any one of pixels 1, 2, 5, and 6 has at least one pixel adjacent to its position, and any one of pixels 4 and 8 has at least one pixel adjacent to its position. That is, two sets of foreground pixels can be obtained from these multiple candidate foreground pixels, and the regions formed by these two sets of foreground pixels are taken as the two candidate foreground regions of the first captured image.

[0086] In some embodiments, before determining multiple candidate foreground regions based on multiple captured images, a second confidence threshold can also be configured. That is, the electronic device can display a third parameter configuration interface that instructs the user to configure the second confidence threshold. In response to the user's completion of the configuration of the second confidence threshold, the second confidence threshold is obtained from the third parameter configuration interface.

[0087] The electronic device displays a third parameter configuration interface, where the user can configure the second confidence threshold. After configuration, the user triggers a second confidence threshold configuration completion operation to indicate that the configuration of the second confidence threshold has been completed. In response to the second confidence threshold configuration completion operation, the electronic device retrieves the second confidence threshold from the third parameter configuration interface.

[0088] Of course, the above-described configuration of the second confidence threshold by the user is merely one implementation method. In practical applications, the electronic device can also store the second confidence threshold, eliminating the need for user configuration. This application does not limit this approach.

[0089] Since foreground objects are those that occlude the target object in the captured image, their object category varies depending on the circumstances. Therefore, before determining multiple candidate foreground regions based on multiple captured images, the object category of the foreground object can be determined.

[0090] There are several ways to determine the object category of a foreground object; two of these methods will be introduced below.

[0091] In the first implementation, the electronic device displays a foreground object configuration interface, which instructs the user to configure the object category of the foreground object. In response to the user's completion of the configuration of the object category of the foreground object, the object category of the foreground object is obtained from the foreground object configuration interface.

[0092] The electronic device displays a foreground object configuration interface, where the user can configure the object category of the foreground object. After configuration, the user can trigger a foreground object configuration completion operation to indicate that the configuration of the object category of the foreground object is complete. In response to the object category configuration completion operation, the electronic device retrieves the object category of the foreground object from the foreground object configuration interface.

[0093] The second implementation method involves determining the target category, which is the object category of the target object, and determining the object category of the foreground object as the object category of the object that is not the target category.

[0094] For example, if the target category is a ship, then the object category of the foreground object is any object category other than a ship. For instance, the object category of the foreground object could be a person, a bicycle, a tree, etc.

[0095] In some embodiments, the electronic device may display a target category configuration interface that instructs a user to configure the object category of a target object, and in response to the user's completion of the target category configuration, retrieve the category of the target object from the target category configuration interface.

[0096] The electronic device displays a target category configuration interface, where the user can configure the object category of the target object. After configuration, the user can trigger a target category configuration completion operation to indicate that the configuration of the target category is complete. In response to the target category configuration completion operation, the electronic device retrieves the object category of the target object from the target category configuration interface.

[0097] In other embodiments, the target category can be determined based on multiple captured images. That is, the multiple captured images are input into an object category detection model to obtain the object category in each captured image output by the object category detection model. The object categories in each captured image are counted to obtain the occurrence frequency of each object category, and the object category with the highest occurrence frequency is taken as the target category.

[0098] In practical applications, before determining multiple candidate foreground regions based on the multiple captured images, the electronic device can also acquire multiple captured images. Optionally, the electronic device can display a captured image upload interface that instructs the user to upload multiple captured images. In response to a user-triggered captured image upload completion operation, the electronic device acquires the multiple captured images from the captured image upload interface. Of course, the electronic device can also acquire multiple captured images through other means, and this application does not limit this.

[0099] Step 202: Determine the three-dimensional surface coordinates and / or depth of pixels in multiple candidate foreground regions.

[0100] There are several ways to determine the 3D surface coordinates of pixels in multiple candidate foreground regions. Two of these methods will be introduced below.

[0101] The first implementation method involves determining a first neural radiation field model based on multiple captured images. Using this model, the volume density of multiple physical points corresponding to the target pixel in the first candidate foreground region in three-dimensional space is determined. The first candidate foreground region is any one of these multiple candidate foreground regions, the target pixel is any pixel within that region, and the multiple physical points are physical points along the imaging optical path of the target pixel. Based on the volume density of these physical points, the three-dimensional surface coordinates of the target pixel are determined. Processing each pixel in the first candidate foreground region in the same manner allows the determination of its three-dimensional surface coordinates. Similarly, processing each candidate foreground region in the same manner allows the determination of the three-dimensional surface coordinates of pixels within the multiple candidate foreground regions.

[0102] It should be noted that the above imaging optical path refers to the optical path through which the sampling light corresponding to the target pixel passes. This sampling light refers to the light emitted from the camera's optical center and passing through the target pixel.

[0103] For example, the first neural radiation field model to be trained can be obtained by training multiple captured images according to relevant algorithms.

[0104] Optionally, the process of determining the three-dimensional surface coordinates of the target pixel based on the volume density of the multiple physical points includes: taking physical points with a volume density greater than a volume density threshold as candidate physical points, and taking the three-dimensional coordinates of the candidate physical point that is closest to the target pixel as the three-dimensional surface coordinates of the target pixel.

[0105] Since volume density indicates the transparency of physical points, a higher volume density corresponds to lower transparency. Therefore, physical points with a volume density greater than a volume density threshold are considered opaque. Opaque physical points can be imaged in an image, and thus, they can be used as candidate physical points. Furthermore, since candidate physical points near the target pixel will occlude distant candidate physical points, thus affecting their image formation, the candidate physical point closest to the target pixel can be considered the physical point imaged on the target pixel in 3D space. Based on this, the 3D coordinates of the candidate physical point closest to the target pixel can be used as the 3D surface coordinates of the target pixel. This ensures the accuracy of the target pixel's 3D surface coordinates, resulting in a more accurate determination of the target foreground region and effectively improving the quality of the 3D reconstruction of the target object.

[0106] It should be noted that the step of determining the first neural radiation field model based on multiple captured images can be performed before, after, or simultaneously with step 201. This application embodiment does not limit this.

[0107] In the second implementation, the captured image is a depth image. In this case, each pixel in the captured image has depth. Therefore, the three-dimensional surface coordinates of the target pixel can be determined based on the depth of the target pixel in the first candidate foreground region. Processing each pixel in the first candidate foreground region in the same way allows the determination of the three-dimensional surface coordinates of each pixel in the first candidate foreground region. Processing each candidate foreground region in the same way allows the determination of the three-dimensional surface coordinates of pixels in multiple candidate foreground regions.

[0108] Optionally, based on the depth of the target pixel, the pixel coordinates of the target pixel, the pose of the camera corresponding to the first candidate foreground region, and the coordinates of the camera optical center in the pixel coordinate system, the three-dimensional surface coordinates of the target pixel are determined according to a relevant algorithm.

[0109] In some embodiments, the captured image is a depth image, in which case each pixel in the captured image also has depth. In this case, the depth of each pixel in the first candidate foreground region can be directly obtained. By processing each candidate foreground region in the same way, the depth of pixels in the multiple candidate foreground regions can be determined.

[0110] Step 203: Based on the three-dimensional surface coordinates and / or depth of pixels in multiple candidate foreground regions, determine at least one target foreground region from the multiple candidate foreground regions, wherein the target foreground region is not planar in three-dimensional space.

[0111] Based on the 3D surface coordinates and / or depth of each pixel in the first candidate foreground region, it is determined whether the shape of the first candidate foreground region in 3D space is planar. The first candidate foreground region can be any one of the multiple candidate foreground regions. If the shape of the first candidate foreground region in 3D space is not planar, then the first candidate foreground region is determined as the target foreground region. If the shape of the first candidate foreground region in 3D space is planar, then the first candidate foreground region is not determined as the target foreground region. By processing each candidate foreground region in the same way, at least one target foreground region can be determined.

[0112] There are several ways to determine whether the shape of the first candidate foreground region is planar in three-dimensional space based on the three-dimensional surface coordinates of each pixel in the first candidate foreground region. Two of these methods will be introduced below.

[0113] In the first implementation, the three-dimensional surface coordinates of each pixel in the first candidate foreground region are input into the foreground discriminator to obtain the first shape output by the foreground discriminator and the confidence level of the first shape. If the first shape is planar and the confidence level of the first shape is greater than the first confidence level threshold, then the shape of the first candidate foreground region in three-dimensional space is determined to be planar. Otherwise, the shape of the first candidate foreground region in three-dimensional space is determined to be non-planar.

[0114] The first confidence threshold is preset and can be adjusted according to needs under different circumstances. For example, the first confidence threshold can be 0.8.

[0115] In some embodiments, before determining whether the shape of the first candidate foreground region in three-dimensional space is planar based on the three-dimensional surface coordinates of each pixel in the first candidate foreground region, a first confidence threshold can also be configured. That is, the electronic device displays a first parameter configuration interface, which instructs the user to configure the first confidence threshold. In response to the user's completion of the configuration of the first confidence threshold, the first confidence threshold is obtained from the first parameter configuration interface.

[0116] The electronic device displays a first parameter configuration interface, where the user can configure a first confidence threshold. After configuration, the user can trigger a configuration completion operation for the first confidence threshold to indicate that the configuration of the first confidence threshold has been completed. In response to the configuration completion operation for the first confidence threshold, the electronic device obtains the first confidence threshold from the first parameter configuration interface.

[0117] Of course, the above-mentioned configuration of the first confidence threshold by the user is merely one implementation method. In practical applications, the first confidence threshold can also be stored in the electronic device, without the need for user configuration. This application embodiment does not limit this.

[0118] The second implementation method involves determining the 3D plane corresponding to the first candidate foreground region based on the 3D surface coordinates of each pixel in the first candidate foreground region. Then, based on the distance between the 3D surface coordinates of each pixel and this 3D plane, a morphological evaluation index for the first candidate foreground region is determined. This morphological evaluation index indicates the difference between the first candidate foreground region and its corresponding 3D plane. If the morphological evaluation index is less than a morphological evaluation index threshold, the first candidate foreground region is determined to be planar in 3D space. If the morphological evaluation index is not less than the morphological evaluation index threshold, the first candidate foreground region is determined to be non-planar in 3D space.

[0119] For ease of description, the distance between the three-dimensional surface coordinates of any pixel in the first candidate foreground region and the three-dimensional plane will be referred to as the first distance in the following text.

[0120] In some embodiments, the three-dimensional plane corresponding to the first candidate foreground region can be determined based on the three-dimensional surface coordinates of each pixel in the first candidate foreground region using the random sample consensus (RANSAC) algorithm. Of course, other algorithms can also be used to determine the three-dimensional plane corresponding to the first candidate foreground region, such as least squares method, singular value decomposition (SVD) algorithm, etc., and this application embodiment does not limit this.

[0121] In some embodiments, the morphological evaluation index of the first candidate foreground region can be determined based on the variance of a plurality of first distances, wherein the plurality of first distances correspond one-to-one with a plurality of pixels in the first candidate foreground region.

[0122] In other embodiments, the average value of multiple first distances can be determined as the morphological evaluation index of the first candidate foreground region. Of course, in practical applications, in addition to determining the variance or average value as the morphological evaluation index of the first candidate foreground region, the maximum or minimum value among multiple first distances can also be determined as the morphological evaluation index. This application does not limit this.

[0123] Among them, the threshold of the morphological evaluation index is set in advance and can be adjusted according to needs under different circumstances.

[0124] In some embodiments, before determining whether the shape of the first candidate foreground region is planar in three-dimensional space based on the three-dimensional surface coordinates of each pixel in the first candidate foreground region, a morphology evaluation index threshold can also be configured. That is, the electronic device displays a second parameter configuration interface, which instructs the user to configure the morphology evaluation index threshold. In response to the user's completion of the configuration of the morphology evaluation index threshold, the morphology evaluation index threshold is obtained from the second parameter configuration interface.

[0125] The electronic device displays a second parameter configuration interface, where the user can configure the morphological evaluation index threshold. After configuration, the user can trigger a configuration completion operation for the morphological evaluation index threshold to indicate that the configuration of the morphological evaluation index threshold has been completed. In response to the configuration completion operation for the morphological evaluation index threshold, the electronic device obtains the morphological evaluation index threshold from the second parameter configuration interface.

[0126] Of course, the above-mentioned configuration of the morphological evaluation index threshold by the user is only one implementation method. In practical applications, the morphological evaluation index threshold can also be stored in the electronic device, without the need for user configuration. This application embodiment does not limit this.

[0127] It should be noted that the first parameter configuration interface, the second parameter configuration interface, and the third parameter configuration interface mentioned above can be the same interface, or they can be different interfaces, or any two of the first parameter configuration interface, the second parameter configuration interface, and the third parameter configuration interface can be the same interface. This application embodiment does not limit this.

[0128] The process of determining whether the shape of the first candidate foreground region in three-dimensional space is planar based on the depth of each pixel in the first candidate foreground region includes: if the depth difference between any two pixels in the first candidate foreground region is less than the difference threshold, then the shape of the first candidate foreground region in three-dimensional space is determined to be planar; otherwise, the shape of the first candidate foreground region in three-dimensional space is determined not to be planar.

[0129] In some embodiments, determining whether the shape of the first candidate foreground region in three-dimensional space is planar based on the three-dimensional surface coordinates and depth of each pixel in the first candidate foreground region includes: determining whether the shape of the first candidate foreground region in three-dimensional space is planar based on the three-dimensional surface coordinates of each pixel in the first candidate foreground region to obtain a first judgment result; determining whether the shape of the first candidate foreground region in three-dimensional space is planar based on the depth of each pixel in the first candidate foreground region to obtain a second judgment result; if at least one of the first judgment result and the second judgment result determines that the shape of the first candidate foreground region in three-dimensional space is planar, then the shape of the first candidate foreground region in three-dimensional space is finally determined to be planar; otherwise, the shape of the first candidate foreground region in three-dimensional space is determined not to be planar.

[0130] The process of determining whether the shape of the first candidate foreground region in three-dimensional space is planar based on the three-dimensional surface coordinates of each pixel in the first candidate foreground region, and the detailed process of determining whether the shape of the first candidate foreground region in three-dimensional space is planar based on the depth of each pixel in the first candidate foreground region, please refer to the relevant content above, which will not be repeated here.

[0131] Step 204: Perform 3D reconstruction of the target object based on pixels in multiple captured images, excluding at least one target foreground region.

[0132] There are several ways to reconstruct a target object in three dimensions based on pixels in multiple captured images, excluding at least one target foreground region. Two of these methods will be introduced below.

[0133] The first implementation method is to determine a second neural radiation field model based on the pixels in multiple captured images, excluding at least one target foreground region. The second neural radiation field model is then used to render the image of the target object from any viewpoint to achieve three-dimensional reconstruction of the target object.

[0134] Light is sampled from pixels in multiple captured images, excluding at least one target foreground region. Then, based on the color of each pixel in the captured image, the sampled light, and the viewing angle, a neural radiation field model is trained to obtain a second neural radiation field model. For detailed implementation methods of training the neural radiation field model, please refer to relevant technologies; they will not be elaborated here.

[0135] For example, please refer to Figure 3 If only image 1 contains a foreground object among multiple captured images, then the target object in image 1 is... Figure 3The ship shown has a person in the foreground. In this case, the embodiments of this application can determine the area where the person is located in the captured image 1 as the target foreground area, and then perform three-dimensional reconstruction of the target object based on the pixels in multiple captured images other than the target foreground area.

[0136] The second implementation involves determining a second neural radiation field model based on pixels in multiple captured images, excluding at least one target foreground region. This second neural radiation field model is then used to render the image of the target object from a target viewpoint, resulting in a rendered image from any of the multiple captured images. If the rendered image contains a problem area, a target image region (either a blurred or blank area) is identified from the target captured image at the same location as the problem area. Based on this target image region, the second neural radiation field model is further trained to obtain a third neural radiation field model. This third neural radiation field model is then used to determine the 3D reconstruction result of the target object. If the rendered image does not contain a problem area, the second neural radiation field model is used to render the image of the target object from any viewpoint to achieve 3D reconstruction of the target object.

[0137] In some embodiments, the rendered image can be input into a problem region identification model to obtain the identification status of the rendered image output by the problem region identification model. This identification status includes whether a problem region exists in the rendered image, and if so, the location of the problem region within the rendered image. The electronic device determines whether a problem region exists in the rendered image based on this identification status.

[0138] Of course, the above-mentioned method of inputting the rendered image into the problem area recognition model to determine whether there is a problem area in the rendered image is only one implementation method. In practical applications, a problem area selection interface can also be displayed. This problem area selection interface instructs the user to select the problem area in the rendered image. In response to the user's completion of the problem area selection operation, the electronic device can determine that there is a problem area in the rendered image and obtain the problem area in the rendered image from the problem area selection interface.

[0139] In some embodiments, the problem area selection interface further includes a no-problem-area button, which indicates that there is no problem area in the rendered image. The electronic device determines that there is no problem area in the rendered image in response to the user's triggering operation of the no-problem-area button.

[0140] The third neural radiation field model determines that there are multiple ways to achieve the three-dimensional reconstruction result of the target object. Two of these methods will be introduced below.

[0141] The first method involves rendering images of the target object from any viewpoint using a third neural radiation field model to achieve 3D reconstruction of the target object.

[0142] The second method uses a third neural radiation field model to render the image of the target object from the target's perspective. If the rendered image has problematic areas, a prompt interface is displayed, indicating that the image quality from the current target perspective is poor and a new image needs to be uploaded. If the rendered image does not have problematic areas, the third neural radiation field model is used to render the image of the target object from any perspective to achieve a 3D reconstruction of the target object.

[0143] In some embodiments, the prompt interface further includes an "End" button and a "Re-upload" button. The "End" button instructs the user not to re-upload the image taken from the target viewpoint, while the "Re-upload" button instructs the user to re-upload the image taken from the target viewpoint. In response to the user's triggering of the "End" button, the electronic device renders the image of the target object from any viewpoint using a third neural radiation field model to achieve 3D reconstruction of the target object. In response to the user's triggering of the "Re-upload" button, the electronic device displays an image re-upload interface instructing the user to re-upload the image taken from the target viewpoint. In response to the user's triggered image upload completion operation, the electronic device retrieves the re-uploaded image taken from the target viewpoint from the image re-upload interface, replaces the image taken from the target viewpoint among the multiple captured images with the re-uploaded image taken from the target viewpoint, and returns to step 201 above.

[0144] Next will be through Figure 4 The three-dimensional reconstruction method provided in the embodiments of this application will be described again.

[0145] Please refer to Figure 4An electronic device acquires multiple captured images, and then, based on these multiple captured images, determines a first neural radiation field model and multiple candidate foreground regions. Using the first neural radiation field model, it determines the volume density of multiple physical points corresponding to target pixels in the first candidate foreground region in three-dimensional space. Based on the volume density of these multiple physical points, it determines the three-dimensional surface coordinates of the target pixel. Based on the three-dimensional surface coordinates of pixels in the multiple candidate foreground regions, it determines at least one target foreground region from these multiple candidate foreground regions. Based on the pixels in the multiple captured images other than at least one target foreground region, it determines a second neural radiation field model. Using the second neural radiation field model, it renders the image of the target object from the target viewpoint to obtain a rendered image from the target viewpoint. If the rendered image contains a problem area, it determines a target image region in the target captured image that is at the same location as the problem area. Based on this target image region, it continues to train the second neural radiation field model to obtain a third neural radiation field model. Using the third neural radiation field model, it renders the image of the target object from any viewpoint to achieve three-dimensional reconstruction of the target object.

[0146] Next will be through Figure 5 The three-dimensional reconstruction method provided in the embodiments of this application will be described again.

[0147] Please refer to Figure 5Multiple images are acquired. Based on these images, a first neural radiation field model is determined. The first image is then input into a semantic segmentation model to obtain the region containing the foreground object and its confidence level in the output image. Regions containing foreground objects with a confidence level greater than a second confidence threshold are then identified as candidate foreground regions. Each image is processed in the same way to obtain multiple candidate foreground regions. Using the first neural radiation field model, the volume density of multiple physical points corresponding to the target pixel in the first candidate foreground region is determined. Based on this volume density, the three-dimensional surface coordinates of the target pixel are determined. Each pixel in the first candidate foreground region is processed in the same way to determine its three-dimensional surface coordinates. This process is repeated for each candidate foreground region to determine the three-dimensional surface coordinates of the pixels within the multiple candidate foreground regions. Based on the 3D surface coordinates of pixels in multiple candidate foreground regions, the 3D surface coordinates of each pixel in the first candidate foreground region are input into a foreground discriminator to obtain the first shape output by the foreground discriminator and the confidence level of the first shape. If the first shape is planar and the confidence level of the first shape is greater than the first confidence threshold, then the shape of the first candidate foreground region in 3D space is determined to be planar. Each candidate foreground region is processed in the same way to determine at least one target foreground region. Based on the pixels in the multiple captured images other than at least one target foreground region, a second neural radiation field model is determined. Using the second neural radiation field model, the image of the target object from the target viewpoint is rendered to obtain a rendered image from the target viewpoint. It is determined whether there is a problem region in the rendered image. If there is a problem region, a target image region with the same location as the problem region is determined from the target captured image. Based on the target image region, the second neural radiation field model is further trained to obtain a third neural radiation field model. Using the third neural radiation field model, the image of the target object from any viewpoint is rendered to achieve 3D reconstruction of the target object.

[0148] This application embodiment can perform 3D reconstruction of a target object based on pixels in multiple captured images, excluding the at least one target foreground region. That is, pixels in the foreground region are excluded during light sampling, avoiding interference from the foreground object in the 3D reconstruction of the target object, thereby effectively improving the 3D reconstruction quality of the target object. Furthermore, since the target foreground region is not planar in 3D space, this application embodiment can also avoid problems such as missing or blurred images of billboards and reflective target objects after 3D reconstruction, thus ensuring the 3D reconstruction quality of the target object. When determining the three-dimensional surface coordinates of a target pixel based on the volume density of multiple physical points, since volume density indicates the transparency of physical points (the higher the volume density, the lower the transparency), physical points with a volume density greater than a volume density threshold are opaque. Opaque physical points can be imaged in the image. Therefore, in this embodiment, physical points with a volume density greater than the volume density threshold are used as candidate physical points. Furthermore, since candidate physical points near the target pixel will occlude candidate physical points far away, thus preventing them from being imaged in the image, the candidate physical point closest to the target pixel can be considered as the physical point imaged on the target pixel in three-dimensional space. Based on this, in this embodiment, the three-dimensional coordinates of the candidate physical point closest to the target pixel are used as the three-dimensional surface coordinates of the target pixel. This ensures the accuracy of the three-dimensional surface coordinates of the target pixel, making the final determined target foreground region more accurate, thereby effectively improving the quality of the three-dimensional reconstruction of the target object.

[0149] Figure 6 This is a schematic diagram of a three-dimensional reconstruction device provided in an embodiment of this application. This three-dimensional reconstruction device can be implemented as part or all of the aforementioned electronic device by software, hardware, or a combination of both. See also... Figure 6 The device includes: a first determining module 601, a second determining module 602, a third determining module 603, and a reconstruction module 604.

[0150] The first determining module 601 is used to determine multiple candidate foreground regions based on multiple captured images. These multiple captured images are images of the target object from different viewpoints, and the candidate foreground region refers to an image region in the captured images where a foreground object exists. Detailed implementation processes are described in the corresponding contents of the above embodiments and will not be repeated here.

[0151] The second determining module 602 is used to determine the three-dimensional surface coordinates and / or depth of pixels in multiple candidate foreground regions. For detailed implementation details, please refer to the corresponding content in the above embodiments; they will not be repeated here.

[0152] The third determining module 603 is used to determine at least one target foreground region from multiple candidate foreground regions based on the three-dimensional surface coordinates and / or depth of pixels in multiple candidate foreground regions. The target foreground region is not planar in three-dimensional space. For detailed implementation, please refer to the corresponding content in the above embodiments, which will not be repeated here.

[0153] The reconstruction module 604 is used to perform three-dimensional reconstruction of the target object based on pixels in multiple captured images, excluding at least one target foreground region.

[0154] Optionally, the second determining module 602 is specifically used for:

[0155] Based on multiple captured images, a first neural radiation field model was determined;

[0156] The volume density of multiple physical points corresponding to the target pixel in the first candidate foreground region in three-dimensional space is determined by the first neural radiation field model.

[0157] Wherein, the first candidate foreground region is any one of the multiple candidate foreground regions, the target pixel is any one of the pixels in the first candidate foreground region, and the multiple physical points are physical points on the imaging optical path of the target pixel;

[0158] The three-dimensional surface coordinates of the target pixel are determined based on the volume density of multiple physical points.

[0159] Optionally, the third determining module 603 is specifically used for:

[0160] Based on the three-dimensional surface coordinates and / or depth of each pixel in the first candidate foreground region, determine whether the shape of the first candidate foreground region in three-dimensional space is planar, and the first candidate foreground region is any one of the multiple candidate foreground regions;

[0161] If the first candidate foreground region is not planar in three-dimensional space, then the first candidate foreground region is determined as the target foreground region.

[0162] Optionally, the third determining module 603 is specifically used for:

[0163] Input the three-dimensional surface coordinates of each pixel in the first candidate foreground region into the foreground discriminator to obtain the first shape output by the foreground discriminator and the confidence level of the first shape;

[0164] If the first morphology is planar and the confidence level of the first morphology is greater than the first confidence level threshold, then the first candidate foreground region is determined to be planar in three-dimensional space.

[0165] Optionally, the third determining module 603 is specifically used for:

[0166] Based on the three-dimensional surface coordinates of each pixel in the first candidate foreground region, determine the three-dimensional plane corresponding to the first candidate foreground region;

[0167] Based on the distance between the three-dimensional surface coordinates of each pixel and the three-dimensional plane, a morphological evaluation index for the first candidate foreground region is determined, which indicates the difference between the first candidate foreground region and the three-dimensional plane.

[0168] If the morphological evaluation index is less than the morphological evaluation index threshold, then the morphology of the first candidate foreground region in three-dimensional space is determined to be planar.

[0169] Optionally, the first determining module 601 is specifically used for:

[0170] The first captured image is input into the foreground prediction model to obtain the region where the foreground object is located in the first captured image output by the foreground prediction model and the confidence level of the foreground object. The first captured image is any one of multiple captured images.

[0171] The regions containing foreground objects in the first captured image with a confidence level greater than the second confidence threshold are identified as candidate foreground regions.

[0172] Optionally, the reconstruction module 604 is specifically used for:

[0173] A second neural radiation field model is determined based on pixels in multiple captured images, excluding at least one target foreground region.

[0174] The second neural radiation field model is used to render the image of the target object from the target viewpoint, and the rendered image from the target viewpoint is any one of the viewpoints corresponding to multiple captured images.

[0175] If there is a problem area in the rendered image, then determine the target image area in the same position as the problem area from the target captured image. The problem area is a blurred area or a blank area. The target captured image is the captured image from the target viewpoint among multiple captured images.

[0176] Based on the target image region, the second neural radiation field model is further trained to obtain the third neural radiation field model.

[0177] The three-dimensional reconstruction result of the target object is determined by the third neural radiation field model.

[0178] Optionally, the device further includes:

[0179] The first display module is used to display a first parameter configuration interface, which instructs the user to configure a first confidence threshold.

[0180] The first acquisition module is used to acquire the first confidence threshold from the first parameter configuration interface in response to the user's completion of the configuration of the first confidence threshold.

[0181] Optionally, the device further includes:

[0182] The second display module is used to display the second parameter configuration interface, which instructs the user to configure the threshold of the morphological evaluation index.

[0183] The second acquisition module is used to acquire the morphological evaluation index threshold from the second parameter configuration interface in response to the user's completed configuration operation of the morphological evaluation index threshold.

[0184] Optionally, the device further includes:

[0185] The third display module is used to display the third parameter configuration interface, which instructs the user to configure the second confidence threshold.

[0186] The third acquisition module is used to acquire the second confidence threshold from the third parameter configuration interface in response to the user's completion of the configuration of the second confidence threshold.

[0187] This application embodiment can perform 3D reconstruction of a target object based on pixels in multiple captured images, excluding the at least one target foreground region. That is, pixels in the foreground region are excluded during light sampling, avoiding interference from the foreground object in the 3D reconstruction of the target object, thereby effectively improving the 3D reconstruction quality of the target object. Furthermore, since the target foreground region is not planar in 3D space, this application embodiment can also avoid problems such as missing or blurred images of billboards and reflective target objects after 3D reconstruction, thus ensuring the 3D reconstruction quality of the target object. When determining the three-dimensional surface coordinates of a target pixel based on the volume density of multiple physical points, since volume density indicates the transparency of physical points (the higher the volume density, the lower the transparency), physical points with a volume density greater than a volume density threshold are opaque. Opaque physical points can be imaged in the image. Therefore, in this embodiment, physical points with a volume density greater than the volume density threshold are used as candidate physical points. Furthermore, since candidate physical points near the target pixel will occlude candidate physical points far away, thus preventing them from being imaged in the image, the candidate physical point closest to the target pixel can be considered as the physical point imaged on the target pixel in three-dimensional space. Based on this, in this embodiment, the three-dimensional coordinates of the candidate physical point closest to the target pixel are used as the three-dimensional surface coordinates of the target pixel. This ensures the accuracy of the three-dimensional surface coordinates of the target pixel, making the final determined target foreground region more accurate, thereby effectively improving the quality of the three-dimensional reconstruction of the target object.

[0188] It should be noted that the 3D reconstruction apparatus provided in the above embodiments is only illustrated by the division of the functional modules described above. In practical applications, the functions described above can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above. In addition, the 3D reconstruction apparatus and the 3D reconstruction method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0189] This application also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the steps of the three-dimensional reconstruction method described in the above embodiments.

[0190] This application also provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform the steps of the three-dimensional reconstruction method described in the above embodiments. Alternatively, a computer program is provided that, when executed on a computer, causes the computer to perform the steps of the three-dimensional reconstruction method described in the above embodiments.

[0191] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital versatile disc (DVD)), or a semiconductor medium (e.g., solid state disk (SSD)). It is worth noting that the computer-readable storage medium mentioned in the embodiments of this application can be a non-volatile storage medium; in other words, it can be a non-transient storage medium.

[0192] It should be understood that "multiple" as mentioned herein refers to two or more. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. In addition, to facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first," "second," etc., are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and the terms "first," "second," etc., do not necessarily imply that they are different.

[0193] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in the embodiments of this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, multiple captured images involved in the embodiments of this application were obtained with full authorization.

[0194] The above descriptions are embodiments provided in this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method of three-dimensional reconstruction, characterized by, The method comprises: determining a plurality of candidate foreground regions based on a plurality of photographed images, the plurality of photographed images being images of a target object under different angles of view, and the candidate foreground region being an image region in which a foreground object exists in the photographed image; determining three-dimensional surface coordinates and / or depths of pixel points in the plurality of candidate foreground regions; determining at least one target foreground region from the plurality of candidate foreground regions based on the three-dimensional surface coordinates and / or depths of the pixel points in the plurality of candidate foreground regions, the target foreground region not being a plane in a three-dimensional space; performing three-dimensional reconstruction on the target object based on pixel points in the plurality of photographed images other than the at least one target foreground region; wherein the determining of the at least one target foreground region from the plurality of candidate foreground regions based on the three-dimensional surface coordinates and / or depths of the pixel points in the plurality of candidate foreground regions comprises: determining whether the first candidate foreground region is a plane in the three-dimensional space based on the three-dimensional surface coordinates and / or depths of the pixel points in the first candidate foreground region, the first candidate foreground region being any one of the plurality of candidate foreground regions; and if the first candidate foreground region is not a plane in the three-dimensional space, determining the first candidate foreground region as the target foreground region.

2. The method of claim 1, wherein, The determining of the three-dimensional surface coordinates of the pixel points in the plurality of candidate foreground regions comprises: determining a first neural radiance field model based on the plurality of photographed images; determining, by the first neural radiance field model, a plurality of physical point densities of a plurality of physical points corresponding to a target pixel point in the first candidate foreground region in the three-dimensional space; wherein the target pixel point is any one of the pixel points in the first candidate foreground region, and the plurality of physical points are physical points on an imaging light path of the target pixel point; determining the three-dimensional surface coordinates of the target pixel point based on the plurality of physical point densities.

3. The method of claim 1 or 2, wherein, The determining of whether the first candidate foreground region is a plane in the three-dimensional space based on the three-dimensional surface coordinates of the pixel points in the first candidate foreground region comprises: inputting the three-dimensional surface coordinates of each pixel point in the first candidate foreground region into a foreground discriminator to obtain a first shape output by the foreground discriminator and a confidence of the first shape; if the first shape is a plane and the confidence of the first shape is greater than a first confidence threshold, determining that the first candidate foreground region is a plane in the three-dimensional space.

4. The method of claim 1 or 2, wherein, The determining of whether the first candidate foreground region is a plane in the three-dimensional space based on the three-dimensional surface coordinates of the pixel points in the first candidate foreground region comprises: determining a three-dimensional plane corresponding to the first candidate foreground region based on the three-dimensional surface coordinates of each pixel point in the first candidate foreground region; determining a shape evaluation index of the first candidate foreground region based on distances between the three-dimensional surface coordinates of each pixel point and the three-dimensional plane, the shape evaluation index indicating a difference between the first candidate foreground region and the three-dimensional plane; and If the shape evaluation index is less than a shape evaluation index threshold value, it is determined that the shape of the first candidate foreground region in the three-dimensional space is a plane.

5. The method of claim 1, wherein, The determining of the multiple candidate foreground regions based on the multiple photographed images comprises: inputting a first photographed image into a foreground prediction model to obtain a region in the first photographed image in which a foreground object is located and a confidence degree of the foreground object output by the foreground prediction model, the first photographed image being any one of the multiple photographed images; determining a region in which a foreground object with a confidence degree greater than a second confidence degree threshold value in the first photographed image as the candidate foreground region.

6. The method of claim 1, wherein, The three-dimensional reconstruction of the target object based on the pixel points in the multiple photographed images except the at least one target foreground region comprises: determining a second neural radiance field model based on the pixel points in the multiple photographed images except the at least one target foreground region; rendering an image of the target object at a target view angle through the second neural radiance field model to obtain a rendered image at the target view angle, the target view angle being any one of the view angles corresponding to the multiple photographed images; if the rendered image has a problem region, determining a target image region having the same position as the problem region from a target photographed image, the problem region being a blur region or a blank region, the target photographed image being a photographed image at the target view angle in the multiple photographed images; continuously training the second neural radiance field model based on the target image region to obtain a third neural radiance field model; determining a three-dimensional reconstruction result of the target object through the third neural radiance field model.

7. The method of claim 3, wherein, Before the determining of whether the shape of the first candidate foreground region in the three-dimensional space is a plane based on the three-dimensional surface coordinates of the pixel points in the first candidate foreground region, the method further comprises: displaying a first parameter configuration interface, the first parameter configuration interface indicating a user to configure the first confidence degree threshold value; in response to a configuration completion operation of the user on the first confidence degree threshold value, acquiring the first confidence degree threshold value from the first parameter configuration interface.

8. The method of claim 4, wherein, Before the determining of whether the shape of the first candidate foreground region in the three-dimensional space is a plane based on the three-dimensional surface coordinates of the pixel points in the first candidate foreground region, the method further comprises: displaying a second parameter configuration interface, the second parameter configuration interface indicating a user to configure the shape evaluation index threshold value; in response to a configuration completion operation of the user on the shape evaluation index threshold value, acquiring the shape evaluation index threshold value from the second parameter configuration interface.

9. The method of claim 5, wherein, Before the determining of the multiple candidate foreground regions based on the multiple photographed images, the method further comprises: displaying a third parameter configuration interface, the third parameter configuration interface indicating a user to configure the second confidence degree threshold value; in response to a configuration completion operation of the user on the second confidence degree threshold value, acquiring the second confidence degree threshold value from the third parameter configuration interface.

10. An apparatus for three-dimensional reconstruction, characterized by The apparatus comprises: The first determining module is configured to determine a plurality of candidate foreground regions based on a plurality of photographed images, the photographed images being images of the target object at different angles, and the candidate foreground region being an image region in which a foreground object exists in the photographed image. The second determining module is configured to determine three-dimensional surface coordinates and / or depth of a pixel point in the plurality of candidate foreground regions. The third determining module is configured to determine at least one target foreground region from the plurality of candidate foreground regions based on the three-dimensional surface coordinates and / or depth of the pixel point in the plurality of candidate foreground regions, the target foreground region not being a plane in a three-dimensional space. The reconstruction module is configured to perform three-dimensional reconstruction on the target object based on pixel points in the plurality of photographed images except the at least one target foreground region. The third determining module is specifically configured to determine whether the first candidate foreground region is a plane in the three-dimensional space based on the three-dimensional surface coordinates and / or depth of the pixel point in the first candidate foreground region, the first candidate foreground region being any one of the plurality of candidate foreground regions; and if the first candidate foreground region is not a plane in the three-dimensional space, the first candidate foreground region is determined as the target foreground region.

11. An electronic device, comprising: The electronic device includes a memory and a processor, the memory is configured to store a computer program, and the processor is configured to execute the computer program stored in the memory to implement the steps of the method of any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The storage medium stores instructions, and when the instructions run on the computer, the computer executes the steps of the method of any one of claims 1-9.

13. A computer program, characterized in that, The computer program includes instructions, and when the instructions run on the computer, the computer executes the steps of the method of any one of claims 1-9.

Images