Three-dimensional image establishing method and related equipment

Abstract

The invention provides a three-dimensional image establishment method and related equipment, and the method comprises the steps: converting a target image into an initial three-dimensional image, and determining the certainty degree of a first Gaussian primitive according to the attribute parameter of the first Gaussian primitive in the initial three-dimensional image; according to the determinacy of each first Gaussian primitive in the initial three-dimensional image, a space body is determined, and the space body is composed of at least one second Gaussian primitive; changing the initial shooting view angle of the space body to obtain a candidate shooting view angle, and generating a target view angle image of the space body under the candidate shooting view angle; according to the target visual angle image and an initial visual angle image of the space body under the initial shooting visual angle, a second Gaussian primitive in the initial three-dimensional image is reconstructed, a target three-dimensional image is obtained, and the determination degree of the reconstructed second Gaussian primitive is larger than or equal to a preset threshold value. According to the invention, the image quality of the three-dimensional image is improved.

Description

Technical Field

[0001] This application relates to the field of image technology, and in particular to a method for creating three-dimensional images and related equipment. Background Technology

[0002] In scenarios such as intelligent transportation, autonomous driving, and underground parking management, image recognition and perception systems play a crucial role in accurately perceiving the environment. However, ambient lighting and ground materials can cause specular reflections, resulting in large areas of specular highlights in the acquired images, which severely interferes with the texture information and semantic content of the images.

[0003] In the exemplary technology, Gaussian sputtering technology is used to render multi-view images to obtain high-fidelity 3D images, thereby solving the problem of interference with texture information and semantic content in the image.

[0004] However, when the camera captures multi-view images along the trajectory, there will be a slight trajectory offset. Therefore, when rendering multi-view images, the rendered images are prone to artifacts, blurring, or structural errors, resulting in poor image quality of the rendered 3D images. Summary of the Invention

[0005] Based on the aforementioned technological status, this application provides a method and related equipment for creating three-dimensional images to address the problem of poor image quality in three-dimensional images.

[0006] To achieve the above-mentioned technical objectives, this application proposes the following technical solution: Firstly, this application provides a method for creating a three-dimensional image, including: The target image is converted into an initial three-dimensional image, and the certainty of the first Gaussian element is determined based on the attribute parameters of the first Gaussian element in the initial three-dimensional image. The certainty is used to indicate the accuracy of the object represented by the first Gaussian element. Based on the certainty of each first Gaussian unit in the initial three-dimensional image, a spatial volume is determined, wherein the spatial volume is composed of at least one second Gaussian unit, and the second Gaussian unit is a first Gaussian unit with a certainty of less than a preset threshold. The initial shooting angle of the space object is changed to obtain a candidate shooting angle, and a target shooting angle image of the space object under the candidate shooting angle is generated; Based on the target viewpoint image and the initial viewpoint image of the spatial object under the initial shooting viewpoint, the second Gaussian element in the initial three-dimensional image is reconstructed to obtain the target three-dimensional image. The certainty of the reconstructed second Gaussian element is greater than or equal to a preset threshold.

[0007] In some implementations, changing the initial shooting angle of the space body to obtain a candidate shooting angle includes: Determine the center of gravity position of the spatial body, and determine the image acquisition position adjacent to the center of gravity position, wherein the image acquisition position is the position of the camera that captures the first object in the target image; Obtain the initial camera pose corresponding to the image acquisition position, wherein the initial camera pose is the pose of the camera when it captures the first object at the image acquisition position; The initial camera pose is modified to obtain candidate shooting angles.

[0008] In some implementations, the pose change of the initial camera pose includes: The pose change conditions are obtained, including at least one of the following: the height range of the camera after pose change, the angle range between the acquisition direction of the camera after pose change and the target direction, and the distance range between the camera and the center of gravity position after pose change. The target direction is the direction of the line connecting the camera and the center of gravity position. The initial camera pose is changed according to the pose change conditions.

[0009] In some implementations, generating a target view image of the spatial volume under the candidate shooting viewpoint includes: Generate Gaussian feature maps and transparency maps under the candidate shooting perspectives; The real-view image of the second object corresponding to the spatial volume taken by the camera, the camera parameters when the camera took the second object, the Gaussian feature map and the transparency map are input into the image generation model to obtain multiple intermediate view images. A scoring model is used to score each of the intermediate view images to obtain a quality score for each intermediate view image; The intermediate view image corresponding to the highest quality score is determined as the target view image.

[0010] In some implementations, after scoring each of the intermediate view images using a scoring model, the process includes: Based on the quality score, a first view image and a second view image are determined among multiple intermediate view images. The first view image is an intermediate view image with a quality score greater than or equal to a score threshold, and the second view image is an intermediate view image with a quality score less than a score threshold. Training samples are generated based on the first viewpoint image and the camera parameters, and supervised samples are generated based on the second viewpoint image. The training samples and the supervision samples are stored. The training samples are used to train the scoring model, and the supervision samples are used to supervise the learning of the scoring model.

[0011] In some implementations, after scoring each of the intermediate view images using a scoring model, the process includes: Extract the target sub-region from the intermediate view image, wherein the confidence level of the target sub-region is higher than a confidence threshold; The target sub-region is used as a supervision signal, and the scoring model is subjected to supervised learning based on the supervision signal.

[0012] In some implementations, determining the certainty of the first Gaussian element based on its attribute parameters in the initial 3D image includes: Based on the attribute parameters of the first Gaussian element, determine the transparency, shape certainty, and color difference of the object represented by the first Gaussian element. The certainty of the first Gaussian element is determined based on the transparency, the shape certainty, and the color difference.

[0013] Secondly, this application provides a three-dimensional image creation apparatus, comprising: A conversion module is used to convert a target image into an initial three-dimensional image, and to determine the certainty of the first Gaussian element based on the attribute parameters of the first Gaussian element in the initial three-dimensional image. The certainty is used to indicate the accuracy of the object represented by the first Gaussian element. The determination module is used to determine a spatial volume based on the certainty of each first Gaussian element in the initial three-dimensional image. The spatial volume is composed of at least one second Gaussian element, and the second Gaussian element is a first Gaussian element whose certainty is less than a preset threshold. The processing module is used to change the initial shooting angle of the space object to obtain a candidate shooting angle, and generate a target perspective image of the space object under the candidate shooting angle. The reconstruction module is used to reconstruct the second Gaussian element in the initial three-dimensional image based on the target view image and the initial view image of the spatial object under the initial shooting view, so as to obtain the target three-dimensional image. The certainty of the reconstructed second Gaussian element is greater than or equal to a preset threshold.

[0014] Thirdly, this application provides an electronic device, including a memory and a processor, wherein, The memory is connected to the processor and is used to store programs; The processor is used to implement the three-dimensional image creation method as described in the first aspect or any implementation thereof by running a program in the memory.

[0015] Fourthly, this application provides a computer program product, which, when executed by a processor, implements the three-dimensional image creation method as described in the first aspect or any implementation thereof.

[0016] Fifthly, this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the three-dimensional image creation method as described in the first aspect or any implementation thereof.

[0017] This application provides a method and related apparatus for creating a three-dimensional image. The method converts a target image into an initial three-dimensional image. It determines the degree of certainty regarding the accuracy of the object represented by the first Gaussian unit through the attribute parameters of the first Gaussian unit in the initial three-dimensional image. Based on the degree of certainty of each first Gaussian unit in the initial three-dimensional image, it determines a spatial volume composed of at least one second Gaussian unit. It then modifies the initial shooting angle of the spatial volume to obtain a candidate shooting angle, and generates a target perspective image of the spatial volume under the candidate shooting angle. Finally, it reconstructs the second Gaussian units in the initial three-dimensional image using the target perspective image and the initial perspective image of the spatial volume under the initial shooting angle, thereby obtaining the target three-dimensional image. In this application, the second Goughbred element is the first Goughbred element with an accuracy lower than a threshold. That is, by obtaining a spatial volume with low certainty from the first Goughbred element with low accuracy, the shooting angle of the spatial volume is changed to generate a target view image of the spatial volume under the changed shooting angle. Based on the target view image and the initial view image under the initial shooting angle, the second Goughbred element is reconstructed, so that the certainty of the reconstructed second Goughbred element is greater than or equal to the preset threshold. That is, the accuracy of the objects represented by each Goughbred element in the reconstructed target 3D image is high, thereby improving the image quality of the 3D image. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0019] Figure 1 A flowchart of a three-dimensional image creation method provided in this application embodiment Figure 1 .

[0020] Figure 2 A flowchart of a three-dimensional image creation method provided in this application embodiment Figure 2 .

[0021] Figure 3 A flowchart of a three-dimensional image creation method provided in this application embodiment Figure 3 .

[0022] Figure 4 A flowchart of a three-dimensional image creation method provided in this application embodiment Figure 4 .

[0023] Figure 5 This is a schematic diagram of the functional modules of a three-dimensional image creation device provided in an embodiment of this application.

[0024] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] It should be noted that the user information (including but not limited to electrical equipment information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0027] In scenarios such as intelligent transportation, autonomous driving, and underground parking management, image recognition and perception systems play a crucial role in accurately perceiving the environment. However, ambient lighting and ground materials can cause specular reflections, resulting in large areas of specular highlights in the acquired images, which severely interferes with the texture information and semantic content of the images.

[0028] In the exemplary technology, Gaussian sputtering technology is used to render multi-view images to obtain high-fidelity 3D images, thereby solving the problem of interference with texture information and semantic content in the image.

[0029] However, when the camera captures multi-view images along the trajectory, there will be a slight trajectory offset. Therefore, when rendering multi-view images, the rendered images are prone to artifacts, blurring, or structural errors, resulting in poor image quality of the rendered 3D images.

[0030] In view of this, the embodiments of this application aim to provide a three-dimensional image creation method and related equipment, which converts a target image into an initial three-dimensional image, determines the degree of certainty of the accuracy of the object represented by the first Gaussian unit through the attribute parameters of the first Gaussian unit in the initial three-dimensional image, determines a spatial volume composed of at least one second Gaussian unit based on the degree of certainty of each first Gaussian unit in the initial three-dimensional image, changes the initial shooting angle of the spatial volume to obtain a candidate shooting angle, and then generates a target perspective image of the spatial volume under the candidate shooting angle. Thus, the second Gaussian unit in the initial three-dimensional image is reconstructed through the target perspective image and the initial perspective image of the spatial volume under the initial shooting angle to obtain the target three-dimensional image. In this application, the second Goughbred element is the first Goughbred element with an accuracy lower than a threshold. That is, by obtaining a spatial volume with low certainty from the first Goughbred element with low accuracy, the shooting angle of the spatial volume is changed to generate a target view image of the spatial volume under the changed shooting angle. Based on the target view image and the initial view image under the initial shooting angle, the second Goughbred element is reconstructed, so that the certainty of the reconstructed second Goughbred element is greater than or equal to the preset threshold. That is, the accuracy of the objects represented by each Goughbred element in the reconstructed target 3D image is high, thereby improving the image quality of the 3D image.

[0031] Figure 1 A flowchart of a three-dimensional image creation method provided in this application embodiment Figure 1 .like Figure 1 As shown, the three-dimensional image creation method provided in this embodiment includes: Step S101: Convert the target image into an initial three-dimensional image, and determine the degree of certainty of the first Gaussian element based on the attribute parameters of the first Gaussian element in the initial three-dimensional image. The degree of certainty is used to indicate the accuracy of the object represented by the first Gaussian element.

[0032] In this embodiment, the execution subject is a three-dimensional image creation device. The three-dimensional image creation device can be any terminal device with Gaussian sputtering function. For ease of description, the device is used to refer to the three-dimensional image creation device below.

[0033] The user uploads an image to be rendered to the device; this image is defined as the target image, which can be a street view image. The device uses Gaussian sputtering to convert the target image into a 3D image, which is defined as the initial 3D image. For example, the device runs the Gaussian sputtering function by inputting the target image and its camera parameters into an application with Gaussian sputtering capabilities. The application outputs the initial 3D image, where the camera parameters include the intrinsic and extrinsic parameters of the camera that acquired the target image. The initial 3D image contains multiple Gaussian primitives represented by Gaussian point clouds, each of which can be considered an object in the target image. To facilitate differentiation from subsequent Gaussian primitives, the Gaussian primitives in the initial 3D image are defined as the first Gaussian primitive.

[0034] The initial 3D image contains a first set of Gaussian elements G, each of which is a first Gaussian element. It contains multiple attribute parameters, for example, multiple attribute parameters are , representing the position, rotation, scale, and opacity of the Gaussian element, respectively. The apparatus determines the determinism of the first Gaussian element based on its attribute parameters. .

[0035] In one example, each attribute parameter is compared with its corresponding attribute threshold to obtain a comparison result for each attribute parameter. Then, a score is determined for each comparison result. The degree of certainty is obtained by weighting the attribute parameter's weight and the score of its corresponding comparison result. If the comparison result indicates that the attribute parameter meets the standard, the score is set to a preset first value for that attribute parameter. If the comparison result indicates that the attribute parameter does not meet the standard, the score is set to a preset second value for that attribute parameter, where the first value is greater than the second value.

[0036] In another example, the device determines the transparency, shape certainty, and color difference of the object represented by the first Gaussian element based on the attribute parameters of the first Gaussian element, and determines the certainty of the first Gaussian element through the transparency, shape certainty, and color difference.

[0037] For example, = Low transparency indicates the stability of sampling and visibility; The covariance matrix is ​​formed by the eigenvectors of the three main directions and their corresponding eigenvalues, which is used to characterize the local position and shape uncertainty. The three main directions refer to the three main shooting directions of the street scene image. The attribute parameters of the three first Gaussian elements in the three main directions form three eigenvectors, and the eigenvalues ​​of the three eigenvectors form the covariance matrix. , refers to the difference values ​​of color and features projected by the first Gaussian element in multiple view images; Therefore, the degree of certainty =1-( ),in, , as well as The weights are used as the basis for determining the certainty of each first Gaussian element.

[0038] Step S102: Based on the certainty of each first Gaussian unit in the initial three-dimensional image, determine the spatial volume. The spatial volume is composed of at least one second Gaussian unit, and the second Gaussian unit is a first Gaussian unit with a certainty of less than a preset threshold.

[0039] After determining the certainty of the first Gaussian unit, a spatial volume is determined based on the certainty of each first Gaussian unit in the initial 3D image. The spatial volume is composed of at least one second Gaussian unit, which is a first Gaussian unit with a certainty of less than a preset threshold.

[0040] For example, the device first identifies a first Gaussian element with a certainty greater than a preset threshold as a second Gaussian element in the initial three-dimensional image, then clusters the second Gaussian elements and clusters adjacent second Gaussian elements into a cluster. Each cluster of second Gaussian elements constitutes a spatial volume, and the device can identify at least one spatial volume.

[0041] Step S103: Modify the initial shooting angle of the space object to obtain a candidate shooting angle, and generate a target perspective image of the space object under the candidate shooting angle.

[0042] After determining the spatial object, the device modifies the initial shooting perspective of the spatial object to obtain candidate shooting perspectives. For example, the target image is captured by the camera along a predetermined trajectory. Therefore, when capturing the spatial object, the camera has a camera pose, which refers to the camera's height, shooting direction, distance between the camera and the Gaussian unit, and the angle between the camera's shooting direction and the plane containing the Gaussian unit; this angle is the camera's rotation angle. The device stores the camera poses captured along the predetermined trajectory and uses these camera poses as the initial shooting perspective. The device can obtain candidate shooting perspectives by modifying the initial shooting perspective. For example, by translating the distance in the initial shooting perspective and changing the rotation angle, candidate shooting perspectives can be obtained. The translation distance is less than a distance threshold, and the rotation angle is less than an angle threshold. Multiple candidate shooting perspectives can be obtained. For example, if the initial shooting angle indicates that the distance between the camera and the object is 5m and the angle between the camera's shooting direction and the plane containing the side of the object being shot is 45°, then 5m can be changed to 5.5m or 4.5m, and 45° can be changed to 44° or 46° to obtain candidate shooting angles.

[0043] After obtaining candidate shooting viewpoints, a viewpoint image of the spatial object under the candidate shooting viewpoints is generated, and this viewpoint image is defined as the target viewpoint image. For example, the device includes a conditional diffusion model. It constructs conditional inputs using candidate shooting viewpoints and uses real viewpoint images from other shooting viewpoints adjacent to the initial shooting viewpoint as semantic guidance. The candidate shooting viewpoints are projected, and the resulting image is rasterized into a Gaussian feature map with several channels. Finally, the Gaussian feature map, the conditional inputs, and the real viewpoint images from other shooting viewpoints are input to the conditional diffusion model. The conditional diffusion model then outputs the viewpoint image of the spatial object under the candidate shooting viewpoints, which is defined as the target viewpoint image. The device can select depth map or normal map estimation as additional conditions, which can be used as conditional inputs.

[0044] Step S104: Based on the target view image and the initial view image of the spatial object under the initial shooting view, the second Gaussian element in the initial three-dimensional image is reconstructed to obtain the target three-dimensional image. The certainty of the reconstructed second Gaussian element is greater than or equal to a preset threshold.

[0045] In this embodiment, the camera takes pictures along a set trajectory, thus capturing a viewpoint image of the space object at an initial shooting angle, which is defined as the initial viewpoint image. The device reconstructs the second Gaussian elements contained within the space object based on the target viewpoint image and the initial viewpoint image.

[0046] For example, the device iteratively optimizes the second Gaussian-rendered image and the real image using pixel losses, supplemented by perceptual loss and regularization loss as constraints. During the differentiable rendering process, the error gradient is backpropagated to each second Gaussian primitive, jointly updating the position, shape, color, and opacity of the second Gaussian primitives, thereby achieving reconstruction of the second Gaussian primitives. Here, the Gaussian-rendered image is obtained by rendering the target viewpoint image, and the real image is the initial viewpoint image. The pixel loss between the Gaussian-rendered image and the real image represents the pixel loss between the image rendered from the target viewpoint image and the initial viewpoint image. The certainty of the reconstructed second Gaussian primitives is greater than or equal to a preset threshold. Once the second Gaussian primitives contained in each spatial volume in the initial 3D image are reconstructed, the target 3D image is obtained.

[0047] In this embodiment, the target image is converted into an initial 3D image. The accuracy of the object represented by the first Gaussian primitive (GG) in the initial 3D image is determined using the attribute parameters of the first GG in the initial 3D image. Based on the accuracy of each first GG in the initial 3D image, a spatial volume composed of at least one second GG is determined. The initial shooting angle of the spatial volume is changed to obtain a candidate shooting angle. A target perspective image of the spatial volume under the candidate shooting angle is then generated. The second GG in the initial 3D image is reconstructed using the target perspective image and the initial perspective image of the spatial volume under the initial shooting angle to obtain the target 3D image. In this embodiment, the second GG is the first GG with an accuracy lower than a threshold. That is, a spatial volume with low accuracy is obtained by using the first GG with low accuracy. The shooting angle of the spatial volume is then changed to generate a target perspective image of the spatial volume under the changed shooting angle. Based on the target perspective image and the initial perspective image under the initial shooting angle, the second GG is reconstructed, ensuring that the accuracy of the reconstructed second GG is greater than or equal to a preset threshold. This means that the accuracy of the objects represented by each GG in the reconstructed target 3D image is high, improving the image quality of the 3D image.

[0048] Figure 2 A flowchart of a three-dimensional image creation method provided in this application embodiment Figure 2 ,based on Figure 1 In the embodiment shown, step S103 includes: Step S201: Determine the center of gravity of the space body and determine the image acquisition position adjacent to the center of gravity. The image acquisition position is the position of the camera that captures the first object in the target image.

[0049] In this embodiment, for the space body Then, the position of the center of gravity is calculated. For example, a spatial volume... The device comprises multiple second Gausky elements. First, the center point of each second Gausky element is determined. These center points constitute a region, and the center of this region is the center of gravity of the spatial volume. Alternatively, each center point has coordinates; the average values ​​of the horizontal, vertical, and triangular coordinates constitute the coordinates of the center of gravity. In this way, the center of gravity of the spatial volume can be determined.

[0050] After determining the center of gravity position, determine the image acquisition position adjacent to the center of gravity position. The image acquisition position is the position of the camera that acquires the first object in the target image.

[0051] Step S202: Obtain the initial camera pose corresponding to the image acquisition position. The initial camera pose is the pose of the camera when it captures the first object at the image acquisition position.

[0052] The device stores the camera poses at different image acquisition positions. The camera pose associated with the image acquisition position is used as the initial camera pose. That is, the initial camera pose refers to the pose of the camera when acquiring an object in the target image. This object is defined as the first object.

[0053] For example, after determining the center of gravity position, the device finds the image acquisition positions adjacent to the center of gravity position. For instance, a circle is constructed with the center of gravity position as the center and a preset value as the radius. The camera poses associated with the image acquisition positions within the circle are used as the initial camera poses. When the camera poses contain timestamp information (timestamp information refers to the timestamp of the image being captured at the image acquisition position), the image acquisition positions within a time window of ±2 seconds are further restricted. For example, if there are 5 camera poses associated with the image acquisition positions within the circle, and the absolute value of the difference between the timestamps of 3 of these camera poses is less than 2 seconds, then these 3 camera poses are used as the initial camera poses.

[0054] Step S203: Modify the initial camera pose to obtain candidate shooting angles.

[0055] After obtaining the initial camera pose, several candidate camera poses are generated by slightly translating and rotating the initial camera pose. The candidate camera poses are the candidate shooting angles.

[0056] In one example, after the initial camera pose image acquisition position is near the centroid, the device makes small changes to the distance and the included angle in the initial camera pose to obtain candidate shooting angles. A small change in distance means that the change in distance between the camera and the object is less than a distance threshold; a small change in rotation means that the change in the angle between the camera's shooting direction and the side of the object being shot is less than an angle threshold.

[0057] In another example, the device acquires pose change conditions and modifies the initial camera pose based on these conditions. For instance, the device obtains candidate camera poses by modifying the initial camera pose using pose change conditions. The pose change conditions include at least one of the following: the camera's height range after pose change; the angle range between the camera's acquisition direction and the target direction after pose change; and the distance range between the camera and the center of gravity position after pose change. The target direction is the direction of the line connecting the camera and the center of gravity position.

[0058] For example, the generated candidate camera poses must meet physical and scene constraints (pose change conditions). The camera height is fixed at 1.5 to 2 meters (typical camera height for street scene vehicles or handheld acquisition) to ensure it is parallel to the ground and maintains a horizontal attitude. The camera's main line of sight must point to the center of gravity of the corresponding spatial object, and the line of sight deviation must be controlled within ±15° to ensure that the spatial object is located in the center of the image field of view. The viewing distance (i.e., the Euclidean distance from the camera center to the center of gravity of the spatial object) should be limited to 3 to 25 meters to avoid occlusion if too close or insufficient features if too far. In addition, after generating candidate camera poses, a fast occlusion estimation is performed by reconstructing the point cloud projection to filter out views that are occluded by surrounding buildings or objects by more than 60%. Only the top 4 candidate camera poses with less occlusion and the most newly added information in the covered area are retained for the generation of subsequent target view images.

[0059] In this embodiment, the center of gravity of the spatial object is determined, and the initial camera pose is determined based on the center of gravity. The initial camera pose is then modified to accurately obtain the candidate shooting angle.

[0060] Figure 3 A flowchart of a three-dimensional image creation method provided in this application embodiment Figure 3 ,based on Figure 1 or Figure 2 In the embodiment shown, step S103 includes: Step S301: Generate Gaussian feature maps and transparency maps under candidate shooting perspectives.

[0061] In this embodiment, after obtaining the candidate shooting view, the device projects the Gaussian point cloud of the second Gaussian element of the spatial volume onto the candidate shooting view and rasterizes it into a multi-channel Gaussian feature map and a transparency map.

[0062] Step S302: Input the real-view image of the second object corresponding to the spatial volume captured by the camera, the camera parameters when the camera captured the second object, the Gaussian feature map, and the transparency map into the image generation model to obtain multiple intermediate view images.

[0063] The device inputs the real-view image of the second object corresponding to the spatial volume captured by the camera, the camera parameters when capturing the second object, the Gaussian feature map, and the transparency map into the image generation model. This produces multiple view images output by the image generation model, which are defined as intermediate view images. A real-view image refers to the actual view image of the second object captured by an image acquisition position near the centroid. There can be multiple image acquisition positions near the centroid, resulting in multiple real-view images.

[0064] For example, for each candidate captured vision Construct conditional inputs to feed into the image generation model (conditional diffusion model), specifically as follows: 1. The second Gaussian element contained in the space volume exist The projection is rasterized into a Gaussian feature map and a transparency map with several channels.

[0065] 2. Select poses from neighboring cameras In A viewpoint image (real viewpoint image) is used as a reference image, and depth map or normal map estimation can be used as an additional condition.

[0066] 3. Output the synthesized image using the conditional diffusion model. The conditional diffusion model's backbone is a U-Net network structure. Depth, normals, and other control conditions are injected into multiple decoding layers of the U-Net, and feature fusion is performed through cross-attention. The sampling phase of the conditional diffusion model uses 50 steps of back-diffusion to achieve a balance between speed and quality. Classless conditional guidance is used during sampling to improve geometric consistency, enabling the conditional diffusion model's generator to output a high-resolution synthetic RGB image and a pixel-level confidence mask as an intermediate viewpoint image.

[0067] Step S303: Using a scoring model, each intermediate-view image is scored to obtain a quality score for each intermediate-view image.

[0068] After obtaining multiple intermediate-view images, a scoring model is used to score each intermediate-view image, thus obtaining a quality score for each image. The scoring model can be trained using real-view images as positive samples and failed-to-synthesize view images as negative samples.

[0069] Step S304: The intermediate view image corresponding to the highest quality score is determined as the target view image.

[0070] After determining the quality score of each intermediate view image, the intermediate view image corresponding to the highest quality score is determined as the target view image.

[0071] In this embodiment, multiple intermediate view images are generated by an image generation model, and then the intermediate view images are scored by a scoring model to obtain a quality score. Based on the quality score, the intermediate view image with the best image quality is determined as the target view image.

[0072] Figure 4 A flowchart of a three-dimensional image creation method provided in this application embodiment Figure 4 .based on Figure 3 In the embodiment shown, after step S303, the method further includes: Step S401: Based on the quality score, determine the first view image and the second view image from multiple intermediate view images. The first view image is an intermediate view image with a quality score greater than or equal to the score threshold, and the second view image is an intermediate view image with a quality score less than the score threshold.

[0073] In this embodiment, the scoring model can be a pre-trained model. After scoring the intermediate view image, the scoring model can be trained based on the scored intermediate view image.

[0074] For example, based on the quality score, a first-view image and a second-view image are determined from multiple intermediate-view images. The first-view image is an intermediate-view image with a quality score greater than or equal to a score threshold, and the second-view image is an intermediate-view image with a quality score less than a score threshold.

[0075] Step S402: Generate training samples based on the first-view image and camera parameters, and generate supervised samples based on the second-view image.

[0076] The device generates training samples based on first-view images and camera parameters, and generates supervised samples based on second-view images.

[0077] Step S403: Store training samples and supervision samples. The training samples are used to train the scoring model, and the supervision samples are used to supervise the learning of the scoring model.

[0078] The device stores training samples and supervised samples. The training samples are used to train the scoring model, and the supervised samples are used to supervise the learning of the scoring model.

[0079] In addition, after scoring the intermediate view image, the target sub-region in the intermediate view image is extracted. The confidence of the target sub-region is higher than the confidence threshold. The target sub-region is used as a supervision signal, and the scoring model is supervised learning based on the supervision signal.

[0080] For example, the scoring model's scoring and training process is as follows: 1. Use a scoring model Give each (Mid-view image) Calculate quality score , In the initial stage, real original perspectives can be used as positive samples, and known failed synthetic samples can be used as negative samples for pre-training. The threshold will then The camera parameters are added to the incremental training set; otherwise, they are discarded or added with low weight. The scoring model can also output a local confidence map, used to select only high-confidence regions in the image as supervision. The discriminator uses a ResNet-50 residual network as the backbone quality regressor, outputting a global quality score s ∈ [0,1] for the synthesized image. Its goal is to make true samples as close to 1 as possible and low-quality samples as close to 0. The training loss uses mean squared error regression loss.

[0081] 2. Re-optimize the Gaussian parameters using the updated training set, optimizing the loss including pixel reconstruction loss, perceptual loss, depth consistency loss, and regularization term (covariance regularization); update the Gaussian parameters via backpropagation gradients. Location, covariance, color / feature, opacity, or hash table features, etc.

[0082] 3. If the average uncertainty of Gaussian elements in the 3D image obtained in step 2 decreases to the threshold or the number of iterations reaches the upper limit, then stop training; otherwise, return to step 1 for retraining.

[0083] In this embodiment, a scoring model is used to score each intermediate view image. Initially, the scoring model is pre-trained using real view samples as positive samples and known failed synthesized images as negative samples. Intermediate view images with scores above a set threshold are added to the incremental training set along with their corresponding camera parameters; intermediate view images with scores below the threshold are either discarded or used as weakly supervised samples with low weight. The scoring model can also output a local confidence map, which is used to adopt only high-confidence regions in the synthesized image as supervisory signals for supervised training of the scoring model.

[0084] Corresponding to the above-described three-dimensional image creation method, this application also provides a three-dimensional image creation apparatus. Figure 5 This is a schematic diagram of a module of a three-dimensional image creation apparatus provided in an embodiment of this application. The three-dimensional image creation apparatus 500 provided in this embodiment includes: The conversion module 510 is used to convert the target image into an initial three-dimensional image, and to determine the certainty of the first Gaussian element based on the attribute parameters of the first Gaussian element in the initial three-dimensional image. The certainty is used to indicate the accuracy of the object represented by the first Gaussian element. The determination module 520 is used to determine a spatial volume based on the certainty of each first high-order unit in the initial three-dimensional image. The spatial volume is composed of at least one second high-order unit, and the second high-order unit is a first high-order unit with a certainty of less than a preset threshold. The processing module 530 is used to change the initial shooting angle of the space object to obtain a candidate shooting angle, and generate a target perspective image of the space object under the candidate shooting angle. The reconstruction module 540 is used to reconstruct the second Gaussian element in the initial three-dimensional image based on the target view image and the initial view image of the spatial object under the initial shooting view, so as to obtain the target three-dimensional image. The certainty of the reconstructed second Gaussian element is greater than or equal to a preset threshold.

[0085] In some implementations, the 3D image building device 500 is also used for: Determine the position of the center of gravity of the space object, and determine the image acquisition position adjacent to the center of gravity. The image acquisition position is the position of the camera that captures the first object in the target image. Obtain the initial camera pose corresponding to the image acquisition location. The initial camera pose is the pose of the camera when it captures the first object at the image acquisition location. The initial camera pose is modified to obtain candidate shooting angles.

[0086] In some implementations, the 3D image building device 500 is also used for: The pose change conditions are obtained. The pose change conditions include at least one of the following: the height range of the camera after pose change, the angle range between the acquisition direction of the camera after pose change and the target direction, and the distance range between the camera and the center of gravity after pose change. The target direction is the direction of the line connecting the camera and the center of gravity. The initial camera pose is changed according to the pose change conditions.

[0087] In some implementations, the 3D image building device 500 is also used for: Generate Gaussian feature maps and transparency maps for candidate shooting perspectives; The real-view image of the second object corresponding to the spatial volume taken by the camera, the camera parameters when the camera took the second object, the Gaussian feature map and the transparency map are input into the image generation model to obtain multiple intermediate view images. A scoring model is used to score each intermediate-view image, and a quality score is obtained for each intermediate-view image. The intermediate view image corresponding to the highest quality score is determined as the target view image.

[0088] In some implementations, the 3D image building device 500 is also used for: Based on the quality score, a first-view image and a second-view image are determined from multiple intermediate-view images. The first-view image is an intermediate-view image with a quality score greater than or equal to the score threshold, and the second-view image is an intermediate-view image with a quality score less than the score threshold. Training samples are generated based on first-view images and camera parameters, and supervised samples are generated based on second-view images. Store training samples and supervised samples. Training samples are used to train the scoring model, and supervised samples are used to supervise the learning of the scoring model.

[0089] In some implementations, the 3D image building device 500 is also used for: Extract the target sub-region from the intermediate view image, where the confidence level of the target sub-region is higher than the confidence threshold; The target sub-region is used as a supervision signal, and the scoring model is subjected to supervised learning based on the supervision signal.

[0090] In some implementations, the 3D image building device 500 is also used for: Based on the attribute parameters of the first Gaussian element, determine the transparency, shape certainty, and color difference of the object represented by the first Gaussian element; The certainty of the first Gaussian element is determined based on transparency, shape certainty, and color difference.

[0091] The three-dimensional image creation apparatus and method provided in the above embodiments of this application belong to the same concept and can execute the three-dimensional image creation method provided in any of the above embodiments of this application. They have the corresponding functional modules and beneficial effects of executing the three-dimensional image creation method. Technical details not described in detail in this embodiment can be found in the specific processing content of the three-dimensional image creation method provided in the above embodiments of this application, and will not be repeated here.

[0092] The functions implemented by each module in the 3D image creation device can be implemented by the same or different processors, and this application embodiment does not limit this.

[0093] It should be understood that the modules in the above-described 3D image creation device can be implemented in the form of processor calling firmware. For example, the system includes a processor connected to a memory containing instructions. The processor calls the instructions stored in the memory to implement any of the above methods or to realize the functions of each module of the device. The processor can be a general-purpose processor, such as a CPU or microprocessor, and the memory can be internal to the device or external to the system. Alternatively, the modules in the system can be implemented in the form of hardware circuits. By designing the hardware circuits, some or all of the module functions can be realized. The hardware circuits can be understood as one or more processors. For example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above modules are realized by designing the logical relationships of the components within the circuit. In another implementation, the hardware circuit can be implemented using a PLD, such as an FPGA, which can include a large number of logic gates. The connection relationships between the logic gates are configured through configuration files, thereby realizing the functions of some or all of the above modules. All modules of the above-described 3D image creation device can be implemented entirely by processor calling firmware, entirely by hardware circuits, or partially by processor calling firmware with the remaining parts implemented by hardware circuits.

[0094] In this application embodiment, a processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction reading and execution capabilities, such as a CPU, microprocessor, GPU, or DSP. In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. These logical relationships are fixed or reconfigurable. For example, the processor may be a hardware circuit implemented as an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the processor loading instructions to implement the functions of some or all of the above modules. Furthermore, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, or DPU.

[0095] As can be seen, each module in the above three-dimensional image creation device can be one or more processors (or processing circuits) configured to implement the above methods, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor types.

[0096] Furthermore, the modules in the above-mentioned 3D image creation device can be integrated in whole or in part, or they can be implemented independently. In one implementation, these modules are integrated together and implemented in the form of a System-on-Chip (SoC). The SoC may include at least one processor for implementing any of the above methods or implementing the functions of the modules of the device. The at least one processor can be of different types, such as CPU and FPGA, CPU and artificial intelligence processor, CPU and GPU, etc.

[0097] This application provides another structural schematic diagram of an electronic device, see [link to schematic diagram]. Figure 6 As shown, the electronic device includes a memory 600 and a processor 610; wherein the memory 600 is connected to the processor 610 and is used to store programs; the processor 610 is used to implement the three-dimensional image creation method disclosed in any of the above embodiments by running the programs stored in the memory 600.

[0098] Specifically, the aforementioned electronic device may further include: a bus, a communication interface 620, an input device 630, and an output device 640. The electronic device may also include a data transceiver module, an image monitoring module, and a signal monitoring module.

[0099] The processor 610, memory 600, communication interface 620, input device 630, and output device 640 are interconnected via a bus. Among them: A bus can include a pathway for transmitting information between various components in an electronic device.

[0100] The processor 610 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present invention. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0101] The processor 610 may include a main processor, as well as a baseband chip, modem, etc.

[0102] The memory 600 stores a program that executes the technical solution of this invention, and may also store an operating system and other key business functions. Specifically, the program may include program code, which includes computer operation instructions. More specifically, the memory 600 may include read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), other types of dynamic storage devices capable of storing information and instructions, disk storage, flash memory, etc.

[0103] Input device 630 may include a device for receiving user input data and information, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer, or gravity sensor.

[0104] Output device 640 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0105] The communication interface 620 may include a device that uses any transceiver to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0106] The processor 610 executes the program stored in the memory 600 and calls other devices, and can be used to implement each step of any of the three-dimensional image creation methods provided in the above embodiments of this application.

[0107] It should be noted that the electronic device can be an in-vehicle terminal, a mobile phone, a wearable device, or a server, etc.; or it can be a vehicle that includes an in-vehicle terminal, etc.

[0108] This application also proposes a chip, which includes a processor and a data interface. The processor reads and runs a program stored in the memory through the data interface to execute the three-dimensional image creation method described in any of the above embodiments. For the specific processing procedure and its beneficial effects, please refer to the embodiments of the three-dimensional image creation method described above.

[0109] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the three-dimensional image creation methods according to various embodiments of this application as described in any of the above embodiments of this specification.

[0110] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the power device, as a standalone firmware package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0111] Furthermore, embodiments of this application may also be storage media storing computer programs, which are executed by a processor to perform the steps of the three-dimensional image creation method according to various embodiments of this application described in any of the above embodiments of this specification, specifically implementing the steps of the three-dimensional image creation method as described above.

[0112] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0113] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0114] The steps in the methods of the various embodiments of this application can be adjusted, merged, or deleted in order according to actual needs, and the technical features described in each embodiment can be replaced or combined.

[0115] The units of the apparatus in the various embodiments of this application can be merged, divided, and deleted according to actual needs.

[0116] It should be understood that the disclosed terminals, devices, and methods can be implemented in other ways, given the several embodiments provided in this application. For example, the terminal embodiments described above are merely illustrative. For instance, the division of modules or sub-modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple sub-modules or modules may be combined or integrated into another module, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0117] The modules or submodules described as separate components may or may not be physically separate. The components that constitute a module or submodule may or may not be physical modules or submodules; that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules can be selected to achieve the purpose of this embodiment's solution, depending on actual needs.

[0118] Furthermore, the functional modules or sub-modules in the various embodiments of this application can be integrated into one processing module, or each module or sub-module can exist physically separately, or two or more modules or sub-modules can be integrated into one module. The integrated modules or sub-modules described above can be implemented in hardware or as firmware functional modules or sub-modules.

[0119] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer firmware, or a combination of both. To clearly illustrate the interchangeability of hardware and firmware, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or firmware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0120] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly using hardware, firmware units executed by a processor, or a combination of both. The firmware unit can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0121] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0122] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for creating a three-dimensional image, characterized in that, include: The target image is converted into an initial three-dimensional image, and the certainty of the first Gaussian element is determined based on the attribute parameters of the first Gaussian element in the initial three-dimensional image. The certainty is used to indicate the accuracy of the object represented by the first Gaussian element. Based on the certainty of each first Gaussian unit in the initial three-dimensional image, a spatial volume is determined, wherein the spatial volume is composed of at least one second Gaussian unit, and the second Gaussian unit is a first Gaussian unit with a certainty of less than a preset threshold. The initial shooting angle of the space object is changed to obtain a candidate shooting angle, and a target shooting angle image of the space object under the candidate shooting angle is generated; Based on the target viewpoint image and the initial viewpoint image of the spatial object under the initial shooting viewpoint, the second Gaussian element in the initial three-dimensional image is reconstructed to obtain the target three-dimensional image. The certainty of the reconstructed second Gaussian element is greater than or equal to a preset threshold.

2. The three-dimensional image creation method according to claim 1, characterized in that, The process of changing the initial shooting angle of the space body to obtain candidate shooting angles includes: Determine the center of gravity position of the spatial body, and determine the image acquisition position adjacent to the center of gravity position, wherein the image acquisition position is the position of the camera that captures the first object in the target image; Obtain the initial camera pose corresponding to the image acquisition position, wherein the initial camera pose is the pose of the camera when it captures the first object at the image acquisition position; The initial camera pose is modified to obtain candidate shooting angles.

3. The three-dimensional image creation method according to claim 2, characterized in that, The pose change of the initial camera pose includes: The pose change conditions are obtained, including at least one of the following: the height range of the camera after pose change, the angle range between the acquisition direction of the camera after pose change and the target direction, and the distance range between the camera and the center of gravity position after pose change. The target direction is the direction of the line connecting the camera and the center of gravity position. The initial camera pose is changed according to the pose change conditions.

4. The three-dimensional image creation method according to claim 1, characterized in that, Generating the target view image of the spatial volume under the candidate shooting view includes: Generate Gaussian feature maps and transparency maps under the candidate shooting perspectives; The real-view image of the second object corresponding to the spatial volume taken by the camera, the camera parameters when the camera took the second object, the Gaussian feature map and the transparency map are input into the image generation model to obtain multiple intermediate view images. A scoring model is used to score each of the intermediate view images to obtain a quality score for each intermediate view image; The intermediate view image corresponding to the highest quality score is determined as the target view image.

5. The three-dimensional image creation method according to claim 4, characterized in that, The process of scoring each intermediate-view image using a scoring model includes: Based on the quality score, a first view image and a second view image are determined among multiple intermediate view images. The first view image is an intermediate view image with a quality score greater than or equal to a score threshold, and the second view image is an intermediate view image with a quality score less than a score threshold. Training samples are generated based on the first viewpoint image and the camera parameters, and supervised samples are generated based on the second viewpoint image. The training samples and the supervision samples are stored. The training samples are used to train the scoring model, and the supervision samples are used to supervise the learning of the scoring model.

6. The three-dimensional image creation method according to claim 4, characterized in that, The process of scoring each intermediate-view image using a scoring model includes: Extract the target sub-region from the intermediate view image, wherein the confidence level of the target sub-region is higher than a confidence threshold; The target sub-region is used as a supervision signal, and the scoring model is subjected to supervised learning based on the supervision signal.

7. The method for establishing a three-dimensional image according to any one of claims 1-6, characterized in that, The step of determining the certainty of the first Gaussian element based on the attribute parameters of the first Gaussian element in the initial three-dimensional image includes: Based on the attribute parameters of the first Gaussian element, determine the transparency, shape certainty, and color difference of the object represented by the first Gaussian element. The certainty of the first Gaussian element is determined based on the transparency, the shape certainty, and the color difference.

8. A three-dimensional image creation device, characterized in that, include: A conversion module is used to convert a target image into an initial three-dimensional image, and to determine the certainty of the first Gaussian element based on the attribute parameters of the first Gaussian element in the initial three-dimensional image. The certainty is used to indicate the accuracy of the object represented by the first Gaussian element. The determination module is used to determine a spatial volume based on the certainty of each first Gaussian element in the initial three-dimensional image. The spatial volume is composed of at least one second Gaussian element, and the second Gaussian element is a first Gaussian element whose certainty is less than a preset threshold. The processing module is used to change the initial shooting angle of the space object to obtain a candidate shooting angle, and generate a target perspective image of the space object under the candidate shooting angle. The reconstruction module is used to reconstruct the second Gaussian element in the initial three-dimensional image based on the target view image and the initial view image of the spatial object under the initial shooting view, so as to obtain the target three-dimensional image. The certainty of the reconstructed second Gaussian element is greater than or equal to a preset threshold.

9. An electronic device, characterized in that, Including memory and processor, among which, The memory is connected to the processor and is used to store programs; The processor is used to implement the three-dimensional image creation method as described in any one of claims 1-7 by running the program in the memory.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the three-dimensional image creation method as described in any one of claims 1-7.

11. A computer program product, wherein when executed by a processor, the computer program implements the three-dimensional image creation method as described in any one of claims 1-7.

Images