Image generation method and device, electronic equipment and storage medium

Abstract

Embodiments of the present disclosure provide an image generation method and device, electronic equipment and storage medium, obtaining initial point cloud data generated based on target image data; based on the initial point cloud data, an initial Gaussian body model is generated, and the following steps are executed in a loop for the Gaussian body model: from the target image data, a target object image and corresponding camera parameters are obtained; the Gaussian body model is rendered based on the camera parameters to obtain a predicted image; the target object image is processed based on a pre-trained super-resolution model to generate a super-resolution image; the predicted image and the super-resolution image are compared to obtain an error loss, and the properties of the Gaussian body model are updated based on the error loss; and a two-dimensional rendering image is generated based on the Gaussian body model and a target rendering angle. Through the super-resolution model, the object image in the target image data is super-resolution resampled, so that the generated Gaussian body model has higher accuracy, thereby improving the image quality of the generated two-dimensional rendering image.

Description

Technical Field

[0001] This disclosure relates to the field of three-dimensional visual rendering technology, and in particular to an image generation method, apparatus, electronic device, and storage medium. Background Technology

[0002] 3D Gaussian Splatting (3DGS) is a novel 3D scene reconstruction and rendering technology. It constructs a Gaussian volume model using a large number of 3D Gaussian functions, and through training and optimization of the Gaussian volume, it accurately reconstructs complex 3D objects and scenes, achieving high-quality, real-time rendering from a new perspective.

[0003] In existing technologies, to achieve a sufficiently detailed and realistic Gaussian body model, it is necessary to deploy a large number of camera arrays to acquire images of the target object, and then use 3DGS technology to perform three-dimensional reconstruction of the acquired images to obtain a high-precision Gaussian body model.

[0004] However, due to cost considerations, camera arrays typically use wide-angle cameras with short focal lengths for image acquisition, resulting in lower image clarity. This degrades the accuracy of the generated Gaussian model and affects the visual effect of the final rendered image. Summary of the Invention

[0005] This disclosure provides an image generation method to overcome the problem of low accuracy in the generated Gaussian body model.

[0006] In a first aspect, embodiments of this disclosure provide an image generation method, including:

[0007] The process involves: acquiring initial point cloud data generated from target image data, wherein the target image data includes object images acquired from different angles of the target object, and the initial point cloud data is used to characterize the coordinates of feature points constituting the target object in three-dimensional space; generating an initial Gaussian volume model based on the initial point cloud data, and iteratively executing the following steps on the Gaussian volume model until a preset stopping condition is met, and then outputting the Gaussian volume model: acquiring the target object image and corresponding camera parameters from the target image data; rendering the Gaussian volume model based on the camera parameters to obtain a predicted image; processing the target object image based on a pre-trained super-resolution model to generate a super-resolution image; comparing the predicted image and the super-resolution image to obtain an error loss, and updating the attributes of the Gaussian volume model based on the error loss; acquiring the target rendering angle, and rendering the Gaussian volume model based on the target rendering angle to generate a two-dimensional rendered image.

[0008] In a second aspect, embodiments of this disclosure provide an image generation apparatus, comprising:

[0009] The acquisition module is used to acquire initial point cloud data generated based on target image data, wherein the target image data includes object images collected from different angles for the target object, and the initial point cloud data is used to characterize the coordinates of feature points constituting the target object in three-dimensional space.

[0010] The training module is used to generate an initial Gaussian body model based on the initial point cloud data, and to iteratively execute the following steps on the Gaussian body model until a preset stopping condition is reached, and then output the Gaussian body model: obtaining the target object image and corresponding camera parameters from the target image data; rendering the Gaussian body model based on the camera parameters to obtain a predicted image; processing the target object image based on a pre-trained super-resolution model to generate a super-resolution image; comparing the predicted image and the super-resolution image to obtain an error loss, and updating the attributes of the Gaussian body model based on the error loss;

[0011] The generation module is used to obtain the target rendering angle and render a Gaussian volume model based on the target rendering angle to generate a two-dimensional rendered image.

[0012] Thirdly, embodiments of this disclosure provide an electronic device, including: a processor and a memory;

[0013] The memory stores computer-executed instructions;

[0014] The processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the image generation method as described in the first aspect and various possible designs of the first aspect.

[0015] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the image generation method described in the first aspect and various possible designs of the first aspect.

[0016] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the image generation method as described in the first aspect and various possible designs of the first aspect.

[0017] The image generation method, apparatus, electronic device, and storage medium provided in this embodiment acquire initial point cloud data generated based on target image data. The target image data includes object images acquired from different angles of the target object. The initial point cloud data is used to characterize the coordinates of feature points constituting the target object in three-dimensional space. Based on the initial point cloud data, an initial Gaussian volume model is generated, and the following steps are iteratively executed on the Gaussian volume model until a preset stopping condition is met, at which point the Gaussian volume model is output: The target object image and corresponding camera parameters are acquired from the target image data; the Gaussian volume model is rendered based on the camera parameters to obtain a predicted image; the target object image is processed based on a pre-trained super-resolution model to generate a super-resolution image; the predicted image and the super-resolution image are compared to obtain an error loss, and the attributes of the Gaussian volume model are updated based on the error loss; a two-dimensional rendered image is generated based on the acquired target rendering angle and the Gaussian volume model rendered based on the target rendering angle. By using a pre-trained super-resolution model during the training of the Gaussian body model, the low-resolution object images in the target image data are super-resampled to obtain super-resolution images. Then, the Gaussian body model is trained using the camera parameters corresponding to the object images and the super-resolution images, thereby giving the generated Gaussian body model higher accuracy and improving the fineness and realism of the final generated rendering image. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is an application scenario diagram of the image generation method provided in the embodiments of this disclosure;

[0020] Figure 2 Flowchart of the image generation method provided in the embodiments of this disclosure Figure 1 ;

[0021] Figure 3 This is a schematic diagram illustrating the process of training a Gaussian body model according to an embodiment of the present disclosure;

[0022] Figure 4 This is a flowchart illustrating the specific implementation process of step S1022;

[0023] Figure 5 A schematic diagram illustrating the process of training a Gaussian body model provided in an embodiment of this disclosure;

[0024] Figure 6 Flowchart of the image generation method provided in the embodiments of this disclosure Figure 2 ;

[0025] Figure 7 A schematic diagram illustrating another process for training a Gaussian body model provided in this embodiment of the disclosure;

[0026] Figure 8 This is a structural block diagram of the image generation apparatus provided in the embodiments of this disclosure;

[0027] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure;

[0028] Figure 10 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0030] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0031] The application scenarios of the embodiments of this disclosure are explained below:

[0032] The image generation method provided in this disclosure can be applied to applications (APPs) with image processing capabilities, such as photo-taking applications and live-streaming applications. More specifically, it can be applied to application scenarios such as video production and 3D portrait modeling. The execution subject of this embodiment can be a terminal device running the aforementioned application with image processing capabilities, a server deploying the server corresponding to the aforementioned application, or other electronic devices that perform similar functions. Specifically, when the execution subject is a terminal device, the terminal device executes the method provided in this embodiment by running the aforementioned application; when the execution subject is a server, the server of the aforementioned application with image processing capabilities can run partially or entirely on the server, and the method provided in this embodiment is executed on the server side, while the terminal device runs the client of the application. Communication between the server and the terminal device is based on server-client communication, thereby enabling the terminal device to obtain the execution result of the method provided in this embodiment and display it as needed.

[0033] In some embodiments, the terminal device or server can implement the image generation method provided in this disclosure by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be program-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system; they can be local applications, i.e., programs that need to be installed in the operating system to run, or small programs embedded in any APP, i.e., programs that run in a browser environment. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application, module, or plugin; the specific implementation can be configured as needed. Furthermore, in implementing the image generation method provided in this disclosure, the terminal device or server can execute the method by running computer-executable instructions or computer programs set locally, or by calling computer-executable instructions or computer programs set in an external server. In some embodiments, the server may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud storage, cloud communication, cloud database, cloud computing, cloud functions, network services, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms. Among these, cloud services may be interactive processing services that can be invoked by terminal devices.

[0034] Figure 1 This is an application scenario diagram of the image generation method provided in the embodiments of this disclosure, with reference to... Figure 1 As shown in the diagram, taking a terminal device as the execution entity as an example, such as a personal computer, the terminal device first communicates with a camera array to control the camera array to capture images of a target, such as the target person shown in the diagram. The camera array consists of multiple image acquisition units that capture images of the target person from different angles, thus obtaining a limited number of images of the target person from different angles, i.e., image data. Then, the terminal device acquires the image data acquired by the camera array. Of course, it is understood that in another possible implementation, other devices could control the camera array to acquire images, generate the aforementioned image data, and store it in a preset storage medium, such as a data server. The terminal device could then obtain this image data, for example, by accessing the data server; this is not a limitation. Next, the terminal device uses this image data to generate a Gaussian body model, which is the 3D reconstruction result of the target person. Finally, the terminal device can generate rendered images of the target person from any angle based on specific needs.

[0035] In existing technologies, to achieve a sufficiently detailed and realistic Gaussian body model, a large array of cameras is required to acquire images of the target object. Then, 3DGS technology is used to reconstruct the three-dimensional images, resulting in a high-precision Gaussian body model. In practical applications, camera arrays typically use short-focal-length wide-angle cameras for image acquisition. Short-focal-length wide-angle cameras have a wider field of view, thus offering higher image information efficiency compared to cameras with varying focal lengths, meaning fewer wide-angle cameras are needed to acquire image data. However, images acquired by wide-angle cameras have lower resolution. Reconstructing a 3D Gaussian body from images acquired by wide-angle cameras degrades the accuracy of the generated Gaussian body model, thus affecting the visual quality of the final rendered image.

[0036] This disclosure provides an image generation method to solve the above-mentioned problems.

[0037] refer to Figure 2 , Figure 2 Flowchart of the image generation method provided in the embodiments of this disclosure Figure 1 The method of this embodiment can be applied in a terminal device. This image generation method includes:

[0038] Step S101: Obtain initial point cloud data generated based on target image data, wherein the target image data includes object images collected from different angles for the target object, and the initial point cloud data is used to characterize the coordinates of feature points constituting the target object in three-dimensional space.

[0039] For example, refer to Figure 1The illustrated application scenario diagram shows that a terminal device can obtain a collection of multi-view images (i.e., target image data) captured in real-time by multiple image acquisition devices (e.g., cameras) of a target object via a direct connection to a camera array. More specifically, the target object is, for example, a person in a specific shooting environment, such as a dancer in a photography studio. The target image data would then include, for example, N images (object images) captured from different angles of the dancer. The angle of each object image is determined by the device pose of each image acquisition device in the camera array. The attribute information corresponding to the object image records the identifier of the image acquisition device that captured the object image. Of course, in other possible implementations, the terminal device can also directly obtain the target image data by accessing a data server or other storage medium, which can be configured as needed. This embodiment does not limit the implementation process of how the terminal device obtains the target image data.

[0040] Subsequently, based on the aforementioned target image data, 3D reconstruction is performed to obtain the corresponding initial point cloud data. This initial point cloud data represents the coordinates of the feature points constituting the target object in 3D space. Specifically, for example, the terminal device processes multi-view images (target image data) using the Structure from Motion (SfM) algorithm to obtain the initial point cloud data. The specific implementation process is existing technology and will not be elaborated here. Of course, it is understood that in another implementation, other devices can generate the initial point cloud data based on the above steps, and the terminal device can directly read this initial point cloud data from a specific location.

[0041] Step S102: Based on the initial point cloud data, generate an initial Gaussian volume model, and iteratively execute the following steps on the Gaussian volume model until a preset stopping condition is met, and then output the Gaussian volume model: Obtain the target object image and the corresponding camera parameters from the target image data; Render the Gaussian volume model based on the camera parameters to obtain the predicted image; Process the target object image based on the pre-trained super-resolution model to generate a super-resolution image; Compare the predicted image and the super-resolution image to obtain the error loss, and update the properties of the Gaussian volume model based on the error loss.

[0042] For example, after obtaining initial point cloud data, the terminal device uses this initial point cloud data to reconstruct a 3D model of the target object, namely a Gaussian body model. The Gaussian body model consists of multiple Gaussian bodies, each corresponding to a feature point of the target object. A feature point can be understood as a "point" or "spot" that constitutes the target object. The attributes of the Gaussian body describe the specific location, shape, and color attributes of that feature point. Therefore, the attributes of the Gaussian body model, i.e., the set of attributes of each Gaussian body, are used to train the Gaussian body model. This process involves adjusting and optimizing the attributes of the Gaussian body model, ultimately resulting in a Gaussian body model that accurately and precisely describes the shape features of the target object. Specifically, the terminal device first creates an initial Gaussian body model based on the initial point cloud data and initializes it, i.e., creating a Gaussian body for each point cloud point and initializing its mean, covariance, color, and opacity attributes. Then, it trains the Gaussian body model using object images extracted from the target image data, gradually updating the attributes of the Gaussian body model.

[0043] For example, Figure 3 This is a schematic diagram illustrating the process of training a Gaussian body model according to an embodiment of the present disclosure, such as... Figure 3 As shown, exemplarily, the steps include:

[0044] Step 1021: Obtain the image of the target object and the corresponding camera parameters from the target image data.

[0045] Step 1022: Render a Gaussian model based on camera parameters to obtain the predicted image;

[0046] Step 1023: Process the target object image based on the pre-trained super-resolution model to generate a super-resolution image;

[0047] Step 1024: Compare the predicted image and the super-resolution image to obtain the error loss, and update the properties of the Gaussian body model based on the error loss.

[0048] Step 1025: If the current loop count reaches the threshold, output the Gaussian body model; otherwise, return to step S1021.

[0049] First, the terminal device acquires the target object image and its corresponding camera parameters from the target image data. More specifically, the object image captured by the image acquisition unit and the camera parameters of that unit are stored in pairs within the target image data. After randomly reading a set of data, the terminal device obtains a frame of the target object image and its corresponding camera parameters. These camera parameters include, for example, camera pose, camera identifier, and shooting parameters such as focal length, aperture, and exposure time. If the camera parameters only include the camera identifier, the camera pose can be determined at least by combining the camera identifier with other information. Then, using the camera parameters and the properties of a Gaussian body model, rendering is performed to obtain an image generated based on the current image, i.e., the predicted image. On the other hand, a pre-trained super-resolution model processes the target object image to generate a higher-resolution super-resolution image. For example, if the original captured object image has a resolution of 720p, after processing by the super-resolution model, a super-resolution image with a resolution of 1080p is generated. Next, by comparing the predicted image and the super-resolution image, the error loss is obtained, and the properties of the Gaussian body model are updated based on the error loss, completing the training process for one loop. Then, the process returns to step S1021 until a preset stopping condition is reached, for example, the number of loops reaches a preset number.

[0050] Furthermore, in one possible implementation, the camera parameters include the device pose of the image acquisition device, such as... Figure 4 As shown, the specific implementation steps of step S1022 include:

[0051] Step S1022-1: Based on the device pose of the image acquisition device, perform differentiable rendering on the Gaussian point cloud represented by the Gaussian volume model to generate a rendering object.

[0052] Step S1022-2: Use a preset color calibration model to perform color calibration on the rendered object and then perform image rendering to generate a predicted image.

[0053] For example, in this embodiment, the camera parameters include the device pose of the image acquisition device, i.e., the camera pose, which characterizes the position and orientation (i.e., the rotation angle in three-dimensional space) of the image acquisition device. The Gaussian volume model can be represented as G:{T,C,R,S,O}, where G represents the Gaussian volume model, T represents the position parameters of the Gaussian volume model, used to describe the position of each Gaussian volume, C represents the color attribute of the Gaussian volume model, used to describe the color of each Gaussian volume, R represents the rotation attribute of the Gaussian volume model, used to describe the rotation angle of each Gaussian volume, S represents the scale attribute of the Gaussian volume model, used to describe the scale of each Gaussian volume, and O represents the density attribute of the Gaussian volume model, used to describe the density of each Gaussian volume. Based on this, the Gaussian volume model constructed based on the device pose is rendered in a differentiable manner, which can be described as I'={T,C,R,S,O,V}.i}, which is the rendering object. Among them, V i Let represent the device pose corresponding to the i-th object image, including camera memory and viewpoint information (extrinsic parameters). After generating the rendering object, an additional color calibration model is used to calibrate the color of the rendering object, followed by image rendering to generate a predicted image. This color calibration model is a learnable model that calibrates the colors of the rendering object to make them closer to the real colors (the object images captured by the image acquisition device). In one possible implementation, the color calibration model is a pre-trained model. Directly using this trained model can improve the convergence speed of the Gaussian model and enhance the training effect.

[0054] In another possible implementation, at a certain stage of the Gaussian body model training process, the color calibration model is trained synchronously, thereby giving the color calibration model (better) color calibration capabilities. When using lenses with different focal lengths to capture images and generate target image data, the white balance is difficult to adjust consistently due to differences in aperture and focal length. Therefore, in practical applications, this can lead to color anomalies and temporal flicker in the reconstructed 3D model. In this embodiment, by setting a color calibration model and training it synchronously during the training of the Gaussian body model using the target image data, the problem of inconsistent white balance affecting the training effect of the Gaussian body model in images captured by image acquisition devices with different focal lengths is avoided, thus improving the quality of the final generated Gaussian body model.

[0055] Further, in one possible implementation, the object image includes a first object image acquired based on the focal length of a first lens and a second object image acquired based on the focal length of a second lens, wherein the focal length of the first lens is greater than the focal length of the second lens. Specifically, the first object image is an image captured by a telephoto camera, and the second object image is an image captured by a wide-angle camera with a short focal length. Due to the characteristics of the telephoto image acquisition device, the first object image has relatively good resolution but a smaller field of view; while the second object image is generated by capturing the target object with a short focal length (second lens focal length), due to the characteristics of the telephoto image acquisition device (wide-angle camera), the first object image has relatively good resolution but a smaller field of view. In this application scenario, the specific implementation of step S1023 in this embodiment includes:

[0056] Step S1023-1: Obtain the lens focal length corresponding to the target object image.

[0057] Step S1023-2: Based on the lens focal length, if the target object image is a second object image, then process the second target object image based on the pre-trained super-resolution model to generate a super-resolution image; if the target object image is a first object image, then output the first object image as a super-resolution image.

[0058] For example, in this embodiment, after obtaining the target object image, the lens focal length corresponding to the target object image can be determined by detecting the target object image or reading the image attribute information corresponding to the target object image. The lens focal length includes at least two types: short focal length and long focal length, which respectively represent the lens focal length used by the image acquisition device to acquire the target object image. Then, based on different lens focal lengths, the resolution of the target object image is determined. That is, if the target object image is a second object image, it is determined to be a low-resolution image and requires super-resolution processing. In this case, the super-resolution model is called to process the second target object image to generate a super-resolution image. If the target object image is a first object image, it is determined to be a high-resolution image itself and does not require super-resolution processing. The first object image is directly output as a super-resolution image, thereby achieving the technical effect of dynamically adjusting and reducing the computational resource overhead of the device.

[0059] Figure 5 This is a schematic diagram illustrating the process of training a Gaussian body model according to an embodiment of this disclosure. The following is in conjunction with... Figure 5 The above steps are described, for example, as follows: Figure 5 As shown, firstly, the terminal device obtains initial point cloud data and initializes it, constructing a Gaussian volume model G{T,C,R,S,O}. Then, it obtains the object image I_i and the corresponding camera parameters V_i from the target image data. On one hand, combining the camera parameters V_i and the Gaussian volume model G{T,C,R,S,O}, it obtains the rendering object F{T,C,R,S,O,V_i}. After processing the rendering object through a color calibration model, it generates a rendering image I_r. On the other hand, it processes the object image I_i through a super-resolution model to obtain a super-resolution image I_s. Based on the rendering image I_r and the super-resolution image I_s, it calculates the error loss L. Finally, it updates the Gaussian volume model G{T,C,R,S,O} using the error loss L and repeats the above steps until the preset number of training loops is reached.

[0060] Step S103: Obtain the target rendering angle, and render the Gaussian volume model based on the target rendering angle to generate a two-dimensional rendering image.

[0061] For example, after generating the Gaussian body model through the above steps, by obtaining the target rendering angle and rendering the Gaussian body model based on the target rendering angle, the Gaussian body model can be projected onto a two-dimensional plane corresponding to the target rendering angle (e.g., a plane perpendicular to the target rendering angle) to generate a two-dimensional rendered image. Specifically, in one possible implementation, after the terminal device generates the Gaussian body model in the image processing application, the terminal device renders the Gaussian body model as a frame of two-dimensional rendered image based on a preset default rendering angle. Then, in response to the user's operation command on the terminal device (e.g., a command to rotate the viewing angle), the target rendering angle is updated, and based on the updated target rendering angle, the Gaussian body model is rendered again, and a new two-dimensional rendered image is generated, thereby achieving the effect of visual preview of the Gaussian body model.

[0062] In another possible implementation, a fixed target rendering angle, such as 45 degrees above, can be used to render the Gaussian model, resulting in a 2D image displaying the target object described by the Gaussian model from a 45-degree angle above. In yet another possible implementation, multiple target rendering angles are used, each corresponding to a different time point. Within a certain period, the corresponding target rendering angle is dynamically determined based on the current system time, and the Gaussian model is rendered accordingly, thus achieving the effect of displaying the target object described by the Gaussian model at different times and from different angles.

[0063] In this embodiment, initial point cloud data generated based on target image data is acquired. The target image data includes object images taken from different angles of the target object. The initial point cloud data is used to characterize the coordinates of feature points constituting the target object in three-dimensional space. Based on the initial point cloud data, an initial Gaussian volume model is generated, and the following steps are iteratively executed on the Gaussian volume model until a preset stopping condition is met, at which point the Gaussian volume model is output: The target object image and corresponding camera parameters are obtained from the target image data; the Gaussian volume model is rendered based on the camera parameters to obtain a predicted image; the target object image is processed based on a pre-trained super-resolution model to generate a super-resolution image; the predicted image and the super-resolution image are compared to obtain an error loss, and the attributes of the Gaussian volume model are updated based on the error loss. By using a pre-trained super-resolution model to super-resample low-resolution object images in the target image data during the training of the Gaussian volume model, obtaining a super-resolution image, and then using the camera parameters corresponding to the object image and the super-resolution image to train the Gaussian volume model, the generated Gaussian volume model has higher accuracy, improving the refinement and realism of the final generated rendered image.

[0064] refer to Figure 6 , Figure 6Flowchart of the image generation method provided in the embodiments of this disclosure Figure 2 This embodiment is in Figure 2 Based on the illustrated embodiment, step S102 is further refined. The object image is acquired through an image acquisition device configured with a rolling shutter. The image generation method includes:

[0065] Step S201: Obtain initial point cloud data generated based on target image data, and generate an initial Gaussian model based on the initial point cloud data. The target image data includes object images collected from different angles for the target object. The initial point cloud data is used to characterize the coordinates of the feature points constituting the target object in three-dimensional space. The object images include a first object image collected based on the focal length of a first lens and a second object image collected based on the focal length of a second lens, wherein the focal length of the first lens is greater than the focal length of the second lens.

[0066] For example, the image generation method provided in this embodiment is applied to an application scenario where images are acquired and three-dimensional models are reconstructed using a multi-focal-length hybrid camera array. Specifically, the camera array consists of a telephoto image acquisition device and a short-focal-length image acquisition device. A first object image is generated by capturing the target object with a telephoto lens (first lens focal length). Due to the characteristics of the telephoto image acquisition device, the first object image has relatively good resolution but a smaller field of view. Conversely, a second object image is generated by capturing the target object with a short-focal-length lens (second lens focal length). Due to the characteristics of the telephoto image acquisition device (wide-angle camera), the first object image has relatively good resolution but a smaller field of view. The number of image acquisition devices with the first lens focal length in the camera array is proportional to the number of image acquisition devices with the second lens focal length. Correspondingly, the first object image and the second object image in the target image data can also have a certain proportional relationship.

[0067] Compared to a solution that uses only telephoto image acquisition equipment for image acquisition and 3D model reconstruction, the target image data acquired using the above method can reduce the number of image acquisition devices and lower costs. At the same time, compared to a solution that uses only short-focus image acquisition equipment for image acquisition and 3D model reconstruction, by combining subsequent super-resolution model and other technical features, the model's precision and quality can be effectively improved.

[0068] Step S202: Obtain the current number of training iterations.

[0069] Step S203: When the current number of training iterations is less than the first quantity threshold, obtain the second object image and the corresponding camera parameters, and use the obtained second object image and the corresponding camera parameters as the target object image and the corresponding camera parameters.

[0070] Step S204: When the current number of training iterations is greater than or equal to the first quantity threshold, based on the target sampling probability, randomly acquire the first object image or the second object image as the target object image, and acquire the corresponding camera parameters.

[0071] For example, in this embodiment, different training samples are selected to train the Gaussian body model based on the current number of training iterations. Specifically, if the current number of training iterations is less than a first threshold, the second object image captured by a short-focal-length image acquisition device in the target image data is obtained as the target object image for subsequent steps. This allows the Gaussian body model to learn more global information in the early stages of training. In the later stages of training, the first object image and the second object image are randomly acquired based on the target sampling probability, for example, in a 3:7 ratio. This allows the model to learn more detailed information, realizing a course-to-fine sampling training strategy for focal length blending, thereby improving the training efficiency and quality of the model.

[0072] Step S205: Render the Gaussian body model based on camera parameters and color calibration model to obtain the predicted image.

[0073] For example, subsequently, a Gaussian volume model is rendered based on machine parameters and a color calibration model. Specifically, based on the device pose of the image acquisition device, the Gaussian point cloud represented by the Gaussian volume model is rendered in a differentiable manner to generate a rendering object. The rendering object is then color-calibrated using a pre-defined color calibration model, and the image is rendered to generate a predicted image. The specific implementation process is detailed in [details omitted]. Figure 2 The embodiments shown have already been described and will not be repeated here.

[0074] Step S206: Process the target object image based on the pre-trained super-resolution model to generate a super-resolution image.

[0075] Next, the pre-trained super-resolution model is used to process the target object image. When the target object image is a low-resolution second object image acquired by a short-focal-length image acquisition device based on the focal length of the second lens, the super-resolution model is used to perform super-resolution processing on the target object image to improve the resolution of the target object image. This improves the effect of subsequent training of the Gaussian body model based on the target object image and enhances the model's precision.

[0076] Optionally, before performing step S205 above, a step of training the super-resolution model is also included, specifically including:

[0077] Step S200A: Obtain the first object image from the target image data, and downsample the first object image to obtain a sample image;

[0078] Step S200B: Based on the sample image and the first object image, train the initial neural network model to obtain the super-resolution model.

[0079] For example, on the other hand, the super-resolution model is implemented based on a neural network model, which can be trained using target image data. Since the object images in the target image data are all images of the same target object, training the super-resolution model using high-resolution images (first object images) from the target image data can yield better training results. This allows the super-resolution model to restore better real image details when performing super-resolution processing on images containing the target object, improving the realism and detail of the output super-resolution image. Therefore, in this embodiment, after obtaining the target image data, the super-resolution model is first trained based on the target image data. Then, the trained super-resolution model is used to perform super-resolution processing on images captured by a short-focal-length image acquisition device, improving the quality of the object image. This allows the images captured by the short-focal-length image acquisition device to achieve the same image resolution as those captured by a long-focal-length image acquisition device, while having a larger field of view. Ultimately, this achieves low-cost 3D model reconstruction based on hybrid focal lengths.

[0080] Step S207: Compare the predicted image and the super-resolution image to obtain the error loss.

[0081] Step S208: Update the parameters of the Gaussian body model based on the error loss, or control the Gaussian body model to split.

[0082] For example, after obtaining the predicted image, the error loss can be obtained by comparing the predicted image with the super-resolution image mentioned above. The properties of the Gaussian volume model are then updated based on the error loss, and the specific implementation method is existing technology, which will not be elaborated here. The property update of the Gaussian volume model includes updating the attribute parameters of the Gaussian volume model and / or controlling the splitting of the Gaussian volume model. The process of splitting the Gaussian point cloud to adjust the Gaussian point cloud model is part of the adaptive density control in 3D Gaussian scattering technology. This process aims to improve the model's ability to represent the details of the scene by increasing the number of Gaussian points, while keeping the model's computational efficiency and storage requirements within an acceptable range.

[0083] In one possible implementation, the object image is acquired using an image acquisition device configured with a rolling shutter. The Gaussian body model's properties include position parameters and motion compensation parameters. The position parameters characterize the spatial location of feature points, while the motion compensation parameters compensate for the motion displacement between different pixel rows in the object image caused by exposure time differences. Specifically, the image acquisition device configured with a rolling shutter refers to an image acquisition device using a rolling shutter technology. This shutter method is similar to a curtain, exposing each row sequentially from top to bottom. Image capture is achieved through row-by-row exposure controlled by a chip. At the start of exposure, the pixels on the sensor are cleared row by row, and then exposure begins row by row until all pixels are exposed. Because data readout is serial, the clearing, exposure, and readout processes can only be performed sequentially row by row, typically from top to bottom. Due to the technical characteristics of rolling shutters, object images acquired using an image acquisition device configured with a rolling shutter suffer from row-by-row asynchronous imaging, which affects the quality of the captured object image and the quality of the Gaussian body model trained based on the object image.

[0084] For example, in step S208, the specific implementation of updating the parameters of the Gaussian body model based on error loss includes:

[0085] Step S208A: Update the position parameters and motion compensation parameters of the Gaussian body model based on the error loss.

[0086] In this embodiment, to solve the above problems, additional motion compensation parameters are set in the attributes of the Gaussian body model. During the training of the Gaussian body model, these motion compensation parameters are simultaneously optimized. This allows cloud-based compensation parameters to compensate for motion displacement between different pixel rows in the object image caused by exposure time differences, thereby correcting the problem of asynchronous rolling shutter programs. Specifically, when calculating the actual position of the Gaussian body model (i.e., the actual spatial position of each Gaussian body), the actual position is determined by calculating the original position parameter T and the motion compensation parameter Tm. The initial value of the motion compensation parameter Tm is 0, and its initial number is related to the number of image acquisition devices. For example, a Gaussian body model G containing N Gaussian bodies has N position parameters T, and the number of learnable parameters is N*3. Assuming there are P image acquisition devices, the number of learnable parameters for Tm is N*P*3. When rendering the predicted image corresponding to the i-th image acquisition device, the actual position of the n-th Gaussian body is: T'n = Tn + Tm. i n .

[0087] Furthermore, in one possible implementation, training of the compensation component (motion compensation parameters) can begin once the current number of training iterations reaches a second threshold.

[0088] For example, the specific implementation of step S208A includes:

[0089] Step S208A-1: When the number of training iterations exceeds the second threshold, update the position parameters and motion compensation parameters of the Gaussian body model based on the error loss.

[0090] Step S208A-2: When the number of training iterations is less than or equal to the second quantity threshold, update the position parameters of the Gaussian body model based on the error loss.

[0091] Furthermore, in step S208, the specific methods for controlling the splitting of the Gaussian body model include:

[0092] Step S208B-1: Obtain the focal length information of the target object image. The focal length information represents the horizontal and vertical focal lengths when acquiring the target object image.

[0093] Step S208B-2 calculates the current gradient of the Gaussian body model based on the error loss and focal length information, and controls the splitting of the Gaussian body model based on the current gradient.

[0094] For example, the classification of the Gaussian body model is controlled based on the partial derivative of the error loss with respect to the pixel. Specifically, the formula for calculating the current gradient of the Gaussian body model is shown in equation (1) below:

[0095] Grad = ||f x / W*dL / du‖+‖f y / H*dL / dv‖ (1)

[0096] Where Grad is the gradient, f x f is the focal length in the horizontal direction. y is the vertical focal length, u and v represent pixel coordinates, W and H are the width and height of the image, and L is the loss error.

[0097] Subsequently, when the gradient of the Gaussian body model at a certain location exceeds a certain threshold, it can be considered that the detail representation capability at that location is insufficient, and the number of Gaussian points needs to be increased to improve the model's accuracy. Therefore, a splitting operation can be triggered at that location to generate new Gaussian points, thereby increasing the model's detail representation capability. In this embodiment, the current gradient of the Gaussian body model is calculated by combining focal length information, so that object images taken with different lens focal lengths have different effects on the Gaussian body model, producing different gradient changes, thereby improving the training effect of the Gaussian body model and the model refinement of the generated Gaussian body model.

[0098] Furthermore, optionally, this embodiment also includes:

[0099] Step S209: When the current number of training iterations exceeds the third quantity threshold, update the color calibration model based on the error loss.

[0100] Furthermore, the color calibration model performs color calibration on the rendered object, thereby increasing the color accuracy of the generated predicted image and avoiding the problem of inaccurate white balance. In this embodiment, when the current number of training iterations exceeds the third threshold, the color calibration model is trained based on the error loss. This achieves the training effect of the color calibration model while avoiding the impact of starting the color calibration model training too early on the training quality and convergence speed of the Gaussian model, thus improving the overall training effect of the Gaussian model and providing the model quality of the final generated Gaussian model.

[0101] Step S210: If the stopping condition is not met, return to step S202; otherwise, output the trained Gaussian body model.

[0102] Step S211: Obtain the target rendering angle and render the Gaussian volume model based on the target rendering angle to generate a two-dimensional rendering image.

[0103] After S208 and S209 are completed, return to step 202 and repeat the above steps until the preset number of loops is reached or other stopping conditions are met, and then output the trained Gaussian body model.

[0104] In this embodiment, the specific implementation of step S211 is the same as... Figure 2 The specific implementation of step S103 in the illustrated embodiment is similar, and the steps are described in detail here.

[0105] Figure 7 This is a schematic diagram illustrating another process for training a Gaussian body model provided in an embodiment of this disclosure. The following is in conjunction with... Figure 7 The steps of the above embodiments will be further described, such as... Figure 7 As shown, firstly, the terminal device obtains initial point cloud data and initializes it, constructing a Gaussian volume model G{T,C,R,S,O}. Then, by detecting the current training iteration count, it obtains the object image I_i and corresponding camera parameters V_i from the target image data based on a coarse-to-fine hybrid focal length strategy. On the one hand, combining the camera parameters V_i and the Gaussian volume model G{T,C,R,S,O}, it obtains a rendering object F{T+T} containing motion compensation parameters. m iThe Gaussian body model G{T,C,R,S,O,V_i} is first processed using a color calibration model to generate a rendered image I_r. Then, after obtaining the object image I_i, a resolution model is used to process it to obtain a super-resolution image I_s. Error calculation is performed based on the rendered image I_r and the super-resolution image I_s to obtain an error loss L. The error loss L is then used to update and split the Gaussian body model G{T,C,R,S,O}, and the above steps are repeated until the preset number of training iterations is reached.

[0106] Corresponding to the image generation method in the above embodiments, Figure 8 This is a structural block diagram of an image generation apparatus provided in an embodiment of this disclosure. The method described in the above embodiments can be executed by this image generation apparatus, which can be implemented by software and / or hardware, and can be integrated into an electronic device with certain data processing capabilities. The electronic device may include, but is not limited to, mobile terminals with big data processing capabilities, as well as fixed terminals with big data processing capabilities such as desktop computers and supercomputers.

[0107] For ease of explanation, only the parts relevant to embodiments of this disclosure are shown. (Refer to...) Figure 8 The image generating device 3 includes:

[0108] The acquisition module 31 is used to acquire initial point cloud data generated based on target image data, wherein the target image data includes object images collected from different angles for the target object, and the initial point cloud data is used to characterize the coordinates of feature points constituting the target object in three-dimensional space.

[0109] Training module 32 is used to generate an initial Gaussian body model based on the initial point cloud data, and to iteratively execute the following steps on the Gaussian body model until a preset stopping condition is met, and then output the Gaussian body model: Obtain the target object image and corresponding camera parameters from the target image data; render the Gaussian body model based on the camera parameters to obtain a predicted image; process the target object image based on the pre-trained super-resolution model to generate a super-resolution image; compare the predicted image and the super-resolution image to obtain the error loss, and update the attributes of the Gaussian body model based on the error loss.

[0110] The generation module 33 is used to obtain the target rendering angle and render the Gaussian volume model based on the target rendering angle to generate a two-dimensional rendering image.

[0111] In one or more embodiments of this disclosure, the object image includes a first object image acquired based on a first lens focal length and a second object image acquired based on a second lens focal length, wherein the first lens focal length is greater than the second lens focal length. When the training module 32 processes the target object image based on a pre-trained super-resolution model to generate a super-resolution image, it is specifically used to: obtain the lens focal length corresponding to the target object image; based on the lens focal length, if the target object image is the second object image, process the second target object image based on the pre-trained super-resolution model to generate a super-resolution image; if the target object image is the first object image, output the first object image as a super-resolution image.

[0112] In one or more embodiments of this disclosure, the training module 32 is further configured to: acquire a first object image from the target image data, and downsample the first object image to obtain a sample image; and train an initial neural network model based on the sample image and the first object image to obtain a super-resolution model.

[0113] In one or more embodiments of this disclosure, when the training module 32 obtains the target object image and corresponding camera parameters from the target image data, it is specifically used to: obtain the current number of training iterations; when the current number of training iterations is less than a first quantity threshold, obtain the second object image and corresponding camera parameters, and use the obtained second object image and corresponding camera parameters as the target object image and corresponding camera parameters; when the current number of training iterations is greater than or equal to the first quantity threshold, randomly obtain the first object image or the second object image as the target object image based on the target sampling probability, and obtain the corresponding camera parameters.

[0114] In one or more embodiments of this disclosure, when updating the properties of the Gaussian body model based on error loss, the training module 32 is specifically used to: obtain the focal length information of the target object image, wherein the focal length information represents the horizontal and vertical focal lengths when the target object image is acquired; calculate the current gradient of the Gaussian body model based on the error loss and the focal length information, and control the Gaussian body model to split based on the current gradient.

[0115] In one or more embodiments of this disclosure, an object image is acquired by an image acquisition device configured with a rolling shutter; the attributes of the Gaussian body model include position parameters and motion compensation parameters, wherein the position parameters characterize the spatial position of feature points, and the motion compensation parameters are used to compensate for the motion displacement between different pixel rows in the object image due to exposure time difference; when the training module 32 updates the attributes of the Gaussian body model based on error loss, it is specifically used to: update the position parameters and motion compensation parameters of the Gaussian body model based on error loss.

[0116] In one or more embodiments of this disclosure, when the training module 32 updates the position parameters and motion compensation parameters of the Gaussian body model based on the error loss, it is specifically used to: obtain the current number of training cycles; when the number of training cycles is greater than a second threshold, update the position parameters and motion compensation parameters of the Gaussian body model based on the error loss; when the number of training cycles is less than or equal to the second threshold, update the position parameters of the Gaussian body model based on the error loss.

[0117] In one or more embodiments of this disclosure, the camera parameters include the device pose of the image acquisition device; when the training module 32 renders the Gaussian volume model based on the camera parameters to obtain the predicted image, it is specifically used to: perform differentiable rendering of the Gaussian point cloud represented by the Gaussian volume model based on the device pose of the image acquisition device to generate a rendering object; perform color calibration on the rendering object using a preset color calibration model, and perform image rendering to generate the predicted image.

[0118] In one or more embodiments of this disclosure, the training module 32 is further configured to: obtain the current number of training iterations; and update the color calibration model based on error loss when the current number of training iterations is greater than a third quantity threshold.

[0119] The acquisition module 31, training module 32, and generation module 33 are connected sequentially. The image generation device 3 provided in this embodiment can execute the technical solution of the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0120] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure, such as... Figure 9 As shown, the electronic device 4 includes:

[0121] Processor 41, and memory 42 communicatively connected to processor 41;

[0122] Memory 42 stores instructions executed by the computer;

[0123] The processor 41 executes computer execution instructions stored in the memory 42 to achieve, for example, Figures 2-7 The image generation method in the illustrated embodiment.

[0124] Optionally, the processor 41 and the memory 42 are connected via a bus 43.

[0125] For relevant instructions, please refer to the corresponding text. Figures 2-7 The relevant descriptions and effects of the steps in the corresponding embodiments are understood, and will not be elaborated on here.

[0126] This disclosure provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement this disclosure. Figures 2-7 The image generation method provided in any of the corresponding embodiments.

[0127] This disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements this disclosure. Figures 2-7 The image generation method provided in any of the corresponding embodiments.

[0128] To implement the above embodiments, this disclosure also provides an electronic device.

[0129] refer to Figure 10 The diagram illustrates a structural schematic of an electronic device 900 suitable for implementing embodiments of the present disclosure. The electronic device 900 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers, portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 10 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0130] like Figure 10 As shown, the electronic device 900 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 into a random access memory (RAM) 903. The RAM 903 also stores various programs and data required for the operation of the electronic device 900. The processing unit 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0131] Typically, the following devices can be connected to I / O interface 905: input devices 906 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 907 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 908 including, for example, magnetic tapes, hard disks, etc.; and communication devices 909. Communication device 909 allows electronic device 900 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 10 An electronic device 900 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0132] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by a processing device 901, it performs the functions defined in the methods of embodiments of this disclosure.

[0133] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0134] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0135] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0136] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0137] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0138] The units or modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units or modules do not necessarily limit the specific unit itself.

[0139] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0140] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0141] In a first aspect, according to one or more embodiments of the present disclosure, an image generation method is provided, comprising:

[0142] Acquire initial point cloud data based on target image data, wherein the target image data includes object images of the target object collected from different angles, and the initial point cloud data is used to characterize the coordinates of feature points constituting the target object in three-dimensional space;

[0143] Based on the initial point cloud data, an initial Gaussian volume model is generated, and the following steps are iteratively executed on the Gaussian volume model until a preset stopping condition is reached, at which point the Gaussian volume model is output: The target object image and corresponding camera parameters are obtained from the target image data; the Gaussian volume model is rendered based on the camera parameters to obtain a predicted image; the target object image is processed based on a pre-trained super-resolution model to generate a super-resolution image; the predicted image and the super-resolution image are compared to obtain an error loss, and the properties of the Gaussian volume model are updated based on the error loss.

[0144] Obtain the target rendering angle, and render the Gaussian volume model based on the target rendering angle to generate a two-dimensional rendered image.

[0145] According to one or more embodiments of this disclosure, the object image includes a first object image acquired based on a first lens focal length and a second object image acquired based on a second lens focal length, wherein the first lens focal length is greater than the second lens focal length. The step of processing the target object image based on a pre-trained super-resolution model to generate a super-resolution image includes: obtaining the lens focal length corresponding to the target object image; based on the lens focal length, if the target object image is the second object image, processing the second target object image based on the pre-trained super-resolution model to generate a super-resolution image; if the target object image is the first object image, outputting the first object image as the super-resolution image.

[0146] According to one or more embodiments of this disclosure, the method further includes: acquiring a first object image from target image data, and downsampling the first object image to obtain a sample image; and training an initial neural network model based on the sample image and the first object image to obtain the super-resolution model.

[0147] According to one or more embodiments of this disclosure, obtaining a target object image and corresponding camera parameters from the target image data includes: obtaining the current number of training iterations; when the current number of training iterations is less than a first quantity threshold, obtaining a second object image and corresponding camera parameters, and using the obtained second object image and corresponding camera parameters as the target object image and corresponding camera parameters; when the current number of training iterations is greater than or equal to the first quantity threshold, randomly obtaining either the first object image or the second object image as the target object image based on the target sampling probability, and obtaining the corresponding camera parameters.

[0148] According to one or more embodiments of this disclosure, updating the properties of the Gaussian body model based on the error loss includes: obtaining focal length information of the target object image, the focal length information representing the horizontal and vertical focal lengths when the target object image was acquired; calculating the current gradient of the Gaussian body model based on the error loss and the focal length information, and controlling the Gaussian body model to split based on the current gradient.

[0149] According to one or more embodiments of this disclosure, the object image is acquired by an image acquisition device configured with a rolling shutter; the attributes of the Gaussian volume model include position parameters and motion compensation parameters, wherein the position parameters characterize the spatial position of the feature points, and the motion compensation parameters are used to compensate for the motion displacement between different pixel rows in the object image due to exposure time difference; updating the attributes of the Gaussian volume model based on the error loss includes: updating the position parameters and the motion compensation parameters of the Gaussian volume model based on the error loss.

[0150] According to one or more embodiments of this disclosure, updating the position parameters and motion compensation parameters of the Gaussian body model based on the error loss includes: obtaining the current number of training iterations; when the number of training iterations is greater than a second threshold, updating the position parameters and motion compensation parameters of the Gaussian body model based on the error loss; when the number of training iterations is less than or equal to the second threshold, updating the position parameters of the Gaussian body model based on the error loss.

[0151] According to one or more embodiments of this disclosure, the camera parameters include the device pose of the image acquisition device; the step of rendering the Gaussian volume model based on the camera parameters to obtain a predicted image includes: performing differentiable rendering on the Gaussian point cloud represented by the Gaussian volume model based on the device pose of the image acquisition device to generate a rendering object; performing color calibration on the rendering object using a preset color calibration model, and performing image rendering to generate the predicted image.

[0152] According to one or more embodiments of this disclosure, the method further includes: obtaining the current number of training iterations; and updating the color calibration model based on the error loss when the current number of training iterations is greater than a third quantity threshold.

[0153] Secondly, according to one or more embodiments of this disclosure, an image generation apparatus is provided, comprising:

[0154] The acquisition module is used to acquire initial point cloud data generated based on target image data, wherein the target image data includes object images collected from different angles for the target object, and the initial point cloud data is used to characterize the coordinates of feature points constituting the target object in three-dimensional space.

[0155] The training module is used to generate an initial Gaussian body model based on the initial point cloud data, and to iteratively execute the following steps on the Gaussian body model until a preset stopping condition is reached, and then output the Gaussian body model: obtaining the target object image and corresponding camera parameters from the target image data; rendering the Gaussian body model based on the camera parameters to obtain a predicted image; processing the target object image based on a pre-trained super-resolution model to generate a super-resolution image; comparing the predicted image and the super-resolution image to obtain an error loss, and updating the attribute data of the Gaussian body model based on the error loss;

[0156] The generation module is used to obtain the target rendering angle and render a Gaussian volume model based on the target rendering angle to generate a two-dimensional rendered image.

[0157] In one or more embodiments of this disclosure, the object image includes a first object image acquired based on a first lens focal length and a second object image acquired based on a second lens focal length, wherein the first lens focal length is greater than the second lens focal length. When the training module processes the target object image based on a pre-trained super-resolution model to generate a super-resolution image, it is specifically configured to: obtain the lens focal length corresponding to the target object image; based on the lens focal length, if the target object image is the second object image, process the second target object image based on the pre-trained super-resolution model to generate a super-resolution image; if the target object image is the first object image, output the first object image as the super-resolution image.

[0158] In one or more embodiments of this disclosure, the training module is further configured to: acquire a first object image from the target image data, and downsample the first object image to obtain a sample image; and train an initial neural network model based on the sample image and the first object image to obtain the super-resolution model.

[0159] In one or more embodiments of this disclosure, when the training module obtains a target object image and corresponding camera parameters from the target image data, it is specifically configured to: obtain the current number of training iterations; when the current number of training iterations is less than a first quantity threshold, obtain a second object image and corresponding camera parameters, and use the obtained second object image and corresponding camera parameters as the target object image and corresponding camera parameters; when the current number of training iterations is greater than or equal to the first quantity threshold, randomly obtain the first object image or the second object image as the target object image based on the target sampling probability, and obtain the corresponding camera parameters.

[0160] In one or more embodiments of this disclosure, when the training module updates the attributes of the Gaussian body model based on the error loss, it is specifically configured to: obtain the focal length information of the target object image, wherein the focal length information represents the horizontal and vertical focal lengths when the target object image is acquired; calculate the current gradient of the Gaussian body model based on the error loss and the focal length information, and control the Gaussian body model to split based on the current gradient.

[0161] In one or more embodiments of this disclosure, the object image is acquired using an image acquisition device configured with a rolling shutter; the attributes of the Gaussian body model include position parameters and motion compensation parameters, wherein the position parameters characterize the spatial position of the feature points, and the motion compensation parameters are used to compensate for the motion displacement between different pixel rows in the object image due to exposure time differences; when the training module updates the attributes of the Gaussian body model based on the error loss, it is specifically used to: update the position parameters and motion compensation parameters of the Gaussian body model based on the error loss.

[0162] In one or more embodiments of this disclosure, when the training module updates the position parameters and motion compensation parameters of the Gaussian body model based on the error loss, it is specifically configured to: obtain the current number of training iterations; when the number of training iterations is greater than a second threshold, update the position parameters and motion compensation parameters of the Gaussian body model based on the error loss; when the number of training iterations is less than or equal to the second threshold, update the position parameters of the Gaussian body model based on the error loss.

[0163] In one or more embodiments of this disclosure, the camera parameters include the device pose of the image acquisition device; when the training module 32 renders the Gaussian volume model based on the camera parameters to obtain a predicted image, it is specifically used to: perform differentiable rendering of the Gaussian point cloud represented by the Gaussian volume model based on the device pose of the image acquisition device to generate a rendering object; perform color calibration on the rendering object using a preset color calibration model, and perform image rendering to generate the predicted image.

[0164] In one or more embodiments of this disclosure, the training module is further configured to: obtain the current number of training iterations; and update the color calibration model based on the error loss when the current number of training iterations is greater than a third quantity threshold.

[0165] Thirdly, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor and a memory;

[0166] The memory stores computer-executed instructions;

[0167] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the image generation method as described in the first aspect and various possible designs of the first aspect.

[0168] Fourthly, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, which, when executed by a processor, implement the image generation method described in the first aspect and various possible designs of the first aspect.

[0169] Fifthly, according to one or more embodiments of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the image generation method as described in the first aspect and various possible designs of the first aspect.

[0170] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0171] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0172] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An image generation method, characterized in that, include: Acquire initial point cloud data based on target image data, wherein the target image data includes object images of the target object collected from different angles, and the initial point cloud data is used to characterize the coordinates of feature points constituting the target object in three-dimensional space; Based on the initial point cloud data, an initial Gaussian volume model is generated, and the following steps are iteratively executed on the Gaussian volume model until a preset stopping condition is reached, at which point the Gaussian volume model is output: The target object image and corresponding camera parameters are obtained from the target image data; the Gaussian volume model is rendered based on the camera parameters to obtain a predicted image; the target object image is processed based on a pre-trained super-resolution model to generate a super-resolution image; the predicted image and the super-resolution image are compared to obtain an error loss, and the properties of the Gaussian volume model are updated based on the error loss. Obtain the target rendering angle, and render the Gaussian volume model based on the target rendering angle to generate a two-dimensional rendered image.

2. The method according to claim 1, characterized in that, The object image includes a first object image acquired based on a first lens focal length and a second object image acquired based on a second lens focal length, wherein the first lens focal length is greater than the second lens focal length. The pre-trained super-resolution model processes the target object image to generate a super-resolution image, including: Obtain the lens focal length corresponding to the image of the target object; Based on the lens focal length, if the target object image is the second object image, then the second target object image is processed based on the pre-trained super-resolution model to generate a super-resolution image; If the target object image is the first object image, then the first object image is output as the super-resolution image.

3. The method according to claim 2, characterized in that, The method further includes: Acquire the first object image from the target image data, and downsample the first object image to obtain a sample image; Based on the sample image and the first object image, an initial neural network model is trained to obtain the super-resolution model.

4. The method according to claim 2, characterized in that, The step of obtaining the target object image and corresponding camera parameters from the target image data includes: Get the current number of training iterations; When the current number of training iterations is less than the first quantity threshold, the second object image and the corresponding camera parameters are obtained, and the obtained second object image and the corresponding camera parameters are used as the target object image and the corresponding camera parameters. When the current number of training iterations is greater than or equal to the first quantity threshold, the first object image or the second object image is randomly selected as the target object image based on the target sampling probability, and the corresponding camera parameters are obtained.

5. The method according to claim 2, characterized in that, The updating of the properties of the Gaussian model based on the error loss includes: The focal length information of the target object image is obtained, wherein the focal length information represents the horizontal and vertical focal lengths when the target object image is acquired; Based on the error loss and the focal length information, the current gradient of the Gaussian body model is calculated, and the Gaussian body model is split based on the current gradient.

6. The method according to claim 1, characterized in that, The object image is acquired by an image acquisition device configured with a rolling shutter; the properties of the Gaussian body model include position parameters and motion compensation parameters, wherein the position parameters characterize the spatial position of the feature points, and the motion compensation parameters are used to compensate for the motion displacement between different pixel rows in the object image due to exposure time difference; Updating the properties of the Gaussian model based on the error loss includes: Based on the error loss, the position parameters and motion compensation parameters of the Gaussian body model are updated.

7. The method according to claim 6, characterized in that, The step of updating the position parameters and motion compensation parameters of the Gaussian body model based on the error loss includes: Get the current number of training iterations; When the number of training iterations exceeds the second threshold, the position parameters and motion compensation parameters of the Gaussian body model are updated based on the error loss. When the number of training iterations is less than or equal to the second quantity threshold, the position parameters of the Gaussian body model are updated based on the error loss.

8. The method according to claim 1, characterized in that, The camera parameters include the device pose of the image acquisition device; the step of rendering the Gaussian model based on the camera parameters to obtain the predicted image includes: Based on the device pose of the image acquisition device, the Gaussian point cloud represented by the Gaussian volume model is rendered in a differentiable manner to generate a rendering object. The color of the rendered object is calibrated using a preset color calibration model, and then the image is rendered to generate the predicted image.

9. The method according to claim 8, characterized in that, The method further includes: Get the current number of training iterations; When the current number of training iterations exceeds the third threshold, the color calibration model is updated based on the error loss.

10. An image generation apparatus, characterized in that, include: The acquisition module is used to acquire initial point cloud data generated based on target image data, wherein the target image data includes object images collected from different angles for the target object, and the initial point cloud data is used to characterize the coordinates of feature points constituting the target object in three-dimensional space. The training module is used to generate an initial Gaussian body model based on the initial point cloud data, and to iteratively execute the following steps on the Gaussian body model until a preset stopping condition is reached, and then output the Gaussian body model: obtaining the target object image and corresponding camera parameters from the target image data; rendering the Gaussian body model based on the camera parameters to obtain a predicted image; processing the target object image based on a pre-trained super-resolution model to generate a super-resolution image; comparing the predicted image and the super-resolution image to obtain an error loss, and updating the attributes of the Gaussian body model based on the error loss; The generation module is used to obtain the target rendering angle and render a Gaussian volume model based on the target rendering angle to generate a two-dimensional rendered image.

11. An electronic device, characterized in that, include: Processor and memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the image generation method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the image generation method as described in any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the image generation method as described in any one of claims 1 to 9.

Images