Model training method based on visual reconstruction technology, device, and cluster

Abstract

Disclosed are a model training method based on visual reconstruction technology, a device, and a cluster, relating to the technical field of visual reconstruction. The method comprises: acquiring a residual between a first image and a second image generated by an initial model on the basis of first photographing parameters; and using the residual to train the initial model to obtain a trained initial model (e.g., an image generation model). The residual between the first image and the second image can faithfully reflect information of first content and first interference information in the first image, thereby increasing the possibility of the image generation model accurately identifying interference information in images, and thus ensuring the production of high-quality images.

Description

A model training method, device, and cluster based on visual reconstruction technology

[0001] This application claims priority to Chinese Patent Application No. 202410985237.3, filed on July 22, 2024, entitled "A Model Training Method, Device and Cluster Based on Visual Reconstruction Technology", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of visual reconstruction technology, and in particular to a model training method, device and cluster based on visual reconstruction technology. Background Technology

[0003] Images are the visual foundation for human perception of the world; humans can use images to acquire, express, and transmit information. To quickly acquire image information, computing devices can process images to obtain images free of interfering information (such as objects briefly appearing in the image, lighting, changes in appearance or color, etc.). For example, computing devices can use artificial intelligence (AI) models (such as RobustNeRF) to process images to obtain images free of interfering information. Specifically, the computing device uses RobustNeRF to obtain a rendered image of the first scene based on the parameters of the capturing device. The computing device obtains the residual between this rendered image and the photograph of the first scene, compares this residual with a user-defined threshold to obtain interfering information in the image, and then uses this interfering information to obtain an image of the first scene free of interfering information.

[0004] In the above process, improperly set thresholds may lead to the inability to accurately identify interference information in the image, resulting in poor image quality. Summary of the Invention

[0005] This application provides a model training method, device, and cluster based on visual reconstruction technology to solve the problem of poor image quality caused by the inability to accurately identify interference information in images.

[0006] In a first aspect, this application provides a model training method based on visual reconstruction technology. The method includes: acquiring training data. The training data includes multiple images of a first scene from different perspectives, and shooting parameters corresponding to each image. Obtaining the residual between a first image and a second image. The first image is one of multiple images. The first image includes first content and first interference information. The second image is an image generated by an initial model based on the first shooting parameters. The first shooting parameters are the shooting parameters corresponding to the first image. The second image includes the first content and second interference information. The second interference information is interference information generated by the initial model based on the first shooting parameters, corresponding to the first interference information. And training the initial model using the residual to obtain an image generation model. The image generation model is used to generate a third image based on the first shooting parameters, and the third image includes the first content.

[0007] In the first aspect of this application, a residual is obtained between a first image and a second image generated by an initial model based on first shooting parameters, and the initial model is trained using the residual to obtain a trained initial model (such as an image generation model). Since the residual obtained using the first image and the second image can accurately reflect the information of the first content and the first interference information in the first image, the possibility of the image generation model accurately identifying interference information in the image is improved, thus ensuring the acquisition of high-quality images.

[0008] In another possible implementation, the initial model is trained using residuals to obtain an image generation model. This includes: obtaining the distribution information of the first content and the first interference information based on the residuals; and training the initial model using the residuals and distribution information to obtain the image generation model. Thus, training the initial model using the distribution information of the residuals, the first content, and the first interference information ensures that the trained initial model can accurately capture the interference information in the image.

[0009] In another possible implementation, an initial model is trained using residual and distribution information to obtain an image generation model. This includes: obtaining a mask for a first image using the residual and distribution information; processing the first image using the mask to obtain a processed first image; the processed first image including the first content; and training the initial model using the processed first image to obtain the image generation model. Thus, obtaining the image mask based on the distribution information that accurately reflects the content and interference information in the image ensures that the obtained mask can accurately select the content in the image.

[0010] In another possible implementation, a mask for the first image is obtained using residual and distribution information. This includes: using a Gaussian Mixture Model (GMM) to obtain a threshold for distinguishing between the first content and the first interference information based on the distribution information; and obtaining the mask for the first image based on the residual and the threshold. Thus, by obtaining a threshold that accurately distinguishes between content and interference information in the image based on distribution information that accurately reflects the image content and interference information, a guarantee is provided for subsequently obtaining a mask that can accurately select the content in the image.

[0011] In another possible implementation, a Gaussian Mixture Model (GMM) is used to obtain the threshold for distinguishing between the first content and the first interference information based on distribution information. This includes: using the GMM, obtaining the first mean of the Gaussian distribution of the residuals corresponding to the first content, and the second mean of the Gaussian distribution of the residuals corresponding to the first interference information. The number of training epochs of the initial model and the expected total number of training epochs are also obtained. Finally, the threshold for distinguishing between the first content and the first interference information is obtained based on the first mean, the second mean, the number of training epochs, and the expected total number of training epochs. This method is simple to calculate, has low complexity, and can quickly obtain the threshold. Compared to waiting for user input of the threshold, it shortens the waiting time and improves the efficiency of model training.

[0012] In another possible implementation, the mask of the first image is obtained based on residuals and a threshold, including: obtaining a first mask based on pixel residuals (including the pixel residuals) and a threshold. The pixel residuals are calculated based on the pixel values ​​of individual pixels in the first and second images. A second mask is obtained based on patch residuals (including the patch residuals) and a threshold. The patch residuals are calculated based on the pixel values ​​of multiple pixels in the first and second images. Finally, the mask of the first image is obtained based on the average of the first and second masks. Thus, the mask of the image is calculated using the first mask calculated based on pixel residuals and the second mask calculated based on image residuals. Obtaining the mask based on information from individual pixels and information from multiple pixels associated with those individual pixels ensures that the obtained image mask can accurately select content within the image.

[0013] In another possible implementation, the first interference information includes one or more of the following: second content, lighting, and color. The shooting parameters include one or more of the following: shooting angle, shooting position, and internal parameters of the shooting device.

[0014] In another possible implementation, the method may also include: constructing a three-dimensional model of the first content based on the third image.

[0015] Secondly, this application provides a model training device based on visual reconstruction technology. The model training device includes a transceiver module and a processing module. The transceiver module is used to: acquire training data. The training data includes multiple images from different perspectives corresponding to a first scene, and shooting parameters corresponding to each image. The transceiver module is also used to: acquire the residual between a first image and a second image. The first image is one of multiple images. The first image includes first content and first interference information. The second image is an image generated by the initial model based on the first shooting parameters. The first shooting parameters are the shooting parameters corresponding to the first image. The second image includes the first content and second interference information. The second interference information is interference information generated by the initial model based on the first shooting parameters, corresponding to the first interference information. The processing module is used to: train the initial model using the residual to obtain an image generation model. The image generation model is used to generate a third image based on the first shooting parameters, and the third image includes the first content.

[0016] In one possible implementation, the processing module is specifically used to: obtain the distribution information of the first content and the first interference information based on the residual. The processing module is also specifically used to: train an initial model using the residual and the distribution information to obtain an image generation model.

[0017] In another possible implementation, the processing module is specifically used to: obtain a mask for the first image using residual and distribution information. The processing module is also specifically used to: process the first image using the mask to obtain a processed first image. The processed first image includes the first content. The processing module is also specifically used to: train the initial model using the processed first image to obtain an image generation model.

[0018] In another possible implementation, the processing module is specifically used to: utilize a Gaussian Mixture Model (GMM) to obtain a threshold for distinguishing between the first content and the first interference information based on the distribution information. The processing module is also specifically used to: obtain a mask for the first image based on the residual and the threshold.

[0019] In another possible implementation, the processing module is specifically configured to: utilize a Gaussian Mixture Model (GMM) to obtain the first mean of the Gaussian distribution of the residual corresponding to the first content and the second mean of the Gaussian distribution of the residual corresponding to the first interference information, based on the distribution information. The processing module is also specifically configured to: obtain the number of training epochs of the initial model and the expected total number of training epochs. Furthermore, the processing module is also specifically configured to: obtain the threshold for distinguishing between the first content and the first interference information based on the first mean, the second mean, the number of training epochs, and the expected total number of training epochs.

[0020] In another possible implementation, the processing module is specifically configured to: obtain a first mask based on the pixel residuals and a threshold included in the residuals. The pixel residuals are calculated based on the pixel values ​​of individual pixels in the first and second images. The processing module is also specifically configured to: obtain a second mask based on the patch residuals and a threshold included in the residuals. The patch residuals are calculated based on the pixel values ​​of multiple pixels in the first and second images. The processing module is further specifically configured to: obtain a mask for the first image based on the average of the first and second masks.

[0021] In another possible implementation, the first interference information includes one or more of the following: second content, lighting, and color. The shooting parameters include one or more of the following: shooting angle, shooting position, and internal parameters of the shooting device.

[0022] In another possible implementation, the processing module is also used to: build a 3D model of the first content based on the third image.

[0023] Thirdly, this application provides a chip. The chip includes an interface circuit and a control circuit, wherein the interface circuit is used to acquire training data and, in conjunction with the control circuit, implement the method in the first aspect or any of the optional implementations of the first aspect.

[0024] Fourthly, this application provides a cluster. The cluster includes at least one computing device, each computing device including a processor and a memory. The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, causing the cluster to perform operational steps of the method in the first aspect or any possible implementation of the first aspect.

[0025] Fifthly, this application provides a computer-readable storage medium, comprising: computer software instructions. When executed in a computing device, the computer software instructions cause the computing device to perform operational steps of the method as described in the first aspect or any possible implementation thereof.

[0026] Sixthly, this application provides a computer program product. When the computer program product is run on a computer, it causes a computing device to perform the operational steps of the method as described in the first aspect or any possible implementation thereof.

[0027] The beneficial effects of aspects two through six above can be described with reference to the first aspect or any implementation thereof, and will not be repeated here. Based on the implementations provided in the above aspects, this application can be further combined to provide more implementations. Attached Figure Description

[0028] Figure 1 is a schematic diagram of the training process of the NeRF model;

[0029] Figure 2 is a schematic diagram of the training process of the 3DGS model;

[0030] Figure 3 is an example of training data;

[0031] Figure 4 is a flowchart illustrating a model training method based on visual reconstruction technology provided in this application;

[0032] Figure 5 is a schematic diagram of a process for obtaining distribution information provided in this application;

[0033] Figure 6 is an example diagram of the first model training process provided in this application;

[0034] Figure 7 is a schematic diagram of a process for obtaining a mask provided in this application;

[0035] Figure 8 is an application example of an image generation model provided in this application;

[0036] Figure 9 is a flowchart of a model training process provided in this application;

[0037] Figure 10 is a flowchart illustrating the training process of the second model provided in this application;

[0038] Figure 11 is a schematic diagram of a model training device based on visual reconstruction technology provided in this application;

[0039] Figure 12 is a schematic diagram of the structure of a model training system provided in this application;

[0040] Figure 13 is a schematic diagram of the structure of a chip provided in this application;

[0041] Figure 14 is a schematic diagram of a cluster structure provided in this application;

[0042] Figure 15 is a schematic diagram of the connection between computing devices provided in this application. Detailed Implementation

[0043] To ensure clarity and brevity in the description of the following embodiments, some concepts that may be involved in this application will be briefly introduced first.

[0044] (1) New perspective synthesis.

[0045] Novel view synthesis refers to the process by which a computing device generates a target image based on a source image, the pose of an object in the source image (or simply source pose), and the desired pose of that object (or target pose). The object in the target image possesses the target pose.

[0046] (2) Nerve radiation field model.

[0047] Neural radiance field (NeRF) models are neural network models that enable novel perspective synthesis. NeRF models can include robust NeRF models. Depending on the specific application scenario, before using a NeRF model to synthesize novel perspectives, computing devices typically need to train an initial NeRF model to suit the application scenario, resulting in a NeRF model that meets the application requirements. Figure 1 illustrates the training process of a NeRF model. As shown in Figure 1, the computing device can generate a rendered image using input information and train an initial NeRF model using the rendered image and real-world scene images to obtain a NeRF model that meets the application scenario. Specifically, the computing device can process the input information (such as the camera position and camera angle used in capturing images) to obtain multiple camera rays. The computing device uses a multilayer perceptron (MLP) in the NeRF model to encode and compute the input information, obtaining the transparency and color values ​​at multiple locations on each camera ray. The computing device then uses integration based on the transparency and color values ​​at multiple locations on each camera ray to obtain the pixel result of each camera ray. The computing device can generate a rendered image based on the pixel results of each camera ray. Furthermore, the computing device uses the rendered image and real-world scene images to adjust the weights of each layer of the MLP network in the initial NeRF model, resulting in a NeRF model that meets the application scenario.

[0048] (3) 3D Gaussian sputtering.

[0049] 3D Gaussian splatting (3DGS) models can achieve new perspective synthesis. Similarly, depending on the actual application scenario, before using a 3DGS model to achieve new perspective synthesis, computing devices usually need to train the initial 3DGS model in conjunction with the actual application scenario to obtain a 3DGS model that meets the application scenario. Figure 2 is a schematic diagram of the 3DGS model training process. As shown in Figure 2, the computing device can generate a rendered image using input information, and train the initial 3DGS model using the rendered image and real scene images to obtain a 3DGS model that meets the application scenario. Specifically, the computing device initializes the scene point cloud in the input information into a 3D Gaussian ellipsoid. The computing device projects an image based on the 3D Gaussian ellipsoid, the camera position used in the camera image taken in the input information, and the camera angle used in the camera image taken in the input information. The computing device rasterizes this image to obtain the rendered image. The computing device adjusts the density of the 3D Gaussian ellipsoid (adaptive density control) based on the differences between the rendered image and the real image. Using the adjusted 3D Gaussian ellipsoid, the camera position and angle used in the input image, it redraws the image. The computing device rasterizes and renders this redrawn image, obtaining a new rendered image. The computing device recalculates the differences between the new rendered image and the real image, and readjusts the density of the 3D Gaussian ellipsoid. This process is repeated until the differences between the rendered image and the real image meet the requirements, at which point a 3DGS model is output.

[0050] (4) Gaussian mixture model.

[0051] A Gaussian Mixture Model (GMM) is a model that decomposes things into multiple Gaussian sub-models. GMM can be represented by the following formula (1).

[0052] Among them, Γ n For Gaussian models, p(Γ) n Let λ be the probability distribution of the Gaussian model, K be the number of Gaussian distributions, and λ be the probability distribution of the model. k Let k be the weight of the k-th Gaussian submodel in the GMM. The k-th Gaussian sub-model, μ k Let be the mean of the k-th Gaussian sub-model. Let be the variance of the k-th Gaussian sub-model.

[0053] The concepts that may be involved in this application have been described above with reference to the accompanying drawings. A brief introduction to the related technologies is given below with reference to the accompanying drawings.

[0054] The computing device can use the NeRF model and 3DGS model described above to generate a target image of an object with the target pose, based on the source image, the pose of the object in the source image (or simply the source pose), and the desired pose of the object (or the target pose). If the source image also contains interference information, the computing device can use the RobustNeRF model within the NeRF model to remove the interference information according to a user-defined threshold, resulting in a target image of the object with the target pose but without interference information.

[0055] The computing device can train the initial RobustNeRF model using training data and a set threshold to obtain the trained RobustNeRF model. Figure 3 shows an example of training data. As shown in Figure 3, the training data includes images 1 to n, as well as the shooting angle and shooting position of each image. Images 1 to n are images from viewpoints 1 to n of the first scene, respectively. Among them, image 1 obtained from viewpoint 1 has interference 1, image 3 obtained from viewpoint 3 has interference 2, and other images obtained from other viewpoints do not have interference.

[0056] The computing device can obtain a mask for an image by using a set threshold to distinguish content and distractors in the images included in the training data. The computing device can then use this mask to process the images in the training data to obtain a processed image. Finally, the processed image can be used to train an initial model to obtain a trained model.

[0057] The computing device uses a fixed threshold to distinguish between content and interference information in an image. Based on this fixed threshold, the computing device obtains an image mask and processes the image using this mask to obtain a processed image. The computing device then uses the processed image to train a model, resulting in a trained model. However, in this process, the interference information in the image may vary depending on the shooting scene and the shooting device. The fixed threshold may not accurately distinguish between interference information and content in the image, thus affecting the image quality generated by the trained model.

[0058] Based on this, this application provides a model training method based on visual reconstruction technology. The method involves obtaining the residual between a first image and a second image generated by an initial model according to first shooting parameters, and using this residual to train the initial model to obtain a trained initial model (such as an image generation model). Since the residual obtained using the first and second images can accurately reflect the information of the first content and the first interference information in the first image, it improves the possibility of the image generation model accurately identifying interference information in the image, thus ensuring the acquisition of high-quality images.

[0059] Figure 4 is a flowchart illustrating a model training method based on visual reconstruction technology provided in this application. This method can be executed by a computing device, a cluster of multiple computing devices, a component within the computing device (such as a processor, chip, etc.), or a logic module or software capable of implementing all or part of the functions. In the case where the method is implemented by a logic module or software, the software can be deployed on a cloud platform, which can then provide the software as a value-added service to users. For the hardware implementation of the computing device, refer to the computing device 1210, client device 1220, or acceleration device 1215 shown in Figure 12 below. For the hardware implementation of the components within the computing device, refer to the chip shown in Figure 13 below.

[0060] The model training method provided in this application is described below. As shown in Figure 4, the model training method provided in this embodiment may include the following steps S410 to S430.

[0061] S410, acquire training data.

[0062] The training data includes multiple images of the first scene from different perspectives, and the shooting parameters corresponding to each image. Depending on the needs of the actual application, each image in the multiple images can correspond to a perspective, and at least one image in the multiple images can also correspond to a perspective; this application does not limit this. For example, the training data includes m images. Images 1 to 5 of the m images are five images of the second scene captured by the shooting device from perspective 1. Image 6 of the m images is one image of the second scene captured by the shooting device from perspective 2.

[0063] The shooting parameters for each image refer to the shooting parameters used by the shooting device to capture that image. The shooting device can be any device capable of acquiring an image, and can include, but is not limited to, cameras, drones with video recording capabilities, virtual reality devices, mobile phones, personal computers, etc. The shooting parameters for each image may include: the shooting position used by the shooting device to capture the image, the shooting angle used by the shooting device to capture the image, and the internal parameters of the shooting device. The shooting position, shooting angle, and internal parameters are explained below.

[0064] (1) Shooting location.

[0065] The shooting position can refer to the location where the shooting device captures the image. The shooting position can be the coordinate position of the shooting device in the geodetic coordinate system when capturing the image. For example, the shooting position in the first shooting parameter can refer to the coordinate position (x1, y1, z1) of the shooting device in the geodetic coordinate system when the shooting device captures the first image, that is, the shooting position used by the shooting device to capture the first image is (x1, y1, z1).

[0066] (2) Shooting angle.

[0067] The shooting angle refers to the angle used by the shooting device to capture an image. The shooting angle can be the angle between the shooting device and the X-axis, Y-axis, and Z-axis of the camera coordinate system when the shooting device captures the image. For example, the shooting angle in the first shooting parameter can refer to the angle θ1, θ2, and θ3 between the shooting device and the X-axis, Y-axis, and Z-axis of the camera coordinate system when the shooting device captures the first image; that is, the shooting angle used by the shooting device to capture the first image is (θ1, θ2, θ3).

[0068] (3) Internal parameters of the shooting equipment.

[0069] The internal parameters of a shooting device may include one or more of the following: the focal length of the shooting device, the size of the pixels, the resolution of the pixels, etc. For more information on internal parameters, please refer to the description of general technology, which will not be repeated here.

[0070] There are several ways to obtain training data; two possible examples are given below.

[0071] Example 1: Receive training data sent by other devices.

[0072] In this scenario, training data can be received from other devices (such as client devices).

[0073] Example 2: Directly collect and obtain training data.

[0074] In this situation, training data can be acquired by utilizing one's own resources.

[0075] The training data may contain images with or without interference information. The residual between each image in the training data and the rendered image generated by the initial model using the shooting parameters corresponding to that image can be obtained. This residual is then used to train the initial model, resulting in the trained initial model. For example, the training data may include a first image and a fourth image. The first image includes first content and first interference information. The first interference information may include one or more of the following: second content, illumination, and color. The fourth image includes the first content. The following description uses the first image in the training data and the shooting parameters corresponding to that image as an example to illustrate the model training method provided in this application. In this case, the following step S420 can be executed.

[0076] S420, Obtain the residual between the first image and the second image.

[0077] The first image is one of multiple images. The first image includes first content and first interference information. The first content and first interference information in the first image are described below.

[0078] (1) First content.

[0079] The first content can refer to an object in the first scene whose duration is greater than or equal to a duration threshold. The first content can be a moving object (such as an animal, vehicle, etc.) or a stationary object (such as a stone, etc.). The first content can include one object or multiple objects. When the first content includes multiple objects, the multiple objects can include moving objects or stationary objects, and this application does not limit this.

[0080] (2) First interference information.

[0081] The first interference information may include one or more of the following: second content, illumination, and color. The second content may refer to an object in the first scene whose duration is less than a duration threshold. The second content may be a moving object (such as an animal, vehicle, etc.). Similarly, the second content may include one moving object or multiple moving objects, which will not be elaborated upon in this application. Illumination may refer to illumination change information on the first content. In some possible cases, illumination can be determined to exist on the first content if the illumination change information is greater than an illumination change threshold. For example, the first content has a first illumination intensity at a first moment, and the first content has a second illumination intensity at a second moment. The first illumination intensity and the second illumination intensity are different, and the first illumination intensity and the second illumination intensity have a first illumination difference. Illumination is determined to exist on the first content if the first illumination difference is greater than an illumination change threshold. Color may refer to color change information on the first content. Similarly, color can be determined to exist on the first content if the color change information is greater than a color change threshold. For example, the first content has a first color at a first moment, and the first content has a second color at a second moment. Taking the three primary colors (red, green, blue, RGB) as an example, the first color has RGB1, and the second color has RGB2. Furthermore, if the difference between each primary color in RGB1 and RGB2 is greater than a color change threshold, then the first content is determined to have a color.

[0082] The second image is generated by the initial model based on the first shooting parameters. The shooting position, shooting angle, and internal parameters of the shooting device, included in the first shooting parameters, can be input into the initial model, which then generates the second image based on these parameters. The first shooting parameters are the shooting parameters corresponding to the first image. The second image includes first content and second interference information. The second interference information is interference information generated by the initial model based on the first shooting parameters, corresponding to the first interference information. In some possible cases, the second image can also be referred to as a rendered image of the first image. The initial model can include, but is not limited to, a NeRF model, a 3DGS model, etc.

[0083] In some possible cases, the residuals of the first and second images can be obtained using the following ① and ②.

[0084] ① Obtain the initial residuals of the first and second images.

[0085] The initial residual can be represented by ∈.

[0086] In some possible scenarios, the initial residuals of the first and second images can be obtained on a single pixel basis, or on a multiple pixel basis. These scenarios will be explained below.

[0087] Case 1: Obtain the initial residual of the first and second images on a per-pixel basis.

[0088] In this scenario, the pixel values ​​of each pixel in the first image and the pixel values ​​of each pixel in the second image can be obtained. The difference between the pixel values ​​of each pixel in the first image and the corresponding pixel values ​​in the second image is calculated to obtain the initial residual between the first and second images. In some possible cases, the initial residual obtained on a per-pixel basis can also be called the pixel-level initial residual, pixel initial residual, etc.

[0089] Scenario 2: Obtain the initial residual of the first and second images in units of multiple pixels.

[0090] In this scenario, the first image can be divided into multiple patches. Each patch contains multiple pixels. The initial residual for each patch is calculated to obtain the initial residuals for the first and second images. The initial residual for each patch can be calculated as follows: calculate the difference between the pixel value of each pixel in each patch and the corresponding pixel in the second image. Calculate the mean of the differences in pixel values ​​across all patches and use this mean as the initial residual for that patch. Patches can be in array form; in this case, the pixels included in a patch are the pixels included in the array. This array can be an array with an equal number of rows and columns (e.g., a 2×2 array) or an array with unequal numbers of rows and columns (e.g., a 2×3 array). In some possible cases, the initial residual obtained in units of multiple pixels can also be called patch-level initial residual, patch initial residual, etc.

[0091] For example, the first image can be divided into multiple patches of size 2×2. The initial residual of each patch is calculated, and the initial residuals of the first and second images are obtained based on the initial residuals of each patch. Taking the calculation of the initial residual of patch 1 as an example, the process of calculating the initial residual of a patch is explained. The differences (e.g., a1, a2, a3, a4) between the four pixel values ​​of patch 1 and the corresponding four pixel values ​​in the second image are calculated. The mean of a1, a2, a3, and a4 is calculated, and this mean is used as the initial residual of patch 1.

[0092] ② Based on the initial residual, obtain the residual between the first image and the second image.

[0093] After obtaining the initial residuals of the first and second images, these initial residuals can be processed to obtain the remaining residuals of the first and second images. Depending on the type of the initial residual, different methods can be used to process it to obtain the remaining residuals of the first and second images. These methods are explained below.

[0094] In case i, the initial residual is the initial residual of the pixel.

[0095] In this scenario, the processed residual of the target pixel can be obtained based on the initial residual of the target pixel and the initial residuals of multiple pixels associated with the target pixel. Furthermore, the residuals of the first and second images can be obtained based on the processed residuals of the target pixel. Specifically, the initial residual of the target pixel and the initial residual of each pixel among the multiple pixels associated with the target pixel are obtained, and the average of the initial residuals of the target pixel and the initial residuals of each pixel associated with the target pixel is calculated. This average is used as the processed residual of the target pixel. The processed residuals of each target pixel included in the first image are calculated using this method, and the residuals of the first and second images are obtained based on the processed residuals of each target pixel. In some possible cases, the residual obtained using the initial pixel residual can also be called the pixel residual or pixel-level residual.

[0096] Multiple pixels associated with the target pixel can refer to: multiple pixels that are geographically adjacent to the target pixel, multiple pixels that meet preset conditions, etc. Multiple pixels geographically adjacent to the target pixel can be in the form of an array. This array can be an array with an equal number of rows and columns, or an array with unequal numbers of rows and columns, such as pixels included in a 1×1 array, a 1×3 array, a 3×3 array, a 3×5 array, a 5×5 array, etc. Preset conditions can include, but are not limited to: pixels whose interval with the target pixel does not exceed a pixel threshold, and pixels whose residual is greater than a first residual threshold and less than a second residual threshold. For example, preset conditions could be pixels whose interval with the target pixel does not exceed one pixel, and pixels whose residual is greater than residual threshold 1 and less than residual threshold 2. In this case, the 8 pixels adjacent to the target pixel and the 8 initial residuals corresponding to these 8 pixels are obtained. Each of the eight initial residuals is compared with residual threshold 1 and residual threshold 2 to identify pixels whose initial residuals are greater than residual threshold 1 and less than residual threshold 2. The residual of the target pixel is calculated based on these pixels. The residuals of the first and second images are then obtained based on the residual of the target pixel. The following explanation uses the example of multiple pixels associated with the target pixel (i.e., multiple pixels arranged in an array that are geographically adjacent to the target pixel) to illustrate the model training method provided in this application.

[0097] Depending on the needs of the application, multiple processed residuals of the target pixel can be obtained from the initial residuals of multiple pixels included in arrays of various sizes and the initial residuals of the target pixel. Each array size corresponds to one processed residual.

[0098] For example, the processed residual 1 of the target pixel can be obtained based on the initial residuals of the pixels in a 1×1 array and the initial residual of the target pixel. Similarly, the processed residual 2 of the target pixel can be obtained based on the initial residuals of the pixels in a 3×3 array and the initial residual of the target pixel. Furthermore, the processed residual n of the target pixel can be obtained based on the initial residuals of the pixels in a (2n-1)×(2n-1) array and the initial residual of the target pixel. Here, n is a positive integer greater than or equal to 4.

[0099] In case j, the initial residual is the initial residual of the patch.

[0100] In this case, the initial residual of the tile can be directly used as the processed residual to obtain the residuals of the first and second images. In some possible cases, the residual obtained from the initial residual of the tile can also be called the tile residual or the tile-level residual.

[0101] For example, the first image can be divided into patches 1 to n. Initial residuals 1 to n corresponding to patches 1 to n are obtained using the method described in scenario 2 above. These initial residuals 1 to n are then used as the residuals for patches 1 to n. Based on the residuals for patches 1 to n, the residuals for the first and second images are obtained.

[0102] In some possible scenarios, after obtaining the processed residuals of the first and second images based on the initial residuals, the dimensions of the processed residuals can be adjusted to obtain the dimension-adjusted residuals. For example, the processed residuals can be adjusted to have a dimension of 1. In some possible examples, the dimension-adjusted residuals can be simply referred to as residuals, and can be represented by Γ. n express.

[0103] S430 uses the residuals to train the initial model, resulting in an image generation model.

[0104] The image generation model is used to generate a third image based on the first shooting parameters. The third image includes the first content.

[0105] (3) The initial model is trained using the residuals to obtain the image generation model.

[0106] The image generation model can be obtained by training the initial model using the residuals using the following methods ① and ②.

[0107] ① Using Gaussian Mixture Model (GMM), the distribution information of the first content and the first interference information is obtained based on the residuals.

[0108] Figure 5 is a schematic diagram of a process for obtaining distribution information provided in this application. As shown in Figure 5, the residual Γ can be used to obtain the distribution information. n The residual distribution histogram is obtained. Using a Gaussian Mixture Model (GMM), the distribution information of the first content and the first interference information is obtained from the residual distribution histogram. This distribution information includes the Gaussian distribution information of the residuals of the first content (distribution a in Figure 5) and the Gaussian distribution information of the residuals of the first interference information (distribution b in Figure 5).

[0109] Having obtained multiple pixel residuals and multiple patch residuals of the first image, the method described in Figure 5 can be used to obtain multiple distribution information of the first content and the first interference information. One distribution information corresponds to one pixel residual or one patch residual.

[0110] In some possible scenarios, a Gaussian Mixture Model (GMM) is used to obtain the first mean of the Gaussian distribution of the residuals corresponding to the first content and the second mean of the Gaussian distribution of the residuals corresponding to the first interference information, based on the distribution information. The number of training epochs of the initial model and the expected total number of training epochs are then obtained. Finally, based on the first mean, the second mean, the number of training epochs, and the expected total number of training epochs, a threshold for distinguishing between the first content and the first interference information is derived.

[0111] Specifically, the residual obtained during the i-th training round can be used to obtain the threshold corresponding to that training round according to the method described in Figure 5. The threshold calculated using the residual of the i-th round is then used as the threshold for training rounds i+1 to i+n. Here, i is a positive integer greater than or equal to 1, and n is a positive integer greater than or equal to 2, set according to actual needs. In this way, it is unnecessary to calculate the threshold corresponding to each training round based on the residual obtained in each round, thus reducing the computational load.

[0112] ② The initial model is trained using residual and distribution information to obtain the image generation model.

[0113] The process may include the following (a) to (c).

[0114] (a) Using residual and distribution information, obtain the mask of the first image.

[0115] The residual can include pixel residuals or tile residuals. Depending on the content of the residual, different methods can be used to obtain the mask of the first image, which will be explained in detail below.

[0116] In case a, the residual includes pixel residuals.

[0117] In this case, the mask for the first image can be obtained using the following methods ① and ②.

[0118] ① Based on the pixel residual, obtain the threshold corresponding to the pixel residual.

[0119] The pixel residual can refer to a single pixel residual or multiple pixel residuals.

[0120] For example, pixel residual refers to pixel residual 1. Pixel residual 1 is the residual of the first image calculated based on the initial residual of the pixel and the initial residuals of the pixels in the 3×3 array associated with that pixel. For details on how to calculate the residual using the initial residual of the pixel and the initial residuals of multiple pixels associated with that pixel, please refer to the relevant explanation above; it will not be repeated here.

[0121] For example, pixel residuals refer to pixel residual 1 and pixel residual 2. Pixel residual 1 is the residual of the first image calculated based on the initial residual of the pixel and the initial residuals of the pixels in the 3×3 array associated with the pixel. Pixel residual 2 is the residual of the first image calculated based on the initial residual of the pixel and the initial residuals of the pixels in the 5×5 array associated with the pixel. For details on how to calculate the residual using the initial residual of the pixel and the initial residuals of multiple pixels associated with the pixel, please refer to the relevant explanation above; it will not be repeated here.

[0122] ② Obtain the mask of the first image based on the pixel residuals and the threshold.

[0123] Depending on the content pointed to by the pixel residual, the mask of the first image is obtained in different ways. The following uses (1) and (2) to explain them respectively.

[0124] (1) Pixel residual refers to the residual of a single pixel.

[0125] In this case, the distribution information corresponding to the pixel residual can be obtained using the method described in Figure 5 above. Based on the distribution information, the threshold corresponding to the pixel residual is calculated using formula (2). This threshold is used to distinguish between the first content and the first interference information in the first image. The pixel residual is then compared with the threshold to obtain the mask of the first image. Formula (2) can be expressed as follows: ρ n =α·μ1(Γ n )+(1-α)·μ2(Γ n ) Formula (2)

[0126] in, ω is a hyperparameter controlling the movement speed, S is the number of training epochs of the initial model, N is the expected total number of training epochs, and Γ n For the residual, μ1(Γ) n ) represents the mean of the Gaussian distribution of the residuals corresponding to the first content, μ2(Γ) n ) represents the mean of the Gaussian distribution of the residual corresponding to the first interference information.

[0127] In some possible scenarios, during the training of the initial model using training data, the initial model is first trained using images in the training data that do not contain interference information, along with the corresponding shooting parameters of those images. Then, the initial model is trained again using images in the training data that do contain interference information, along with the corresponding shooting parameters of those images. Figure 6 is an example diagram of the first model training process provided in this application. As shown in Figure 6, in the training rounds of round 1 and round 5×10... 3 In the case of 2.5 × 10^6 training epochs, the initial model is trained by prioritizing images in the training data that do not contain interference information, along with the corresponding shooting parameters of those images. 5 In this case, the initial model is trained using the image containing interference information and the corresponding shooting parameters of that image. In this scenario, if the residual of a pixel is greater than a threshold, the pixel is considered to be included in the first interference information; conversely, if the residual of a pixel is less than or equal to the threshold, the pixel is considered to be included in the first content. Thus, the mask for pixels with residuals greater than the threshold can be set to 0 to eliminate the first interference information in the first image, and the mask for pixels with residuals less than or equal to the threshold can be set to 1 to retain the first content in the first image.

[0128] Depending on the needs of the application, if the residual of a pixel is greater than the threshold, the mask of that pixel can be set to 1. If the residual of a pixel is less than or equal to the threshold, the mask of that pixel can be set to 0; this application does not limit this. In some possible cases, the mask can also be set to other content that has the same meaning as false, such as 0. And the mask can also be set to other content that has the same meaning as true, such as 1.

[0129] For example, the pixel residual includes pixel residual 1. Using the method described in Figure 5, the distribution information 1 of pixel residual 1 is obtained. The threshold 1 corresponding to pixel residual 1 is obtained using formula (2). The residual corresponding to pixel residual 1 of each pixel in the first image is compared with the threshold 1. According to the content described in Figure 6 above, if the residual of a pixel is greater than the threshold 1, the mask of that pixel is set to 0; if the residual of a pixel is less than or equal to the threshold 1, the mask of that pixel is set to 1, thus obtaining the mask of the first image.

[0130] (2) Pixel residual refers to the residual of multiple pixels.

[0131] In this case, the method described in (1) can be used to obtain multiple initial masks corresponding to multiple pixel residuals. An OR operation is then performed on the multiple initial masks to obtain the mask of the first image. Specifically, the initial mask of each pixel indicated by each initial mask of the first image is obtained, and an OR operation is performed on the initial mask of each pixel indicated by each initial mask to obtain the OR operation result of each pixel in the first image. Based on the OR operation result of each pixel, the mask of the first image is obtained. In some possible cases, the mask calculated using the pixel residuals of the image can also be called the first mask.

[0132] For example, the first image includes pixels 1 to m. The initial mask of the first image includes initial mask 1 and initial mask 2. The initial masks for pixels 1 to m indicated by initial mask 1 and the initial masks for pixels 1 to m indicated by initial mask 2 are obtained. An OR operation is performed on the initial masks for pixels 1 indicated by initial mask 1 and the initial masks for pixels 1 indicated by initial mask 2. Similarly, an OR operation is performed on the other initial masks indicated by initial mask 1 and the other initial masks for pixels indicated by initial mask 2, obtaining the result of the OR operation for each pixel. Based on the result of the OR operation for each pixel, the mask of the first image is obtained.

[0133] Figure 7 is a schematic diagram of a mask acquisition process provided in this application. As shown in Figure 7, image 1 includes pixels 1 to 4. The initial mask of the image includes initial mask 1 and initial mask 2. Initial mask 1 indicates that the masks for pixels 1 to 4 are 0011. Initial mask 2 indicates that the masks for pixels 1 to 4 are 0101. In this case, based on initial masks 1 to initial masks 2, the mask of image 1 is obtained as 0111.

[0134] In case b, the residuals include pixel residuals and tile residuals.

[0135] In one possible scenario, a first mask is obtained based on the pixel residuals included in the residuals and a threshold. The pixel residuals are calculated based on the pixel values ​​of individual pixels in the first and second images. A second mask is obtained based on the patch residuals included in the residuals and a threshold. The patch residuals are calculated based on the pixel values ​​of multiple pixels in the first and second images. The mask for the first image is obtained by averaging the first and second masks.

[0136] Similarly, tile residuals can refer to a single tile residual or multiple tile residuals. Multiple tile residuals can include tile residuals of multiple tiles of various sizes. Among them, one tile residual corresponds to a tile of one size. Depending on the content referred to by the tile residual, the method of obtaining the mask of the first image is also different, which will be explained below using (1) to (3).

[0137] (1) Plot residual refers to the residual of a single plot.

[0138] The tile residuals can be calculated using the initial residuals of each tile included in the first image. In this case, steps a through c can be performed to obtain the mask of the first image.

[0139] a) Using the method described in case a above, obtain the mask (or first mask) of the first image based on the pixel residuals.

[0140] b. Using the method described in case a above, obtain the mask (or second mask) of the first image based on the tile residual.

[0141] Specifically, using the method described in Figure 5 above, the distribution information of the first content and the first interference information in the first image is calculated based on the patch residuals. The threshold corresponding to the patch residuals is then calculated using formula (2) based on this distribution information. The residuals of each patch in the first image are compared with the threshold. If the residual of a patch is greater than the threshold, the mask of that patch is set to 0. If the residual of a patch is less than or equal to the threshold, the mask of that patch is set to 1. The mask of each pixel included in each patch is set as the mask of that patch, thus obtaining the second mask of the first image.

[0142] c. Based on the first mask and the second mask, obtain the mask of the first image.

[0143] Calculate the average of the first mask and the second mask, and obtain the mask of the first image based on the average. Specifically, calculate the average of the mask of each pixel in the first image indicated by the first mask and the mask of the corresponding pixel indicated by the second mask, and use the average as the mask of the corresponding pixel in the first image.

[0144] For example, the third image includes pixels 1 to n. The residual includes pixel residual 1 and patch residual 1. Patch residual 1 is the residual calculated after dividing the first image into multiple patches of size 2×2. These multiple patches include patch 1, which includes pixels 1 to 4. Using the method described above, the mask corresponding to pixels 1 to 4 is calculated as 0011 based on pixel residual 1, and the mask of patch 1 is calculated as 1 based on patch residual 1. In this case, the masks of pixels 1 to 4 are all set to 1 based on the mask of patch 1. The average value of the mask 0 of pixel 1 calculated based on pixel residual 1 and the mask 1 of pixel 1 calculated based on patch residual 1 is calculated to be 0.5, and this average value is used as the mask of pixel 1 of the first image. Similarly, the mask 0.5 of pixel 2, the mask 1 of pixel 3, and the mask 1 of pixel 4 of the first image are calculated. Similarly, the method described above can be used to obtain the mask for pixels 4 to n of the first image. Based on the mask for each pixel, the mask for the first image is obtained.

[0145] (2) Plot residuals refer to the residuals of multiple plots.

[0146] These multiple tile residuals are tile residuals of multiple tiles of various sizes. Each tile residual corresponds to a specific type of stored tile residual.

[0147] For example, the plurality of tile residuals include tile residual 1 to tile residual n. Among them, tile residual 1 is calculated by dividing the first image into tiles of size 2×2, and tile residual i is calculated by dividing the first image into tiles of size j×j, where i is a positive integer greater than or equal to 2 and less than or equal to n, and j is a positive integer greater than or equal to 3.

[0148] In this case, the mask of the first image corresponding to each patch residual can be obtained by using the method described in (1) above, thus obtaining multiple masks of the first image. The average of the masks obtained based on pixel residuals and the masks obtained based on multiple patch residuals is calculated, and the mask of the first image is obtained based on this average.

[0149] For example, the first image includes pixels 1 to n. The residuals include pixel residual 1, patch residual 1, and patch residual 3. Patch residual 1 is the residual of the first image calculated by dividing the first image into multiple first patches of size 2×2. Patch residual 2 is the residual of the first image calculated by dividing the first image into multiple second patches of size 3×3. Patch residual 3 is the residual of the first image calculated by dividing the first image into multiple first patches of size 4×4. The multiple first patches include patch 1, which includes pixels 1 to 4. The multiple second patches include patch 2, which includes pixels 1 to 9. The multiple third patches include patch 3, which includes pixels 1 to 16. Using the method described above, the mask corresponding to pixels 1 to 4 is calculated as 0011 based on pixel residual 1, the mask for patch 1 is calculated as 1 based on patch residual 1, the mask for patch 2 is calculated as 0 based on patch residual 2, and the mask for patch 3 is calculated as 0 based on patch residual 3. In this case, the average value of the mask 0 for pixel 1 corresponding to pixel residual 1, the mask 1 for pixel 1 corresponding to patch residual 1, the mask 0 for pixel 1 corresponding to patch residual 2, and the mask 0 for pixel 1 corresponding to patch residual 3 is calculated to be 0.25, and this average value is used as the mask for pixel 1 of the first image. Similarly, the masks for pixels 2 to 4 are obtained as 0.25, 0.5, and 0.5, respectively. The mask corresponding to each pixel in the first image is calculated using the above method to obtain the mask of the first image. In some possible cases, a weighted calculation can also be performed on the mask of the first image obtained from a single pixel, or on the mask of the first image obtained from multiple pixels. And the mask of the third image can be obtained from the result of the weighted calculation.

[0150] (b) The first image is processed using a mask to obtain the processed first image.

[0151] The processed first image includes the first content.

[0152] In some possible cases, the first image can be processed using a mask according to formula (3) to minimize the difference between the first image and the second image, thus obtaining the processed first image.

[0153] in, The difference between the first image and the second image is represented by R, where R is the total number of pixels in the first image, r is the r-th pixel in the first image, and M is the number of pixels in the second image. r Let r be the mask of the r-th pixel in the first image. Let I be the pixel value of the r-th pixel in the second image. r Let r be the pixel value of the first image. The difference between the pixel value of the r-th pixel in the first image and the pixel value of the r-th pixel in the second image.

[0154] (c) The initial model is trained using the processed first image to obtain the image generation model.

[0155] The processed first image is used as the input initial model, and the initial model is trained to obtain the image generation model.

[0156] The above description uses an example of a first image including first interference information to illustrate the model training method provided in this application. In some possible cases, when the initial model is trained using an image in the training data that does not include interference information, and the corresponding shooting parameters of that image, the model training method described above can still be used to train the initial model. The difference is that when the training data consists of an image that does not include interference information and the corresponding shooting parameters of that image, the mask of the first image calculated using this application is 1.

[0157] In some possible scenarios, Figure 8 illustrates an application example of an image generation model provided by this application. As shown in Figure 8, after training the initial model using training data to obtain the image generation model, the first shooting parameters can be input into the image generation model to generate a third image. The first shooting parameters are the parameters used by the shooting device to capture the first image. The first image includes: first content and first interference information. The third image includes the first content.

[0158] For example, the shooting position (x1, y1, z1), shooting angle (θ1, θ2, θ3), and internal parameters of the shooting device included in the first shooting parameters are input into the image generation model, and the image generation model generates a third image that includes the first content but does not include the first interference information based on the first shooting parameters.

[0159] In some possible scenarios, a 3D model of the first content can also be built using the second image. Specifically, using the image processing method described above, multiple images, including the second image, are processed to obtain a processed image that excludes interference information. These multiple images are images of the same scene from different perspectives. A 3D model of the first content is built using this processed image. For details on building 3D models using images, please refer to the descriptions of related technologies; they will not be repeated here. In this way, a 3D model excluding interference information can be generated based on the shooting parameters used to capture the images.

[0160] Based on the above functions, the model training method provided in this application can be applied to the fields of virtual reality (VR), autonomous driving, digital asset game modeling, and cultural relic protection. The usage of the model training method provided in this application in the above fields will be explained below.

[0161] In the VR field: The model training method provided in this application can be used to obtain an image generation model adapted to VR scenes using training data, and then use the image generation model to obtain an image without interference information, and reconstruct a scene without interference information based on the image. Thus, even when interference information (such as moving people, moving vehicles, pets, etc.) exists in the scene, a scene without interference information can be displayed, improving the user experience.

[0162] In the field of autonomous driving: the model training method provided in this application can be used to obtain an image generation model adapted to autonomous driving scenarios using training data, and the image generation model can be used to obtain images that do not contain interference information (such as moving objects such as people and vehicles), and the scene that does not contain interference information can be reconstructed from the image. In this way, a clean and tidy road survey environment for autonomous driving is guaranteed.

[0163] In the field of digital asset modeling: the model training method provided in this application can be used to obtain an image generation model adapted to the digital asset reconstruction scenario using training data. This model can then be used to obtain an image free of interference information (such as lighting, occlusion, etc.), and based on this image, object assets free of interference information can be reconstructed. This ensures the generation of high-quality object assets through digital asset modeling.

[0164] In the field of cultural relic protection: the model training method provided in this application can be used to obtain an image generation model of the cultural relic protection scene using training data, and the image generation model can be used to obtain an image that does not include interference information (such as color, lighting, occlusion, etc.), and the cultural relic that does not include interference information can be reconstructed based on the image.

[0165] The above-mentioned fields are merely optional methods for applying the technical solutions provided in the embodiments of this application to specific fields, and should not be construed as limiting this application. In some optional implementation methods, the technical solutions provided in the embodiments of this application can also be applied to fields such as robot navigation, which will not be elaborated here.

[0166] Figure 9 is a flowchart of a model training method provided in this application. As shown in Figure 9, an initial model is used to generate a first image based on the first shooting parameters of a first image captured by a shooting device. The residual between the first image and the second image is calculated. The distribution information of the first content and the first interference information in the first image is calculated using the Gaussian Mixture Model (GMM) based on the residual. The mask of the first image is calculated based on the distribution information of the first content and the first interference information in the first image. The first image is processed using the mask to obtain the processed first image. The initial model is trained using the processed first image to obtain the image generation model.

[0167] In the above process, the initial residual ε between the first image and the second image can be calculated first. Then, the initial residual ε is processed to obtain the residual between the first image and the second image. The dimension of this residual is adjusted to obtain the dimension-adjusted residual. The dimension of the dimension-adjusted residual is 1. The distribution information of the first content and the first interference information in the first image is obtained from this residual. Based on this residual and the distribution information of the first content and the first interference information in the first image, a mask for the first image is obtained. This mask is then used to process the first image to obtain the processed first image. The processed first image is then used to train an initial model to obtain an image generation model.

[0168] Figure 10 is a flowchart illustrating the training process of the second model provided in this application. As shown in Figure 10, the processed residual is obtained based on the initial residual ε. The processed residual includes pixel residuals 1 to Pixel residual i and patch residuals 1 to Patch residual m. Pixel residual a is calculated using the initial residuals of each pixel and the initial residuals of multiple pixels included in an array of size a associated with each pixel. Pixel residual a is one of pixel residuals 1 to Pixel residual i, and one pixel residual corresponds to an array of one size. Patch residuals 1 to Patch residual m are the residuals calculated by dividing the first image into patches of m sizes. One patch residual corresponds to a patch of one size. The dimensions of each residual are adjusted to obtain the dimension-adjusted residual. The dimension of the dimension-adjusted residual is 1; in this case, the dimension-adjusted residual is also called a one-dimensional residual. Based on each residual, the distribution information 1 of the first content and the distribution information 2 of the first interference information in the first image are obtained. Based on the distribution information of each residual, the first content in the first image, and the first interference information, a threshold corresponding to each residual is obtained. Based on each residual and its corresponding threshold, the residuals of the first image are processed to obtain multiple masks for the first image. Based on each mask, a mask for the first image is obtained. This mask is then used to process the first image to obtain a processed first image. The processed first image is then used to train an initial model to obtain an image generation model.

[0169] In this embodiment, the residual between the first image and the second image generated by the initial model based on the first shooting parameters is obtained, and the initial model is trained using the residual to obtain the trained initial model (such as an image generation model). Since the residual obtained using the first image and the second image can truly reflect the information of the first content and the first interference information in the first image, the possibility of the image generation model accurately identifying interference information in the image is improved, thus ensuring the acquisition of high-quality images.

[0170] It is understood that, in order to achieve the functions in the above embodiments, the computing device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and method steps described in conjunction with the embodiments disclosed in this application, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0171] The model training method provided according to the embodiments of this application has been described in detail above with reference to Figures 4 to 10. The model training device based on visual reconstruction technology provided according to the embodiments of this application will be described below with reference to Figure 11.

[0172] This application also provides a model training device based on visual reconstruction technology. Figure 11 is a schematic diagram of the structure of a model training device based on visual reconstruction technology provided in this application. This model training device can be used to implement the functions of the computing device in the above-described model training method embodiments to obtain an image generation model using training data, thus achieving the beneficial effects of the above-described method embodiments. In this embodiment, the model training device can be any device shown in Figure 12 below, such as computing device 1210, or the computing device shown in subsequent embodiments, or it can be a module (such as a chip) applied to the device.

[0173] As shown in Figure 11, the model training device 1100 based on visual reconstruction technology includes a transceiver module 1110 and a processing module 1120. The transceiver module 1110 is used to: acquire training data. The training data includes multiple images from different perspectives corresponding to a first scene, and a set of shooting parameters corresponding to each image. The transceiver module 1110 is also used to: acquire the residual between the first image and the second image. The first image is one of multiple images. The first image includes first content and first interference information. The second image is an image generated by the initial model based on the first shooting parameters. The first shooting parameters are the shooting parameters corresponding to the first image. The second image includes the first content and the second interference information. The second interference information is interference information generated by the initial model based on the first shooting parameters and corresponding to the first interference information. The processing module 1120 is used to: train the initial model using the residual to obtain an image generation model. The image generation model is used to generate a third image based on the first shooting parameters, and the third image includes the first content.

[0174] For more information on the functions of the transceiver module 1110 and the processing module 1120, please refer to the description of the image processing method above; they will not be repeated here.

[0175] The model training device 1100 based on visual reconstruction technology according to this application embodiment can be implemented by software modules. The model training device 1100 based on visual reconstruction technology according to this application embodiment can correspond to the execution of the model training method described in this application embodiment, and the above and other operations and / or functions of each module in the model training device 1100 based on visual reconstruction technology are respectively for implementing the process of the model training method in the foregoing figures. For the sake of brevity, they will not be described again here.

[0176] It is worth noting that if the model training device 1100 based on visual reconstruction technology is implemented through a software module, for example, this software module can be provided to users through a cloud service subscription model, and users can choose different subscription levels according to their needs. Alternatively, this software module can also provide enterprise-level customized services with professional domain customization, interface personalization, and extended functions according to the needs of users or enterprises.

[0177] Furthermore, the image processing device and the model training device based on visual reconstruction technology provided in this application can also be provided to users as value-added services, and this application does not limit this.

[0178] The model training device 1100 based on visual reconstruction technology in this application embodiment can also be implemented by hardware, such as a computing device. The specific implementation method of the computing device can be referred to the description in FIG12, which will not be repeated here.

[0179] The model training method provided in this application can also be applied to a model training system. Figure 12 is a schematic diagram of the structure of a model training system provided in this application. As shown in Figure 12, the model training system 1200 includes a computing device 1210. The computing device 1210 is a common computer device. Users can input data (such as training data) into the computing device 1210, which processes the input data to obtain processing results. The computing device 1210 can also output processing results (such as an image generation model).

[0180] The computing device 1210 includes a communication interface 1214, a processor 1211, and a memory 1212. The communication interface 1214 is used to communicate with devices located outside the computing device 1210. For example, a user inputs first imaging parameters to the computing device 1210 through the communication interface 1214 to obtain a second image that does not include interference information. Based on the received first imaging parameters, the computing device 1210 obtains a processing result (such as an image generation model) and then sends the second image through the communication interface 1214. This communication interface can be an input / output (I / O) interface.

[0181] Processor 1211 is the core of computing device 1210 for both computation and control. It may include: a central processing unit (CPU), a specific integrated circuit, other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. In practical applications, computing device 1210 may also include multiple processors. Processor 1211 may include one or more processor cores. An operating system and other software programs are installed in processor 1211, enabling it to access memory 1212 and various peripheral component interconnect (PCIe) devices.

[0182] The processor 1211 is connected to the memory 1212 via a double data rate (DDR) bus or other types of bus. The memory 1212 is the main memory of the computing device 1210. The memory 1212 is typically used to store various running software in the operating system. To improve the access speed of the processor 1211, the memory 1212 needs to have a high access speed. In conventional computer devices, dynamic random access memory (DRAM) is typically used as the memory 1212. Besides DRAM, the memory 1212 can also be other random access memories, such as static random access memory (SRAM). Alternatively, the memory 1212 can also be a read-only memory (ROM). For example, a read-only memory could be a programmable read-only memory (PROM) or an erasable programmable read-only memory (EPROM). This embodiment does not limit the number or type of the memory 1212.

[0183] In some possible scenarios, computing device 1210 may also include an acceleration device 1215. The acceleration device 1215 can be used to perform image processing tasks, model training tasks, etc. Processor 1211 sends the received task and input data (such as training data) to acceleration device 1215. After completing the task based on the input data, acceleration device 1215 sends the processing result (such as an image generation model) back to processor 1211. As shown in Figure 12, acceleration device 1215 can be directly inserted into a slot on the motherboard of computing device 1210 and exchanges data with processor 1211 via bus 1216. It should be noted that bus 1216 in Figure 12 can also be replaced with a bus using Compute Express Link (CXL), Universal Serial Bus (USB), or other protocols for data transmission.

[0184] Alternatively, the aforementioned acceleration device 1215 may not be directly inserted into a slot on the motherboard of the computing device 1210, but may be located within an acceleration device. This acceleration device could be a separate device from the computing device 1210, such as an acceleration card. In this case, the computing device 1210 can connect to the acceleration device 1215 via a wired network such as a network cable, or via a wireless network such as a Wi-Fi hotspot or Bluetooth. If the acceleration device 1215 is used to perform image processing tasks, such as generating a second image based on first shooting parameters, the acceleration device can be implemented by one or more chips. These chips may include any one of the following: CPU, graphics processing unit (GPU), neural network processing unit (NPU), tensor processing unit (TPU), FPGA, or ASIC. A GPU, also known as a display core, visual processor, or display chip, is a microprocessor specifically designed for image processing in personal computers, workstations, game consoles, and some mobile devices (such as tablets and smartphones). NPUs simulate human neurons and synapses at the circuit level and directly process large-scale neurons and synapses using deep learning instruction sets, with a single instruction completing the processing of a group of neurons. ASICs are suitable for integrated circuit products with a single purpose.

[0185] Optionally, to persistently store data (such as image generation models), the model training system 1200 may also include a data storage system 1213. The data storage system 1213 may be located external to the computing device 1210 (as shown in Figure 12) and exchange data with the computing device 1210 via a network. Alternatively, the data storage system 1213 may also be located internally to the host computer, such as exchanging data with the processor 1211 via a bus 1216. In this case, the data storage system 1213 functions as a hard disk.

[0186] In some possible scenarios, the model training system 1200 may also include a client device 1220. In this case, the user uses the client device 1220 to send data (such as training data) to the computing device 1210 and receives the processing results (such as an image generation model) output by the computing device 1210 based on the data. The client device 1220 is a terminal device, including but not limited to a personal computer, server, mobile phone, tablet computer, or smart car.

[0187] For example, the processor 1211 in FIG12 can be implemented by a chip, as shown in FIG13. FIG13 is a schematic diagram of the structure of a chip provided in this application. For example, the chip 1300 includes a core 1301, a CPU 1302, a system buffer 1303, and a DDR 1306.

[0188] The CPU 1302 is used to accept tasks (such as compression tasks, decompression tasks, image processing tasks, model training tasks, etc.) and call core 1301 to execute the task. When chip 1300 has multiple cores 1301, the CPU 1302 is also used to handle scheduling tasks. For example, the CPU 1302 can be implemented by an ARM processor, which is small in size, low in power consumption, uses a 34-bit reduced instruction set, and has simple and flexible addressing. Of course, in some implementations, the CPU 1302 can also be implemented by other processors.

[0189] Core 1301 provides the computational power required for tasks such as model training. In one optional configuration, core 1301 includes a load / store unit (LSU), a cube computation unit, a scalar computation unit, a vector computation unit, and a buffer. The LSU loads data to be processed and stores processed data. It also manages read / write operations between different buffers within the core and performs format conversions. The cube computation unit provides the core computational power for matrix multiplication. The scalar computation unit is a single-instruction, single-data-stream unit.

[0190] A Single Instruction Single Data (SISD) processor processes only one data item (usually an integer or floating-point number) at a time. A vector computation unit, also known as an array processor, is a processor capable of directly manipulating an array or vector for computation. There may be one or more buffers; for example, this buffer primarily refers to the level 1 cache (L1 buffer). The buffer temporarily stores data that the core 1301 needs to access repeatedly, thus reducing bus reads and writes. Additionally, the implementation of certain data format conversion functions also requires the source data to be located in the buffer. In this embodiment, because the buffer is located in the core, the distance between the cube computation unit in the core and the storage area where the data is located is reduced, decreasing the cube computation unit's access to the DDR 1306, thereby reducing data access latency and core data processing latency.

[0191] System buffer 1303 mainly refers to the level 4 buffer (L1 buffer or L4 cache), which is used to temporarily store input data, intermediate results or final results that have passed through the chip.

[0192] DDR 1306 is an off-chip memory that can be replaced by high bandwidth memory (HBM) or other off-chip memory. Located between the chip and external memory, DDR 1306 overcomes the access speed limitations of shared memory read / write operations in computing resource sharing.

[0193] The input / output (I / O) device 1305 included in chip 1300 refers to the hardware that performs data transmission, or the device that interfaces with the I / O interface. Common I / O devices include network cards, printers, keyboards, and mice. All external storage devices can also be used as I / O devices, such as hard drives, floppy disks, and optical discs.

[0194] Core 1301, CPU 1302, system buffer 1303, I / O devices 1305, and DDR 1306 are connected via a bus. The bus may include a pathway for transferring information between the aforementioned components (such as CPU 1302 and system buffer 1303). In addition to a data bus, the bus may also include a power bus, control bus, and status signal bus. However, for clarity, the bus may be a PCIe bus, or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc. For example, core 1301 can access these I / O devices 1305 via the PCIe bus. Core 1301 is connected to system buffer 1303 via the DDR bus. Here, different system buffers 1303 may use different data buses to communicate with the core 1301. Therefore, the DDR bus can also be replaced with other types of data buses. This application embodiment does not limit the bus type.

[0195] For example, after the CPU 1302 loads the data to be processed by the image processing task (such as training data) into the DDR 1306, the LSU in the core 1301 reads (loads) the data from the DDR 1306, processes the data, and obtains the processing result (such as an image generation model). After obtaining the processing result, the LSU then loads (stores) the processing result into the DDR 1306, and the network interface card sends the processing result to the client device or to the data storage system 1213 for persistent storage.

[0196] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the computing device. In other embodiments, the computing device and chip may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0197] This application also provides a cluster. The cluster includes at least one computing device, which may be a server. In some embodiments, the computing device may also be a desktop computer, laptop computer, or smartphone, or other terminal device. The cluster can be used to execute the image processing method described in the above method embodiments, and can also be used to execute the model training method described in the above method embodiments. The following description uses the example of a cluster executing a model training method to illustrate the cluster.

[0198] As shown in Figure 14, which is a schematic diagram of a cluster structure provided in this application, the cluster includes at least one computing device 1210. The memory 1212 of one or more computing devices 1210 in the cluster may store the same instructions for executing model training methods.

[0199] In some possible implementations, the memory 1212 of one or more computing devices 1210 in the cluster may also store partial instructions for executing model training methods. In other words, a combination of one or more computing devices 1210 can jointly execute instructions for executing model training methods.

[0200] It should be noted that the memory 1212 in different computing devices 1210 in the cluster can store different instructions, which are used to execute part of the functions of the computing devices. That is, the instructions stored in the memory 1212 in different computing devices 1210 can realize the functions of one or more units in the transceiver module 1110 and the processing module 1120.

[0201] In some possible implementations, one or more computing devices in the cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN). Figure 15 illustrates one possible implementation. As shown in Figure 15, this application provides a schematic diagram of the connection between computing devices, where two computing devices 1210A and 1210B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device. In this type of possible implementation, the instructions stored in the memory 1212 of computing device 1210A can implement the functions implemented by the transceiver module 1110. Simultaneously, the instructions stored in the memory 1212 of computing device 1210B can implement the functions implemented by the processing module 1120.

[0202] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform an image processing method, or causes the at least one computing device to perform a model training method.

[0203] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform an image processing method, or instructions that instruct the computing device to perform a model training method.

[0204] This application also provides a chip. The chip includes an interface circuit and a control circuit. The interface circuit is used to acquire first imaging parameters, and the control circuit is used to implement the functions of a computing device in an image processing method; alternatively, the interface circuit is used to acquire training data, and the control circuit is used to implement the functions of a computing device in a model training method.

[0205] The method steps in this embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a computing device. Of course, the processor and storage medium can also exist as discrete components in a network device or terminal device.

[0206] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of this application are performed entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable device. The computer program or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape. It can also be an optical medium, such as a digital video disc (DVD). It can also be a semiconductor medium, such as a solid-state drive (SSD).

[0207] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A model training method based on a visual reconstruction technique, characterized in that, The method comprises: obtaining training data; the training data comprising multiple images of different perspectives corresponding to a first scene, and shooting parameters corresponding to each image; obtaining a residual of a first image and a second image; wherein the first image is one of the multiple images, the first image comprises first content and first interference information, the second image is an image generated by an initial model according to a first shooting parameter, the first shooting parameter is a shooting parameter corresponding to the first image, the second image comprises the first content and second interference information, and the second interference information is interference information corresponding to the first interference information generated by the initial model according to the first shooting parameter; training the initial model using the residual to obtain an image generation model; wherein the image generation model is used to generate a third image according to the first shooting parameter, and the third image comprises the first content.

2. The method of claim 1, wherein the training of the initial model using the residual to obtain the image generation model comprises: obtaining distribution information of the first content and the first interference information according to the residual; training the initial model using the residual and the distribution information to obtain the image generation model.

3. The method of claim 2, wherein the training of the initial model using the residual and the distribution information to obtain the image generation model comprises: obtaining a mask of the first image using the residual and the distribution information; processing the first image using the mask to obtain a processed first image; the processed first image comprises the first content; training the initial model using the processed first image to obtain the image generation model.

4. The method of claim 3, wherein the obtaining of the mask of the first image using the residual and the distribution information comprises: obtaining a threshold value for distinguishing the first content and the first interference information according to the distribution information using a Gaussian Mixture Model (GMM); obtaining the mask of the first image according to the residual and the threshold value.

5. The method of claim 4, wherein the obtaining of the threshold value for distinguishing the first content and the first interference information according to the distribution information using the Gaussian Mixture Model (GMM) comprises: obtaining a first mean value of a Gaussian distribution of a residual corresponding to the first content and a second mean value of a Gaussian distribution of a residual corresponding to the first interference information according to the distribution information using the Gaussian Mixture Model (GMM); obtaining a trained number of rounds of the initial model and a total number of expected training rounds; obtaining the threshold value for distinguishing the first content and the first interference information according to the first mean value, the second mean value, the trained number of rounds, and the total number of expected training rounds.

6. The method of claim 4 or 5, wherein the obtaining of the mask of the first image according to the residual and the threshold value comprises: obtain a first mask according to the pixel residual included in the residual and the threshold; the pixel residual is calculated according to pixel values of single pixel points in the first image and the second image; obtain a second mask according to the tile residual included in the residual and the threshold; the tile residual is calculated according to pixel values of multiple pixel points in the first image and the second image; obtain a mask of the first image according to a mean value of the first mask and the second mask.

7. The method of any one of claims 1-6, wherein the first interference information comprises one or more of the following: second content, illumination, color; and / or the shooting parameter comprises one or more of the following: shooting angle, shooting position, internal parameter of the shooting device. The method further comprises: establishing a three-dimensional model of the first content according to the third image.

8. The method according to any one of claims 1-7, characterized in that, The apparatus comprises: a transceiver module, configured to: obtain training data; the training data comprises multiple images of different perspectives corresponding to a first scene, and shooting parameters corresponding to each image; 9.A model training apparatus based on a visual reconstruction technique, characterized by comprising: the transceiver module is further configured to: obtain a residual of a first image and a second image; wherein the first image is one of the multiple images, the first image comprises first content and first interference information, the second image is generated by an initial model according to a first shooting parameter, the first shooting parameter is a shooting parameter corresponding to the first image, the second image comprises the first content and second interference information, the second interference information is interference information corresponding to the first interference information generated by the initial model according to the first shooting parameter; a processing module, configured to: train the initial model by using the residual to obtain an image generation model; wherein the image generation model is used to generate a third image according to the first shooting parameter, and the third image comprises first content.

10. The apparatus of claim 9, wherein the processing module is specifically configured to: obtain distribution information of the first content and the first interference information according to the residual; the processing module is further specifically configured to: train the initial model by using the residual and the distribution information to obtain the image generation model.

11. The apparatus of claim 10, wherein the processing module is specifically configured to: obtain a mask of the first image by using the residual and the distribution information; the processing module is further specifically configured to: process the first image by using the mask to obtain a processed first image; the processed first image comprises the first content; the processing module is further specifically configured to: train the initial model by using the processed first image to obtain the image generation model.

12. The apparatus of claim 11, wherein the processing module is specifically configured to: obtain a threshold for distinguishing the first content and the first interference information according to the distribution information by using a Gaussian Mixture Model (GMM). ​ ​ ​ ​ ​ The processing module is further configured to obtain a mask of the first image according to the residual error and the threshold.

13. The apparatus of claim 12, wherein, The processing module is configured to obtain a first mean of a Gaussian distribution of the residual error corresponding to the first content and a second mean of a Gaussian distribution of the residual error corresponding to the first interference information according to the distribution information by using a Gaussian Mixture Model (GMM). The processing module is further configured to obtain a trained round number of the initial model and a total round number of expected training. The processing module is further configured to obtain a threshold value for distinguishing the first content and the first interference information according to the first mean, the second mean, the trained round number, and the total round number of expected training.

14. The apparatus of claim 12 or 13, wherein, The processing module is configured to obtain a first mask according to pixel residual errors included in the residual error and the threshold value, the pixel residual errors being calculated according to pixel values of single pixel points in the first image and the second image. The processing module is further configured to obtain a second mask according to tile residual errors included in the residual error and the threshold value, the tile residual errors being calculated according to pixel values of multiple pixel points in the first image and the second image. The processing module is further configured to obtain a mask of the first image according to a mean of the first mask and the second mask.

15. The apparatus of any one of claims 9-14, wherein, The first interference information includes one or more of the following: a second content, illumination, and color. And / or, the shooting parameter includes one or more of the following: a shooting angle, a shooting position, and an internal parameter of the shooting device.

16. The apparatus of any one of claims 9-15, wherein, The processing module is further configured to establish a three-dimensional model of the first content according to the third image.

17. A chip, characterized by The chip includes an interface circuit and a control circuit; the interface circuit is configured to obtain training data, and the control circuit is configured to execute the method in any one of claims 1-8 in cooperation with the interface circuit.

18. A computing device, comprising: The computing device includes a memory and a processor, and the memory stores program code; when the processor executes the program code, the processor is configured to execute the method in any one of claims 1-8.

19. A cluster, characterized by The cluster includes at least one computing device, and each computing device includes a processor and a memory. The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the cluster executes the method in any one of claims 1-8.

20. A computer-readable storage medium, characterized in that, The computer readable storage medium includes computer instructions; when the computer instructions are executed in a computing device, the computing device executes the method in any one of claims 1-8.

21. A computer program product, characterised in that, When the computer program product is executed in a computing device, the computing device executes the method in any one of claims 1-8.

Images