Model training method, image generation method and device

Abstract

The invention discloses a model training method and device and an image generation method and device. Belongs to the technical field of artificial intelligence. The method comprises the steps that multiple frames of first images are input into a first model, the multiple frames of first images are corrected through the first model, multiple frames of second images corresponding to the multiple frames of first images are obtained, and the multiple frames of first images comprise images obtained by shooting the same object at different visual angles in the same scene; inputting camera pose information and multiple frames of second images when multiple frames of first images are shot into a second model, and modeling and rendering the scene through the second model to obtain multiple frames of third images corresponding to the multiple frames of second images; and training the first model and the second model based on the multiple frames of first images and the multiple frames of third images.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and specifically to a model training method, an image generation method, and a device thereof. Background Art

[0002] 3D (3D) representation is a technology for modeling and rendering scenes. In the fields of robotics, augmented reality, and virtual world generation, 3D representation can provide a highly realistic and interactive three-dimensional virtual world, enhancing user experience and work efficiency. Usually, 3D representation can be achieved through 3D representation models such as NeRF (Neural Radiance Fields) and 3DGS (3D Gaussian Splats). During the training process, the 3D representation model needs to use images from different perspectives to learn the three-dimensional representation of the scene.

[0003] In the prior art, in the process of training a 3D representation model, the images input to the 3D representation model are required to have multi-view consistency, that is, the features of the same object in the scene in images taken from different viewpoints have no obvious differences. However, if some input images contain interfering shadows such as the photographer's figure, it will lead to a lack of multi-view consistency between the input images, resulting in distorted rendering results of the 3D representation model. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a model training method, an image generation method and a device thereof, which can improve the authenticity of the rendering results of the 3D representation model.

[0005] In a first aspect, an embodiment of the present application provides a model training method, which includes: inputting multiple frames of first images into a first model, correcting the multiple frames of first images through the first model to obtain multiple frames of second images corresponding to the multiple frames of first images, the multiple frames of first images include images of the same object captured from different perspectives in the same scene, and the multiple frames of first images include images containing interfering shadows and images without interfering shadows; inputting camera posture information when shooting the multiple frames of first images and the multiple frames of second images into a second model, rendering through the second model to obtain multiple frames of third images corresponding to the multiple frames of second images; training the first model and the second model based on the multiple frames of first images and the multiple frames of third images.

[0006] In a second aspect, an embodiment of the present application provides a model training method, the method comprising: obtaining at least one camera pose information; inputting the at least one camera pose information into a second model trained using the model training method described in the first aspect, performing image rendering through the second model, and obtaining at least one frame of image corresponding to the at least one camera pose information, wherein each frame of the at least one frame of image does not contain interfering shadows.

[0007] In the third aspect, an embodiment of the present application provides a model training device, which includes: a correction unit, used to input multiple frames of first images into a first model, and correct the multiple frames of first images through the first model to obtain multiple frames of second images corresponding to the multiple frames of first images, wherein the multiple frames of first images include images of the same object captured from different perspectives in the same scene, and the multiple frames of first images include images with interfering shadows and images without interfering shadows; a rendering unit, used to input camera posture information when capturing the multiple frames of first images and the multiple frames of second images into a second model, and model the scene through the second model and render to obtain multiple frames of third images corresponding to the multiple frames of second images; a training unit, used to train the first model and the second model based on the multiple frames of first images and the multiple frames of third images.

[0008] In a fourth aspect, an embodiment of the present application provides a model training device, which includes: an acquisition unit for acquiring at least one camera pose information; a generation unit for inputting the at least one camera pose information into a second model trained by the model training device described in the first aspect, performing image rendering through the second model, and obtaining at least one frame of image corresponding to the at least one camera pose information, wherein each frame of the at least one frame of image does not contain interfering shadows.

[0009] In a fifth aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0010] In a sixth aspect, an embodiment of the present application provides a readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first aspect above are implemented.

[0011] In a seventh aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0012] In an eighth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0013] In the embodiment of the present application, firstly, multiple frames of first images are input into the first model, and the multiple frames of first images are corrected by the first model to obtain multiple frames of second images corresponding to the multiple frames of first images, wherein the multiple frames of first images are images of the same object captured from different perspectives in the same scene, and the multiple frames of first images include images containing interference shadows and images without interference shadows; then, the camera posture information when the multiple frames of first images are captured and the multiple frames of second images are input into the second model, and the scene is modeled and rendered by the second model to obtain multiple frames of third images corresponding to the multiple frames of second images; finally, the first model and the second model can be trained based on the multiple frames of first images and the multiple frames of third images, thereby completing a training process. Since the first model is introduced, the multiple frames of first images can be corrected in each training process and then input into the second model for modeling and rendering. By iterating the training multiple times in this way, the parameters of the first model and the second model can be gradually optimized, so that the final first model can remove the interfering shadows in the first image and retain other content, and finally input the image without interfering shadows to the second model to ensure the multi-perspective consistency of the input image, thereby avoiding the distortion of the rendering result of the second model due to interfering shadows, and improving the realism of the second model's scene modeling and rendering. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a flow chart of the model training method provided in an embodiment of the present application;

[0015] Figure 2 It is a schematic diagram of an application scenario of the model training method provided in an embodiment of the present application;

[0016] Figure 3 It is a schematic diagram of the processing process of the model training method provided in the embodiment of the present application;

[0017] Figure 4 It is a schematic diagram of the processing process of the first model in the model training method provided in the embodiment of the present application;

[0018] Figure 5 It is a schematic diagram of the shooting process of multiple frames of first images in the model training method provided in an embodiment of the present application;

[0019] Figure 6 It is a schematic diagram of the parameter optimization process when the first model is not introduced in the model training method provided in the embodiment of the present application;

[0020] Figure 7It is a schematic diagram of the parameter optimization process after the first model is introduced in the model training method provided in the embodiment of the present application;

[0021] Figure 8 is a flow chart of an image generation method provided in an embodiment of the present application;

[0022] Fig. 9 It is a structural schematic diagram of a model training device provided in an embodiment of the present application;

[0023] Fig.10 is a schematic diagram of the structure of an image generating device provided in an embodiment of the present application;

[0024] Fig.11 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0025] Fig.12 It is a schematic diagram of the hardware structure of an electronic device suitable for implementing the embodiments of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0027] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0028] The model training method and device provided in the embodiments of the present application are described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0029] Please refer to Figure 1 , which shows a flow chart of the model training method provided in the embodiment of the present application. The model training method provided in the embodiment of the present application can be applied to electronic devices. In practice, the above electronic devices can be electronic devices such as smart phones, tablet computers, laptops, and wearable devices.

[0030] The process of the model training method provided in the embodiment of the present application includes the following steps:

[0031] Step 101: input a plurality of frames of first images into a first model, and correct the plurality of frames of first images through the first model to obtain a plurality of frames of second images corresponding to the plurality of frames of first images. The plurality of frames of first images include images obtained by photographing the same object at different viewing angles in the same scene, and the plurality of frames of first images include images containing interfering shadows and images without interfering shadows.

[0032] In this embodiment, the multiple frames of first images are images of the same object captured from different perspectives in the same scene. The scene may be a scene captured under any light source, for example, a scene captured under an indoor light source, a scene captured under an outdoor light source, etc. The object may be any object in the scene. During shooting, the first images of the object captured from different perspectives in the same scene may be obtained by rotating the camera angle.

[0033] In this embodiment, the plurality of first frames include images with interference shadows and images without interference shadows. Interference shadows are shadows introduced by a photographer or a shooting device. For example, the shadow of a photographer, the shadow of a camera, etc. As an example, see Figure 2 In a shooting scene with sunlight as the light source, the photographer holds a camera and shoots the shooting object 201 at multiple positions. When the photographer shoots the shooting object 201 at positions T0, T1, and T2, the photographer's figure is also included in the captured images, as shown by reference numerals 202, 203, and 204.

[0034] In this embodiment, the first model may be a correction model for correcting the pixel brightness of an image. The first model may adopt a model structure such as a convolutional neural network, a multilayer perceptron (MLP, Multilayer Perceptron), etc. Figure 3 , multiple frames of first images can be input into the first model, and multiple frames of second images can be obtained by correcting the multiple frames of first images through the first model. The multiple frames of second images correspond one to one with the multiple frames of first images.

[0035] In practical applications, the first model can be constructed through the following steps:

[0036] First, based on the rendering method and its discrete form, the following formula 1 is constructed:

[0037] L=L e +TL e +T 2 L e +…+T N L e (1)

[0038] The rendering equation is a basic equation in computer graphics, which is used to simulate the interaction between light and the surface of objects in the scene. e is the radiance of the light source in the scene; TL e Refers to the radiance of the light source after one refraction in the scene; T 2 L e is the radiance of the light source after two refractions in the scene; T N L e It is the radiance of the light source after n refractions in the scene; N is a positive integer; T is the light transmission operator; L is the amount of light received by the camera plane.

[0039] Since light is lost during transmission, the size of the modulus of T (which can be recorded as Norm(T N )) is less than 1. As the number of refractions increases, the contribution of the refracted light to the rendering result decreases by orders of magnitude. The light source that is refracted once in the scene is regarded as direct light, and the radiance of the direct light is recorded as L direct , then L direct =TL e The light source that is refracted at least twice in the scene is regarded as indirect light, and the radiance of the indirect light is recorded as L indirect , then L indirect =T 2 L e +…+T N L e Direct light has the greatest intensity and has the greatest impact on the formation of shadows.

[0040] Assume that there is no light source captured in the image, that is, L e =0, then the following imaging equation is obtained: I = ρ × L. Assuming that the imaging time of each frame is the same, which is the unit time, its imaging is I, that is, the outgoing radiation rate; ρ is the material of the object, reflecting the characteristics of the object's ability to reflect light in each band; L is the incident radiation rate.

[0041] Assume that shadows are mainly caused by the blocking of direct light, that is, I cap =ρ×(αL direct +L indirect ), and I free =ρ×(L direct +L indirect ). Among them, I cap I represents the original captured image, which may contain interfering shadows; free Indicates that the first model is used to capCorrected image; α∈[0,1], represents the amount of direct light passing through, i.e., the shadow mask. When α=1, it means that the direct light is completely blocked; when α∈(0,1), it means that the direct light is partially blocked; when α=0, no direct light source is blocked, i.e., there is no shadow at that position.

[0042] The ratio of the direct light radiance to the indirect light radiance is taken as an intermediate parameter, denoted as t, that is, t = (L direct / L indirect ), t∈(0,+∞), we can get the following formula 2:

[0043]

[0044] Assuming C = (1 + t) / (1 + tα), C ∈ [1, + ∞], it can be seen that for the same coordinate point, the brightness value of the point in the original image and the brightness value of the point in the corrected image are in linear proportion. The reciprocal of C is C -1 , then we have the following formula three:

[0045] I cap =C -1 I free (3)

[0046] In the image domain, the correspondence between the original captured image and the corrected image is established. cap After the linear transformation of C, it can be corrected to I free The object material is eliminated as an intermediate variable, and due to I cap and I free With the same camera parameters, there is no light angle affecting the rendering equation. C is the correction parameter matrix connecting the original captured image and the corrected image. It can be seen that under the above assumptions, the interfering shadows in the original captured image can be corrected by C. The above formula 2 or formula 3 can be used as the expression of the first model.

[0047] Furthermore, C -1 parameterized to the range (0,1). As an example, Among them, Sigmoid() is an S-type function, which is a function used to map the value of a variable in the range of (0,1); x is the correction parameter matrix of the first model; (i, j) is the coordinate of the correction parameter in the correction parameter matrix, and i and j are both positive integers; x i,j The initial value of can be 0.5, because the Sigmoid function has a higher gradient transfer at this value.

[0048] Alternatively, α and t can be parameterized. As an example, α can be i,j =Sigmoid(x i,j ), and make ti,j =Abs(x i,j ). Where Abs() is the absolute value function.

[0049] By parameterization, C -1 It is in the range of (0,1) to ensure that the brightness value of the interfering shadow area in the original captured image can be improved after being corrected by the first model.

[0050] In some optional implementations of this embodiment, for each first image in the multiple first image frames, the following steps may be performed by the first model to obtain a second image corresponding to the first image:

[0051] The first step is to obtain the first pixel matrix of the first image of the frame and the correction parameter matrix corresponding to the first image of the frame. Among them, multiple frames of first images can be expressed as W1×H1×n, W represents the width of the first image, H represents the height of the first image, and n represents the number of channels. C of the first model is a three-dimensional correction parameter matrix, which can be expressed as W2×H2×n, that is, a two-dimensional correction parameter matrix with n channels. The first image and the two-dimensional correction parameter matrix have a one-to-one correspondence.

[0052] In the second step, based on the correction parameter matrix, the first pixel matrix is ​​corrected to obtain a second pixel matrix of the first image of the frame.

[0053] In some examples, W1=W2 and H1=H2. At this time, the first pixel matrix of the first image of the frame may be multiplied with the elements at the same position of the correction parameter matrix corresponding to the first image of the frame to obtain the second pixel matrix of the first image of the frame.

[0054] In other examples, W1>W2 and H1>H2. For example, W1×H1=512×512, W1×H1=64×64, then each parameter in the correction parameter matrix corresponds to an 8×8 image block in the first image of the frame, that is, the pixels in an 8×8 image block have the same correction parameter. At this time, the pixel matrix corresponding to each 8×8 image block in the first image of the frame can be multiplied by the corresponding correction parameter in the correction parameter matrix to obtain the second pixel matrix of the first image of the frame.

[0055] The third step is to determine the image corresponding to the second pixel matrix as the second image corresponding to the first image of the frame.

[0056] By using the correction parameter matrix corresponding to each frame of the first image to correct the pixel matrix of the corresponding first image, the multi-view images input to the second model in each training process can be corrected images. Since the first model is jointly trained with the second model, and the second model can tend to perform the same representation of the same object under multiple perspectives, multiple iterations of training can gradually bring the interfering shadow area in the image containing interfering shadows closer to the representation of the same area in the image without interfering shadows, so that the final first model can effectively remove the interfering shadows in the first image containing interfering shadows, and the 3D model can model and render the scene more realistically.

[0057] Step 102, inputting the camera posture information when taking the multiple frames of first images and the multiple frames of second images into the second model, modeling the scene through the second model and rendering to obtain multiple frames of third images corresponding to the multiple frames of second images.

[0058] In this embodiment, the camera position information when each frame of the first image is captured can be obtained through an open source computer vision tool. The camera position information may include but is not limited to the position coordinates and posture information of the camera in the scene.

[0059] In this embodiment, the second model may be a 3D representation model. The 3D representation model is a data model of an algorithm based on multi-view geometry estimation and scene reconstruction, which can be recorded as F(V, S). Among them, V refers to the viewing angle, and S refers to the coordinates in the scene. In practice, the 3D representation model may be a model such as NeRF (Neural Radiance Fields), 3DGS (3D GaussianSplats), etc. Since the shooting viewing angles of each frame of the first image are different, the camera pose information when shooting each frame of the first image is different. Since the above-mentioned multiple frames of the first image have a one-to-one correspondence with the above-mentioned multiple frames of the second image, and the above-mentioned multiple frames of the first image have a one-to-one correspondence with the camera pose information, the camera pose information also has a one-to-one correspondence with the above-mentioned multiple frames of the second image.

[0060] After the camera pose information when taking each frame of the first image and each frame of the second image are input into the second model, the second model can learn the 3D representation of the scene based on the second image corresponding to each camera pose information, that is, the second image from different perspectives, so as to model the scene. After the scene modeling is performed, the scene under the perspective corresponding to the above camera pose information can be rendered, so as to output the third image under the perspective. As an example, see Figure 3 After the camera posture information when taking multiple frames of first images and multiple frames of second images output by the first model are input into the second model, the second model can output multiple frames of third images.

[0061] In some optional implementations of this embodiment, after the scene is modeled by the second model and rendered to obtain multiple frames of third images corresponding to the multiple frames of second images, for each frame of the multiple frames of third images, the second image corresponding to the frame of the third image may be used as the target second image, and the following steps may be performed:

[0062] In the first step, if the difference in brightness between the third image of the frame and the second image of the target is greater than the threshold, the correction parameter matrix corresponding to the second image of the target (which can be recorded as C i ), the correction parameter matrix corresponding to the previous frame image of the target second image in the multiple frames of the second image (which can be recorded as C i-1 ) and the correction parameter matrix corresponding to the next frame of the target second image in the multiple frames of the second image (which can be recorded as C i+1 ), determine the brightness compression coefficient (which can be recorded as G). Specifically, refer to the following formula 4:

[0063] G=λMedian(C i )+(1-λ)(Median(C i-1 )+Median(C i+1 )) (4)

[0064] Among them, Median(C i ) is C i Median (C i-1 ) is C i-1 Median (C i+1 ) is C i+1 The median or mean of ; λ is the weight, and its value can be set in advance.

[0065] The second step is to perform brightness compression processing on the frame of the third image based on the brightness compression coefficient G to update the frame of the third image. Here, the brightness compression coefficient G can be multiplied by the pixel matrix of the frame of the third image to update the frame of the third image.

[0066] By performing brightness compression on a third image of a certain frame when the difference in brightness value between a third image of the frame and its corresponding second image is greater than a threshold, it is possible to avoid a significant increase in brightness of the third image output by the second model, thereby making the rendering effect of the second model closer to the original scene and improving the realism of the rendering.

[0067] Step 103: training the first model and the second model based on the multiple frames of the first image and the multiple frames of the third image.

[0068] In this embodiment, the value of the loss function, i.e., the loss value, can be first calculated based on the above-mentioned multiple frames of the first image and the above-mentioned multiple frames of the third image. The loss function is a non-negative real-valued function that can be used to characterize the difference between the detection result and the true result. In general, the smaller the loss value, the better the robustness of the model. The loss function can be set according to actual needs. Then, based on the loss value, the parameters of the first model and the second model are updated to complete a joint training process of the first model and the second model.

[0069] As an example, the loss function Loss can be seen in the following formula 5:

[0070] Loss=L(I cap ,C -1 I render ) (5)

[0071] Among them, I cap represents the input data, i.e., the first image of the above-mentioned multiple frames; C -1 I render represents the rendered image output by the second model after being corrected by the first model, that is, the above-mentioned multiple frames of the third image. The above-mentioned loss function can adopt a function such as the minimum absolute deviation, the minimum square deviation function, etc., which is not specifically limited here.

[0072] It should be noted that the above steps 101 to 103 can be iteratively executed multiple times to complete the training of the first model and the second model. The training process is iteratively executed multiple times, and the parameters of the first model and the second model will be gradually optimized. Since the first model is jointly trained with the second model, and the second model can tend to perform the same representation of the same object under multiple perspectives, the interfering shadow area in the image containing interfering shadows will be gradually brought closer to the representation of the same area in the image without interfering shadows, so as to gradually achieve the effect of removing interfering shadows. The final first model can achieve the effect of removing interfering shadows in the input image. See Figure 4 , the image containing interference shadows in the multiple frames of the first image is shown as 401, and the correction parameter matrix corresponding to the image can be represented by an image mask, as shown in 402. Through the trained first model, the pixel matrix of the image containing interference shadows can be multiplied element by element with the correction parameter matrix corresponding to the image to update the pixel matrix of the image containing interference shadows, thereby correcting the image containing interference shadows. The first model can output the corrected image, that is, the second image, as shown in 403. In the second image, the interference shadows are removed.

[0073] It should be noted that for non-interfering shadows in multiple frames of the first image, for example, the shadows of still objects in the scene, since they have the same representation under multiple perspectives, the non-interfering shadows will not be corrected and will be directly modeled and rendered by the second model. Therefore, the final first model can achieve the effect of only removing the interfering shadows in the input image while retaining other content, so that the image that only removes the interfering shadows and retains other content can be input to the second model in the end, so as to ensure the authenticity of the input image while ensuring the multi-perspective consistency of the input image, making the modeling and rendering results of the 3D model more realistic.

[0074] The following is an explanation using a specific scenario:

[0075] See also Figure 5 , the photographer shoots the subject S at different viewing angles V0, V1, V2, V3, and V4 in an outdoor shooting scene, and obtains 5 frames of first images. Among them, the first image shot at viewing angle V2 includes the photographer's figure, that is, the interfering shadow. The first images shot at viewing angles V0, V1, V3, and V4 do not include interfering shadows. Images containing interfering shadows will produce obvious intensity loss, and the training goal of the second model is to correct the intensity of the image to an intensity similar to that of an image without interfering shadows.

[0076] The second model tends to maintain geometric consistency between multiple views, that is, to reuse similar representations for the same position locally, so that the parameters of the second model are compact. For example, NeRF uses only one layer of MLP, which is intended to limit the large changes in multi-view representations; 3DGS uses a third-order harmonic function, which is easy to learn low-frequency signals and high-frequency signals are easy to fall into local minima. Based on the loss function, the rendered image and the original captured image are required to be overfitted.

[0077] When the first model is not introduced, that is, when the second model is trained alone, see Figure 6 During the iterative training process, the second model first learns the low-frequency signal, and the curve of F(V,S) is relatively smooth between multiple perspectives. Then it gradually overfits the high-frequency signal, and F(V,S) gradually has high-frequency characteristics. When observing point S from different perspectives, the gradient optimization causes the representation function F(V,S) to fluctuate up and down, causing the observed data to compete when expressed in the 3D representation function. When the difference in perspective is relatively small, the interference shadow will cause the overfitting F(V,S) to be too steep locally, and the representation function of the second model is too complex to be fitted.

[0078] After the first model is introduced, the training process is iterated, and the parameters of the first model and the second model are optimized at the same time. Figure 7, the image containing interfering shadows will be gradually optimized by the correction parameters in the first model, and the light intensity value will be increased to be consistent with other adjacent images from other perspectives. Since the correction layer parameter C≥1, only the area below the local brightness can be increased in brightness, and the high light intensity area will not be compressed into a low light brightness area. Therefore, the brightness of the interfering shadow area can be increased without reducing the brightness of the non-shadow area.

[0079] In some optional implementations of this embodiment, the multiple frames of first images include m frames of images containing interference shadows and n frames of images without interference shadows, where m and n are both positive integers, and m<n. Thus, when the same interference shadow exists in multiple first images of adjacent perspectives, since there are more images without interference shadows than images containing interference shadows in the multiple frames of first images, it is still possible to ensure that the images containing interference shadows are gradually optimized to a light intensity value consistent with adjacent images of other perspectives through the correction parameters in the first model, thereby eliminating the interference shadow.

[0080] In some optional implementations of this embodiment, the training of the first model and the second model based on the multiple frames of the first image and the multiple frames of the third image may include the following steps:

[0081] In the first step, based on the first loss function, the multiple frames of the first image and the multiple frames of the third image, a first loss value is determined. The first loss value can be calculated using the above formula 5, which will not be described in detail here.

[0082] The second step is to determine the second loss value based on the second loss function, the parameter value matrix of the first model and the target parameter value matrix. Here, the second loss function can be: Loss = λ c (C-1) 2 Among them, λ c is a hyperparameter of the first model, for example, it can be a learning rate.

[0083] The third step is to determine a target loss value based on the first loss value and the second loss value. Here, the target loss value may be determined as the sum of the first loss value and the second loss value, or the first loss value and the second loss value may be weighted and summed to obtain the target loss value.

[0084] The fourth step is to update the parameters of the first model and the second model based on the target loss value. Thus, the joint training process of the first model and the second model can be completed.

[0085] It can be understood that if there is no interfering shadow in a certain area of ​​the input first image, it is necessary to make C -1 The value of this area in is as close to 1 as possible, so that the second model can learn accurate 3D representation. The introduction of the second loss function enables the 3D representation structure to represent more scene content without causing excessive brightness.

[0086] In some optional implementations of this embodiment, the parameter learning rate of the first model can be lower than the parameter learning rate of the second model. Thus, the second model can be quickly fitted in the non-interference shadow area, and the interference shadow can be gradually recognized by the first model, so as to avoid introducing the second loss function to affect the elimination effect of the interference shadow.

[0087] The method provided by the above-mentioned embodiment of the present application first inputs multiple frames of first images into the first model, and corrects the multiple frames of first images through the first model to obtain multiple frames of second images corresponding to the multiple frames of first images, wherein the multiple frames of first images are images of the same object captured from different perspectives in the same scene, and the multiple frames of first images include images containing interfering shadows and images without interfering shadows; then the camera posture information when capturing the multiple frames of first images and the multiple frames of second images are input into the second model, and the scene is modeled and rendered through the second model to obtain multiple frames of third images corresponding to the multiple frames of second images; finally, the first model and the second model can be trained based on the multiple frames of first images and the multiple frames of third images, thereby completing a training process. Since the first model is introduced, the multiple frames of first images can be corrected in each training process and then input into the second model for modeling and rendering. By iterating the training multiple times in this way, the parameters of the first model and the second model can be gradually optimized, so that the final first model can remove the interfering shadows in the first image and retain other content, and finally can input the image without interfering shadows to the second model to ensure the multi-perspective consistency of the input image, thereby avoiding the distortion of the rendering result of the second model due to interfering shadows, and improving the realism of the second model's scene modeling and rendering.

[0088] Further references Figure 8 , which shows a flow chart of the image generation method provided in the embodiment of the present application. The image generation method provided in the embodiment of the present application can be applied to electronic devices. In practice, the above electronic devices can be electronic devices such as smart phones, tablet computers, laptops, and wearable devices.

[0089] The process of the image generation method provided in the embodiment of the present application includes the following steps:

[0090] Step 801, obtaining at least one camera position information.

[0091] In this embodiment, the at least one camera pose information may be any camera pose information. The at least one camera pose information may include Figure 1 The camera posture information in the corresponding embodiment can also be different from Figure 1 The new camera pose information corresponding to the camera pose information in the embodiment is not specifically limited here.

[0092] Step 802: input at least one camera pose information into the trained second model, perform image rendering through the second model, and obtain at least one frame of image corresponding to the at least one camera pose information, wherein each frame of the at least one frame of image does not contain interfering shadows.

[0093] In this embodiment, the second model can be trained by the model training method of the above embodiment. The process of the second model performing image rendering can refer to the relevant description in the above embodiment. To avoid repetition, it will not be repeated here.

[0094] It should be noted that the image generation method of this embodiment can be used to test the second model trained in the above embodiment, and then the second model can be continuously optimized according to the test results. The image generation method of this embodiment can also be a practical application method of the second model trained in the above embodiment. Using the second model trained in the above embodiment, image rendering can be performed based on the camera pose information to obtain an image of the perspective corresponding to the camera pose information. Since the above-mentioned second model performs joint parameter optimization with the first model during the training process, the input image finally obtained by the above-mentioned second model is an image with interfering shadows removed by the first model, which ensures the multi-perspective consistency of the input image, so the modeling results and image rendering effects of the second model on the scene are more realistic.

[0095] It should be noted that the model training method provided in the embodiment of the present application can be executed by a model training device. In the embodiment of the present application, the model training device executing the model training method is taken as an example to illustrate the model training device provided in the embodiment of the present application.

[0096] like Fig. 9 As shown, the model training device 900 of this embodiment includes: a correction unit 901, used to input multiple frames of first images into a first model, and correct the multiple frames of first images through the first model to obtain multiple frames of second images corresponding to the multiple frames of first images, wherein the multiple frames of first images are images of the same object captured from different perspectives in the same scene; a rendering unit 902, used to input camera posture information when capturing the multiple frames of first images and the multiple frames of second images into a second model, and model the scene through the second model and render to obtain multiple frames of third images corresponding to the multiple frames of second images; a training unit 903, used to train the first model and the second model based on the multiple frames of first images and the multiple frames of third images.

[0097] In some optional implementations of this embodiment, the correction unit 901 is further used to: for each first image in the multiple first images, perform the following steps through the first model: obtain a first pixel matrix of the first image and a correction parameter matrix corresponding to the first image; based on the correction parameter matrix, correct the first pixel matrix to obtain a second pixel matrix of the first image; determine the image corresponding to the second pixel matrix as the second image corresponding to the first image. By correcting the pixel matrix of the corresponding first image using the correction parameter matrix corresponding to each first image, the multi-view images input to the second model in each training process can be corrected images. Since the first model is jointly trained with the second model, and the second model can tend to perform the same representation of the same object under multiple perspectives, iterative training for multiple times can make the interfering shadow area in the image containing interfering shadows gradually close to the representation of the same area of ​​the image without interfering shadows, so that the final first model can effectively remove the interfering shadows in the first image containing interfering shadows, and make the 3D model model and render the scene more realistically.

[0098] In some optional implementations of this embodiment, the device further includes an updating unit, which is further used to: for each third image in the multiple third images, take the second image corresponding to the third image as the target second image, and perform the following steps: if the difference in brightness between the third image and the target second image is greater than a threshold, then determine a brightness compression coefficient based on a correction parameter matrix corresponding to the target second image, a correction parameter matrix corresponding to a previous frame of the target second image in the multiple second images, and a correction parameter matrix corresponding to a subsequent frame of the target second image in the multiple second images; and perform brightness compression processing on the third image based on the brightness compression coefficient to update the third image. By performing brightness compression on a third image when the difference in brightness between a third image and its corresponding second image is greater than a threshold, it is possible to avoid a significant increase in brightness of the third image output by the second model, so that the rendering effect of the second model is closer to the original scene, thereby improving the realism of the rendering.

[0099] In some optional implementations of this embodiment, the training unit 903 is further used to: determine a first loss value based on a first loss function, the multiple frames of first images, and the multiple frames of third images; determine a second loss value based on a second loss function, a parameter value matrix of the first model, and a target parameter value matrix; determine a target loss value based on the first loss value and the second loss value; and update the parameters of the first model and the second model based on the target loss value. The introduction of the second loss function enables the 3D representation structure to represent more scene content without causing excessive brightness.

[0100] In some optional implementations of this embodiment, the multiple frames of first images include m frames of images containing interference shadows and n frames of images without interference shadows, wherein m and n are both positive integers, and m<n. Thus, when the same interference shadow exists in multiple first images of adjacent perspectives, since there are more images without interference shadows than images containing interference shadows in the multiple frames of first images, it is still possible to ensure that the images containing interference shadows are gradually optimized to a light intensity value consistent with adjacent images of other perspectives through the correction parameters in the first model, thereby eliminating the interference shadow.

[0101] In some optional implementations of this embodiment, the parameter learning rate of the first model is lower than the parameter learning rate of the second model. Thus, the second model can be quickly fitted in the non-interference shadow area, and the interference shadow can be gradually recognized by the first model, so as to avoid introducing the second loss function to affect the elimination effect of the interference shadow.

[0102] The device provided by the above-mentioned embodiment of the present application first inputs multiple frames of first images into the first model, corrects the multiple frames of first images through the first model, and obtains multiple frames of second images corresponding to the multiple frames of first images, wherein the multiple frames of first images are images of the same object captured from different perspectives in the same scene, and the multiple frames of first images include images containing interfering shadows and images without interfering shadows; then the camera posture information when capturing the multiple frames of first images and the multiple frames of second images are input into the second model, and the scene is modeled and rendered through the second model to obtain multiple frames of third images corresponding to the multiple frames of second images; finally, the first model and the second model can be trained based on the multiple frames of first images and the multiple frames of third images, thereby completing a training process. Since the first model is introduced, the multiple frames of first images can be corrected in each training process and then input into the second model for modeling and rendering. By iterating the training multiple times in this way, the parameters of the first model and the second model can be gradually optimized, so that the final first model can remove the interfering shadows in the first image and retain other content, and finally can input the image without interfering shadows to the second model to ensure the multi-perspective consistency of the input image, thereby avoiding the distortion of the rendering result of the second model due to interfering shadows, and improving the realism of the second model's scene modeling and rendering.

[0103] It should be noted that the image generation method provided in the embodiment of the present application can be executed by an image generation device. In the embodiment of the present application, the image generation device executing the image generation method is taken as an example to illustrate the image generation device provided in the embodiment of the present application.

[0104] like Fig.10As shown, the image generating device 1000 of this embodiment includes: an acquiring unit 1001, used to acquire at least one camera pose information; a generating unit 1002, used to input the at least one camera pose information into a trained second model, perform image rendering through the second model, and obtain at least one frame of image corresponding to the at least one camera pose information, wherein each frame of the at least one frame of image does not contain interfering shadows.

[0105] It is understandable that the units recorded in the device 1000 correspond to the steps in the above-mentioned image generation method. Therefore, the operations, features and beneficial effects described above for the image generation method are also applicable to the device 1000 and the units contained therein, and will not be repeated here.

[0106] The model training device in the embodiment of the present application can be an electronic device, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplary, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (Augmented Reality, AR) / virtual reality (Virtual Reality, VR) device, a robot, a wearable device, a super mobile personal computer (Ultra-Mobile Personal Computer, UMPC), a netbook or a personal digital assistant (Personal Digital Assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (Personal Computer, PC), a television (Television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0107] The model training device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0108] The model training device provided in the embodiment of the present application can achieve Figure 1 or Figure 8 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0109] Alternatively, if Fig.11As shown, an embodiment of the present application also provides an electronic device 1100, including a processor 1101 and a memory 1102, wherein the memory 1102 stores programs or instructions that can be executed on the processor 1101, and when the program or instructions are executed by the processor 1101, the various steps of the above-mentioned model training method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0110] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0111] Fig.12 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0112] The electronic device 1200 includes but is not limited to: a radio frequency unit 1201, a network module 1202, an audio output unit 1203, an input unit 1204, a sensor 1205, a display unit 1206, a user input unit 1207, an interface unit 1208, a memory 1209, and a processor 1210 and other components.

[0113] Those skilled in the art will appreciate that the electronic device 1200 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1210 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.12 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0114] Among them, the processor 1210 is used to input multiple frames of first images into the first model, correct the multiple frames of first images through the first model, and obtain multiple frames of second images corresponding to the multiple frames of first images, wherein the multiple frames of first images include images of the same object captured from different perspectives in the same scene; input the camera posture information when shooting the multiple frames of first images and the multiple frames of second images into the second model, model the scene through the second model and render to obtain multiple frames of third images corresponding to the multiple frames of second images; the training unit 903 is used to train the first model and the second model based on the multiple frames of first images and the multiple frames of third images.

[0115] Since the first model is introduced, multiple frames of the first image can be corrected and then input into the second model for modeling and rendering during each training process. By iterating the training multiple times in this way, the parameters of the first model and the second model can be gradually optimized, so that the final first model can achieve the effect of removing interfering shadows in the first image and retaining other content, and finally can input images without interfering shadows into the second model to ensure the multi-view consistency of the input image, thereby avoiding the distortion of the rendering results of the second model due to interfering shadows, and improving the realism of the second model in scene modeling and rendering.

[0116] In some optional implementations of this embodiment, the processor 1210 is further configured to perform the following steps for each first image in the multiple first images through the first model: obtaining a first pixel matrix of the first image and a correction parameter matrix corresponding to the first image; based on the correction parameter matrix, correcting the first pixel matrix to obtain a second pixel matrix of the first image; and determining the image corresponding to the second pixel matrix as the second image corresponding to the first image. By correcting the pixel matrix of the corresponding first image using the correction parameter matrix corresponding to each first image, the multi-view images input to the second model in each training process can be corrected images. Since the first model is jointly trained with the second model, and the second model can tend to perform the same representation of the same object under multiple perspectives, iterative training for multiple times can gradually bring the interfering shadow area in the image containing interfering shadows closer to the representation of the same area as the image without interfering shadows, so that the final first model can effectively remove the interfering shadows in the first image containing interfering shadows, and the 3D model can model and render the scene more realistically.

[0117] In some optional implementations of the present embodiment, the processor 1210 is further configured to, for each third image in the plurality of third images, use the second image corresponding to the third image as the target second image and perform the following steps: if the difference in brightness between the third image and the target second image is greater than a threshold, determine a brightness compression coefficient based on a correction parameter matrix corresponding to the target second image, a correction parameter matrix corresponding to a previous frame of the target second image in the plurality of second images, and a correction parameter matrix corresponding to a subsequent frame of the target second image in the plurality of second images; and perform brightness compression processing on the third image based on the brightness compression coefficient to update the third image. By performing brightness compression on the third image when the difference in brightness between a third image and its corresponding second image is greater than a threshold, it is possible to avoid a significant increase in brightness of the third image output by the second model, so that the rendering effect of the second model is closer to the original scene and the realism of the rendering is improved.

[0118] In some optional implementations of this embodiment, the processor 1210 is further configured to determine a first loss value based on a first loss function, the multiple frames of the first image, and the multiple frames of the third image; determine a second loss value based on a second loss function, a parameter value matrix of the first model, and a target parameter value matrix; determine a target loss value based on the first loss value and the second loss value; and update the parameters of the first model and the second model based on the target loss value. The introduction of the second loss function enables the 3D representation structure to represent more scene content without causing excessive brightness.

[0119] In some optional implementations of this embodiment, the multiple frames of first images include m frames of images containing interference shadows and n frames of images without interference shadows, wherein m and n are both positive integers, and m<n. Thus, when the same interference shadow exists in multiple first images of adjacent perspectives, since there are more images without interference shadows than images containing interference shadows in the multiple frames of first images, it is still possible to ensure that the images containing interference shadows are gradually optimized to a light intensity value consistent with adjacent images of other perspectives through the correction parameters in the first model, thereby eliminating the interference shadow.

[0120] In some optional implementations of this embodiment, the parameter learning rate of the first model is lower than the parameter learning rate of the second model. Thus, the second model can be quickly fitted in the non-interference shadow area, and the interference shadow can be gradually recognized by the first model, so as to avoid introducing the second loss function to affect the elimination effect of the interference shadow.

[0121] It should be noted that the image generation method provided in the embodiment of the present application can be executed by an image generation device. In the embodiment of the present application, the image generation device executing the image generation method is taken as an example to illustrate the image generation device provided in the embodiment of the present application.

[0122] In some optional implementations of this embodiment, the processor 1210 is also used to obtain at least one camera pose information; input the at least one camera pose information into a trained second model, perform image rendering through the second model, and obtain at least one frame of image corresponding to the at least one camera pose information, and each frame of the at least one frame of image does not contain interfering shadows.

[0123] It should be understood that in the embodiment of the present application, the input unit 1204 may include a graphics processor (Graphics Processing Unit, GPU) 12041 and a microphone 12042, and the graphics processor 12041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1206 may include a display panel 12061, and the display panel 12061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1207 includes a touch panel 12071 and at least one of other input devices 12072. The touch panel 12071 is also called a touch screen. The touch panel 12071 may include two parts: a touch detection device and a touch controller. Other input devices 12072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0124] The memory 1209 can be used to store software programs and various data. The memory 1209 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1209 may include a volatile memory or a non-volatile memory, or the memory 1209 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1209 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0125] The processor 1210 may include one or more processing units; optionally, the processor 1210 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1210.

[0126] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned model training method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0127] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0128] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned model training method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0129] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0130] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned model training method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0131] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0132] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0133] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Inputting a plurality of frames of first images into a first model, and correcting the plurality of frames of first images by using the first model to obtain a plurality of frames of second images corresponding to the plurality of frames of first images, wherein the plurality of frames of first images include images obtained by photographing the same object at different viewing angles in the same scene; Inputting the camera pose information when taking the multiple frames of the first image and the multiple frames of the second image into a second model, modeling the scene through the second model and rendering to obtain multiple frames of third images corresponding to the multiple frames of the second image; The first model and the second model are trained based on the multiple frames of first images and the multiple frames of third images.

2. The method according to claim 1, characterized in that The method of correcting the multiple frames of first images by using the first model to obtain multiple frames of second images corresponding to the multiple frames of first images includes: For each first image frame in the multiple first image frames, the following steps are performed by the first model: Obtaining a first pixel matrix of the first image of the frame and a correction parameter matrix corresponding to the first image of the frame; Based on the correction parameter matrix, the first pixel matrix is ​​corrected to obtain a second pixel matrix of the first image of the frame; The image corresponding to the second pixel matrix is ​​determined as the second image corresponding to the first image of the frame.

3. The method according to claim 1, characterized in that After the scene is modeled by the second model and rendered to obtain a plurality of frames of third images corresponding to the plurality of frames of second images, the method further includes: For each third image in the plurality of third images, the second image corresponding to the third image is used as the target second image, and the following steps are performed: If the difference between the brightness values ​​of the third image of the frame and the target second image is greater than a threshold, a brightness compression coefficient is determined based on a correction parameter matrix corresponding to the target second image, a correction parameter matrix corresponding to a previous frame image of the target second image in the multiple frames of second images, and a correction parameter matrix corresponding to a subsequent frame image of the target second image in the multiple frames of second images; Based on the brightness compression coefficient, brightness compression processing is performed on the frame of the third image to update the frame of the third image.

4. The method according to claim 1, characterized in that: The training of the first model and the second model based on the multiple frames of first images and the multiple frames of third images includes: Determine a first loss value based on a first loss function, the plurality of first image frames, and the plurality of third image frames; Determine a second loss value based on a second loss function, a parameter value matrix of the first model, and a target parameter value matrix; Determining a target loss value based on the first loss value and the second loss value; Based on the target loss value, parameters of the first model and the second model are updated.

5. An image generation method, characterized in that: The method comprises: Obtain at least one camera pose information; The at least one camera pose information is input into a second model trained by the model training method described in one of claims 1-4, and image rendering is performed through the second model to obtain at least one frame of image corresponding to the at least one camera pose information, and each frame of the at least one image does not contain interfering shadows.

6. A model training device, characterized in that: The device comprises: a correction unit, configured to input a plurality of frames of first images into a first model, and correct the plurality of frames of first images by using the first model to obtain a plurality of frames of second images corresponding to the plurality of frames of first images, wherein the plurality of frames of first images include images obtained by photographing the same object at different viewing angles in the same scene; A rendering unit, used for inputting the camera posture information when shooting the multiple frames of the first image and the multiple frames of the second image into a second model, modeling the scene through the second model and rendering to obtain multiple frames of third images corresponding to the multiple frames of the second image; A training unit is used to train the first model and the second model based on the multiple frames of first images and the multiple frames of third images.

7. The device according to claim 6, characterized in that The correction unit is further used for: For each first image frame in the multiple first image frames, the following steps are performed by the first model: Obtaining a first pixel matrix of the first image of the frame and a correction parameter matrix corresponding to the first image of the frame; Based on the correction parameter matrix, the first pixel matrix is ​​corrected to obtain a second pixel matrix of the first image of the frame; The image corresponding to the second pixel matrix is ​​determined as the second image corresponding to the first image of the frame.

8. The device according to claim 7, characterized in that The device also includes an updating unit, further configured to: For each third image in the plurality of third images, the second image corresponding to the third image is used as the target second image, and the following steps are performed: If the difference between the brightness values ​​of the third image of the frame and the target second image is greater than a threshold, a brightness compression coefficient is determined based on a correction parameter matrix corresponding to the target second image, a correction parameter matrix corresponding to a previous frame image of the target second image in the multiple frames of second images, and a correction parameter matrix corresponding to a subsequent frame image of the target second image in the multiple frames of second images; Based on the brightness compression coefficient, brightness compression processing is performed on the frame of the third image to update the frame of the third image.

9. The device according to claim 7, characterized in that The training unit is further used for: Determine a first loss value based on a first loss function, the plurality of first image frames, and the plurality of third image frames; Determine a second loss value based on a second loss function, a parameter value matrix of the first model, and a target parameter value matrix; Determining a target loss value based on the first loss value and the second loss value; Based on the target loss value, parameters of the first model and the second model are updated.

10. An image generating device, characterized in that: The device comprises: An acquisition unit, used to acquire at least one camera position information; A generation unit, used to input the at least one camera pose information into a second model trained by the model training method described in one of claims 1 to 4, and perform image rendering through the second model to obtain at least one frame of image corresponding to the at least one camera pose information, wherein each frame of the at least one frame of image does not contain interfering shadows.

Images