Image processing method and related equipment

Abstract

The embodiments of the present application disclose an image processing method and related equipment. They relate to the field of artificial intelligence, and more specifically, to the field of computer vision. The method includes: acquiring a first image and a second image, mapping the first image and the second image based on a spatial mapping model to obtain a first expanded image and a second expanded image; fusing the first expanded image and the second expanded image to obtain a third image; and inputting the third image into a trained image prediction network for image prediction to obtain a predicted image, which is used for ambient light rendering. By mapping and fusing expanded images from multiple perspectives using a spatial mapping model, the predicted image obtained after inputting into the image prediction network has more texture, thereby improving the quality of the ambient light rendering of virtual objects in the subsequent scene.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer vision, and in particular to an image processing method and related equipment. Background Art

[0002] Computer vision is an integral part of various intelligent / autonomous systems in various application areas, such as manufacturing, inspection, document analysis, medical diagnosis, and the military. It involves the use of cameras and computers to acquire the data and information we need about the objects being photographed. Figuratively speaking, it's like equipping computers with eyes (cameras) and brains (algorithms) to replace the human eye in identifying, tracking, and measuring objects, thus enabling computers to perceive their environment. Because perception can be viewed as extracting information from sensory signals, computer vision can also be considered the study of how artificial systems can "perceive" from images or multidimensional data. In short, computer vision uses various imaging systems to replace the visual organs to acquire input information, and then computers replace the brain to process and interpret this input information. The ultimate goal of computer vision research is to enable computers to observe and understand the world through vision like humans do, and to have the ability to autonomously adapt to their environments.

[0003] In fields such as AR image editing and filmmaking, it is necessary to synthesize 3D virtual objects into images of real scenes. To achieve realistic rendering effects, it is necessary to estimate the lighting conditions of the real scene so that the virtual objects in the synthesized image can display shading, shadows, and reflections consistent with the real objects in the scene, and the synthesized image can correctly present the projection or occlusion between the virtual objects and the real objects.

[0004] Therefore, for object surface coloring and environmental reflection, it is particularly important to determine the ambient light parameters. Summary of the Invention

[0005] The embodiments of the present application provide an image processing method and related devices, which can improve the quality of subsequent ambient light rendering.

[0006] The first aspect of the embodiment of the present application provides an image processing method, which can be executed by an image processing device or by a component of the image processing device (such as a processor, a chip, or a chip system, etc.), wherein the image processing device can be a local device (such as a mobile phone, a camera, etc.) or a cloud device. The method can also be jointly executed by a local device and a cloud device. The method includes: acquiring a first image and a second image, the first image and the second image being images captured from different perspectives of the same scene; mapping the first image and the second image based on a spatial mapping model to obtain a first expanded image and a second expanded image; fusing the first expanded image and the second expanded image to obtain a third image; inputting the third image into a trained image prediction network for image prediction to obtain a predicted image, which is used for ambient light rendering of virtual objects in the aforementioned scene.

[0007] In an embodiment of the present application, the spatial mapping model is used to map and fuse the expanded images of multiple perspectives so that the predicted image obtained after inputting the image prediction network has more texture, thereby improving the subsequent ambient light rendering quality of the virtual object in the scene.

[0008] Optionally, in a possible implementation of the first aspect, the above steps map the first image and the second image based on the spatial mapping model to obtain the first expanded image and the second expanded image, including: constructing a spatial mapping model according to the optical center of the first device, the first device is a device for capturing the first image, and the second device is a device for capturing the second image; determining a first projection area of the first image on the spatial mapping model according to the first posture of the first device, the first posture being the posture of the first device when capturing the first image; mapping each pixel point in the first image to the first projection area to obtain the first expanded image; determining a second projection area of the second image on the spatial mapping model according to the second posture of the second device, the second posture being the posture of the second device when capturing the second image; mapping each pixel point in the second image to the second projection area to obtain the second expanded image.

[0009] In this possible implementation, a spatial mapping model is constructed by the posture of the first device when capturing the first image. After subsequent images are mapped to the spatial mapping model based on the posture, they are texture mapped and fused to obtain texture information similar to the captured images (the first image and the second image). This allows the third image output by the subsequent image prediction network to have more texture information, thereby improving the realistic effect of ambient light rendering.

[0010] Optionally, in a possible implementation of the first aspect, the above steps also include: obtaining a historical image and a historical posture of the historical image from a server, the acquisition time of the historical image is before the acquisition time of the first image or the second image, the historical posture is the posture of the historical device when acquiring the historical image, and the historical image stores an image at the same position as the first image and / or the second image; determining a historical projection area of the historical image on the spatial mapping model according to the historical posture, mapping each pixel in the historical image to the historical projection area, and obtaining a historical expanded image; fusing the first expanded image and the second expanded image to obtain a third image, including: fusing the first expanded image, the second expanded image and the historical expanded image to obtain the third image.

[0011] In this possible implementation, historical images on the cloud can be fused to obtain a third image, so that the texture information of the scene stored in the cloud can be referenced, which can improve the texture details of subsequent predicted images and the quality of ambient light rendering.

[0012] Optionally, in a possible implementation of the first aspect, the image prediction network trained in the above steps is obtained by using the training image as the input of the image prediction network and training the image prediction network with the goal of having a loss function value less than a first threshold; the loss function is used to indicate the difference between the image output by the image prediction network and a third target image, where the third target image is the captured image.

[0013] In this possible implementation, the training process of the image prediction network is realized through the training image and the third target image, thereby providing a more optimized image prediction network for the subsequent process and improving the precision of the output image (ie, the predicted image).

[0014] Optionally, in a possible implementation manner of the first aspect, the weight of the loss function in the above steps is controlled by a mask image corresponding to the training image.

[0015] In this possible implementation, the weight of the loss function is controlled by a mask image. For example, the weight of the area with a scene is 1, and the weight of the area without a scene is 0. This can remove invalid parts, reduce interference in invalid areas, and improve the texture details of the output image.

[0016] Optionally, in a possible implementation manner of the first aspect, the above steps further include: acquiring spherical harmonic coefficients of the predicted image; and performing ambient light rendering on the virtual object using the spherical harmonic coefficients.

[0017] In this possible implementation, the virtual object can be rendered by obtaining spherical harmonic coefficients, so that the lighting of the virtual object is more realistic.

[0018] Optionally, in a possible implementation manner of the first aspect, the field of view of the third image in the above step is greater than that of the first image or the second image.

[0019] In this possible implementation, a third image with a larger viewing angle is obtained by inputting an image with a small viewing angle into an image prediction model, which is beneficial for increasing the area of subsequent ambient light rendering.

[0020] A second aspect of the present application provides an image processing device, which may be a local device (e.g., a mobile phone, a camera, etc.) or a cloud device. The image processing device includes:

[0021] An acquisition unit, configured to acquire a first image and a second image, where the first image and the second image are images captured from different viewing angles of the same scene;

[0022] A mapping unit, configured to perform mapping processing on the first image and the second image based on a spatial mapping model to obtain a first expanded image and a second expanded image;

[0023] a fusion unit, configured to fuse the first expanded image and the second expanded image to obtain a third image;

[0024] The prediction unit is used to input the third image into the trained image prediction network to perform image prediction to obtain a predicted image, and the predicted image is used for ambient light rendering.

[0025] Optionally, in a possible implementation manner of the second aspect, the mapping unit in the image processing apparatus includes:

[0026] a construction subunit, configured to construct a spatial mapping model according to an optical center of a first device, wherein the first device is a device for acquiring a first image, and the second device is a device for acquiring a second image;

[0027] a determination subunit, configured to determine a first projection area of the first image on the spatial mapping model according to a first pose of the first device, where the first pose is a pose of the first device when the first image is captured;

[0028] A mapping subunit, configured to map each pixel in the first image to a first projection area to obtain a first expanded image;

[0029] The determination subunit is further configured to determine a second projection area of the second image on the spatial mapping model according to a second posture of the second device, where the second posture is the posture of the second device when capturing the second image;

[0030] The mapping subunit is further configured to map each pixel point in the second image into the second projection area to obtain a second expanded image.

[0031] Optionally, in a possible implementation of the second aspect, the acquisition unit of the image processing apparatus is further configured to acquire historical images and historical poses of the historical images from the cloud, where the historical poses are poses of historical devices when capturing the historical images, and the historical images store texture information at the same position as that in the first image and / or the second image.

[0032] The determination subunit is further used to determine the historical projection area of the historical image on the spatial mapping model according to the historical posture, and the mapping subunit is further used to map each pixel in the historical image to the historical projection area to obtain the historical unfolded image;

[0033] The fusion unit is specifically used to fuse the first expanded image, the second expanded image and the historical expanded image to obtain a third image.

[0034] Optionally, in a possible implementation of the second aspect, the above-mentioned trained image prediction network is obtained by using the training image as the input of the image prediction network and training the image prediction network with the goal of having a loss function value less than a first threshold; the loss function is used to indicate the difference between the image output by the image prediction network and a third target image, and the third target image is the captured image.

[0035] Optionally, in a possible implementation manner of the second aspect, the weight of the above-mentioned loss function is controlled by a mask image corresponding to the training image.

[0036] Optionally, in a possible implementation manner of the second aspect, the acquisition unit of the image processing apparatus is further configured to acquire spherical harmonic coefficients of the predicted image;

[0037] The image processing device further includes:

[0038] The rendering unit is used to perform ambient light rendering on virtual objects using spherical harmonic coefficients.

[0039] Optionally, in a possible implementation manner of the second aspect, a field of view angle of the third image is greater than that of the first image or the second image.

[0040] A third aspect of the present application provides an image processing device, which may be a mobile phone or a camera, or a cloud device (such as a server), that executes the method of the first aspect or any possible implementation of the first aspect.

[0041] A fourth aspect of an embodiment of the present application provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run a computer program or instruction so that the chip implements the method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0042] A fifth aspect of an embodiment of the present application provides a computer-readable storage medium, which stores instructions. When the instructions are executed on a computer, the computer executes the method in the aforementioned first aspect or any possible implementation of the first aspect.

[0043] A sixth aspect of the embodiments of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the method in the aforementioned first aspect or any possible implementation of the first aspect.

[0044] The seventh aspect of an embodiment of the present application provides an image processing device, including: a processor, the processor is coupled to a memory, the memory is used to store programs or instructions, when the program or instructions are executed by the processor, the image processing device implements the method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0045] Among them, the technical effects brought about by the second, third, fourth, fifth, sixth, and seventh aspects or any possible implementation methods thereof can be referred to the technical effects brought about by the first aspect or different possible implementation methods of the first aspect, and will not be repeated here.

[0046] As can be seen from the above technical solution, the embodiments of the present application have the following advantages: obtaining a first image and a second image, mapping the first image and the second image based on a spatial mapping model to obtain a first expanded image and a second expanded image; fusing the first expanded image and the second expanded image to obtain a third image; inputting the third image into a trained image prediction network for image prediction to obtain a predicted image, which is used for ambient light rendering. Mapping and fusing the expanded images from multiple perspectives using the spatial mapping model allows the predicted image obtained after input into the image prediction network to have more texture, thereby improving the quality of the subsequent ambient light rendering of virtual objects in the scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic diagram of the system architecture provided in the embodiment of the present application;

[0048] Figure 2 A schematic diagram of a convolutional neural network structure provided by an embodiment of the present invention;

[0049] Figure 3 A schematic diagram of another convolutional neural network structure provided by an embodiment of the present invention;

[0050] Figure 4 A schematic diagram of a chip hardware structure provided in an embodiment of the present application;

[0051] Figure 5 A schematic flowchart of a method for training an image prediction model provided in an embodiment of the present application;

[0052] Figure 6 A schematic diagram of a training image provided in an embodiment of the present application;

[0053] Figure 7 A schematic diagram of a mask image corresponding to a training image provided in an embodiment of the present application;

[0054] Figure 8 A schematic diagram of a third target image provided in an embodiment of the present application;

[0055] Figure 9 A schematic diagram of an output image provided in an embodiment of the present application;

[0056] Figure 10 A schematic diagram of the structure of an image prediction model provided in an embodiment of the present application;

[0057] Figure 11 A schematic diagram of the structure of another image prediction model provided in an embodiment of the present application;

[0058] Figure 12 A schematic diagram of the structure of another image prediction model provided in an embodiment of the present application;

[0059] Figure 13 A flowchart of the image processing method provided in an embodiment of the present application;

[0060] Figure 14A A schematic diagram of two overlapping perspectives provided in an embodiment of the present application;

[0061] Figure 14B Another schematic diagram of two overlapping perspectives provided in an embodiment of the present application;

[0062] Figure 15A A schematic diagram of a first image provided in an embodiment of the present application;

[0063] Figure 15B A schematic diagram of a second image provided in an embodiment of the present application;

[0064] Figure 16 A schematic diagram of a spatial mapping model constructed based on a first image provided in an embodiment of the present application;

[0065] Figure 17 A schematic diagram of mapping the second image in the spatial mapping model provided in an embodiment of the present application;

[0066] Figure 18 A schematic diagram of a second expanded image provided in an embodiment of the present application;

[0067] Figure 19A schematic diagram of a third image provided in an embodiment of the present application;

[0068] Figure 20 A schematic diagram of vertical fusion of two images provided in an embodiment of the present application;

[0069] Figure 21 A schematic diagram of a historical image provided in an embodiment of the present application;

[0070] Figure 22 Another schematic diagram of the third image or the fourth image provided in the embodiment of the present application;

[0071] Figure 23-26 Several schematic diagrams of user interfaces provided in embodiments of the present application;

[0072] Figure 27 A schematic diagram of the structure of another image prediction model provided in an embodiment of the present application;

[0073] Figure 28 A schematic diagram of the structure of another image prediction model provided in an embodiment of the present application;

[0074] Figure 29 A schematic diagram of a predicted image provided in an embodiment of the present application;

[0075] Figure 30 A schematic diagram of the structure of another image prediction model provided in an embodiment of the present application;

[0076] Figure 31A A schematic diagram of a unit sphere model of a predicted image provided in an embodiment of the present application;

[0077] Figure 31B Schematic diagram of illumination for spherical harmonic coefficient recovery provided in an embodiment of the present application;

[0078] Figure 32 A schematic structural diagram of an image processing device provided in an embodiment of the present application;

[0079] Figure 33 Another structural diagram of the image processing device provided in an embodiment of the present application;

[0080] Figure 34 This is another structural schematic diagram of the image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0081] The technical solution in this application will be described below with reference to the accompanying drawings.

[0082] The image processing method provided in the embodiments of the present application can be applied to augmented reality (AR), game production, movie production, and other scenarios that require ambient light rendering. The following briefly introduces AR scenarios and movie production scenarios.

[0083] AR scenarios:

[0084] AR technology is a new technology developed on the basis of virtual reality. It is a technology that increases the user's perception of the real world through the information provided by the computer system, and superimposes computer-generated virtual objects, scenes or system prompt information on the real scene, thereby achieving "enhancement" of reality. It is a new technology that "seamlessly" integrates real-world information and virtual-world information. Therefore, how to coordinate the rendering effect of virtual objects with the environment is of great significance to the user experience of AR products. Rendering virtual objects using lighting estimation is an important part of "seamless" AR. The image processing method provided in the embodiment of the present application can make the lighting of virtual objects in AR scenes more realistic.

[0085] Film Production:

[0086] Capturing the lighting of non-realistic shots in filmmaking requires estimating the lighting conditions of the real scene to make the non-realistic shots in the film more realistic and able to present the shading, shadows, and reflection effects of the real scene. The image processing method provided in the embodiments of the present application can make the lighting of non-realistic shots in filmmaking more realistic.

[0087] Since the embodiments of the present application involve the application of neural networks, for ease of understanding, the relevant terms and concepts of the neural networks mainly involved in the embodiments of the present application are first introduced below.

[0088] (1) Neural Network

[0089] A neural network can be composed of neural units, which can be represented by X s The output of the operation unit with the intercept 1 as input can be:

[0090]

[0091] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s For X sThe weight of the neural unit, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0092] (2) Deep Neural Networks

[0093] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with many hidden layers. The "many" here does not have a specific metric. Based on the position of different layers in a DNN, the neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, is the bias vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer simply performs such a simple operation on the input vector to obtain the output vector. Since the number of DNN layers is large, the coefficient W and the bias vector The definition of these parameters in DNN is as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as W 24 3 The superscript 3 represents the layer number of the coefficient, while the subscript corresponds to the output of the third layer index 2 and the input of the second layer index 4. In summary, the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as W jk L. It should be noted that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better depict complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (a weight matrix formed by vectors W of many layers).

[0094] (3) Convolutional Neural Networks

[0095] A convolutional neural network (CNN) is a deep neural network with a convolutional architecture. A CNN consists of a feature extractor consisting of convolutional layers and subsampling layers. This feature extractor can be viewed as a filter, and the convolution process can be thought of as convolving an input image or feature map with a trainable filter. A convolutional layer is the layer of neurons in a CNN that performs convolution on the input signal. Within a convolutional layer, a neuron can only connect to a subset of neurons in adjacent layers. A convolutional layer typically contains several feature planes, each composed of a rectangular arrangement of neurons. Neurons within the same feature plane share weights, which are referred to as the convolution kernel. Shared weights can be understood as position-independent extraction of image information. The underlying principle is that the statistical information of one part of the image is the same as that of another. This means that image information learned in one part can also be applied to other parts. Therefore, the same learned image information can be used at all locations in the image. In the same convolutional layer, multiple convolution kernels can be used to extract different image information. Generally speaking, the more convolution kernels there are, the richer the image information reflected by the convolution operation.

[0096] Convolution kernels can be initialized as matrices of random size, and during the training process of the convolutional neural network, the convolution kernels can be learned to obtain reasonable weights. In addition, the direct benefit of shared weights is that they reduce the number of connections between the layers of the convolutional neural network, while also reducing the risk of overfitting.

[0097] (4) Loss function

[0098] During the training of a deep neural network, because we want the output of the deep neural network to be as close as possible to the desired predicted value, we can compare the current network's predicted value with the desired target value and then update the weight vector of each layer of the neural network based on the difference between the two. (Of course, before the first update, there is usually an initialization process, which pre-configures the parameters for each layer in the deep neural network.) For example, if the network's predicted value is too high, the weight vector is adjusted to make it predict a lower value. This adjustment is continued until the deep neural network can predict the desired target value or a value very close to the desired target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value." This is the loss function (or objective function), which is an important equation used to measure the difference between the predicted value and the target value. For example, the loss function output value (loss) indicates a greater difference, so training a deep neural network becomes a process of minimizing this loss.

[0099] (5) Backpropagation algorithm

[0100] Convolutional neural networks can use the back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model during training, reducing the reconstruction error loss of the super-resolution model. Specifically, the forward propagation of the input signal to the output generates an error loss. This error loss information is then backpropagated to update the parameters of the initial super-resolution model, thereby converging the error loss. The BP algorithm is a backward propagation movement dominated by the error loss, aiming to obtain the optimal super-resolution model parameters, such as the weight matrix.

[0101] (6) Generative Adversarial Networks

[0102] Generative adversarial networks (GANs) are a type of deep learning model. They consist of at least two modules: a generative model and a discriminative model. These two modules learn from each other through interaction to produce better outputs. Both the generative and discriminative models can be neural networks, specifically deep neural networks or convolutional neural networks. The basic principle of a GAN is as follows: For example, consider a GAN that generates images. Suppose there are two networks, G (Generator) and D (Discriminator). G is the image generator network, which receives random noise z and generates an image from it, denoted as G(z). D is the discriminator network, which determines whether an image is "real." Its input parameter is x, representing an image. Its output, D(x), represents the probability that x is real. A value of 1 indicates a 100% probability of authenticity, while a value of 0 indicates a high probability of non-authenticity. During the training of this generative adversarial network, the goal of the generative network G is to generate realistic images as much as possible to deceive the discriminative network D, while the goal of the discriminative network D is to distinguish the images generated by G from real images as much as possible. This creates a dynamic "game" between G and D, which is the "game" in "generative adversarial network." Ultimately, under ideal conditions, G can generate images G(z) that are sufficiently realistic, while D has difficulty determining whether the images generated by G are real, i.e., D(G(z)) = 0.5. This results in an excellent generative model G that can be used to generate images.

[0103] (7) Pixel value

[0104] The pixel value of an image can be a red, green, and blue (RGB) color value, which can be a long integer representing the color. For example, a pixel value is 256*Red+100*Green+76Blue, where Blue represents the blue component, Green represents the green component, and Red represents the red component. For each color component, smaller values indicate lower brightness, while larger values indicate higher brightness. For grayscale images, pixel values can be grayscale values.

[0105] (8) Encoder and decoder

[0106] The encoder is used to extract features of the input image. Specifically, the encoder can use a neural network, such as a convolutional neural network.

[0107] The decoder is used to restore the extracted features to an image. Specifically, the decoder can adopt a neural network, for example, a convolutional neural network.

[0108] (9) Upsampling

[0109] In the field of deep learning applied in computer vision, after the input image is extracted through a convolutional neural network (CNN), the output size often becomes smaller. Sometimes we need to restore the image to its original size for further calculations (for example, semantic segmentation of the image). This operation of enlarging the image size and mapping the image from a small resolution to a large resolution is called upsampling.

[0110] Among them, there are three common methods for upsampling: bilinear interpolation (bilinear), deconvolution (TransposedConvolution) and unpooling (Unpooling).

[0111] The following describes the system architecture provided by the embodiments of the present application.

[0112] See attached Figure 1 , an embodiment of the present invention provides a system architecture 100. As shown in the system architecture 100, the data acquisition device 160 is used to collect training data. In the embodiment of the present application, the training data includes: training images. Furthermore, the training images and the first image and / or the second image are images collected for the same scene. The training data is stored in the database 130, and the training device 120 obtains the target model / rule 101 based on the training data maintained in the database 130. The following will describe in more detail how the training device 120 obtains the target model / rule 101 based on the training data using the first embodiment. The target model / rule 101 can be used to implement the image processing method provided in the embodiment of the present application, that is, the third image is input into the target model / rule 101 after relevant pre-processing to obtain a predicted image. The target model / rule 101 in the embodiment of the present application can specifically be an image prediction network. In the embodiment provided in the present application, the image prediction network is obtained by training the training images. It should be noted that in actual applications, the training data maintained in the database 130 does not necessarily come from the collection of the data acquisition device 160, but may also be received from other devices. It should also be noted that the training device 120 does not necessarily train the target model / rule 101 entirely based on the training data maintained by the database 130. It is also possible to obtain training data from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.

[0113] The target model / rule 101 obtained by training the training device 120 can be applied to different systems or devices, such as Figure 1 The execution device 110 shown in the figure can be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, AR / VR, a vehicle terminal, etc., or a server or a cloud terminal. Figure 1 In the embodiment, the execution device 110 is configured with an I / O interface 112 for interacting with external devices. The user can input data to the I / O interface 112 through the client device 140. The input data in the embodiment of the present application may include: a first image and a second image, which can be input by the user or uploaded by the user through a shooting device. Of course, it can also come from a database, which is not limited here.

[0114] The preprocessing module 113 is used to preprocess the input data (such as the first image and the second image) received by the I / O interface 112. In an embodiment of the present application, the preprocessing module 113 can be used to map the first image and the second image based on a spatial mapping model to obtain a first expanded image and a second expanded image.

[0115] When the execution device 110 preprocesses the input data, or when the computing module 111 of the execution device 110 performs calculations and other related processing, the execution device 110 can call the data, code, etc. in the data storage system 150 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing in the data storage system 150.

[0116] Finally, the I / O interface 112 returns the processing result, such as the predicted image obtained above, to the client device 140 for providing to the user.

[0117] It is worth noting that the training device 120 can generate corresponding target models / rules 101 based on different training data for different goals or different tasks. The corresponding target models / rules 101 can be used to achieve the above goals or complete the above tasks, thereby providing users with the desired results.

[0118] In the attached Figure 1In the case shown in FIG, the user can manually input data, which can be operated through the interface provided by I / O interface 112. In another case, client device 140 can automatically send input data to I / O interface 112. If the automatic transmission of input data by client device 140 requires user authorization, the user can set the corresponding permissions in client device 140. The user can view the results output by execution device 110 on client device 140, which can be presented in the form of display, sound, action, etc. Client device 140 can also serve as a data acquisition terminal, collecting input data input into I / O interface 112 and output results from I / O interface 112 as new sample data and storing them in database 130. Of course, the collection can also be performed without client device 140, and instead the input data input into I / O interface 112 and output results from I / O interface 112 as new sample data can be directly stored in database 130 by I / O interface 112.

[0119] It is worth noting that the Figure 1 This is only a schematic diagram of a system architecture provided by an embodiment of the present invention. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in the attached Figure 1 In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 can also be placed in the execution device 110.

[0120] like Figure 1 As shown, the target model / rule 101 is obtained through training according to the training device 120. The target model / rule 101 can be an image prediction network in the embodiment of the present application. Specifically, in the network provided in the embodiment of the present application, the image prediction network can be a convolutional neural network.

[0121] Since CNN is a very common neural network, the following Figure 2 This section focuses on a detailed introduction to the structure of CNNs. As mentioned in the previous basic concepts, a convolutional neural network is a deep neural network with a convolutional structure and a deep learning architecture. Deep learning architectures use machine learning algorithms to perform multiple levels of learning at different levels of abstraction. As a deep learning architecture, a CNN is a feed-forward artificial neural network in which individual neurons respond to input images.

[0122] like Figure 2 As shown, a convolutional neural network (CNN) 100 may include an input layer 110 , a convolutional layer / pooling layer 120 , wherein the pooling layer is optional, and a neural network layer 130 .

[0123] Convolutional layer / pooling layer 120:

[0124] Convolutional layer:

[0125] like Figure 2 The convolutional layer / pooling layer 120 shown may include layers 121-126, for example. In one implementation, layer 121 is a convolutional layer, layer 122 is a pooling layer, layer 123 is a convolutional layer, layer 124 is a pooling layer, layer 125 is a convolutional layer, and layer 126 is a pooling layer. In another implementation, layers 121 and 122 are convolutional layers, layer 123 is a pooling layer, layers 124 and 125 are convolutional layers, and layer 126 is a pooling layer. That is, the output of a convolutional layer can be used as the input of a subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation.

[0126] Taking convolution layer 121 as an example, convolution layer 121 can include many convolution operators, also known as kernels. Their role in image processing is equivalent to a filter that extracts specific information from the input image matrix. The convolution operator can essentially be a weight matrix, which is usually predefined. During the convolution operation on the image, the weight matrix is usually processed horizontally on the input image one pixel at a time (or two pixels at a time... depending on the value of the stride), thereby completing the task of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It is important to note that the depth dimension of the weight matrix is the same as the depth dimension of the input image. During the convolution operation, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a convolution output with a single depth dimension. However, in most cases, a single weight matrix is not used, but multiple weight matrices of the same dimension are applied. The output of each weight matrix is stacked to form the depth dimension of the convolved image. Different weight matrices can be used to extract different features in the image. For example, one weight matrix is used to extract image edge information, another weight matrix is used to extract specific colors of the image, and another weight matrix is used to blur unwanted noise in the image... The multiple weight matrices have the same dimensions, and the feature maps extracted by the multiple weight matrices with the same dimensions also have the same dimensions. The multiple feature maps with the same dimensions extracted are then merged to form the output of the convolution operation.

[0127] The weight values in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values obtained through training can extract information from the input image, thereby helping the convolutional neural network 100 to make correct predictions.

[0128] When the convolutional neural network 100 has multiple convolutional layers, the initial convolutional layer (for example, 121) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 100 increases, the features extracted by the later convolutional layers (for example, 126) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0129] Pooling layer:

[0130] Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer, such as Figure 2 The layers 121-126 illustrated in Figure 120 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In the image processing process, the sole purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer can include an average pooling operator and / or a maximum pooling operator to sample the input image to obtain a smaller image. The average pooling operator can calculate the average value of the pixel values in the image within a specific range. The maximum pooling operator can take the pixel with the largest value within a specific range as the result of maximum pooling. In addition, just as the size of the weight matrix used in the convolutional layer should be related to the image size, the operators in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer can be smaller than the size of the image input to the pooling layer. Each pixel in the image output by the pooling layer represents the average value or maximum value of the corresponding sub-region of the image input to the pooling layer.

[0131] Neural Network Layer 130:

[0132] After being processed by the convolution layer / pooling layer 120, the convolution neural network 100 is not sufficient to output the required output information. As mentioned above, the convolution layer / pooling layer 120 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network 100 needs to use the neural network layer 130 to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer 130 may include multiple hidden layers (such as Figure 2 131, 132 to 13n) and the output layer 140 shown, the parameters contained in the multiple hidden layers can be pre-trained based on relevant training data of specific task types, for example, the task types may include image recognition, image classification, image super-resolution reconstruction, etc.

[0133] After the multiple hidden layers in the neural network layer 130, that is, the last layer of the entire convolutional neural network 100 is the output layer 140, which has a loss function similar to the classification cross entropy, specifically for calculating the prediction error. Once the forward propagation of the entire convolutional neural network 100 (such as Figure 2 The propagation from 110 to 140 is forward propagation), and the reverse propagation (such as Figure 2 The propagation from 140 to 110 is back propagation) and then starts to update the weight values and biases of the aforementioned layers to reduce the loss of the convolutional neural network 100 and the error between the result output by the convolutional neural network 100 through the output layer and the ideal result.

[0134] It should be noted that if Figure 2 The convolutional neural network 100 shown is only an example of a convolutional neural network. In specific applications, the convolutional neural network can also exist in the form of other network models, such as Figure 3 The multiple convolutional layers / pooling layers shown are operated in parallel, and the features extracted from each layer are input to the full neural network layer 130 for processing.

[0135] The following describes a chip hardware structure provided by an embodiment of the present application.

[0136] Figure 4 The embodiment of the present invention provides a chip hardware structure, which includes a neural network processor 40. The chip can be set as follows Figure 1 The execution device 110 shown in FIG. 1 is used to complete the calculation work of the calculation module 111. The chip can also be set in Figure 1 The training device 120 shown in FIG. 1 is used to complete the training work of the training device 120 and output the target model / rule 101. Figure 2 The algorithms of each layer in the convolutional neural network shown in the figure can be used in Figure 4 is implemented in the chip shown.

[0137] The neural network processor 40 can be a neural network processing unit (NPU), a tensor processing unit (TPU), or a graphics processing unit (GPU), any processor suitable for large-scale XOR operation processing. Taking the NPU as an example: the neural network processor NPU40 is mounted on the main central processing unit (CPU) (host CPU) as a coprocessor, and the main CPU assigns tasks. The core part of the NPU is the operation circuit 403, and the controller 404 controls the operation circuit 403 to extract data from the memory (weight memory or input memory) and perform operations.

[0138] In some implementations, the arithmetic circuit 403 includes multiple processing units (PEs). In some implementations, the arithmetic circuit 403 is a two-dimensional systolic array. The arithmetic circuit 403 can also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 403 is a general-purpose matrix processor.

[0139] For example, assume there are input matrix A, weight matrix B, and output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from weight memory 402 and caches it on each PE in the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from input memory 401 and performs a matrix operation on matrix B. The partial or final matrix result is stored in accumulator 408.

[0140] The vector calculation unit 407 can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 407 can be used for network calculations of non-convolutional / non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, etc.

[0141] In some implementations, the vector calculation unit 407 can store the processed output vector to the unified buffer 406. For example, the vector calculation unit 407 can apply a nonlinear function to the output of the operation circuit 403, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 407 generates a normalized value, a merged value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 403, for example, for use in a subsequent layer in a neural network.

[0142] The unified memory 406 is used to store input data and output data.

[0143] The weight data is directly transferred from the external memory to the input memory 401 and / or the unified memory 406 through the direct memory access controller 405 (DMAC), the weight data in the external memory is stored in the weight memory 402, and the data in the unified memory 506 is stored in the external memory.

[0144] The bus interface unit (BIU) 410 is used to implement interaction between the main CPU, DMAC and instruction fetch memory 409 through the bus.

[0145] An instruction fetch buffer 409 connected to the controller 404 is used to store instructions used by the controller 404 .

[0146] The controller 404 is used to call the instructions cached in the memory 409 to control the working process of the computing accelerator.

[0147] Generally, the unified memory 406, the input memory 401, the weight memory 402 and the instruction fetch memory 409 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM) or other readable and writable memory.

[0148] in, Figure 2 or Figure 3 The operations of each layer in the convolutional neural network shown can be performed by the operation circuit 403 or the vector calculation unit 407.

[0149] In fields such as AR image editing and filmmaking, it is necessary to synthesize 3D virtual objects into images of real scenes. To achieve realistic rendering effects, it is necessary to estimate the lighting conditions of the real scene so that the virtual objects in the synthesized image can display shading, shadows, and reflections consistent with the real objects in the scene, and the synthesized image can correctly present the projection or occlusion between the virtual objects and the real objects.

[0150] Therefore, how to determine more realistic ambient light parameters becomes an urgent problem to be solved.

[0151] An embodiment of the present application provides an image processing method that can improve the quality of ambient light rendering.

[0152] The following is a detailed introduction to the image prediction network training method and image processing method of the embodiment of the present application in conjunction with the accompanying drawings.

[0153] First, combine Figure 5 The training method 500 of the image prediction network according to an embodiment of the present application is introduced in detail. Figure 5 The method shown can be executed by a training device for an image prediction network. The training device for an image prediction network can be a cloud service device or a terminal device, for example, a computer, a server, or other device with sufficient computing power to execute the training method for an image prediction network, or a system composed of a cloud service device and a terminal device. For example, the method 500 can be performed by Figure 1 Training equipment 120, Figure 4 The neural network processor 40 in is executed.

[0154] Optionally, method 500 may be processed by a CPU, or may be processed jointly by a CPU and a GPU, or may use other processors suitable for neural network calculations without using a GPU, and this application does not impose any restrictions.

[0155] Method 500 includes step 501 and step 502. Step 501 and step 502 are described in detail below.

[0156] Step 501: Acquire a training image.

[0157] Optionally, the training image may be a partial viewing angle image selected from a captured panoramic image, or may be obtained by fusing at least two images captured by a terminal device, which is not specifically limited here.

[0158] Optionally, during the training process, training samples can be generated from the panoramic image. The training samples include: an input image (that is, the above-mentioned training image, for example: a training image such as Figure 6 As shown in ), the mask image corresponding to the input image (for example: the mask image corresponding to the training image is as shown in Figure 7 As shown) and the third target image (the third target image can be a captured panoramic image or an image selected from the panoramic image, and the viewing angle range of the third target image is larger than the training image. For example: if the third target image is a panoramic image, the third target image is as shown Figure 8 As shown, Figure 6 The training images shown can be obtained from Figure 8 The panoramic image shown is obtained by selecting a partial viewing angle).

[0159] The mask image in the embodiment of the present application can be used to distinguish between valid areas and invalid areas. The mask image can also be understood as a black and white image, where black and white represent different areas. For example, the scene part in the training image is the valid area (or white area), and the area outside the scene is the invalid area (black area).

[0160] Optionally, the acquired training images can be in the same scene as the first image and / or the second image, or in different scenes. Of course, if the training is performed in the same scene, the training effect is better. The interpretation of the same scene can refer to the subsequent interpretation of the first and second images in the same scene. Optionally, the number of acquired training images can be one or more, and the specific number is not limited here.

[0161] Step 502: Using the training image as input of the image prediction network, the image prediction network is trained with the goal of making the value of the loss function less than a first threshold, thereby obtaining a trained image prediction network.

[0162] Among them, the loss function is used to indicate the output image of the image prediction network (for example: the output image is Figure 9 The difference between the image (shown) and the third target image.

[0163] In this case, the image prediction network is trained with the goal of minimizing the loss function. Specifically, the difference between the output image of the image prediction network and the third target image is continuously reduced. This training process can be understood as a prediction task. The loss function can be understood as the loss function corresponding to the prediction task. The output image has a larger viewing angle than the input image.

[0164] Optionally, the penalty weight in the loss function is controlled by the mask image corresponding to the training image. Add a weight before the general loss function, which is 0 or 1. For example: Figure 6 The weight of the area with scenes (also called valid areas) is 1, and the weight of the area without scenes (also called invalid areas) is 0, that is, Figure 7The weight of the white area (i.e., valid area) in the image is 1, and the weight of the black area (i.e., invalid area) is 0. This means that the black area does not participate in subsequent calculations, which can reduce computing power consumption during training. The weight of the loss function is controlled by the mask image. For example, the weight of the area with scene is 1, and the weight of the area without scene is 0. This can remove the invalid part, reduce the interference of the invalid area, and improve the texture detail of the output image.

[0165] In one possible implementation, the training image is used as the input of the image prediction network, and the image prediction network is trained with the loss function value being less than a first threshold as the goal, to obtain a trained image prediction network. Optionally, the image prediction network can be a CNN. For example, the image prediction network is as follows: Figure 10 As shown, the image prediction network includes an encoder and a decoder. The encoder may include convolution, activation, and pooling. The decoder may include convolution and upsampling. Of course, the decoder may also include deconvolution. The specific structure of the image prediction network is not limited here.

[0166] In another possible implementation, Figure 11 As shown, GAN can be introduced. The generator in GAN is an image prediction network that generates output images; the discriminator determines whether the output images are "real." During the training process of this generative adversarial network, the goal of the generator is to generate images that are as realistic as possible to deceive the discriminator, while the goal of the discriminator is to distinguish the output images generated by the generator from real images as much as possible. The real image is the third target image. Under ideal conditions, the generator can generate output images that are "real" enough, while the discriminator has difficulty determining whether the output images generated by the generator are real. This results in an excellent generator that can be used to generate output images.

[0167] For example, continuing with the above example, Figure 12 It can be understood as Figure 11 An example of .

[0168] Optionally, in the above two methods, the encoder and the corresponding layers of the decoder of the image prediction network establish cross-layer connections, that is, after feature extraction, details may be lost. Through cross-layer connections, an image without feature extraction can be provided as a reference, so that the result has more texture details.

[0169] It should be noted that the training process may also use other training methods instead of the aforementioned method 500, which is not limited here.

[0170] The image processing method of the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0171] See also Figure 13 , an embodiment of the image processing method in the embodiment of the present application, the method 1300 includes steps 1301 to 1306.

[0172] Step 1301: An image processing apparatus obtains a first image and a second image.

[0173] The image processing device in the embodiment of the present application can be a cloud service device or a terminal device, for example, a computer, a server, or other device with sufficient computing power to execute the image processing method, or a system composed of a cloud service device and a terminal device. Figure 1 The execution device 110 in Figure 4 The neural network processor 40 in.

[0174] Optionally, the image processing device may be a CPU, a CPU and a GPU, or may not be a GPU but use other processors suitable for neural network calculations, which are not specifically limited here.

[0175] In the embodiments of the present application, the first image and the second image are images captured from different perspectives of the same scene. The first image is an image captured by the first device from the first perspective, and the second image is an image captured by the second device from the second perspective. The first device and the second device can be the same device or different devices. The time when the first device captures the first image may be the same as or different from the time when the second device captures the second image, and the specific time is not limited here.

[0176] The same scenario in the embodiments of the present application can be understood as at least one of the following attributes being satisfied:

[0177] 1. The same scene may refer to two images (eg, a first image and a second image) having partially identical screen contents, for example, the overlapping content (or region, area) of the first image and the second image is greater than or equal to 30%.

[0178] 2. The same scene can mean that the distance between the first position of the device when capturing one image and the position of the second device when capturing another image is less than a certain threshold (for example, the distance between the position of the first device when capturing the first image and the position of the second device when capturing the second image is 1 meter, and the threshold is 2 meters, that is, the distance is less than the threshold, then it can be determined that the first image and the second image are images captured in the same scene); and / or the overlapping angle of the field of view of the two images (for example, the first image and the second image) is greater than a certain threshold (for example, the overlapping angle of the first perspective and the second perspective is greater than 30 degrees); and / or the difference in the rotation angle of the devices that captured the two images is less than a certain threshold. The rotation angle can be the angle value of the device's horizontal rotation angle or the angle value of the camera's downward rotation angle.

[0179] For example, the above-mentioned overlap angle can be as follows: Figure 14A Or as shown by the arrow in 14B, the first image captured by the first device at the first viewing angle and the second image captured by the second device at the second viewing angle, the angle of overlap in between is called the overlap angle.

[0180] The position mentioned above can be a relative position or a geographical location, etc. If the position is a relative position, the relative position of the first device and the second position can be determined by establishing a scene model, etc.; if the position is a geographical location, it can be based on the global positioning system (GPS) or Beidou navigation system, etc. The position of the first device and the position of the second device can be determined, and then the distance between the two positions can be obtained.

[0181] 3. The same scene can also be determined based on light intensity. For example, whether two images are of the same scene can be determined based on whether the weather type when one image was captured is similar to the weather type when the other image was captured. For example, if the first image was captured on a sunny day and the second image was also captured on a sunny day, the first and second images can be determined to be of the same scene. However, if the first image was captured on a sunny day and the second image was captured on a rainy day, the first and second images can be determined to be different scenes.

[0182] 4. The same scene may also mean that the texture similarity between the first image and the second image is greater than or equal to a certain threshold. Generally, this method needs to be combined with the other methods mentioned above to make a judgment.

[0183] It is understandable that the above determination of whether the first image and the second image are of the same scene is only an example. In actual applications, there may be other methods, which are not specifically limited here.

[0184] The above examples can be determined individually or jointly. For example, after determining that the distance is less than a certain threshold and the weather type is consistent, the first image and the second image are determined to be images captured from the same scene. Alternatively, after determining that the distance is less than a certain threshold and the texture similarity of the two images is greater than or equal to a certain threshold, the first image and the second image can be determined to be images captured from the same scene. Alternatively, after determining that the distance is less than a certain threshold, it can also be determined whether the overlapping angle of the field of view of the two images is greater than a certain threshold. If so, the first image and the second image are determined to be images captured from the same scene.

[0185] Optionally, the above-mentioned attributes (position, overlap angle, overlap area and / or light intensity information) can be stored together with the first image or the second image. The image processing device can obtain the attributes corresponding to the image before and after obtaining the image, so as to facilitate the judgment of whether the images are the same scene, and then correctly perform subsequent fusion operations.

[0186] In one possible implementation, the image processing apparatus may capture or shoot the first image and the second image at different viewing angles, that is, the image processing apparatus, the first device, and the second device are the same device.

[0187] In another possible implementation, an image processing apparatus receives a first image and a second image sent by another device. The other device may be a camera or an inertial measurement unit (IMU), etc., which has an acquisition or capture function. That is, the image processing apparatus is a different device from the first device or the second device.

[0188] It is understandable that, in addition to acquiring the first image and the second image, the image processing device can also acquire images from other perspectives. The embodiment of the present application only uses the first image and the second image as examples for schematic illustration.

[0189] The embodiments of the present application are only schematically described by taking the example that the device for capturing the first image or the second image is a camera.

[0190] For example, the first image is Figure 15A As shown, the second image is Figure 15B shown.

[0191] Step 1302: The image processing apparatus performs mapping processing on the first image and the second image based on the spatial mapping model to obtain a first expanded image and a second expanded image.

[0192] In the embodiments of the present application, the pose of the first device when capturing the first image is referred to as the first pose, and the pose of the second device when capturing the second image is referred to as the second pose. The pose can be understood as the position and orientation of the device when capturing the image. For example, the pose can be described using parameters such as 6-DOF pose data (the positions of the three X, Y, and Z axes and the rotation angles of the three axes) or a transformation matrix.

[0193] In the embodiment of the present application, the poses of the first image and the second image are based on the poses in their respective spatial coordinate systems, that is, the first image corresponds to one spatial coordinate system, and the second image corresponds to another spatial coordinate system. The function of the spatial mapping model is to map the first image and the second image to the same spatial coordinate system according to the first pose and the second pose, respectively, and to expand them to obtain the first expanded image and the second expanded image. The first expanded image and the second expanded image are two-dimensional images based on the same spatial coordinate system. This is conducive to improving the realism of the subsequent ambient light rendering of virtual objects in the same scene.

[0194] The function of the spatial mapping model in the embodiment of the present application is to map the first image and the second image into the same spatial coordinate system according to the first pose and the second pose. The following is only schematically explained using the spatial mapping model being a spherical model as an example. It can be understood that the spatial mapping model can also be a cube model, etc., which is not specifically limited here.

[0195] The first step is to build a ball model of the scene.

[0196] For example, Figure 16 As shown, a world coordinate system is constructed based on the first image, and a spherical model is constructed with the optical center of the first device as the origin O. The horizontal projection of the optical axis OF is the X-axis, the vertical direction is the Z-axis, and the horizontal direction is the Y-axis. The center point of the world coordinate system coincides with the optical center of the first device. The theoretical optical center is the center of a single convex mirror; the actual optical center may be the virtual center of a combination of multiple convex mirrors.

[0197] The radius of the sphere can be set as needed. For example, the radius can be set to more than 3 meters for indoor scenes and more than 10 meters for outdoor scenes. The specific value of the radius is not limited here.

[0198] Step 2: Determine the projection area.

[0199] The first projection area of the first image on the spherical model is determined based on the position and orientation of the first image on the spherical model. This can be understood as follows: the position and orientation of the first image on the spherical model are determined based on the first vector. Based on the pinhole imaging principle, the pixel data in the first image can be mapped onto the spherical model to obtain the first projection area. The following details how the second projection area of the second image on the spherical model is determined.

[0200] The image processing device determines the position and orientation of the second image on the ball model created above according to the second posture. For example, the projection method of the second image is as follows: Figure 17 As shown, O is the coordinate of the center point of the world coordinate system (also the optical center position when the first device captures the first image), O1 is the position when the second device captures the second image, and A1B1C1D1 is the camera imaging plane. Among them, P1 is any pixel point in the second image, and the projection point of P1 in the second image on the spherical model can be determined as P2 through the intersection of OP1 and the surface of the spherical model. The angle of OP2 relative to the XOZ plane is Φ, and the angle of the projection of OP2 on the XOZ plane relative to OX is θ. Of course, the projection point of P1 in the second image on the spherical model can also be determined as P2 through the intersection of OP1 and the surface of the spherical model, and the specific method is not limited here. Other points are similar to the method of determining P2 at P1, and then the second projection area A2B2C2D2 of the second image A1B1C1D1 on the spherical model can be obtained.

[0201] The third step is texture mapping.

[0202] After the second projection area corresponding to the second image is determined, the position of the pixel point in the second image in the second projection area can be determined through the straight line where OP1 or O1P1 is located.

[0203] The following briefly introduces how to determine the correspondence between the second image and each pixel in the second expanded image using the following formula:

[0204] Assume the internal parameters of the second device are: focal length fx, fy, pixel size dx, dy; the width of the second image is w, and the height is h. The current position of the second device is O1 (x1, y1, z1), and the rotation angles of the optical axis around the X-axis, Y-axis, and Z-axis are α, β, and γ respectively.

[0205] For any point P1 on the second image, the pixel coordinates are (x, y), then the coordinates of point P1 in the camera coordinate system are: P cam (x*dx, y*dy, fx*dx), according to the transformation from the camera coordinate system to the world coordinate system, the coordinate P of point P1 in the world coordinate system can be obtained world (x world ,y world , z world ). The conversion from the camera coordinate system to the world coordinate system is a rigid body transformation (the object does not change, only rotation and translation are required), that is, the camera coordinate system can be obtained after rotation and translation.

[0206]

[0207] in: P cam The transposed matrix of Since the rotation of the camera coordinate system is related to the rotation of the X, Y, and Z axes, the rotation matrix of the camera coordinate system to the world coordinate system includes three components, namely R = R x R y R z , T cam→world Represents the translation matrix from the camera coordinate system to the world coordinate system.

[0208]

[0209]

[0210]

[0211] Among them, pitch=-α, yaw=-β, roll=-γ.

[0212] Determine the coordinate P of point P1 in the world coordinate systemworld After that, we can calculate the value of point O and point P. world The equation of the line of OP1 can be determined, that is, solving The equation of the line of OP1 is obtained. The specific calculation will not be repeated here. After solving the equation of the line of OP1, the intersection of the line equation and the surface of the sphere model close to P1 is the mapping point P2. The angular coordinates of the point P2 on the sphere are marked as (θ, Φ). According to the coordinates of the point P2, the pixel coordinates P3 (x exp ,y exp ).

[0213]

[0214]

[0215] The above method can be used to obtain the pixel coordinates of any point in the second image in the second expanded image. The method for determining other points is similar to determining P3 from P1. In this way, the values of each pixel in the second image are filled into the second expanded image, which can give the second expanded image a texture similar to that of the first image.

[0216] Optionally, after mapping each pixel in the second image to the second expanded image using the line equation of OP1, the values of some pixels in the second expanded image may be uncertain due to distance, i.e., these pixels are not associated with pixels in the second image. To ensure the authenticity of subsequent rendering, the values of each pixel in the second expanded image can be determined by reversing the correspondence between pixels in the second expanded image and pixels in the second image, using the inverse process of determining P3 from P1.

[0217] Continuing with the above example, assuming that P2 is any point in the second expanded image, the straight line equation of OP2 can be determined by point O and point P2. By solving the plane equation of the second image, the intersection of the straight line equation and the plane equation of OP2 can be solved as mapping point P4.

[0218] The plane equation can be solved by knowing a point on the plane and the normal vector of the point. The intersection of the optical axis of the optical center O1 and the phase plane is O c (x c ,y c ,z c ):

[0219] O c =OO1+f*n=(x1+f*cosαcosβ, y1+f*cosαsinβ, z1+f*sinα).

[0220] Wherein, f is the physical length of the focal length: f = fx * dx. The normal vector n of the plane where the second image is located = (cosαcosβ, cosαsinβ, sinα).

[0221] Then the plane equation of the second image is:

[0222] (cosβcosγ, cosβsinγ, sinβ)*(x1+f*cosβcosγ-x, y1+f*cosβsinγ-y, y1+sinβ-z)=0.

[0223] Therefore, by solving the linear equation of OP2 and the plane equation of the second image simultaneously, the intersection of the linear equation and the plane equation of OP2 can be obtained as the three-dimensional coordinates of the mapping point P4 in the world coordinate system.

[0224] According to the above formula The coordinates of the mapped point P4 in the world coordinate system can be converted to the coordinates of point P4 in the camera coordinate system. The x value in the camera coordinate system is divided by dx to obtain the x value of point P4 in the second image. The y value in the camera coordinate system is divided by dy to obtain the y value of point P4 in the second image. That is, the three-dimensional coordinates in the camera coordinate system are converted to two-dimensional coordinates in the image coordinate system, thereby obtaining the pixel coordinates of the mapped point in the second image. The pixel value of the point in the second image is then assigned to the pixel value of the corresponding point in the second expanded image, so that the pixel values of the second expanded image are closer to the second image.

[0225] After the mapping of each pixel point in the second image is completed, the second projection area can be expanded according to the longitude and latitude to obtain a second expanded image. For example, the second expanded image is as follows: Figure 18 Similarly, a first projection area corresponding to the first image is determined, and the first projection area is expanded after mapping to obtain a first expanded image.

[0226] Optionally, if a server (e.g., a cloud server) stores an image (hereinafter referred to as a historical image) at the same location as the first image and / or the second image, the image processing device can also obtain the historical image from the server. The historical image may be taken at different times and lighting conditions, and the acquisition time of the historical image is before the acquisition time of the first image and / or the second image. The image processing device can also obtain the historical image and the historical posture of the historical device when the historical image was acquired (e.g., the position and direction of the historical device when the historical image was acquired) from the server. The historical image is then placed in the aforementioned spherical model using the historical posture, and a historical expanded image corresponding to the historical image is acquired similarly to the acquisition of the second expanded image of the second image. Among them, the historical device, the first device, and the second device are devices with image acquisition functions. The historical device, the first device, and the second device can be the same device or different devices, and the specific details are not limited here.

[0227] Optionally, if the server stores a large number of images, the historical images that need to be fused can be determined by matching based on the attributes. For example, the historical image can be determined based on the posture (for example, if the distance between the position of the device when collecting the image and the position of the first device when collecting the first image or the position of the second device when collecting the second image is less than a certain threshold, then the image is determined to be a historical image, indicating that the image and the first image and / or the second image are likely to be in the same scene). The historical image can also be determined based on the weather type (for example, determining that the image with the same weather type as the first image or the second image is a historical image). The historical image can also be determined based on attributes such as time intervals (for example, determining that the image with a time interval within a certain threshold is a historical image, or the overlapping angle / area of the perspective). Of course, the historical image can also be selected from the server by a combination of the above attributes (for example, the distance between the position of the device when collecting the image and the position of the first device when collecting the first image or the position of the second device when collecting the second image is less than a certain threshold, and the overlapping area of the first image and the second image is greater than a certain threshold). The specific selection method is not limited here.

[0228] Step 1303: The image processing device fuses the first expanded image and the second expanded image to obtain a third image.

[0229] After acquiring the first expanded image and the second expanded image, the first expanded image and the second expanded image are fused to obtain a third image with a larger field of view (or viewing angle range) (for example, the third image is as follows Figure 19 shown).

[0230] For example, see Figure 20 , I1 is a portion of the first expanded image, and I2 is a portion of the second expanded image. Since the first image and the second image are images captured from different viewing angles of the same scene, there will be an overlapping area between the first expanded image and the second expanded image.

[0231] Optionally, different fusion methods can be used depending on the shape of the overlapping area. For example, if the width of the overlapping area is greater than its height, vertical fusion is performed; if the width of the overlapping area is less than or equal to its height, horizontal fusion is performed. The principles of vertical fusion are similar to those of horizontal fusion, so the following description will only use vertical fusion as an example.

[0232] Determine the height Δh of each column of pixels in the overlapping area and the pixel height threshold δ of the fusion area. If the fusion area is too large, the blurred area will be too large. If the fusion area is too small, the fusion will not be sufficient. The setting of δ needs to be moderate. For example, the value range of δ is 0 to 150. δ is set according to actual needs and is not limited here. The actual height of the fusion area can be min(δ, Δh). For example, Figure 20The height of the fusion area in the middle. The Y coordinate of the upper boundary point of the column where a point P(x, y) in the fusion area is located is y min , the Y coordinate of the lower boundary point is y max , then the pixel value I of point P can be determined by the following formula:

[0233] I=α*I1+(1-α)*I2;

[0234]

[0235] In the above formula, I1 is the pixel value of point P in the first expanded image, and I2 is the pixel value of point P in the second expanded image.

[0236] It is understandable that the above formula is only an example, and the pixel value of point P can be obtained by the above weighted method or the average method, and the specific method is not limited here.

[0237] Optionally, the image processing device fuses the first expanded image, the second expanded image, and the historical expanded image to obtain a third image. Alternatively, the image processing device fuses the historical expanded image with the third image to obtain a fourth image, and uses the fourth image as the third image to perform subsequent steps.

[0238] For example, Figure 21 For historical images, Figure 22 This is the fourth or third image after the historical expansion images are fused.

[0239] Optionally, when there are multiple images, subsequent image processing is performed in a similar manner to the previous image, and a third image is obtained by fusion. Of course, if the acquisition time of the subsequent image to be fused is within a certain threshold range with the first image, the subsequent image can be directly fused with the target fused image again to obtain a fused image, and the fused image can be used as the third image to perform subsequent steps.

[0240] Optionally, the above method is an operation performed on a ball model established based on the first image. It is understandable that the above method is not necessarily an operation performed on a ball model established based on the first image. Since the camera may undergo rotation and displacement posture changes at the same time, when the displacement change between the device that captures one image and the device that captures another image is greater than or equal to a second threshold, it will affect the subsequent fusion quality. The fused image can be cleared, and a new ball model can be rebuilt based on the current image (i.e., the new position of the camera), and then the above mapping and fusion steps can be performed on the new ball model for subsequent images.

[0241] Optionally, the fusion of multi-view images can also be controlled based on the difference in the rotation angle of the device that captured the two images and the size of the third threshold. Optionally, the user can use the user interface (UI) to set the image processing for the above historical images, displacement transformation, and perspective transformation. That is, the user inputs instructions and the image processing device executes the corresponding steps. Figure 23 As shown, cloud data fusion allows users to select whether to integrate image fusion with historical images from the server. High quality and high performance in multi-view fusion allow users to set the second and third thresholds mentioned above. The second threshold corresponding to high performance is greater than the second threshold corresponding to high quality, and the third threshold corresponding to high performance is greater than the third threshold corresponding to high quality. In other words, when using high performance, the second and third thresholds are slightly larger, eliminating more images and increasing the efficiency of the image processing device. When using high quality, the second and third thresholds are slightly smaller, resulting in a more accurate third image.

[0242] Exemplarily, the second threshold corresponding to high performance is 2 meters, and the second threshold corresponding to high quality is 1 meter. Assume that the user inputs an instruction to determine high performance, that is, the second threshold is 2 meters. If the distance between the position when the device captures one image and the position when the device captures another image is less than 2 meters, operations such as projection and fusion can be performed on the two images. If the distance between the position when the device captures one image and the position when the device captures another image is greater than or equal to 2 meters, the previous image can be cleared, and a new spherical model can be reconstructed based on the current image (that is, the new position of the camera), and then the above-mentioned mapping, fusion and other operations are performed on the new spherical model for subsequent images.

[0243] For example, the third threshold corresponding to high performance is 30 degrees, and the third threshold corresponding to high quality is 5 degrees. Assume that the user inputs an instruction to determine high quality, that is, the third threshold is 5 degrees. If the difference in rotation angle is greater than or equal to 5 degrees, mapping, fusion, and other operations are performed on the two images. If the difference in rotation angle is less than 5 degrees, the current image is discarded, and the mapping, fusion, and other operations are performed on the subsequent image based on the sphere model constructed from the previous image.

[0244] For example, the user can select in many ways through the UI. In one possible implementation, Figure 24 As shown, the user selects cloud data fusion, that is, selects to fuse historical images, and then executes the above steps of fusing historical images. In another possible implementation method, as Figure 25 As shown in FIG, the user selects high performance, then the second threshold and the third threshold are slightly larger. In another possible implementation, as shown in FIG. Figure 26As shown, if the user selects high quality, the second threshold and the third threshold are slightly smaller. If the user selects cloud data fusion, that is, selects to fuse historical images, the above steps of fusing historical images are performed. Of course, there are many ways for users to select, and the image processing device performs corresponding operations according to the user's settings. Figures 23 to 26 These are just a few examples.

[0245] Step 1304: The image processing apparatus inputs the third network into the trained image prediction network to perform image prediction and obtain a predicted image.

[0246] The image prediction model used in this step can be the one Figure 5 It can be constructed by the method in , or it can be constructed by other methods, which are not limited here.

[0247] One possible implementation method is Figure 27 As shown, after the image processing device obtains the third image, it inputs the third image into the trained image prediction network to perform image prediction to obtain a predicted image.

[0248] Another possible implementation method is Figure 28 As shown, after the image processing device obtains the third image, the third image and the mask image corresponding to the third image are input into the trained image prediction network for image prediction to obtain a predicted image.

[0249] For example, continuing with the above example, Figure 29 To predict the image, Figure 30 for Figure 27 An example of .

[0250] After the image processing device obtains the predicted image, it can use the predicted image to perform operations such as ambient light rendering on virtual objects in the same scene. The virtual object is the object for which ambient light rendering is required. The virtual object can be a virtual object or a virtual scene, and the specific details are not limited here. The following describes ambient light rendering using the spherical harmonic coefficients of the predicted image through steps 1305 and 1306. It is understood that the predicted image can also be used for ambient light rendering using methods such as spherical Gaussian or image-based lighting (IBL), and the specific details are not limited here.

[0251] Step 1305: The image processing apparatus obtains the spherical harmonic coefficients of the predicted image. This step is optional.

[0252] Optionally, after acquiring the predicted image, the image processing device maps the predicted image to a unit space mapping model, for example, a unit sphere model ( Figure 31AAs shown), the spherical harmonic coefficients of the predicted image are obtained, and the spherical harmonic coefficients can be used to describe the ambient light data of the predicted image.

[0253] Spherical harmonic lighting actually samples the ambient light into several coefficients (i.e., several spherical harmonic coefficients), and then uses these spherical harmonic coefficients to restore the lighting during rendering. This process can be regarded as a simplification of the ambient light. Every time a pixel is sampled, the corresponding spherical harmonic basis is calculated, and the pixel is multiplied by the corresponding spherical harmonic basis and then summed. This is equivalent to the integration of each spherical harmonic basis over all pixels. However, in order to obtain the average light intensity on the spherical harmonic basis, it is also necessary to multiply the value obtained by the integration by the solid angle and divide it by the total pixels. Simply put, the spherical harmonic coefficients are obtained using the following formula:

[0254]

[0255] Among them, i is the spherical harmonic coefficient index, N is the number of sampling points, and for order n, the spherical harmonic coefficients corresponding to a single image channel are n 2 For a three-channel environment map, the spherical harmonic coefficient is 3n 2 , light(x j ) is the RGB value of the sample point. i (x j ) is a spherical harmonic basis, for order n, y i (x j ) is divided into multiple bands: 0, ..., l, ..., n-1. Band l includes 2l+1 spherical harmonics. The spherical harmonics can be expressed by the formula Calculated, where i=l(l+1)+m, m takes the following values: -l, -(l-1),…, 0,…, (l-1), l.

[0256]

[0257]

[0258]

[0259] in, Equivalent to a scaling factor for normalization, P l m (x) is the Legendre polynomial, meaning P l (x)The m-th derivative with respect to x.

[0260]

[0261] If the image processing device maps the predicted image onto the unit sphere model, a set of spherical harmonic coefficients can be obtained using the above formula. For example, the spherical harmonic coefficients are of third order, and the three RGB channels each include 9 coefficients, for a total of 27 coefficients.

[0262] Step 1306: The image processing apparatus uses the spherical harmonic coefficients to perform ambient light rendering on the virtual object. This step is optional.

[0263] The following takes the R channel in RGB as an example for schematic explanation.

[0264] Calculate the RGB value of any point P on the virtual object. The R value in the RGB value is equal to the sum of the first value, the second value, and the third value. First, determine that the normal vector of the point P is (x, y, z, 1). Then the first value is equal to the dot product of the vector consisting of the first four spherical harmonic coefficients of the R channel and the normal vector. The second value is equal to the dot product of the vector consisting of the 5th to 8th spherical harmonic coefficients of the R channel and VB, where VB is (xy, yz, zz, zx). The third value is equal to the product of the 9th spherical harmonic coefficient of the R channel and VC, where VC is x. 2 -y 2 Through the above method, we can get the R channel value corresponding to point P. The calculation method of the G channel value and the B channel value is similar to that of the R channel, so we will not repeat it here.

[0265] After determining the RGB value of point P, the RGB values of other points on the sphere are calculated similarly to point P, thereby completing the rendering of the virtual object. The virtual object can be a virtual object or a virtual scene in the scene, and the specific details are not limited here.

[0266] For example, the spherical harmonic lighting effect is as follows Figure 31B .

[0267] In one possible implementation, the image processing method includes steps 1301 to 1304. In another possible implementation, the image processing method includes steps 1301 to 1306.

[0268] Optionally, Figure 13 The illustrated method can be executed in a loop. That is, after constructing the spatial mapping model, multiple images can be acquired and mapped to the spatial mapping model to obtain multiple expanded images. These images are then fused to obtain a third image to be input into the image prediction network. In other words, the third image can be obtained by fusing two images from different perspectives, or by fusing multiple images from different perspectives.

[0269] Optionally, after the third image is fused and before it is input into the image prediction network, the third image can be continuously updated, that is, the third image of the fusion target and the subsequent expanded image (the method of obtaining the subsequent expanded image is similar to the method of obtaining the aforementioned second expanded image) to obtain a new third image, and then the new third image is input into the image prediction network for prediction to obtain a predicted image.

[0270] In an embodiment of the present application, the predicted image obtained after inputting the image prediction network by mapping and fusing the expanded images of multiple perspectives through a spatial mapping model has more textures, thereby improving the quality of the ambient light rendering of the virtual object in the subsequent scene. For example, the realism of the virtual object rendering is enhanced. Furthermore, the input image can also refer to the image of the same position stored by the server (i.e., the texture information of a certain position in the scene). And the user can flexibly set the fusion method through the user interface, for example, whether it is high performance or high quality.

[0271] Corresponding to the method provided in the above method embodiment, the present application embodiment also provides a corresponding device, including a module for executing the corresponding module of the above embodiment. The module can be software, hardware, or a combination of software and hardware.

[0272] See also Figure 32 In one embodiment of the image processing device of the present application, the image processing device may be a local device (e.g., a mobile phone, a camera, etc.) or a cloud device. The image processing device includes:

[0273] An acquisition unit 3201 is configured to acquire a first image and a second image, where the first image and the second image are images captured from different viewing angles of the same scene.

[0274] A mapping unit 3202 is configured to perform mapping processing on the first image and the second image based on a spatial mapping model to obtain a first expanded image and a second expanded image;

[0275] A fusion unit 3203 is configured to fuse the first expanded image and the second expanded image to obtain a third image;

[0276] The prediction unit 3204 is configured to input the third image into a trained image prediction network for image prediction to obtain a predicted image, which is used for ambient light rendering of the virtual object in the aforementioned scene.

[0277] In this embodiment, the operations performed by each unit in the image processing device are the same as those in the aforementioned Figures 5 to 13 The description in the illustrated embodiment is similar and will not be repeated here.

[0278] In this embodiment, the multi-perspective image is mapped into a spatial mapping model by the mapping unit 3202, and the expanded images of multiple perspectives are fused by the fusion unit 3203 so that the predicted image obtained after the prediction unit 3204 inputs the image prediction network has more texture, thereby improving the subsequent ambient light rendering quality of the virtual object in the scene.

[0279] See also Figure 33In another embodiment of the image processing device in the present application, the image processing device may be a local device (e.g., a mobile phone, a camera, etc.) or a cloud device. The image processing device includes:

[0280] An acquisition unit 3301 is configured to acquire a first image and a second image, where the first image and the second image are images captured from different viewing angles of the same scene.

[0281] A mapping unit 3302 is configured to perform mapping processing on the first image and the second image based on a spatial mapping model to obtain a first expanded image and a second expanded image;

[0282] A fusion unit 3303 is configured to fuse the first expanded image and the second expanded image to obtain a third image;

[0283] The prediction unit 3304 is configured to input the third image into a trained image prediction network for image prediction to obtain a predicted image, which is used for ambient light rendering of the virtual object in the aforementioned scene.

[0284] The mapping unit 3302 includes:

[0285] A construction subunit 33021 is configured to construct a spatial mapping model based on an optical center of a first device, where the first device is a device for acquiring a first image and the second device is a device for acquiring a second image;

[0286] a determining subunit 33022, configured to determine a first projection area of the first image on the spatial mapping model according to a first pose of the first device, where the first pose is a pose of the first device when the first image is captured;

[0287] A mapping subunit 33023 is configured to map each pixel in the first image to a first projection area to obtain a first expanded image;

[0288] The determining subunit 33022 is further configured to determine a second projection area of the second image on the spatial mapping model according to a second posture of the second device, where the second posture is a posture of the second device when the second image is captured;

[0289] The mapping subunit 33023 is further configured to map each pixel in the second image into the second projection area to obtain a second expanded image.

[0290] The acquisition unit 3301 is further configured to acquire a historical image and a historical position and posture of the historical image from the server, wherein the historical image is acquired before the first image or the second image, the historical position and posture are the positions of the historical device when the historical image is acquired, and the historical image stores an image at the same position as the first image and / or the second image;

[0291] The acquisition unit 3301 is further used to obtain the spherical harmonic coefficients of the predicted image;

[0292] The image processing device in this embodiment further includes:

[0293] The rendering unit 3305 is used to perform ambient light rendering on the virtual object using spherical harmonic coefficients.

[0294] The determination subunit 33022 is further configured to determine the historical projection area of the historical image on the spatial mapping model according to the historical posture.

[0295] The mapping subunit 33023 is further used to map each pixel in the historical image to the historical projection area to obtain a historical expanded image;

[0296] The fusion unit 3303 is specifically configured to fuse the first expanded image, the second expanded image, and the historical expanded image to obtain a third image.

[0297] In this embodiment, the operations performed by each unit in the image processing device are the same as those in the aforementioned Figures 5 to 13 The description in the illustrated embodiment is similar and will not be repeated here.

[0298] In this embodiment, the multi-view image is mapped to a spatial mapping model by the mapping unit 3302, and the expanded images of multiple perspectives are fused by the fusion unit 3303 so that the predicted image obtained after the prediction unit 3304 inputs the image prediction network has more texture, thereby improving the quality of the ambient light rendering of the virtual object in the subsequent scene. For example, the realism of the virtual object rendering is enhanced. Furthermore, the input image can also refer to the image of the same position stored by the server (that is, the texture information of a certain position in the scene). And the user can flexibly set the fusion method through the user interface, for example, whether it is high performance or high quality.

[0299] Figure 34 Schematic diagram of the hardware structure of the image processing device provided in an embodiment of the present application. Figure 34 The image processing apparatus 3400 shown (the apparatus 3400 may be a computer device) includes a memory 3401, a processor 3402, a communication interface 3403, and a bus 3404. The memory 3401, the processor 3402, and the communication interface 3403 are connected to each other via the bus 3404.

[0300] Memory 3401 can be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). Memory 3401 can store programs. When the program stored in memory 3401 is executed by processor 3402, processor 3402 and communication interface 3403 are used to perform the various steps of the image processing method of the embodiment of the present application.

[0301] The processor 3402 can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the functions required to be performed by the units in the image processing device of the embodiment of the present application, or to execute the image processing method of the method embodiment of the present application.

[0302] The processor 3402 may also be an integrated circuit chip with signal processing capabilities. During implementation, the various steps of the image processing method of the present application may be performed by hardware integrated logic circuits or software instructions in the processor 3402. The aforementioned processor 3402 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. The various methods, steps, and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present application may be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or the like. The storage medium is located in the memory 3401, and the processor 3402 reads the information in the memory 3401 and combines its hardware to complete the functions required to be performed by the units included in the image processing device of the embodiment of the present application, or executes the image processing method of the method embodiment of the present application.

[0303] The communication interface 3403 uses a transceiver such as, but not limited to, a transceiver to enable communication between the device 3400 and other devices or a communication network. For example, training data (such as the training image described in the embodiment of the present application) can be obtained through the communication interface 3403.

[0304] The bus 3404 may include a path for transmitting information between various components of the device 3400 (eg, the memory 3401 , the processor 3402 , and the communication interface 3403 ).

[0305] It should be noted that although Figure 34 The device 3400 shown only shows a memory, a processor, and a communication interface. However, in the specific implementation process, those skilled in the art should understand that the device 3400 also includes other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the device 3400 may also include hardware devices that implement other additional functions. In addition, those skilled in the art should understand that the device 3400 may also include only the devices necessary to implement the embodiments of the present application, and does not necessarily include Figure 34 All devices shown in .

[0306] An embodiment of the present application also provides a computer program product, which, when executed on a computer, enables the computer to execute the steps executed by the aforementioned image processing apparatus, or enables the computer to execute the steps executed by the aforementioned image processing apparatus.

[0307] An embodiment of the present application further provides a computer-readable storage medium, which stores a program for performing signal processing. When the computer-readable storage medium is run on a computer, the computer executes the steps executed by the aforementioned image processing device.

[0308] The image processing device or terminal device provided in the embodiments of the present application may specifically be a chip, and the chip includes: a processing unit and a communication unit. The processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin, or a circuit. The processing unit may execute computer-executable instructions stored in the storage unit so that the chip in the image processing device executes the image processing method described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0309] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0310] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0311] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0312] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0313] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

Claims

1. An image processing method, characterized in that: include: Acquire a first image and a second image, where the first image and the second image are images captured from different viewing angles of the same scene; Performing mapping processing on the first image and the second image based on a spatial mapping model to obtain a first expanded image and a second expanded image; fusing the first expanded image and the second expanded image to obtain a third image; Inputting the third image into a trained image prediction network for image prediction to obtain a predicted image, wherein the predicted image is used for ambient light rendering of a virtual object in the scene; The mapping process is performed on the first image and the second image based on the spatial mapping model to obtain a first expanded image and a second expanded image, including: constructing the spatial mapping model according to the optical center of a first device, where the first device is a device for acquiring the first image, and the second device is a device for acquiring the second image; determining a first projection area of the first image on the spatial mapping model according to a first posture of the first device, where the first posture is a posture of the first device when the first image is captured; Mapping each pixel in the first image to the first projection area to obtain the first expanded image; determining a second projection area of the second image on the spatial mapping model according to a second posture of the second device, where the second posture is the posture of the second device when capturing the second image; Map each pixel in the second image into the second projection area to obtain the second expanded image.

2. The method according to claim 1, characterized in that The method further comprises: Acquire a historical image and a historical pose of the historical image from a server, where the historical image is acquired before the first image or the second image, the historical pose is the pose of a historical device when acquiring the historical image, and the historical image stores an image at the same position as that in the first image and / or the second image; determining a historical projection area of the historical image on the spatial mapping model according to the historical posture; Mapping each pixel in the historical image to the historical projection area to obtain a historical expanded image; The fusing the first expanded image and the second expanded image to obtain a third image includes: The first expanded image, the second expanded image and the historical expanded image are fused to obtain the third image.

3. The method according to claim 1, characterized in that The trained image prediction network is obtained by using a training image as an input of the image prediction network and training the image prediction network with a loss function value being less than a first threshold as a goal; The loss function is used to indicate the difference between an image output by the image prediction network and a third target image, where the third target image is the captured image.

4. The method according to claim 3, characterized in that The weight of the loss function is controlled by the mask image corresponding to the training image.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Obtaining spherical harmonic coefficients of the predicted image; The spherical harmonic coefficients are used to perform ambient light rendering on the virtual object.

6. The method according to any one of claims 1 to 4, characterized in that The field of view of the third image is greater than that of the first image or the second image.

7. An image processing device, characterized in that: include: An acquisition unit, configured to acquire a first image and a second image, where the first image and the second image are images captured from different viewing angles of the same scene; a mapping unit, configured to perform mapping processing on the first image and the second image based on a spatial mapping model to obtain a first expanded image and a second expanded image; a fusion unit, configured to fuse the first expanded image and the second expanded image to obtain a third image; A prediction unit, configured to input the third image into a trained image prediction network to perform image prediction to obtain a predicted image, wherein the predicted image is used for ambient light rendering of a virtual object in the scene; The mapping unit includes: a construction subunit, configured to construct the spatial mapping model according to the optical center of a first device, where the first device is a device for acquiring the first image, and a second device is a device for acquiring the second image; a determining subunit, configured to determine a first projection area of the first image on the spatial mapping model according to a first posture of the first device, where the first posture is a posture of the first device when the first image is acquired; a mapping subunit, configured to map each pixel in the first image to the first projection area to obtain the first expanded image; The determining subunit is further configured to determine a second projection area of the second image on the spatial mapping model according to a second posture of the second device, where the second posture is a posture of the second device when the second image is acquired; The mapping subunit is further configured to map each pixel point in the second image into the second projection area to obtain the second expanded image.

8. The device according to claim 7, characterized in that The acquisition unit is further configured to acquire, from a server, a historical image and a historical position and posture of the historical image, wherein the historical image is acquired before the first image or the second image, the historical position and posture are positions of historical devices when the historical image is acquired, and the historical image stores an image at the same position as that in the first image and / or the second image; The determining subunit is further configured to determine a historical projection area of the historical image on the spatial mapping model according to the historical posture; The mapping subunit is further configured to map each pixel in the historical image to the historical projection area to obtain a historical expanded image; The fusion unit is specifically configured to fuse the first expanded image, the second expanded image, and the historical expanded image to obtain the third image.

9. The device according to claim 7, characterized in that The trained image prediction network is obtained by using a training image as an input of the image prediction network and training the image prediction network with a loss function value being less than a first threshold as a goal; The loss function is used to indicate the difference between an image output by the image prediction network and a third target image, where the third target image is the captured image.

10. The device according to claim 9, characterized in that The weight of the loss function is controlled by the mask image corresponding to the training image.

11. The device according to any one of claims 7 to 10, characterized in that The acquisition unit is further configured to acquire spherical harmonic coefficients of the predicted image; The image processing device further includes: A rendering unit is used to perform ambient light rendering on a virtual object using the spherical harmonic coefficients.

12. The device according to any one of claims 7 to 10, characterized in that The field of view of the third image is greater than that of the first image or the second image.

13. An image processing device, characterized in that: include: A processor is coupled to a memory, wherein the memory is used to store programs or instructions, and when the programs or instructions are executed by the processor, the image processing device performs the method according to any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the computer is caused to perform the method according to any one of claims 1 to 6.

Images