Training method of image generation model, and new perspective image generation method and device

Abstract

A training method of an image generation model, according to the residual color, the image generation model is trained, the residual color belongs to low frequency information, easy to represent and memory, therefore can improve the definition of the new view image generated by the image generation model. The embodiment method of the application comprises: determining at least one reference image according to the position and view direction of the target observation point. Then according to the reference color of the space position of the light from the target observation point and the real color of the pixel in the target observation point corresponding view image in at least one reference image, the residual color of the space position is determined, and finally the image generation model is trained according to the residual color.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of artificial intelligence, and in particular to a method for training an image generation model, a method and apparatus for generating a new-view image. BACKGROUND

[0002] Artificial intelligence (AI) is the use of digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use the knowledge to obtain the best results. In simple terms, artificial intelligence researches the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making. In practice, AI technology can be used for image generation, so that intelligent machines generate new-view images based on existing images of different views.

[0003] In a method for training an image generation model, points on the observed light rays are sampled, and the geometric information and texture information of the image are stored in the neural network, and a new-view view is obtained using volume rendering technology.

[0004] In this method, since the texture information belongs to high-frequency information, it is difficult for the neural network to remember and represent high-frequency information, so the new-view view obtained is relatively blurred and not clear enough. SUMMARY

[0005] Embodiments of the present application provide a method for training an image generation model, a method and apparatus for generating a new-view image. The image generation model learns the residual color of the space position passed by the light ray from any observation point, and generates a new-view image. Since the residual color belongs to low-frequency information, it is easy to represent and remember, so the clarity of the new-view image can be improved.

[0006] The first aspect of the present application provides a method for training an image generation model, comprising:

[0007] The training device can receive the position and viewing direction of the target observation point input by a person. The target observation point is any observation point for observing the observed object. Each observation point has its own position and viewing direction. Usually, the position of an observation point is represented by three-dimensional coordinates (x, y, z), and the viewing direction of an observation point is represented by .

[0008] After obtaining the position and the perspective direction of the target observation point, the training device can determine at least one reference image from the pre-input N images according to the position and the perspective direction. N is an integer greater than or equal to 2. Then, the training device can predict the reference color of the spatial position through which the light from the target observation point passes.

[0009] When training the image generation model, there is actually a perspective image corresponding to the target observation point, and the training device can obtain the ground-truth of each pixel in the perspective image. Then, the training device can determine the residual color of the spatial position through which the light from the target observation point passes according to the ground-truth and the reference color. Finally, the training device can train the image generation model using the residual color.

[0010] In the embodiments of the present application, the image generation model is trained according to the residual color, and the residual color belongs to low-frequency information and is easy to represent and remember, so that the clarity of the new perspective image obtained based on the image generation model can be improved.

[0011] In combination with the first aspect, in the first implementation manner of the first aspect of the embodiments of the present application, due to interference factors such as occlusion and illumination, the images predicted at different observation points are not completely the same, and the reference colors of the same spatial position in different reference images can also be different. Therefore, the training device can select the mode of the color of the spatial position through which the light from the target observation point passes in at least one reference image as the first reference color, which is used for subsequent determination of the residual color.

[0012] In the embodiments of the present application, selecting the mode of the color of the spatial position in at least one reference image as the first reference color can reduce the influence of interference factors on the accuracy of the image generation model to a certain extent, and improve the accuracy of the technical solution.

[0013] In combination with the first aspect or the first implementation manner of the first aspect, in the second implementation manner of the first aspect of the embodiments of the present application, the loss function for training the image generation model can be multiple.

[0014] In the case where the ground-truth is not a transparent color, the image generation model can be trained only using the loss function of the residual color.

[0015] When the true color is transparent, training the image generation model using only the residual color loss function without allowing the model to learn the true color can easily lead to overfitting. This causes points that are transparent in space to appear opaque in the predicted image, resulting in artifacts and affecting the sharpness of the generated image. Therefore, when the true color is transparent, the training device can be based on a joint network, using both the residual color loss function and the direct prediction loss function to jointly train the image generation model.

[0016] In this embodiment, multiple loss functions are used to train the image generation model, which improves the robustness of the algorithm and allows the trained image generation model to be applicable to various situations, thus enhancing the flexibility of the solution.

[0017] In combination with the first aspect and any of the first to second implementations of the first aspect, in the third implementation of the first aspect of this application, the training device can continuously optimize the image generation model based on the training results of the previous training cycle, so that the image generation model is closer to the real situation. The optimization can be performed in the following ways.

[0018] After training the image generation model for one iteration, the training device can obtain a new perspective image corresponding to the target observation point. Then, by comparing the new perspective image with each of the at least one reference image used during training, a second reference color is determined, which is then used as the first reference color for training the image generation model in the next iteration.

[0019] In conjunction with the third implementation of the first aspect, in the fourth implementation of the first aspect of this application, the training device can determine the second reference color in the following manner.

[0020] First, the training device can select any pixel in the new viewpoint image obtained in the previous iteration as a reference point for the target observation point. Then, it selects an image patch of the same pixel size centered on this reference point as the basis for comparison. By comparing the similarity of this image patch in the new viewpoint image and the reference image, the second reference color is determined. Several situations may occur:

[0021] If the similarity of image patches of the same pixel size centered on the reference point meets the preset conditions in the new view image and each reference image, it means that the reference point is not occluded in at least one reference image, and the first reference color used in the previous training cycle can continue to be used in subsequent iteration cycles.

[0022] If in at least one reference image, there is a part of the reference image such that the similarity of the image block in the new view image and the part of the reference image does not satisfy the preset condition, it means that the reference point is occluded in these reference images. Therefore, the training device can determine that the second reference color is the mode of the color of the reference point in the reference image satisfying the preset condition.

[0023] If the similarity of the image block in the new view image and each of the at least one reference image does not satisfy the preset condition, it means that the reference point is occluded in these reference images, and the training device can determine the color value of the second reference color as 0.

[0024] In the embodiments of the present application, the training device compares the view image of the target observation point predicted through the last iteration period and the at least one reference image used during training, so as to remove the inappropriate parameter value used for generating the training image model in the last iteration period, thereby reducing the influence of occlusion on the new view image, improving the robustness of the algorithm, and improving the accuracy of the technical solution.

[0025] In combination with any one of the first aspect, the first to fourth implementation manners of the first aspect, in the fifth implementation manner of the first aspect of the embodiments of the present application, the training device can determine at least one reference observation point according to the position and the view direction of the target observation point, and then determine the image corresponding to the at least one reference observation point as the reference image.

[0026] Wherein, the distance between each reference observation point and the target observation point needs to satisfy a preset condition. Since the closer the positions of two points are, the more similar the view directions are, the higher the similarity of the images observed by the two points will be, therefore, the distance here is determined by the positions and the view directions of the two points, that is, the positions of the two points need to satisfy the preset condition, and the view directions of the two points also need to satisfy the preset condition. Satisfying the preset condition can be less than or equal to a preset threshold.

[0027] In the embodiments of the present application, the reference observation point is determined by the position and the view direction, and the accuracy of the new view image obtained according to the image generation model is within the allowable error range under the condition of satisfying the preset condition, thereby improving the accuracy of the scheme.

[0028] In combination with any one of the first aspect, the first to fifth implementation manners of the first aspect, in the sixth implementation manner of the first aspect of the embodiments of the present application, the loss function of the residual color can be:

[0029]

[0030] Wherein,

[0031]

[0032] is used to represent the new view image predicted according to the residual color and the first reference color, is used to represent the first reference color of each spatial point, is used to represent the residual color of each spatial point, i is used to represent the opacity of a spatial point in the spatial position, i is used to represent the distance between spatial points on a light ray, and C(r) is used to represent the true color.

[0033] The loss function of the residual color is used to make the new view image predicted according to the first reference color and the residual color as close as possible to the true image.

[0034] With any one of the first aspect, the first to sixth implementation manners of the first aspect, in the seventh implementation manner of the first aspect of the embodiments of the present application, the loss function of the direct prediction can be:

[0035]

[0036] wherein,

[0037]

[0038] is used to represent the color of the pixel in the view image corresponding to the target observation point predicted directly, i is used to represent the true color of each spatial point, i is used to represent the opacity of a spatial point in the spatial position, i is used to represent the distance between spatial points on a light ray. The loss function of the direct prediction is used to make the predicted new view image as close as possible to the true image in the case of only learning the true color.

[0039] The second aspect of the embodiments of the present application provides a method for generating a new view image, comprising:

[0040] The virtual observation point is actually an observation point that has not observed the observed object, and can be randomly selected by human. After the virtual observation point is selected by human, the execution device can receive the position and the view direction of the virtual observation point input by human.

[0041] After the execution device obtains the position and the view direction of the virtual observation point, the position and the view direction can be input into the image generation model to obtain the residual color of the spatial position passed by the light from the virtual observation point. Then, the new view image corresponding to the virtual observation point is generated in combination with the obtained reference color. The reference color is determined according to at least one reference image.

[0042] In the embodiments of the present application, the image generation model is trained according to the residual color, the residual color belongs to low-frequency information, is easy to represent and remember, and therefore the clarity of the new view image obtained based on the image generation model can be improved.

[0043] In combination with the second aspect, in the first implementation manner of the second aspect of the embodiments of the present application, the image generation model can be trained according to the loss function of the residual color. The image generation model can also be trained according to the loss function of the residual color and the loss function of direct prediction. The image generation model trained according to the loss function of the residual color and the loss function of direct prediction can predict the new view image more accurately.

[0044] In the embodiments of the present application, the image generation model used by the execution device can be an image generation model trained using multiple loss functions, which improves the clarity of the generated image.

[0045] In combination with the second aspect or the first implementation manner of the second aspect, in the second implementation manner of the second aspect of the embodiments of the present application, the reference color includes a first reference color, and the first reference color refers to the mode of the color of the space position through which the light from the virtual observation point passes. The execution device can obtain the first reference color in the manner of receiving the first reference color sent by the training device.

[0046] In combination with the second aspect or the first implementation manner of the second aspect, in the third implementation manner of the second aspect of the embodiments of the present application, the execution device does not obtain the first reference color from the training device, but can determine the first reference color according to the position and the view direction of the virtual observation point. The determination process can be as follows:

[0047] The execution device can determine at least one reference observation point according to the position and the view direction of the virtual observation point, and then determine the reference image corresponding to each reference observation point in the at least one reference observation point. The distance between the reference observation point and the virtual observation point needs to meet a preset condition. As long as the position or the view direction of two observation points is different, the two points are different observation points. Since the closer the positions of two points are, the more similar the view directions are, the more similar the images observed by the two points are, therefore, the distance is determined by the position and the view direction of the two points, that is, the position between the two points meets the preset condition, and the view direction between the two points also meets the preset condition. The preset condition can be less than or equal to a preset threshold.

[0048] In the embodiments of the present application, the reference observation point is determined by the position and the direction of the view angle, and in the case of meeting the preset condition, the accuracy of the new view angle image obtained according to the image generation model is within the allowable error range, thereby improving the accuracy of the scheme.

[0049] With any one of the first to third implementation manners of the second aspect, in the fourth implementation manner of the second aspect of the embodiments of the present application, the loss function of the residual color can be:

[0050]

[0051] wherein,

[0052]

[0053] is used to represent the color of the pixel in the view angle image corresponding to the target observation point predicted directly, represents the first reference color of each spatial point, represents the residual color of each spatial point, and i represents the opacity of a spatial point in the spatial position, i represents the distance of each spatial point on a light ray, and C(r) is used to represent the true color.

[0054] The loss function of the residual color is to make the new view angle image predicted according to the first reference color and the residual color as close to the true image as possible.

[0055] With any one of the first to fourth implementation manners of the second aspect, in the fifth implementation manner of the second aspect of the embodiments of the present application, the loss function of the direct prediction can be:

[0056]

[0057] wherein,

[0058]

[0059] is used to represent the color of the pixel in the view angle image corresponding to the target observation point predicted directly, i represents the true color of each spatial point, i represents the opacity of a spatial point in the spatial position, i represents the distance of each spatial point on a light ray. The loss function of the direct prediction is to make the predicted new view angle image as close to the true image as possible in the case of only learning the true color.

[0060] The third aspect of the embodiments of the present application provides a training device of an image generation model, comprising:

[0061] The determining unit is configured to determine a position and a perspective direction of the target observation point, and then determine at least one reference image from the N input images according to the position and the perspective direction of the target observation point, where N is an integer greater than or equal to 2. Then, the reference color of the spatial position is determined according to the at least one reference image. The spatial position is a position through which a light ray from the target observation point passes.

[0062] The acquiring unit is configured to acquire the real color of the pixel in the perspective image corresponding to the target observation point.

[0063] The determining unit is further configured to determine the residual color of the spatial position according to the reference color and the real color.

[0064] The processing unit is configured to train the image generation model according to the residual color.

[0065] The beneficial effects of the present aspect are similar to those of the first aspect, and details are shown in the first aspect, which will not be repeated here.

[0066] In the first implementation manner of the third aspect, the reference color includes: a first reference color, and the first reference color is a mode of the color of the position through which the light ray from the target observation point passes.

[0067] In the second implementation manner of the third aspect, the processing unit is configured to:

[0068] If the real color is not a transparent color, the image generation model is trained according to the loss function of the residual color.

[0069] If the real color is a transparent color, a loss function of direct prediction is acquired, and the image generation model is trained according to the loss function of the residual color and the loss function of direct prediction.

[0070] In the third implementation manner of the third aspect, the acquiring unit is further configured to acquire a new perspective image corresponding to the target observation point, where the new perspective image is predicted by the execution device according to the image generation model.

[0071] The determining unit is further configured to determine a second reference color according to the new perspective image and each reference image in the at least one reference image, and then take the second reference color as the first reference color.

[0072] In the fourth implementation manner of the third aspect, the determining unit is specifically configured to:

[0073] Determine any pixel point in the new view image corresponding to the target observation point as the reference point.

[0074] If the similarity of the image block of the same pixel size centered on the reference point in the new view image and each reference image satisfies the preset condition, determine the second reference color as the first reference color.

[0075] If the similarity of the image block of the same pixel size centered on the reference point in the new view image and the part of the reference images in at least one reference image does not satisfy the preset condition, determine the second reference color as the mode of the reference color in the spatial position of the reference image satisfying the preset condition.

[0076] If the similarity relationship of the image block of the same pixel size centered on the reference point in the new view image and each reference image does not satisfy the preset condition, determine the color value of the second reference color as 0.

[0077] In the fifth implementation manner of the third aspect, the determining unit is specifically configured to:

[0078] According to the position and the view direction of the target observation point, determine at least one reference observation point, wherein the distance between each reference observation point in the at least one reference observation point and the target observation point satisfies a preset condition.

[0079] According to the at least one reference observation point, obtain at least one reference image, wherein each reference observation point in the at least one reference observation point corresponds to each reference image in the at least one reference image.

[0080] In the sixth implementation manner of the third aspect, the loss function of the residual color can be:

[0081]

[0082] wherein,

[0083]

[0084] is used to represent the new view image predicted according to the residual color and the first reference color, represents the first reference color of each spatial point, represents the residual color of each spatial point, σ i represents the opacity of a certain spatial point in the spatial position, δ i represents the distance of each spatial point on a light ray. C(r) is used to represent the true color.

[0085] The loss function of the residual color is to make the new view image predicted according to the first reference color and the residual color as close to the real image as possible.

[0086] With any one of the first to sixth implementation manners of the third aspect, in a seventh implementation manner of the third aspect of the embodiments of the present application, the loss function of the direct prediction can be:

[0087]

[0088] wherein,

[0089]

[0090] c represents the color of the pixel in the view image corresponding to the target observation point predicted directly, i σ represents the real color of each spatial point, i δ represents the opacity of a spatial point in the spatial position, i r represents the distance of each spatial point on a light ray. The loss function of the direct prediction is to make the predicted new view image as close to the real image as possible under the condition of only learning the real color.

[0091] The fourth aspect of the embodiments of the present application provides a new view image generation device, comprising:

[0092] A determination unit is configured to determine the position and the view direction of the virtual observation point.

[0093] An acquisition unit is configured to input the position and the view direction of the virtual observation point into the image generation model, and acquire the residual color of the spatial position passed by the light ray from the virtual observation point.

[0094] The acquisition unit is further configured to acquire a reference color, which is the color of the spatial position determined according to at least one reference image.

[0095] A processing unit is configured to generate a new view image corresponding to the virtual observation point according to the residual color and the reference color of the spatial position.

[0096] The beneficial effects shown in the present aspect are similar to those of the first aspect, and details are shown in the first aspect, which will not be repeated here.

[0097] With the fourth aspect, in a first implementation manner of the fourth aspect of the embodiments of the present application, the image generation model comprises: an image generation model trained according to the loss function of the residual color. Alternatively, the image generation model trained according to the loss function of the residual color and the loss function of the direct prediction.

[0098] In a second implementation manner of the fourth aspect or the first implementation manner of the fourth aspect, the reference color includes a first reference color, and the first reference color is a mode of the color of the spatial position through which the light from the virtual observation point passes.

[0099] The acquisition unit is specifically configured to receive the first reference color sent by the training device.

[0100] In a second implementation manner of the fourth aspect or the first implementation manner of the fourth aspect, the reference color includes a first reference color.

[0101] The acquisition unit is specifically configured to:

[0102] According to the position and the perspective direction of the virtual observation point, at least one reference observation point is determined, and a distance between each reference observation point in the at least one reference observation point and the virtual observation point satisfies a preset condition.

[0103] At least one reference image is determined from N reference pictures, each reference observation point in the at least one reference observation point corresponds to each reference image in the at least one reference image, and N is an integer greater than or equal to 2.

[0104] According to the at least one reference image, the first reference color is determined.

[0105] The fifth aspect of the present application provides an image processing system, including a training device and an execution device.

[0106] The training device includes a first processor and a first memory, the first processor is configured to execute the method of the first aspect, and the first memory is configured to store a training picture set, and the training picture set includes at least two images.

[0107] The execution device includes a second processor and a second memory, the second processor is configured to execute the method of the second aspect, and the second memory is configured to store a new perspective image.

[0108] The sixth aspect of the present application provides a computer readable storage medium, the computer readable storage medium stores a program, and when a computer executes the program, the method of the first aspect or the second aspect is executed.

[0109] The seventh aspect of the present application provides a computer program product, when the computer program product is executed on a computer, the computer executes the method of the first aspect or the second aspect.

[0110] The eighth aspect of the present application provides a computer device, including:

[0111] A processor, a memory, an input / output device, and a bus. The processor, the memory, the input / output device, and the bus are connected. The processor stores computer instructions, and the processor is configured to execute the computer instructions to enable the computer device to perform the following steps:

[0112] Determine the position and the perspective direction of the target observation point.

[0113] According to the position and the perspective direction, determine at least one reference image from N input images, where N is an integer greater than or equal to 2.

[0114] According to the at least one reference image, determine a reference color of a spatial position, the spatial position being a position through which a light ray from the target observation point passes.

[0115] Obtain a real color of a pixel in a perspective image corresponding to the target observation point.

[0116] According to the reference color and the real color, determine a residual color of the spatial position.

[0117] According to the residual color, train an image generation model.

[0118] The computer device is configured to perform the method of the first aspect.

[0119] The ninth aspect of the embodiments of the present application provides a computer device, comprising:

[0120] A processor, a memory, an input / output device, and a bus. The processor, the memory, the input / output device, and the bus are connected. The processor stores computer instructions, and the processor is configured to execute the computer instructions to enable the computer device to perform the following steps:

[0121] Determine the position and the perspective direction of the target observation point.

[0122] Input the position and the perspective direction into the image generation model to obtain a residual color of a spatial position through which a light ray from the virtual observation point passes.

[0123] Obtain a reference color, the reference color being a color of a spatial position determined according to at least one reference image.

[0124] According to the residual color and the reference color of the spatial position, generate a new perspective image corresponding to the virtual observation point.

[0125] The computer device is configured to perform the method of the second aspect. BRIEF DESCRIPTION OF DRAWINGS

[0126] Figure 1 is a structural schematic diagram of an artificial intelligence subject framework of the embodiments of the present application;

[0127] Figure 2 An application scenario diagram of an image processing system according to an embodiment of the present application;

[0128] Figure 3 A system architecture diagram of an image processing system according to an embodiment of the present application;

[0129] Figure 4 A flow diagram of a training method of an image generation model according to an embodiment of the present application;

[0130] Figure 5 Another flow diagram of a training method of an image generation model according to an embodiment of the present application;

[0131] Figure 6 A flow diagram of a new-viewpoint image generation method according to an embodiment of the present application;

[0132] Figure 7 A structure diagram of a training device of an image generation model according to an embodiment of the present application;

[0133] Figure 8 A structure diagram of a new-viewpoint image generation device according to an embodiment of the present application;

[0134] Figure 9 A structure diagram of an image processing system according to an embodiment of the present application. DETAILED DESCRIPTION

[0135] The embodiments of the present application provide a training method of an image generation model, a new-viewpoint image generation method and device. The image generation model learns the residual color of a space point through which a light ray passes, and generates a new-viewpoint image, thereby improving the definition of the new-viewpoint image.

[0136] First, the imaging process is briefly introduced. When a light ray passes through an object, it is affected by related factors such as the opacity of the object, and then undergoes refraction and reflection of light, and finally presents the color observed by the human eye. Using a mathematical model to simulate the process of observing an object by the human eye can be understood as the process of integrating the color of each light ray emitted by a certain observation point at each space point.

[0137] Next, related concepts that may be involved in the embodiments of the present application are explained.

[0138] (1) Loss function.

[0139] The loss function is used to measure the difference between the predicted value and the true value, and the size of the difference can well reflect the difference between the model and the actual data. The role of the training model is to make the predicted results as close to the true results as possible, so the loss function can be set to evaluate and continuously optimize the trained model. The larger the output value (loss) of the loss function, the greater the difference between the predicted results and the true results. The process of training the model is to minimize the loss as much as possible.

[0140] (2) Target observation point, reference observation point, virtual observation point.

[0141] Objectively speaking, the target observation point, the reference observation point and the virtual observation point are all a perspective for observing an object. An observation point can be simply understood as the pose of a camera. Observing the same object from different observation points may result in different images, because each observation point has its own position and viewing direction. Usually, the position of an observation point is represented as a three-dimensional coordinate point (x, y, z), and the viewing direction of an observation point includes the angles of rotation around the axes of the observation point, which can include three angles. Since some observation points are not sensitive to the rotation direction of one of the axes, the viewing direction of an observation point can also be represented in two dimensions, such as The specific embodiments are not limited here.

[0142] In the embodiments of the present application, the target observation point refers to an arbitrarily selected perspective by a person when training an image generation model. The reference observation point refers to an observation point that has observed the object to be observed and generated a corresponding perspective image. The virtual observation point refers to an observation point that has not observed the object to be observed before, and there is no new perspective image corresponding to the virtual observation point in the existing image set.

[0143] (3) Reference color.

[0144] The reference color refers to the color of the space position through which the light from the target observation point passes, which needs to be determined according to the reference image. The process of determining the reference color is actually a prediction process. Under the condition of obtaining the reference image,

[0145] The overall workflow of the artificial intelligence system will be described below. Please refer to Figure 1 , Figure 1 The structure of the artificial intelligence main body framework of the embodiments of the present application is shown in the figure, which describes the overall workflow of the artificial intelligence system and is suitable for general artificial intelligence field requirements.

[0146] Figure 1 The embodiments shown include two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis).

[0147] The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data undergoes a condensation process of "data-information-knowledge-wisdom".

[0148] The "IT value chain" is the industrial ecological process from the underlying infrastructure of human intelligence, information (provision and processing technology implementation) to the system, reflecting the value brought by artificial intelligence to the information technology industry.

[0149] (1) Infrastructure.

[0150] The infrastructure provides computing power support for the artificial intelligence system, realizes communication with the outside world, and realizes support through the underlying platform. The infrastructure communicates with the outside world through sensors; computing power is provided by intelligent chips (CPU, NPU, GPU, ASIC, FPGA, etc. Hardware acceleration chips); the underlying platform includes distributed computing framework and network-related platform support and support, which can include cloud storage and computing, interconnection network, etc. For example, sensors and external communication obtain data, which are provided to intelligent chips in the distributed computing system provided by the underlying platform for calculation.

[0151] (2) Data.

[0152] The data on the upper layer of the infrastructure is used to represent the data source in the field of artificial intelligence. Data involves graphics, images, speech, text, and also involves Internet of Things data of traditional devices, including business data of existing systems and sensing data such as force, displacement, liquid level, temperature, humidity, etc.

[0153] (3) Data processing.

[0154] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.

[0155] Among them, machine learning and deep learning can symbolize and formalize intelligent information modeling, extraction, preprocessing, training, etc.

[0156] Reasoning refers to the process of simulating human intelligent reasoning methods in computers or intelligent systems, using formalized information to solve problems with machine thinking and solving problems according to reasoning control strategies, and the typical function is search and matching.

[0157] Decision-making refers to the process of decision-making after intelligent information reasoning, which usually provides functions such as classification, sorting, prediction, etc.

[0158] (4) General capabilities.

[0159] After the data is processed by the above-mentioned data processing, some general capabilities can be formed based on the results of the data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0160] (5) Intelligent product and industry application.

[0161] Intelligent product and industry application refers to the product and application of artificial intelligence system in various fields, which is the packaging of the overall solution of artificial intelligence, and realizes the application of intelligent information decision productization. The application fields mainly include intelligent manufacturing, intelligent transportation, smart home, intelligent medical treatment, intelligent security, automatic driving, safe city, intelligent terminal, etc.

[0162] The application scenarios of the embodiments of the present application will be briefly described below. Please refer to Figure 2 , Figure 2 for an application scenario diagram of the image processing system of the embodiments of the present application.

[0163] A communication connection is established between the camera 201, the processor 202 and the smart phone 203. The processor 202 can receive the photos or videos sent by the camera 201. Each frame of picture in the photos and videos can be regarded as an image in the training picture set. The processor 202 determines the reference image corresponding to the reference observation point from the received images according to the position and the direction of the virtual observation point. The new view image is generated by using the trained image generation model. Then, the processor 202 can integrate the new view image and send it to the smart phone 203, or only send the new view image without integration to the smart phone 203. The integrated image can be a 360° panoramic photo or a 720° panoramic photo, which is selected according to the actual application needs, and the specific limitation is not made here. The smart phone 203 displays the received image.

[0164] It should be noted that, Figure 2The shown embodiment is only one application scenario of the image processing system of the present embodiment. In actual applications, the camera 201 can also be replaced by other devices, which can be a notebook computer or a tablet computer, as long as the device has a camera function and can take photos or videos. The processor 202 does not necessarily exist outside the smart phone 203, but can be a processor in the smart phone 203. The smart phone 203 can also be replaced by other devices, which can be a virtual reality (VR) device or an augmented reality (AR) device, or a mixed reality (MR) device, as long as the device can display the new view image. The specific implementation is not limited here.

[0165] The image processing system provided by the present embodiment is described below. Please refer to Figure 3 , Figure 3 The image processing system provided by the present embodiment is described below. Please refer to Figure 3 The image processing system shown in the embodiment includes an execution device 310, a training device 320, a database 330, a client device 340 and a data storage system 350, wherein the execution device 310 includes a computing module 311.

[0166] The database 330 stores a set of training images, so that the training device 320 can predict the reference color from the position of the light ray according to at least one reference image in the set of training images. The training device 320 is configured to generate an image generation model 301 and iteratively train the image generation model 301 using at least one image in the database 330, so as to obtain an optimal image generation model 301. After the execution device 310 generates a new view image according to the image generation model 301, the new view image can be sent to different devices, which can be sent to the client device 340 or the data storage system 350. The specific implementation is not limited here.

[0167] The image generation model 301 can be applied to different devices, such as mobile phones, tablets, notebook computers, VR devices, AR devices, monitoring systems, etc. The specific implementation is not limited here. The training device 320 can configure the image generation model 301 in the execution device 310 by wireless communication, wired communication or removable storage device. The actual configuration method is selected according to the actual application requirements. The specific implementation is not limited here.

[0168] The training picture set in the database 330 has multiple images. When training the image generation model 301, the training device 320 determines at least one picture as a reference picture from the input training picture set according to the position and the perspective direction of the input target observation point. The multiple images in the training picture set have various forms, which can be a photo obtained by using a shooting device or at least one image in a video frame, and the specific form is not limited here. The multiple images in the training picture set have various acquisition manners, which can be acquired from the data acquisition device 360 or sent by the client device 340. The data acquisition device 360 can be a notebook computer or a camera, as long as the device has a camera function and can shoot a photo or a video, and the specific form is not limited here.

[0169] In the embodiment of the application, the client device 340 and the execution device 310 can be independent devices or an integral whole, and the specific form is not limited here. The execution device 310 is configured with an I / O interface 312 for data interaction with the client device 340. The user can input the spatial position and the perspective direction of the virtual observation point to the I / O interface 312 through the client device 340, and the execution device 310 sends the generated new perspective image to the client device 340 through the I / O interface 312 to provide the user.

[0170] It should be noted that, Figure 3 The position relationship between the devices and the elements shown in the figure does not constitute any limitation. For example, if the execution device 310 is configured in the client device 340, when the client device 340 is a mobile phone, the execution device 310 can also be a graphics processing unit (GPU) or a neural-network processing unit (NPU) in the mobile phone, and the specific form is not limited here.

[0171] The training method of the image generation model provided by the embodiment of the application is described below. Please refer to Figure 4 , Figure 4 The training method of the image generation model provided by the embodiment of the application is described below. Please refer to

[0172] 401. The training device determines the position and the perspective direction of the target observation point.

[0173] When training the image generation model, the relevant information of the target observation point needs to be input into the training device. The relevant information of the target observation point includes the position of the target observation and the perspective direction of the target observation point. In the embodiment of the present application, the position of the target observation point is a three-dimensional coordinate point (x, y, z), and the perspective direction of the observation point is two-dimensional For example, the coordinate point of the target observation point and the perspective direction are selected by artificial selection and input into the training device.

[0174] 402, the training device determines at least one reference image.

[0175] Before training the image generation model, the same object can be observed at multiple different observation points using a shooting device to obtain multiple images. The shooting device can be a smart phone, a camera or a Polaroid, as long as it has a shooting function, and the specific place is not limited. The multiple images have various forms, which can be photos obtained by the shooting device using the photo function, or each frame image in the video obtained by the shooting device using the video function, and the specific place is not limited.

[0176] The multiple images obtained by the shooting device can be referred to as a training picture set, and these images can be input into the training device to select at least one reference image from the training device. The process of selecting the reference image by the training device is described below.

[0177] The training device can determine at least one reference observation point according to the input position of the target observation point and the perspective direction of the target observation point, and then determine the reference image corresponding to each reference observation point from the training picture set. The distance between each reference observation point in the at least one reference observation point and the target observation point needs to meet the preset condition. The distance mentioned here is determined by the position and perspective direction of the observation point, because the closer the positions of two observation points and the more similar the perspective directions of two observation points, the larger the overlapping area of the perspective images corresponding to the two observation points, and the higher the similarity of the two perspective images. Therefore, the distance between the reference observation point and the target observation point needs to meet the preset condition, which includes that the positions of the two points meet the preset condition and the perspective directions of the two points meet the preset condition. The preset condition can be less than or equal to a certain preset threshold.

[0178] It should be noted that the perspective direction of each observation point can be represented by the coordinate system of the observation point itself, but when compared with other observation points, each observation point should be based on the same coordinate system, so that the comparison result has reference significance.

[0179] In the embodiments of the present application, the training device can select the reference image from the training image set within the error tolerance range, thereby improving the realizability of the scheme.

[0180] 403. The training device determines a first reference color of the spatial position according to the at least one reference image.

[0181] After the training device determines the at least one reference image, the training device can back-project the spatial point in the spatial position through which the light from the target observation point passes, to different pixel positions in the reference image, to obtain the reference color of the spatial point in the reference image. Due to the influence of factors such as occlusion and illumination, the reference colors back-projected from different observation points at the same spatial position can be different. In order to reduce the interference of these factors, the training device can take the mode of the reference color as a parameter used in the subsequent training process, that is, the first reference color. To some extent, the influence of the interference factors on the prediction result can be reduced, and the accuracy of the technical scheme of the present application can be improved.

[0182] 404. The training device obtains the real color of the pixel in the perspective image corresponding to the target observation point.

[0183] When training the image generation model, the target observation point actually observes the observed object, and there is a perspective image corresponding to the target observation point. Therefore, the real color of the pixel in the perspective image corresponding to the target observation point can be manually input to the training device in advance, and used to compare with the first reference color to determine the residual color.

[0184] It should be noted that steps 403 and 404 do not have a certain order, and step 403 can be performed first, or step 404 can be performed first, and the selection is made according to the actual application needs, and the specific place is not limited.

[0185] 405. The training device determines the residual color of the spatial position.

[0186] The residual color can be understood as a function of the position and perspective direction of the target observation point, and the parameters of the neural network, and the first reference color and the real color are related parameters of the neural network. The training device can obtain the residual color and opacity of a certain spatial point in the spatial position through which the light from the target observation point passes, by the neural network, under the condition that the position and perspective direction of the target observation point, and the first reference color and the real color are known.

[0187] 406. The training device determines whether the real color is transparent, and if so, step 408 is performed, and if not, step 407 is performed.

[0188] Since the influence of the transparency of the color on the finally generated image is different, after the training device obtains the real color, the type of the loss function used for training the image generation model can be determined according to the transparency of the real color.

[0189] It should be noted that steps 405 and 406 do not have a certain order, and step 405 can be performed first, or step 406 can be performed first, which is selected according to actual application needs, and the specific place is not limited.

[0190] 407、The training device trains the image generation model according to the loss function of the residual color.

[0191] When the training device determines that the real color is transparent, the image generation model can be trained according to the preset loss function of the residual color. The loss function of the residual color can be:

[0192]

[0193] Wherein,

[0194]

[0195] is used to represent the color of the pixel in the view image corresponding to the target observation point predicted according to the residual color and the first reference color, represents the first reference color of each spatial point, represents the residual color of each spatial point, and i represents the opacity of a spatial point in the spatial position, i represents the distance of each spatial point on a light ray. C(r) is used to represent the real color.

[0196] 408、The training device obtains the loss function of direct prediction.

[0197] When the training device determines that the real color is not transparent, the loss function of direct prediction can be obtained. The loss function of direct prediction can be:

[0198]

[0199] Wherein,

[0200]

[0201] is used to represent the color of the pixel in the view image corresponding to the target observation point predicted according to the residual color and the first reference color, i represents the real color of each spatial point, i represents the opacity of a spatial point in the spatial position, iThe distance of each spatial point on a light ray is represented. The role of the direct prediction loss function is to make the predicted new view image as close to the real image as possible under the condition of only learning the real color.

[0202] The training device can directly predict the real color of the target observation point corresponding to the view image in different ways, such as using MPI technology or using NeRF technology to predict the real color. This process is not the focus of the technical solution of the present application, so it is not described in detail.

[0203] 409、The training device trains the image generation model according to the residual color loss function and the direct prediction loss function.

[0204] After determining the direct prediction loss function, the training device can jointly train the image generation model according to the residual color loss function and the direct prediction loss function. The loss function of joint training can be represented as

[0205] Loss=Loss whole +Loss resi

[0206] In the embodiments of the present application, the image generation model is trained according to the residual color, and the residual color belongs to low-frequency information, which is easy to represent and remember, so the clarity of the new view image obtained based on the image generation model can be improved.

[0207] Further, the training device can train the image generation model jointly with the direct prediction loss function, so as to avoid overfitting of the image generation model when the real color is transparent, reduce the probability of errors in the new view image, and improve the robustness of the algorithm and the reliability of the technical solution of the present application.

[0208] Since the images observed by different reference observation points are not the same, some reference observation points may observe the color of an occluder, which affects the values of the first reference color and the residual color, and also affects the accuracy of the image generation model.

[0209] Therefore, the training device needs to optimize the image generation model, please refer to Figure 5 , Figure 5 is an embodiment of the training method of the image generation model in the embodiments of the present application.

[0210] 501、The training device determines the position and view direction of the target observation point.

[0211] 502、The training device determines at least one reference image.

[0212] 503、The training device determines a first reference color of the spatial position according to at least one reference image.

[0213] 504、The training device obtains a real color of a pixel in the view image corresponding to the target observation point.

[0214] 505、The training device determines a residual color of the spatial position.

[0215] 506、The training device determines whether the real color is transparent, and if so, step 508 is performed, and if not, step 507 is performed.

[0216] 507、The training device trains the image generation model according to the loss function of the residual color.

[0217] 508、The training device obtains a loss function of direct prediction.

[0218] 509、The training device trains the image generation model according to the loss function of the residual color and the loss function of direct prediction.

[0219] Steps 501 to 509 are similar to steps 401 to 409 in the embodiment shown in Figure 4 , and will not be described here.

[0220] 510、The training device determines whether the new view image and each reference image meet a preset condition, and if so, step 511 is performed, and if not, step 512 is performed.

[0221] After the training device has trained the image generation model for one iteration cycle, the accuracy of the image generation model needs to be detected, and the problems are corrected, so as to continuously optimize the image generation model, so that the new view image obtained according to the image generation model is as close to the real image as possible. The optimization process of the image generation model is introduced below.

[0222] The execution device can obtain the residual color of the spatial position corresponding to the virtual observation point according to the image generation model after the last iteration cycle, and then predict the new view image corresponding to the virtual observation point in combination with the reference color of the spatial position. Then the new view image is input into the training device, and the training device judges whether the reference image used in the training process is accurate by judging the similarity between the new view image and the reference image.

[0223] The spatial point is projected onto the image to correspond to a pixel position. Therefore, the judgment manner can be that a pixel point in the new view image is selected as a reference point, and the similarity of the image blocks of the same pixel size centered on the reference point in the new view image and each reference image is compared to determine whether the similarity meets a preset condition. If the preset condition is met, it is indicated that the reference image does not exist in the occlusion condition, and can be continuously used in the training process of the image generation model in the next iteration period.

[0224] The size of the image block can be 3px*3px or 5px*5px, and px is the abbreviation of pixel. The pixel is selected according to the actual application requirement, and the specific pixel is not limited. The similarity of the two image blocks meeting the preset condition can be that the color similarity of the two image blocks is less than or equal to a preset threshold.

[0225] 511. The training device determines the second reference color as the first reference color.

[0226] If each reference image selected by the training device meets the preset condition, it is indicated that the first reference color used in the last iteration period is correct, and the first reference color can be continuously used in the subsequent training process.

[0227] 512. The training device determines the mode of the reference colors of the reference images meeting the condition as the second reference color.

[0228] If at least one reference image selected by the training device does not meet the condition, the training device needs to remove the reference image not meeting the condition, and re-determine the reference color used in the training process.

[0229] At least one reference image not meeting the condition can have the following two cases:

[0230] In one case, part of the reference images in the at least one reference image do not meet the condition. Assuming that there are Y reference images in total, and X reference images do not meet the preset condition, the training device can determine that only the remaining (Y-X) reference images are used to determine the second reference color. In this case, the second reference color is the mode of the reference colors of the (Y-X) reference images at the observation point. The second reference color can be the same as the first reference color used in the last iteration period, or can be different, which is related to the occlusion condition of the observed object. The specific case is not limited. Y is an integer greater than or equal to 1, and X is an integer greater than or equal to 1 and less than Y.

[0231] In another case, each reference image in the N reference images does not meet the condition. In this case, the training device can determine that the color value of the second reference color is 0.

[0232] Optionally, in practical applications, there can be multiple first reference colors, and there can be multiple reasons for this. The following examples are given separately. Assume that there are 18 reference images, and the true color of the observation point is red.

[0233] One possible case is that the reference color in 9 reference images is the color of the occluder (yellow), and the color in 9 reference images is the true color (red), at this time, the mode of the reference color has two.

[0234] One possible case is that the reference color in 6 reference images is the color of the occluder 1 (yellow), the reference color in 6 reference images is the color of the occluder 2 (green), and the reference color in 6 reference images is the true color (red), at this time, the mode of the reference color has three.

[0235] In these cases, due to the existence of occlusion, the first reference color appears to be inconsistent with the actual needs, and only using Figure 4 The method of the embodiment shown in the figure is difficult to remove the influence of the wrong first reference color. The significance of step 512 is to eliminate the adverse effects of the wrong reference image on the image generation model and improve the robustness of the algorithm.

[0236] 513、The training device optimizes the image generation model by taking the second reference color as the first reference color.

[0237] After determining the second reference color, the training device can input the second reference color as the first reference color into the image generation model, thereby adjusting the parameters of the image generation model and optimizing the image generation model.

[0238] In the embodiment of the application, the image generation model is trained according to the residual color, and the residual color belongs to low-frequency information and is easy to represent and remember, so the clarity of the new view image obtained based on the image generation model can be improved.

[0239] Further, the training device compares the new view image with at least one reference image used in training, thereby removing the inappropriate parameter values used in training the image generation model in the last iteration period, thereby reducing the influence of the occluded observation point on the new view image, improving the robustness of the algorithm, and improving the accuracy of the technical solution.

[0240] It should be noted that in an embodiment of the application, in the embodiment shown in the figure Figure 5 In the embodiment shown in the figure, steps 506, 508 and 509 can not be executed, and step 507 is executed directly after step 505.

[0241] In this implementation, the training device directly optimizes the image generation model according to the loss function of the residual color, and continuously optimizes the image generation model according to the training result of the last iteration period, so that the accuracy of the new-view image obtained according to the trained image generation model is higher than that of the image generation model in the embodiment shown in Figure 4 Meanwhile, compared with the embodiment shown in Figure 5 the operation steps can be saved, the operation process can be simplified, and the consumption of operation resources can be reduced.

[0242] Since the quality of the image generation model is related to the accuracy of the dense matching, and the dense matching of the image is based on the similarity of the texture, the area without texture in the image is difficult to provide matching information, and the area with rich texture can provide accurate matching information. The area with rich texture mentioned herein refers to the area with color change, such as the transition from red to yellow, which can be regarded as the edge of the texture. In addition, since the human senses are more sensitive to the area with rich texture, the image generation model provided in the embodiment of the present application will be trained more on the area with rich texture during training, so that the finally obtained image generation model is more practical.

[0243] The embodiment of the present application further provides a method for generating a new-view image, which can use the image generation model described above to generate a new-view image. Please refer to Figure 6 , Figure 6 for an embodiment of the method for generating a new-view image in the embodiment of the present application.

[0244] 601, the execution device determines the position and the view direction of the virtual observation point.

[0245] The virtual observation point is actually an observation point that has not observed the observed object, and can be randomly selected by human. After the virtual observation point is selected by human, the execution device can receive the position and the view direction of the virtual observation point input by human.

[0246] 602, the execution device obtains the residual color according to the image generation model.

[0247] After the execution device obtains the position and the view direction of the virtual observation point, the execution device can input the position and the view direction of the virtual observation point into the image generation model to obtain the residual color of the spatial position corresponding to the virtual observation point. The spatial position corresponding to the virtual observation point refers to the position passed by the light from the virtual observation point.

[0248] In this embodiment, the image generation model used by the execution device includes Figures 3 to 5The image generation model in the illustrated embodiment can be an image generation model that is not fully trained or an image generation model that is trained, and the selection is made according to the actual application needs, and the specific details are not limited here.

[0249] Although the residual color obtained by using the image generation model that is not trained has a large error, the new view image obtained based on the residual color can be used to remove the occluded reference image, thereby optimizing the image generation model, which has a meaning.

[0250] The residual color obtained by using the trained image generation model is the ideal state residual color of the embodiments of the present application, and the new view image generated based on the residual color is also relatively accurate.

[0251] 603, the execution device obtains the first reference color.

[0252] After the execution device receives the position and the view direction of the virtual observation point, at least one reference image can be determined from the training image set, so as to obtain the first reference color.

[0253] It should be noted that the execution subject of determining at least one reference image from the training image set can also be the training device, and the process of selecting the reference image by the training device is similar to the process of selecting the reference image by the execution device. Figure 4 The step 402 in the illustrated embodiment is similar, except that the basis for determining the reference observation point is the position and the view direction of the virtual observation point, instead of the position and the view direction of the target observation point, and the specific details are not repeated here.

[0254] It should be noted that the first reference color in the present embodiment includes Figure 4 and Figure 5 the first reference color in the illustrated embodiment.

[0255] When the execution device obtains the first reference color, the spatial position opacity can also be obtained, because the spatial position opacity will affect the final imaging effect, and therefore the execution device also needs to obtain the opacity.

[0256] 604, the execution device generates a new view image according to the residual color and the first reference color.

[0257] In the image, the color of each pixel position is obtained by integrating the colors of multiple space points on a light ray, and after the execution device obtains the first reference color, the residual color and the opacity of each space point, the new view image corresponding to the virtual observation point can be integrated, and the integration process can have the following cases.

[0258] One of the cases is to integrate the residual color and the first reference color of each space point respectively, and then add the integration results to obtain the new view image.

[0259] Another case is to add the first reference color and the residual color of each spatial point first, and then integrate together to obtain the new view image. The functions used in the above two integration methods can be

[0260]

[0261] wherein, represents the predicted new view image, represents the first reference color of each spatial point, represents the residual color of each spatial point, σ i represents the opacity of a spatial point in the spatial position, δ i is used to represent the distance of each spatial point on a light ray.

[0262] In the embodiments of the present application, the image generation model is trained according to the residual color. The residual color belongs to low-frequency information and is easy to represent and remember. Therefore, the new view image generated by the image generation model used by the execution device has high clarity.

[0263] The training device of the image generation model provided in the embodiments of the present application is described below. Please refer to Figure 7 , Figure 7 An embodiment of the training device 700 of the image generation model provided in the embodiments of the present application includes:

[0264] The determining unit 701 is configured to determine the position and the view direction of the target observation point, and then determine at least one reference image from the N input images according to the position and the view direction of the target observation point, wherein N is an integer greater than or equal to 2. Then, the reference color of the spatial position is determined according to the at least one reference image. The spatial position is the position passed by the light from the target observation point.

[0265] The obtaining unit 702 is configured to obtain the true color of the pixel in the view image corresponding to the target observation point.

[0266] The determining unit 701 is further configured to determine the residual color of the spatial position according to the reference color and the true color.

[0267] The processing unit 703 is configured to train the image generation model according to the residual color.

[0268] In some optional embodiments of the present application, the reference color includes a first reference color, and the first reference color is the mode of the color of the position passed by the light from the target observation point.

[0269] In some optional embodiments of the present application, the processing unit 703 is configured to:

[0270] If the real color is not a transparent color, the image generation model is trained according to the loss function of the residual color.

[0271] If the real color is a transparent color, the loss function of the direct prediction is obtained, and the image generation model is trained according to the loss function of the residual color and the loss function of the direct prediction.

[0272] In some optional embodiments of the present application, the obtaining unit 702 is further configured to obtain a new-view image corresponding to the target observation point, wherein the new-view image is predicted by the execution device according to the image generation model.

[0273] The determining unit 701 is further configured to determine a second reference color according to the new-view image and each of the at least one reference image, and then take the second reference color as the first reference color.

[0274] In some optional embodiments of the present application, the determining unit 701 is specifically configured to:

[0275] determine any pixel point in the new-view image as a reference point.

[0276] If the similarity of the image block of the same pixel size centered on the reference point in the new-view image and each of the reference images meets a preset condition, the second reference color is determined as the first reference color.

[0277] If the similarity of the image block of the same pixel size centered on the reference point in the new-view image and part of the reference images in the at least one reference image does not meet the preset condition, the second reference color is determined as the mode of the reference colors in the spatial positions of the reference images that meet the preset condition.

[0278] If the similarity of the image block of the same pixel size centered on the reference point in the new-view image and each of the reference images does not meet the preset condition, the color value of the second reference color is determined as 0.

[0279] In some optional embodiments of the present application, the determining unit 701 is specifically configured to:

[0280] determine at least one reference observation point according to the position and the perspective direction of the target observation point, wherein the distance between each of the at least one reference observation point and the target observation point meets a preset condition.

[0281] obtain at least one reference image according to the at least one reference observation point, wherein each of the at least one reference observation point corresponds to each of the at least one reference image.

[0282] In some optional embodiments of the present application, the loss function of the residual color can be:

[0283]

[0284] wherein,

[0285]

[0286] is used to represent the color of the pixel in the new view image predicted according to the residual color and the first reference color, represents the first reference color of each spatial point, represents the residual color of each spatial point, σ i represents the opacity of a spatial point in the spatial position, δ i represents the distance of each spatial point on a light ray. C(r) is used to represent the real color.

[0287] In some optional embodiments of the present application, the loss function of the direct prediction can be:

[0288]

[0289] wherein,

[0290]

[0291] is used to represent the color of the pixel in the new view image predicted according to the residual color and the first reference color, i represents the first reference color of each spatial point, i represents the opacity of a spatial point in the spatial position, δ i represents the distance of each spatial point on a light ray. Direct prediction loss function is used to make the predicted new view image as close to the real image as possible under the condition of only learning the real color.

[0292] In the present embodiment, the training device 700 of the image generation model can perform the operations of the foregoing Figures 3 to 5 The training device performs the operations of the foregoing

[0293] The generation device of the new view image provided in the embodiments of the present application will be introduced below. Please refer to Figure 8 , Figure 8 An embodiment of the generation device 800 of the new view image provided in the embodiments of the present application comprises:

[0294] The determining unit 801 is configured to determine the position and the view direction of the virtual observation point.

[0295] The acquisition unit 802 is configured to input the position and the view direction of the virtual observation point into the image generation model, and acquire residual colors of spatial positions through which light from the virtual observation point passes.

[0296] The acquisition unit 802 is further configured to acquire reference colors, which are colors of the spatial positions determined according to at least one reference image.

[0297] The processing unit 803 is configured to generate a new view image corresponding to the virtual observation point according to the residual colors of the spatial positions and the reference colors.

[0298] In some optional embodiments of the present application, the image generation model comprises: the image generation model trained according to the loss function of the residual color. Alternatively, the image generation model is trained according to the loss function of the residual color and a direct prediction loss function.

[0299] In some optional embodiments of the present application, the reference color comprises: a first reference color, which is a mode of the color of the spatial position through which the light from the virtual observation point passes.

[0300] The acquisition unit 802 is specifically configured to receive the first reference color sent by the training device.

[0301] In some optional embodiments of the present application, the reference color comprises: the first reference color.

[0302] The acquisition unit 802 is specifically configured to:

[0303] According to the position and the view direction of the virtual observation point, at least one reference observation point is determined, wherein a distance between each reference observation point in the at least one reference observation point and the virtual observation point satisfies a preset condition.

[0304] At least one reference image is determined from N reference pictures, wherein each reference observation point in the at least one reference observation point corresponds to each reference image in the at least one reference image, and N is an integer greater than or equal to 2.

[0305] According to the at least one reference image, the first reference color is determined.

[0306] In the present embodiment, the new view image generation apparatus 800 can perform the operations of the processor in the foregoing embodiments, or the operations of the execution device in the foregoing embodiments, and details are not repeated here. Figure 2 In the present embodiment, the new view image generation apparatus 800 can perform the operations of the processor in the foregoing embodiments, or the operations of the execution device in the foregoing embodiments, and details are not repeated here. Figure 6 In the present embodiment, the new view image generation apparatus 800 can perform the operations of the processor in the foregoing embodiments, or the operations of the execution device in the foregoing embodiments, and details are not repeated here.

[0307] The present application also provides an image processing system, please refer to Figure 9 , Figure 9An embodiment of the image processing system 900 provided in the embodiments of the present application comprises:

[0308] The training device 910 and the execution device 920.

[0309] The training device 910 comprises a first processor 911 and a first memory 912.

[0310] The first memory 912 is configured to store a training picture set, and the training picture set comprises at least two images.

[0311] The first processor 911 is configured to perform the operations of the training device in the foregoing Figures 3 to 5 embodiments, or the operations of the training apparatus 700 of the image generation model in the foregoing Figure 7 embodiments, which are not described herein again in detail.

[0312] The training device 920 comprises a second processor 921 and a second memory 922.

[0313] The second memory 922 is configured to store a new-view image.

[0314] The second processor 921 is configured to perform the operations of the processor in the foregoing Figure 2 embodiments, the operations of the execution device in the foregoing Figure 6 embodiments, or the operations of the new-view image generation apparatus 800 in the foregoing Figure 8 embodiments, which are not described herein again in detail.

[0315] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the system, apparatus and unit described above can refer to the corresponding processes in the foregoing method embodiments, which are not described herein again in detail.

[0316] In several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method can be implemented in other ways. For example, the apparatus embodiments described above are merely schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, apparatuses or units, and can be electrical, mechanical or other forms.

[0317] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0318] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present alone, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0319] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part of the prior art that contributes to the technical solutions or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

Claims

1. A training method for an image generation model, characterized in that, include: Determine the location and viewing direction of the target observation point; Based on the position and the viewing direction, at least one reference image is determined from N input images, where N is an integer greater than or equal to 2; Based on the at least one reference image, a reference color is determined for a spatial location, where the spatial location is the position through which light rays from the target observation point pass. Obtain the true color of the pixel in the view image corresponding to the target observation point; The residual color of the spatial location is determined based on the reference color and the actual color. Train the image generation model based on the residual color; The step of determining at least one reference image from N input images based on the position and the viewing direction includes: Based on the location and the viewing direction, at least one reference observation point is determined, wherein the distance between each of the at least one reference observation point and the target observation point satisfies a preset condition; The at least one reference image is obtained based on the at least one reference observation point, wherein each of the at least one reference observation point corresponds to each of the at least one reference image.

2. The method according to claim 1, characterized in that, The reference color includes: a first reference color, which is the mode of the color at the spatial location.

3. The method according to claim 1, characterized in that, The step of training the image generation model based on the residual color includes: If the true color is not transparent, then the image generation model is trained according to the loss function of the residual color; If the true color is transparent, then obtain the loss function for direct prediction; The image generation model is trained based on the loss function of the residual color and the loss function of the direct prediction.

4. The method according to any one of claims 1 to 3, characterized in that, After training the image generation model based on the residual color, the method further includes: Acquire a new perspective image corresponding to the target observation point, wherein the new perspective image is predicted by the execution device based on the image generation model; A second reference color is determined based on the new perspective image and each of the at least one reference image; Use the second reference color as the first reference color.

5. The method according to claim 4, characterized in that, Determining the second reference color based on the new perspective image and each of the at least one reference image includes: Any pixel in the new perspective image is designated as a reference point; If the similarity of image blocks of the same pixel size centered at the reference point in the new view image and each of the reference images meets a preset condition, then the second reference color is determined to be the first reference color; If the similarity of image blocks of the same pixel size centered at the reference point in the new view image and some reference images in the at least one reference image does not meet the preset condition, then the second reference color is determined to be the mode of the reference color of the reference image that meets the preset condition at the spatial location. If the similarity relationship between image blocks of the same pixel size centered at the reference point and each of the reference images does not meet the preset conditions in the new perspective image and each of the reference images, then the color value of the second reference color is determined to be 0.

6. A method for generating images from a new perspective, characterized in that, include: Determine the location and viewing direction of the virtual observation point; The location and the viewing direction are input into the image generation model to obtain the residual color of the spatial location through which the light from the virtual observation point passes; A reference color is obtained, which is the color of the spatial location determined based on at least one reference image; each of the at least one reference image corresponds to each of the at least one reference observation points, and the distance between each of the at least one reference observation points and the virtual observation point satisfies a preset condition; A new perspective image corresponding to the virtual observation point is generated based on the residual color of the spatial location and the reference color.

7. The method according to claim 6, characterized in that, The image generation model includes: The image generation model is trained based on the loss function of the residual color; or, The image generation model is trained based on the loss function of the residual color and the loss function of direct prediction.

8. The method according to claim 6 or 7, characterized in that, The reference color includes: a first reference color, wherein the first reference color is the mode of the color at the spatial location; The acquisition of the reference color includes: Receive the first reference color sent by the training device.

9. The method according to claim 6 or 7, characterized in that, The reference color includes: a first reference color; The process of obtaining the reference color includes: Based on the location and viewing direction, determine the at least one reference observation point; The at least one reference image is determined from N reference images; The first reference color is determined based on the at least one reference image.

10. A training device for an image generation model, characterized in that, include: Determine the unit, used for: Determine the location and viewing direction of the target observation point; Based on the position and the viewing direction, at least one reference image is determined from N input images, where N is an integer greater than or equal to 2; Based on the at least one reference image, a reference color is determined for a spatial location, where the spatial location is the position through which light rays from the target observation point pass. The acquisition unit is used to acquire the true color of the pixels in the view image corresponding to the target observation point; The determining unit is further configured to determine the residual color of the spatial position based on the reference color and the real color; A processing unit is used to train an image generation model based on the residual color; The determining unit is specifically used for: Based on the location and the viewing direction, at least one reference observation point is determined, wherein the distance between each of the at least one reference observation point and the target observation point satisfies a preset condition; The at least one reference image is obtained based on the at least one reference observation point, wherein each of the at least one reference observation point corresponds to each of the at least one reference image.

11. The apparatus according to claim 10, characterized in that, The reference color includes: a first reference color, which is the mode of the color at the spatial location.

12. The apparatus according to claim 10, characterized in that, The processing unit is specifically used for: If the true color is not transparent, then the image generation model is trained according to the loss function of the residual color; If the true color is transparent, then obtain the loss function for direct prediction; The image generation model is trained based on the loss function of the residual color and the loss function of the direct prediction.

13. The apparatus according to any one of claims 10 to 12, characterized in that, The acquisition unit is further configured to acquire a new perspective image corresponding to the target observation point, wherein the new perspective image is predicted by the execution device based on the image generation model; The determining unit is further configured to determine a second reference color based on the new perspective image and each of the at least one reference image; The determining unit is further configured to use the second reference color as the first reference color.

14. The apparatus according to claim 13, characterized in that, The determining unit is specifically used for: Any pixel in the new perspective image is designated as a reference point; If the similarity of image blocks of the same pixel size centered at the reference point in the new view image and each of the reference images meets a preset condition, then the second reference color is determined to be the first reference color; If the similarity of image blocks of the same pixel size centered at the reference point in the new view image and some reference images in the at least one reference image does not meet the preset condition, then the second reference color is determined to be the mode of the reference color of the reference image that meets the preset condition at the spatial location. If the similarity relationship between image blocks of the same pixel size centered at the reference point and each of the reference images does not meet the preset conditions in the new perspective image and each of the reference images, then the color value of the second reference color is determined to be 0.

15. An apparatus for generating images from a new perspective, characterized in that, include: The unit is used to determine the position and viewing direction of the virtual observation point; The acquisition unit is used to input the position and the viewing direction into the image generation model to acquire the residual color of the spatial position through which the light from the virtual observation point passes; The acquisition unit is further configured to acquire a reference color, which is the color of the spatial location determined based on at least one reference image; The processing unit is configured to generate a new perspective image corresponding to the virtual observation point based on the residual color of the spatial location and the reference color.

16. The apparatus according to claim 15, characterized in that, The image generation model includes: The image generation model is trained based on the loss function of the residual color; or, The image generation model is trained based on the loss function of the residual color and the loss function of direct prediction.

17. The apparatus according to claim 15 or 16, characterized in that, The reference color includes: a first reference color, wherein the first reference color is the mode of the color at the spatial location; The acquisition unit is specifically used to receive the first reference color sent by the training device.

18. The apparatus according to claim 15 or 16, characterized in that, The reference color includes: a first reference color; The acquisition unit is specifically used for: Based on the position and viewing direction, at least one reference observation point is determined, wherein the distance between each of the at least one reference observation point and the virtual observation point satisfies a preset condition; At least one reference image is determined from N reference images, wherein each of the at least one reference observation point corresponds to each of the at least one reference image, and N is an integer greater than or equal to 2; A first reference color is determined based on the at least one reference image.

19. An image processing system, characterized in that, include: Training equipment, execution equipment; The training device includes a first processor and a first memory, the first processor being used to execute the method of any one of claims 1 to 5, and the first memory being used to store a training image set, the training image set including at least two images; The execution device includes a second processor and a second memory, the second processor being used to execute the method of any one of claims 6 to 9, and the second memory being used to store images from new perspectives.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains a program that, when executed by the computer, performs the method according to any one of claims 1 to 9.

21. A computer program product, characterized in that, When the computer program product is executed on a computer, the computer performs the method according to any one of claims 1 to 9.

Images