Image processing method and device, electronic equipment and storage medium

Abstract

Embodiments of the present disclosure provide an image processing method and device, electronic equipment and storage medium. The method comprises: obtaining first position information corresponding to a plurality of to-be-combined sampling points and first view angle information corresponding to the plurality of to-be-combined sampling points; obtaining a plurality of local three-dimensional semantic features corresponding to the to-be-combined sampling points based on the first position information, the first view angle information and a plurality of first models; determining a first two-dimensional feature map corresponding to the plurality of to-be-combined sampling points based on the plurality of local three-dimensional semantic features; and determining a target image corresponding to the first view angle information based on the first two-dimensional feature map. The above technical solution realizes regional processing of the plurality of to-be-combined sampling points, improves the refinement degree of the generated image, and optimizes the image processing effect; the multi-view image synthesis result can be obtained by controlling the input view angle information; and the image style of the local region can be changed by controlling the parameters of the first model corresponding to the local region of the image.

Description

Technical Field

[0001] This disclosure relates to image processing technology, and more particularly to an image processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of artificial intelligence technology, it is widely used in various scenarios. One common application scenario is the combination of artificial intelligence technology and image processing technology.

[0003] In related technologies, image generation techniques can only manipulate the pose of a person as a whole, but cannot manipulate the local features of the generated image, or are not precise enough when generating local features. Summary of the Invention

[0004] This disclosure provides an image processing method, apparatus, electronic device, and storage medium to improve the detail of generated images and optimize image processing effects.

[0005] In a first aspect, embodiments of this disclosure provide an image processing method, including:

[0006] Obtain first position information corresponding to multiple sampling points to be synthesized and first view information corresponding to the multiple sampling points to be synthesized;

[0007] Based on the first location information, the first viewpoint information, and multiple first models, multiple local three-dimensional semantic features corresponding to the sampling points to be synthesized are obtained, wherein the local three-dimensional semantic features generated by the first model correspond to the local image information of the target image.

[0008] A first two-dimensional feature map corresponding to the multiple sampling points to be synthesized is determined based on multiple local three-dimensional semantic features;

[0009] The target image corresponding to the first viewpoint information is determined based on the first two-dimensional feature map.

[0010] Secondly, embodiments of this disclosure also provide an image processing apparatus, comprising:

[0011] The information input module is used to acquire first position information corresponding to multiple sampling points to be synthesized and first view information corresponding to the multiple sampling points to be synthesized;

[0012] The semantic feature acquisition module is used to obtain multiple local three-dimensional semantic features corresponding to the sampling point to be synthesized based on the first location information, the first viewpoint information, and multiple first models, wherein the local three-dimensional semantic features generated by the first model correspond to the local image information of the target image.

[0013] The intermediate image determination module is used to determine a first two-dimensional feature map corresponding to the multiple sampling points to be synthesized based on multiple local three-dimensional semantic features;

[0014] The target image generation module is used to determine the target image corresponding to the first viewpoint information based on the first two-dimensional feature map.

[0015] Thirdly, embodiments of this disclosure also provide an electronic device, the electronic device comprising:

[0016] One or more processors;

[0017] Storage device for storing one or more programs.

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in any embodiment of this disclosure.

[0019] Fourthly, embodiments of this disclosure also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the image processing method as described in any embodiment of this disclosure.

[0020] In this embodiment, by acquiring first position information corresponding to multiple sampling points to be synthesized and first view information corresponding to the multiple sampling points to be synthesized, the data of multiple sampling points to be synthesized is obtained. Then, based on the first position information, the first view information, and multiple first models, multiple local three-dimensional semantic features corresponding to the sampling points to be synthesized are obtained, realizing the extraction of local three-dimensional semantic features. The local three-dimensional semantic features generated by the first model correspond to the local image information of the target image. Then, based on the multiple local three-dimensional semantic features, a first two-dimensional feature map corresponding to the multiple sampling points to be synthesized is determined, realizing the extraction of two-dimensional feature maps. Then, based on the first two-dimensional feature map, a target image corresponding to the first view information is determined, realizing the acquisition of images under the viewpoints corresponding to the multiple sampling points to be synthesized. In other words, multiple sampling points to be synthesized can be processed by region, improving the refinement of the generated image and optimizing the image processing effect. By controlling the input view information, multi-view image synthesis results can be obtained. By controlling the parameters of the first model corresponding to the local region of the image, the image style of the local region can be changed. Attached Figure Description

[0021] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0022] Figure 1 This is a schematic flowchart of an image processing method provided in an embodiment of this disclosure;

[0023] Figure 2 This is a schematic flowchart of another image processing method provided in this embodiment of the present disclosure;

[0024] Figure 3 This is a schematic flowchart of another image processing method provided in this embodiment of the present disclosure;

[0025] Figure 4 This is a schematic flowchart of another image processing method provided in this embodiment of the present disclosure;

[0026] Figure 5 This is a schematic flowchart of another image processing method provided in this embodiment of the present disclosure;

[0027] Figure 6 This is a schematic diagram of the architecture of a local semantic 3D generator provided in an embodiment of this disclosure;

[0028] Figure 7 This is a schematic diagram of the architecture of a combined neural radiation field network provided in an embodiment of this disclosure;

[0029] Figure 8 This is a schematic diagram of a model training scenario for performing the image processing method of the present disclosure embodiment;

[0030] Figure 9 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this disclosure;

[0031] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0032] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0033] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0034] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0035] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0036] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0037] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0038] Figure 1 This is a flowchart illustrating an image processing method provided in an embodiment of the present disclosure. The embodiments of the present disclosure are applicable to the situation of automatically generating an image of an object from a target perspective. The method can be executed by an image processing device, which can be implemented in the form of software and / or hardware. Optionally, it can be implemented by an electronic device, such as a mobile terminal, a PC, or a server.

[0039] like Figure 1 As shown, the method in this embodiment may specifically include:

[0040] S110. Obtain the first position information corresponding to the multiple sampling points to be synthesized and the first view information corresponding to the multiple sampling points to be synthesized.

[0041] In this embodiment of the disclosure, the sampling points to be synthesized refer to spatial sampling points used for synthesizing the image. The image can be a three-dimensional or other multi-dimensional image, and may contain one or more target objects. For example, the target object may be a person, an animal, or other three-dimensional object. The sampling points to be synthesized can be uniformly sampled according to a set field of view. The set field of view can be a field of view with a set length and width, specifically a grid structure. The first position information corresponding to multiple sampling points to be synthesized can be the position of any local region of the target object in the image. The local region may include, but is not limited to, local regions such as eyes, mouth, eyebrows, and nose. The first viewpoint information corresponding to multiple sampling points to be synthesized can be camera viewpoint direction information, which may include pitch angle and yaw angle.

[0042] Optionally, the sampling points to be synthesized are spatial points sampled from a random distribution. Specifically, the sampling points to be synthesized can be spatial points uniformly sampled within a preset field of view.

[0043] For example, the first position information corresponding to the sampling point to be synthesized can be the three-dimensional spatial coordinates of the sampling point to be synthesized. The first position information and the first viewpoint information can be information input by the user through the software interface.

[0044] S120. Based on the first position information, the first viewpoint information, and multiple first models, obtain multiple local three-dimensional semantic features corresponding to the sampling points to be synthesized, wherein the local three-dimensional semantic features generated by the first model correspond to the local image information of the target image.

[0045] In this embodiment, the first model refers to a network model capable of generating local three-dimensional semantic features. The first model is used to generate local three-dimensional semantic features corresponding to local image information of the target image. In this embodiment, there are multiple first models, each capable of generating features for its corresponding local image region; in other words, different first models can correspond to different local image regions of the target image. The number of local three-dimensional semantic features can be the same as the number of first models. Local image information refers to image information corresponding to local three-dimensional semantic features. Taking a facial image as an example, local image information may include, but is not limited to, image information corresponding to local regions such as eyes, mouth, eyebrows, and nose. Local three-dimensional semantic features refer to three-dimensional semantic features corresponding to local image information of the target image.

[0046] Specifically, the first position information and the first view information are used as input information for each first model. Each first model extracts semantic features of multiple local regions of the three-dimensional image based on the first position information and the first view information, thereby obtaining multiple local three-dimensional semantic features corresponding to the sampling points to be synthesized.

[0047] For example, the local three-dimensional semantic features may include, but are not limited to, at least one of the semantic features such as the first three-dimensional feature value, the first three-dimensional color value, the first three-dimensional semantic mask value, and the first residual directed distance field value.

[0048] S130. Determine a first two-dimensional feature map corresponding to the multiple sampling points to be synthesized based on the multiple local three-dimensional semantic features.

[0049] In this embodiment of the disclosure, the first two-dimensional feature map refers to the two-dimensional feature map obtained by processing local three-dimensional semantic features.

[0050] In some embodiments, multiple local three-dimensional semantic features can be combined to obtain a first two-dimensional feature map corresponding to multiple sampling points to be synthesized. In some embodiments, multiple local three-dimensional semantic features can be used as input information of the model, and the model can fuse and aggregate the local three-dimensional semantic features based on the multiple local three-dimensional semantic features to obtain the first two-dimensional feature map corresponding to multiple sampling points to be synthesized. The specific implementation method of determining the first two-dimensional feature map is not limited here.

[0051] S140. Determine the target image corresponding to the first viewpoint information based on the first two-dimensional feature map.

[0052] In this embodiment of the disclosure, the target image refers to an image of the target object from the target perspective, wherein the target perspective can be the perspective corresponding to the first perspective information. For example, high-resolution image synthesis processing can be performed based on the first two-dimensional feature map to obtain the target image corresponding to the first perspective information.

[0053] In this embodiment of the disclosure, optionally, a target image corresponding to the first viewpoint information is determined based on the first two-dimensional feature map. Based on this, a first two-dimensional feature map and a first two-dimensional mask map corresponding to multiple sampling points to be synthesized can be determined based on multiple first local three-dimensional semantic features and a second model; further, a target image corresponding to the first viewpoint information can also be determined based on the first two-dimensional feature map and the first two-dimensional mask map. Even further, a target mask image corresponding to the target image can be determined based on the first two-dimensional feature map and the first two-dimensional mask map.

[0054] It should be noted that in the embodiments of this disclosure, the "first model," "second model," "third model," and "fourth model" can all be pre-trained artificial intelligence models. These artificial intelligence models include, but are not limited to, machine learning models and deep learning models. Optionally, the first and second models can be constructed based on neural radiation fields. For example, the first model can be composed of a modulated multilayer perceptron and fully connected layers. The third and fourth models can also be constructed based on a multilayer perceptron.

[0055] The technical solution of this disclosure acquires multiple sampling point data by obtaining first position information and first view information corresponding to multiple sampling points to be synthesized. Then, based on the first position information, first view information, and multiple first models, multiple local three-dimensional semantic features corresponding to the sampling points to be synthesized are obtained, achieving the extraction of local three-dimensional semantic features. The local three-dimensional semantic features generated by the first model correspond to the local image information of the target image. Based on the multiple local three-dimensional semantic features, a first two-dimensional feature map corresponding to the multiple sampling points to be synthesized is determined, achieving the extraction of the two-dimensional feature map. Furthermore, based on the first two-dimensional feature map, a target image corresponding to the first view information is determined, achieving the acquisition of images from the viewpoints corresponding to the multiple sampling points to be synthesized. In other words, multiple sampling points to be synthesized can be processed by region, improving the refinement of the generated image and optimizing the image processing effect. Multi-view image synthesis results can be obtained by controlling the input view information. The image style of the local region can be changed by controlling the parameters of the first model corresponding to the local region of the image.

[0056] Figure 2 This is a flowchart illustrating another image processing method provided in this disclosure. Based on the above embodiments, the technical solution of this disclosure further refines how to obtain multiple local three-dimensional semantic features corresponding to the sampling point to be synthesized, based on the first position information, the first viewpoint information, and multiple first models. Optionally, obtaining multiple local three-dimensional semantic features corresponding to the sampling point to be synthesized based on the first position information, the first viewpoint information, and multiple first models includes: determining a first modulation latent code corresponding to the first model, wherein the first modulation latent code includes a shape modulation latent code for modulating a shape-aware network in the first model and a texture modulation latent code for modulating a texture-aware network in the first model; inputting the first position information, the first viewpoint information, and the first modulation latent code into multiple first models to obtain multiple local three-dimensional semantic features corresponding to the sampling point to be synthesized. For detailed implementation methods, please refer to the description of the embodiments of this disclosure. Technical features that are the same as or similar to those in the foregoing embodiments will not be repeated here.

[0057] like Figure 2 As shown, the method in this embodiment may specifically include:

[0058] S210. Obtain the first position information corresponding to the multiple sampling points to be synthesized and the first view information corresponding to the multiple sampling points to be synthesized.

[0059] S220. Determine a first modulation latent code corresponding to the first model, wherein the first modulation latent code includes a shape modulation latent code for modulating the shape-aware network in the first model and a texture modulation latent code for modulating the texture-aware network in the first model.

[0060] In this embodiment of the disclosure, the first modulation latent coding refers to the encoded information obtained by sampling the latent vector space of the image. The first modulation latent coding can be used to indicate the style corresponding to the local three-dimensional semantic features generated by the first model; in other words, it can be used to indicate the image style corresponding to the local image information of the target image. Optionally, the first modulation latent coding includes a shape modulation latent coding for modulating the shape-aware network in the first model and a texture modulation latent coding for modulating the texture-aware network in the first model, wherein the shape-aware network or the texture-aware network can be composed of a multilayer perceptron (MLP). In other words, the shape features and texture features of the local region of the target object can be learned through the shape-aware network and the texture-aware network in the first model.

[0061] It is understandable that each first model can have a corresponding modulation latent code, meaning the first modulation latent code must be set correspondingly to the first model. Different first models can correspond to the same or different first modulation latent codes. It should be noted that the first position information and the first view information can be shared parameters, meaning that the first position information and the first view information input to each first model are the same.

[0062] S230. Input the first position information, the first viewpoint information, and the first modulation latent code into multiple first models to obtain multiple local three-dimensional semantic features corresponding to the sampling points to be synthesized.

[0063] Specifically, for each first model, the first position information, the first view information, and the first modulation latent code can be used as the input information of the first model. Then, the first position information, the first view information, and the first modulation latent code are input into the first model, and the first model outputs the local three-dimensional semantic features corresponding to multiple sampling points to be synthesized.

[0064] It should be noted that in this embodiment of the present disclosure, the first modulation latent coding is used as the input information of the first model, so that the first model can learn the shape features and texture features of the local region of the target object, thereby improving the accuracy of the extracted local three-dimensional semantic features.

[0065] In some alternative implementations, determining the first modulation latent code corresponding to the first model includes: determining the first modulation latent code corresponding to the first model based on a noise vector sampled in a standard Gaussian distribution and a fourth model.

[0066] The fourth model refers to a network model capable of latent coding. For example, the fourth model can be composed of a multilayer perceptron. Specifically, the first modulation latent code can be obtained by passing a noise vector sampled from a standard Gaussian distribution through a multilayer perceptron mapping layer.

[0067] S240. Determine a first two-dimensional feature map corresponding to the multiple sampling points to be synthesized based on the multiple local three-dimensional semantic features.

[0068] S250. The target image based on the first two-dimensional feature map and the first viewpoint information.

[0069] The technical solution of this disclosure embodiment inputs first position information, first view information and first modulation latent coding into a first model, so that the first model can learn the shape features and texture features of the local area of ​​the target object in a targeted manner, thereby improving the generation accuracy of local three-dimensional semantic features. Moreover, the style of the local area of ​​the image can be adjusted through the first modulation latent coding to achieve image diversification and enrich the generation effect of the target image.

[0070] Figure 3 This is a flowchart illustrating another image processing method provided in an embodiment of this disclosure. Based on the above embodiments, the technical solution of this disclosure further refines the step of "determining a first two-dimensional feature map corresponding to the plurality of sampling points to be synthesized based on the local three-dimensional semantic features." Optionally, determining the first two-dimensional feature map corresponding to the plurality of sampling points to be synthesized based on the plurality of local three-dimensional semantic features includes: determining the first two-dimensional feature map corresponding to the plurality of sampling points to be synthesized based on the plurality of local three-dimensional semantic features and a second model. For detailed implementation methods, please refer to the description of the embodiments of this disclosure. Technical features that are the same as or similar to those in the foregoing embodiments will not be repeated here.

[0071] like Figure 3 As shown, the method in this embodiment may specifically include:

[0072] S310. Obtain the first position information corresponding to the multiple sampling points to be synthesized and the first view information corresponding to the multiple sampling points to be synthesized.

[0073] S320. Based on the first position information, the first viewpoint information, and multiple first models, obtain multiple local three-dimensional semantic features corresponding to the sampling points to be synthesized, wherein the local three-dimensional semantic features generated by the first model correspond to the local image information of the target image.

[0074] S330. Based on the multiple local three-dimensional semantic features and the second model, determine the first two-dimensional feature map corresponding to the multiple sampling points to be synthesized.

[0075] In this embodiment of the disclosure, the second model refers to a network model that processes local three-dimensional semantic features. The first two-dimensional feature map refers to the two-dimensional feature map obtained after processing by the second model.

[0076] Specifically, multiple local three-dimensional semantic features are used as input information for the second model. The second model realizes global fusion and volume aggregation of the local three-dimensional semantic features based on the multiple local three-dimensional semantic features, thereby obtaining the first two-dimensional feature map corresponding to multiple sampling points to be synthesized.

[0077] In some optional implementations, the second model includes a weighted fusion layer and a volume aggregation layer. The local three-dimensional semantic features include a first three-dimensional feature value, a first three-dimensional color value, a first three-dimensional semantic mask value, and a first residual directed distance field value. Determining the first two-dimensional feature map corresponding to the plurality of sampling points to be synthesized based on multiple local three-dimensional semantic features and the second model includes: inputting multiple first three-dimensional feature values, first three-dimensional color values, and first three-dimensional semantic mask values ​​into the weighted fusion layer of the second model to obtain second three-dimensional feature values ​​and second three-dimensional color values ​​corresponding to the plurality of sampling points to be synthesized; determining a first image volume density value corresponding to the plurality of sampling points to be synthesized based on multiple first residual directed distance field values; and inputting the second three-dimensional feature value, second three-dimensional color value, first three-dimensional semantic mask value, and first image volume density value into the volume aggregation layer of the second model to obtain the first two-dimensional feature map, first two-dimensional color map, and first two-dimensional mask map corresponding to the plurality of sampling points to be synthesized.

[0078] In this embodiment of the disclosure, the first three-dimensional feature value refers to the three-dimensional feature value corresponding to the sampling point to be synthesized. The number of first three-dimensional feature values ​​is the same as the number of sampling points to be synthesized. For example, the dimension of the first three-dimensional feature value can be 64×64×24×128, where 24 represents the number of sampling points. The first three-dimensional color value refers to the three-dimensional color value corresponding to the sampling point to be synthesized. The number of first three-dimensional color values ​​is the same as the number of sampling points to be synthesized. For example, the dimension of the first three-dimensional color value can be 64×64×24×3, where 24 represents the number of sampling points. The first three-dimensional semantic mask value refers to the three-dimensional semantic mask value corresponding to the sampling point to be synthesized. The number of first three-dimensional semantic mask values ​​is the same as the number of sampling points to be synthesized. For example, the dimension of the first three-dimensional semantic mask value can be 64×64×24×3, where 24 represents the number of sampling points. The first residual directed distance field value refers to the residual directed distance field value corresponding to the sampling point to be synthesized. The number of first residual directed distance field values ​​is the same as the number of sampling points to be synthesized. For example, the dimension of the first residual directed distance field value can be 64×64×24×1, where 24 represents the number of sampling points. The directed distance field (SDF) is a surrogate representation of the volume density function.

[0079] In this embodiment, the weighted fusion layer of the second model refers to a network layer that performs weighted fusion processing on three-dimensional feature values, three-dimensional color values, and three-dimensional semantic mask values. Specifically, multiple first three-dimensional feature values, first three-dimensional color values, and first three-dimensional semantic mask values ​​are input to the weighted fusion layer of the second model, so that the weighted fusion layer performs weighted fusion on the first three-dimensional feature values ​​and first three-dimensional color values ​​corresponding to the multiple sampling points to be synthesized output by each first model based on the first three-dimensional semantic mask values, to obtain second three-dimensional feature values ​​and second three-dimensional color values ​​corresponding to the multiple sampling points to be synthesized. The second three-dimensional feature value refers to the three-dimensional feature value corresponding to the weighted fusion processing of multiple first three-dimensional feature values. Similarly, the second three-dimensional color value refers to the three-dimensional color value corresponding to the weighted fusion processing of multiple first three-dimensional color values.

[0080] In some optional implementations, inputting multiple first three-dimensional feature values, first three-dimensional color values, and first three-dimensional semantic mask values ​​into the weighted fusion layer of the second model to obtain second three-dimensional feature values ​​and second three-dimensional color values ​​corresponding to multiple sampling points to be synthesized includes: in the weighted fusion layer of the second model, weighting and fusing multiple first three-dimensional feature values ​​and first three-dimensional color values ​​respectively using the first three-dimensional semantic mask values ​​to obtain second three-dimensional feature values ​​and second three-dimensional color values ​​corresponding to multiple sampling points to be synthesized.

[0081] Specifically, for any set of sampling points to be synthesized, the first three-dimensional feature value corresponding to the set of sampling points to be synthesized output by each first model is multiplied by the first three-dimensional semantic mask value to obtain the second three-dimensional feature value corresponding to the set of sampling points to be synthesized; similarly, the first three-dimensional color value corresponding to the set of sampling points to be synthesized output by each first model is multiplied by the first three-dimensional semantic mask value to obtain the second three-dimensional color value corresponding to the set of sampling points to be synthesized.

[0082] In this embodiment of the disclosure, the first image volume density value refers to the volume density function value obtained by transforming multiple first residual directional distance field values ​​through a volume density function.

[0083] In some optional implementations, determining the first image volume density value corresponding to the multiple sampling points to be synthesized based on multiple first residual directed distance field values ​​includes: determining a first initial directed distance field value initialized as a sphere corresponding to the first residual directed distance field value; adding the first residual directed distance field value to the first initial directed distance field value to obtain the first image directed distance field value corresponding to the multiple sampling points to be synthesized; and determining the first image volume density value corresponding to the multiple sampling points to be synthesized based on the first image directed distance field value and a preset volume density function.

[0084] Here, the first initial directed range field value refers to the global directed range field value initialized as a sphere, and is an initialization parameter. The preset volume density function can transform the directed range field value into a volume density value.

[0085] For example, the first residual directed distance field value is added to the first initial directed distance field value to obtain the first image directed distance field value. The first image directed distance field value is then substituted into a preset volume density function to obtain the first image volume density value. The volume density function can be set according to the actual situation and is not specifically limited here.

[0086] In this embodiment of the disclosure, the volume aggregation layer of the second model refers to a network layer that performs volume aggregation processing on three-dimensional feature values, three-dimensional color values, three-dimensional semantic mask values, and image volume density values.

[0087] For example, the volume aggregation layer can use the neural radiation field volume aggregation method to aggregate the second three-dimensional feature value using the first image volume density value to obtain a two-dimensional first two-dimensional feature map; and aggregate the second three-dimensional color value using the first image volume density value to obtain a first two-dimensional color map; and aggregate the first three-dimensional semantic mask value using the first image volume density value to obtain a first two-dimensional mask map.

[0088] S340. Determine the target image corresponding to the first viewpoint information based on the first two-dimensional feature map.

[0089] The technical solution of this disclosure embodiment inputs multiple first three-dimensional feature values, first three-dimensional color values, and first three-dimensional semantic mask values ​​into the weighted fusion layer of the second model to obtain second three-dimensional feature values ​​and second three-dimensional color values ​​corresponding to multiple sampling points to be synthesized; based on multiple first residual directional distance field values, a first image volume density value corresponding to multiple sampling points to be synthesized is determined; the second three-dimensional feature values, second three-dimensional color values, first three-dimensional semantic mask values, and first image volume density values ​​are input into the volume aggregation layer of the second model to obtain a first two-dimensional feature map, a first two-dimensional color map, and a first two-dimensional mask map corresponding to multiple sampling points to be synthesized. This process performs global fusion and volume aggregation on the local features corresponding to multiple sampling points to be synthesized, improving the generation accuracy of global image features.

[0090] Figure 4 This is a flowchart illustrating another image processing method provided in this disclosure. Based on the above embodiments, the technical solution of this disclosure further refines the step of "determining the target image corresponding to the first viewpoint information based on the first two-dimensional feature map." Optionally, determining the target image corresponding to the first viewpoint information based on the first two-dimensional feature map includes: determining the target image corresponding to the first viewpoint information based on the first two-dimensional feature map and a third model. For detailed implementation methods, please refer to the description of the embodiments of this disclosure. Technical features that are the same as or similar to those in the foregoing embodiments will not be repeated here.

[0091] like Figure 4 As shown, the method in this embodiment of the disclosure may specifically include:

[0092] S410. Obtain the first position information corresponding to the multiple sampling points to be synthesized and the first view information corresponding to the multiple sampling points to be synthesized.

[0093] S420. Based on the first position information, the first viewpoint information, and multiple first models, obtain multiple local three-dimensional semantic features corresponding to the sampling points to be synthesized, wherein the local three-dimensional semantic features generated by the first model correspond to the local image information of the target image.

[0094] S430. Determine a first two-dimensional feature map corresponding to the multiple sampling points to be synthesized based on the multiple local three-dimensional semantic features.

[0095] S440. Based on the first two-dimensional feature map and the third model, determine the target image corresponding to the first viewpoint information.

[0096] In this embodiment of the disclosure, the third model refers to a network model that performs high-resolution synthesis based on the first two-dimensional feature map. Specifically, the first two-dimensional feature map is used as input information for the third model, and the third model performs high-resolution synthesis of the image based on the first two-dimensional feature map to obtain a target image corresponding to the first viewpoint information.

[0097] In some optional implementations, determining the target image corresponding to the first viewpoint information based on the first two-dimensional feature map and the third model includes: determining the second modulation latent code corresponding to the third model; inputting the second modulation latent code, the first two-dimensional feature map, and the first two-dimensional mask map into the third model to generate the target image corresponding to the first viewpoint information and the target mask image corresponding to the target image.

[0098] In this embodiment, the third model refers to a network model that performs high-resolution synthesis of the second modulation latent code, the first two-dimensional feature map, and the first two-dimensional mask map. The first two-dimensional mask map refers to the two-dimensional mask map obtained by processing the second model. It should be noted that by processing the second modulation latent code, the first two-dimensional feature map, and the first two-dimensional mask map through the third model, a high-resolution target image and a target mask image can be output, thereby improving image quality.

[0099] In this embodiment of the disclosure, the second modulation latent coding refers to the coding information obtained by sampling the latent vector space of the image, which may include the shape features and texture features of the local region of the target object.

[0100] For example, the third model can be a trained StyleGAN2 model. Specifically, the second modulation latent code, the first two-dimensional feature map, and the first two-dimensional mask map can be input into the StyleGAN2 model to obtain the target image and the target mask image corresponding to the target image.

[0101] In some optional implementations, determining the second modulation latent code corresponding to the third model includes: using the first modulation latent code corresponding to the first model as the second modulation latent code corresponding to the third model; or, determining the second modulation latent code corresponding to the third model based on the first modulation latent code corresponding to the first model and the fifth model; or, determining the second modulation latent code corresponding to the third model based on the noise vector sampled in the standard Gaussian distribution and the sixth model.

[0102] In this embodiment of the disclosure, the fifth or sixth model can be a multilayer perceptron.

[0103] For example, the first modulation latent code can be directly used as the second modulation latent code, which can reduce the image processing steps; or, the first modulation latent code can be input into a multilayer perceptron to obtain a second modulation latent code that is more accurate than the first modulation latent code; or, the second modulation latent code can be obtained by passing a noise vector sampled from a standard Gaussian distribution through a multilayer perceptron mapping layer.

[0104] In some optional implementations, the third model is trained as follows: the sample two-dimensional feature map and the sample two-dimensional mask map are input into the training model corresponding to the third model to obtain the model output image, wherein the model output image includes the model synthesized image and the output mask image corresponding to the model synthesized image; the model output image is input into the third discriminator to obtain the third discrimination result; and the model parameters of the training model corresponding to the third model are adjusted based on the third discrimination result to obtain the third model.

[0105] In this embodiment of the disclosure, the sample two-dimensional feature map and the sample two-dimensional mask map can be the output data of the trained second model. In other words, the sample two-dimensional feature map can be the two-dimensional feature map output by the trained second model, and the sample two-dimensional mask map can be the two-dimensional mask map output by the trained second model.

[0106] For example, the third model can be a high-resolution two-dimensional image generator, which can be composed of a StyleGAN2 model. The third discriminator can be a high-resolution discriminator used to supervise the training of the third model. Specifically, the output data of the trained second model is used as training samples and input into the StyleGAN2 model. The StyleGAN2 model outputs a model-synthesized image and an output mask image corresponding to the model-synthesized image. Then, the model-synthesized image and the output mask image corresponding to the model-synthesized image are input into the high-resolution discriminator to determine whether the model-synthesized image and the output mask image corresponding to the model-synthesized image are real or fake. Based on the judgment result, the model parameters of the StyleGAN2 model are adjusted to obtain the trained StyleGAN2 model.

[0107] The technical solution of this disclosure embodiment processes the second modulation latent coding, the first two-dimensional feature map, and the first two-dimensional mask map using a third model, which can output a high-resolution target image and a target mask image, thereby improving image quality.

[0108] Figure 5 This is a schematic flowchart illustrating another image processing method provided in this disclosure. The technical solution of this disclosure further refines the first model training process based on the above embodiments. Detailed implementation methods can be found in the description of the embodiments of this disclosure. Technical features that are the same as or similar to those in the foregoing embodiments will not be repeated here.

[0109] like Figure 5 As shown, the method in this embodiment of the disclosure may specifically include:

[0110] S510. Obtain the first position information corresponding to the multiple sampling points to be synthesized and the first view information corresponding to the multiple sampling points to be synthesized.

[0111] S520. Based on the first location information, the first viewpoint information, and multiple first models, obtain multiple local three-dimensional semantic features corresponding to the sampling point to be synthesized, wherein the model parameters of the multiple first models are adjusted based on the discrimination results of the first discriminator and the second discriminator.

[0112] The first discriminator is used for global parameter correction of the first model, and the second discriminator is used for local parameter correction of the first model. It should be noted that the second discriminator can correct the local parameters of multiple first models separately to obtain first models capable of extracting features from different local regions.

[0113] In some optional implementations, the first discriminator is used to output a first discrimination result of the second two-dimensional color map and the second two-dimensional mask map output by the second model for multiple sample sampling points, as well as third-view information corresponding to the second two-dimensional color map and the second two-dimensional mask map; the second discriminator is used to output a second discrimination result of the semantic color map and a judgment of the semantic category corresponding to the semantic color map, wherein the semantic color map is obtained by multiplying the second two-dimensional color map and the second two-dimensional mask map output by the second model for multiple sample sampling points.

[0114] Here, the second two-dimensional color map refers to the two-dimensional color map corresponding to the sample sampling points; the second two-dimensional mask map refers to the two-dimensional mask map corresponding to the sample sampling points. The first discrimination result can be determined based on the second two-dimensional color map and the second two-dimensional mask map. The third viewpoint information refers to the viewpoint direction information obtained by the first discriminator from the second two-dimensional color map and the second two-dimensional mask map. The semantic color map can be obtained by multiplying the second two-dimensional color map and the second two-dimensional mask map. The second discrimination result can be determined based on the semantic color map. Determining the semantic category refers to the semantic category information obtained by the second discriminator from the semantic color map.

[0115] In some embodiments, the first model is trained as follows: acquiring second position information corresponding to multiple sample sampling points and second view information corresponding to the multiple sample sampling points, and using the second position information and the second view information as second input information; determining multiple sample local three-dimensional semantic features corresponding to the second input information based on multiple first training models corresponding to the first model; inputting the multiple sample local three-dimensional semantic features into the second model to obtain a second two-dimensional feature map, a second two-dimensional color map, and a second two-dimensional mask map corresponding to the multiple sample sampling points; determining the target identification result corresponding to the second two-dimensional feature map and the second two-dimensional mask map based on a target discriminator; and adjusting the model parameters of the first model based on the target identification result to obtain the first model.

[0116] In this embodiment of the disclosure, the second position information corresponding to the multiple sample sampling points used to train the first model can be obtained by uniform sampling according to a set field of view. The second viewpoint information corresponding to the multiple sample sampling points can be camera viewpoint direction information, which may include pitch angle and yaw angle. The second viewpoint information can be obtained by random sampling within a preset sampling range.

[0117] In this embodiment of the disclosure, the sample local three-dimensional semantic features refer to the result of extracting three-dimensional semantic features from the second input information. The sample local three-dimensional semantic features may include second three-dimensional feature values, second three-dimensional color values, and second three-dimensional semantic mask values.

[0118] For example, taking the generation of a face image as an example, the first model can be a local semantic 3D generator. Each local semantic 3D generator can be used to generate multiple local regions of the face image. The local regions can include at least one of the following: background, face, eyes, eyebrows, nose, mouth, ears, hair, and face. Figure 6 This is a schematic diagram of the architecture of a local semantic 3D generator provided in this embodiment. Each local semantic 3D generator can be composed of a modulotable multilayer perceptron (including a shape-aware network and a texture-aware network) and a fully connected layer (FC), and outputs local 3D semantic features of the sample. The local 3D semantic features of the sample generated by the local semantic 3D generator may include a second 3D feature value (f) of the local region. i (x,v)), second three-dimensional color value (c) i (x,v)), the second three-dimensional semantic mask value (m) i (x,v)), and residual SDF value (Δd) i (x)). Where w i Let represent the modulation latent coding, x represent the second position information, v represent the second view information, and i represent the i-th local semantic 3D generator.

[0119] In this embodiment of the disclosure, the second two-dimensional feature map refers to the two-dimensional feature map corresponding to the sample sampling points.

[0120] For example, Figure 7 This is a schematic diagram of a combined neural radiation field network architecture provided in an embodiment of this disclosure; the combined neural radiation field network architecture consists of a first model (G1…G ... k The model consists of a first model and a second model (weighted fusion layer and volume aggregation layer). The output of each local semantic 3D generator can be fused using a weighted method based on the second 3D semantic mask values ​​to obtain the overall 3D feature values ​​(f(x,v)) and 3D color values ​​(c(x,v)) of the face image. Furthermore, the output of each local semantic 3D generator (Δd) can be further fused. i Add (x) to the global SDF value (d0(x)) initialized to a sphere, and then add this to the global SDF value (d(x)) to obtain the true learnable global image SDF value (d(x)). Substitute this global image SDF value into K. a The bulk density value σ can be calculated. Furthermore, using the volumetric polymerization method, the density σ is calculated based on the pairs of (f(x,v)), (c(x,v)) and (m). i Volume aggregation is performed on (x, v) to obtain a second 2D feature map, a second 2D color map, and a second 2D mask map. Here, w represents the modulation latent coding, and k represents the number of local semantic region categories. During model training, the second positional information and second viewpoint information input to each local semantic 3D generator are shared. The second 2D feature map, second 2D color map, and second 2D mask map can be rendered at a 64×64 resolution.

[0121] The target discriminator can supervise the training of the first model and can be used to determine the authenticity of the model's output data. In some optional implementations, the target discriminator includes a first discriminator and a second discriminator; determining the target identification result corresponding to the second two-dimensional feature map and the second two-dimensional mask map based on the target discriminator includes: concatenating the second two-dimensional color map and the second two-dimensional mask map and inputting them into the first discriminator to obtain a first identification result, and outputting the third-view information corresponding to the second two-dimensional color map and the second two-dimensional mask map; multiplying the second two-dimensional color map by the second two-dimensional mask map to obtain a semantic color map, inputting the semantic color map into the second discriminator to obtain a second identification result, and outputting the judgment semantic category corresponding to the semantic color map; correspondingly, adjusting the model parameters of the training model corresponding to the first model based on the target identification result includes: adjusting the model parameters of the training model corresponding to the first model based on the first identification result, the second identification result, the third-view information, the second-view information corresponding to the third-view information, the judgment semantic category, and the expected semantic category corresponding to the judgment semantic category.

[0122] The first discriminator is used to determine the authenticity of the input second 2D color map and second 2D mask map. The second discriminator is used to determine the authenticity of the input semantic color map. Third-view information refers to the viewpoint direction information obtained by the first discriminator. The semantic category determination refers to the semantic category of the local region obtained by the second discriminator. It is understood that the second-view information can be known in advance during the training of the first discriminator. The expected semantic category can be known in advance during the training of the second discriminator. For example, if the third-view information differs from the second-view information, the model loss of the first model and the corresponding second-view information is calculated based on the third-view information and the corresponding second-view information. This loss is used to adjust the model parameters of the training model corresponding to the first model, so that the first model can better handle the input viewpoint information. If the determined semantic category differs from the expected semantic category, the model loss of the training model of the first model is calculated based on the determined semantic category and the corresponding expected semantic category. This loss is used to adjust the model parameters of the training model corresponding to the first model, so that the first model can better handle local semantics.

[0123] As mentioned earlier, different first models are used to generate local three-dimensional semantic features of different local regions of the image. Therefore, the model loss of the training model of the first model can be calculated based on the judgment semantic category and the expected semantic category corresponding to each semantic category, that is, the model parameters of the training model of the first model can be adjusted according to the semantic category corresponding to the first model to obtain the adjusted first model.

[0124] For example, Figure 8 This is a schematic diagram of a model training scenario for performing the image processing method of this disclosure, provided in an embodiment of the present disclosure. The first discriminator can be a global discriminator, and the second discriminator can be a semantic discriminator. Specifically, a second two-dimensional color image and a second two-dimensional mask image can be concatenated and input into the first discriminator to obtain a first discrimination result, and output third-view information corresponding to the second two-dimensional color image and the second two-dimensional mask image; the second two-dimensional color image is multiplied by the second two-dimensional mask image to obtain a semantic color image, which can then be input into the second discriminator to obtain a second discrimination result, and output the determined semantic category corresponding to the semantic color image; based on the first discrimination result, the second discrimination result, the third-view information, the second-view information corresponding to the third-view information, the determined semantic category, and the expected semantic category corresponding to the determined semantic category, the model parameters of the model to be trained corresponding to the first model are adjusted to obtain the first model. Furthermore, Figure 8In this context, z ~ N(0,1) represents a noise vector sampled from a standard Gaussian distribution. The Style-based 2D Generator represents the third model in the above embodiments, and the High-Res Discriminator represents the third discriminator in the above embodiments.

[0125] After training the two-stage model, it is possible to generate a three-dimensional viewpoint consistent face image. By inputting viewpoint information, a face image under that viewpoint information can be generated.

[0126] In some embodiments, the shape and texture of each local region can be manipulated simultaneously and individually. Specifically, shape parameters and / or texture parameters in the modulation latent coding can be manipulated to achieve individual control over the shape and texture of the local region, thereby improving the flexibility of image manipulation.

[0127] S530. Based on the multiple local three-dimensional semantic features, determine the first two-dimensional feature map corresponding to the multiple sampling points to be synthesized.

[0128] S540. Determine the target image corresponding to the first viewpoint information based on the first two-dimensional feature map.

[0129] The technical solution of this disclosure adjusts the model parameters of multiple first models based on the discrimination results of the first discriminator and the second discriminator. The first discriminator is used for global parameter correction of the first model, and the second discriminator is used for local parameter correction of the first model, so as to obtain a first model with higher prediction accuracy.

[0130] Figure 9 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of the present disclosure, as shown below. Figure 9 As shown, the device includes: an information input module 610, a semantic feature acquisition module 620, an intermediate image determination module 630, and a target image generation module 640.

[0131] The system includes: an information input module 610 for acquiring first position information corresponding to multiple sampling points to be synthesized and first view information corresponding to the multiple sampling points to be synthesized; a semantic feature acquisition module 620 for acquiring multiple local three-dimensional semantic features corresponding to the sampling points to be synthesized based on the first position information, the first view information, and multiple first models, wherein the local three-dimensional semantic features generated by the first models correspond to the local image information of the target image; an intermediate image determination module 630 for determining a first two-dimensional feature map corresponding to the multiple sampling points to be synthesized based on the multiple local three-dimensional semantic features; and a target image generation module 640 for determining a target image corresponding to the first view information based on the first two-dimensional feature map.

[0132] The technical solution of this disclosure acquires multiple sampling point data by obtaining first position information and first view information corresponding to multiple sampling points to be synthesized. Then, based on the first position information, first view information, and multiple first models, multiple local three-dimensional semantic features corresponding to the sampling points to be synthesized are obtained, achieving the extraction of local three-dimensional semantic features. The local three-dimensional semantic features generated by the first model correspond to the local image information of the target image. Based on the multiple local three-dimensional semantic features, a first two-dimensional feature map corresponding to the multiple sampling points to be synthesized is determined, achieving the extraction of the two-dimensional feature map. Furthermore, based on the first two-dimensional feature map, a target image corresponding to the first view information is determined, achieving the acquisition of images from the viewpoints corresponding to the multiple sampling points to be synthesized. In other words, multiple sampling points to be synthesized can be processed by region, improving the refinement of the generated image and optimizing the image processing effect. Multi-view image synthesis results can be obtained by controlling the input view information. The image style of the local region can be changed by controlling the parameters of the first model corresponding to the local region of the image.

[0133] Optionally, the semantic feature acquisition module 620 is specifically used for:

[0134] A first modulation latent code corresponding to the first model is determined, wherein the first modulation latent code includes a shape modulation latent code for modulating the shape-aware network in the first model and a texture modulation latent code for modulating the texture-aware network in the first model;

[0135] The first position information, the first viewpoint information, and the first modulation latent code are input into multiple first models to obtain multiple local three-dimensional semantic features corresponding to the sampling points to be synthesized.

[0136] Optionally, the intermediate image determination module 630 includes:

[0137] The second model processing unit is used to determine a first two-dimensional feature map corresponding to the multiple sampling points to be synthesized, based on the multiple local three-dimensional semantic features and the second model.

[0138] Optionally, the second model includes a weighted fusion layer and a volume aggregation layer, wherein the local three-dimensional semantic features include a first three-dimensional feature value, a first three-dimensional color value, a first three-dimensional semantic mask value, and a first residual directed distance field value; the second model processing unit includes:

[0139] The weighted fusion subunit is used to input multiple first three-dimensional feature values, first three-dimensional color values ​​and first three-dimensional semantic mask values ​​into the weighted fusion layer of the second model to obtain second three-dimensional feature values ​​and second three-dimensional color values ​​corresponding to the multiple sampling points to be synthesized;

[0140] The volume density value determination subunit is used to determine the first image volume density value corresponding to the multiple sampling points to be synthesized based on multiple first residual directional distance field values;

[0141] The volume aggregation subunit is used to input the second three-dimensional feature value, the second three-dimensional color value, the first three-dimensional semantic mask value and the first image volume density value into the volume aggregation layer of the second model to obtain the first two-dimensional feature map, the first two-dimensional color map and the first two-dimensional mask map corresponding to the plurality of sampling points to be synthesized.

[0142] Optional, weighted fusion subunit, specifically used for:

[0143] In the weighted fusion layer of the second model, multiple first three-dimensional feature values ​​and first three-dimensional color values ​​are weighted and fused using the first three-dimensional semantic mask value to obtain second three-dimensional feature values ​​and second three-dimensional color values ​​corresponding to the multiple sampling points to be synthesized.

[0144] Optionally, the volume density value determines the sub-unit, specifically for:

[0145] Determine the initial directional distance field value corresponding to the first residual directional distance field value as the first initial directional distance field value of the sphere;

[0146] Add the first residual directed distance field value to the first initial directed distance field value to obtain the first image directed distance field value corresponding to the plurality of sampling points to be synthesized;

[0147] Based on the directional distance field value of the first image and the preset volume density function, the volume density value of the first image corresponding to the plurality of sampling points to be synthesized is determined.

[0148] Optionally, the target image generation module 640 includes:

[0149] The third model processing unit is used to determine the target image corresponding to the first viewpoint information based on the first two-dimensional feature map and the third model.

[0150] Optionally, the third model processing unit includes:

[0151] The modulation latent coding determination subunit is used to determine the second modulation latent coding corresponding to the third model;

[0152] The image prediction subunit is used to input the second modulation latent coding, the first two-dimensional feature map and the first two-dimensional mask map into the third model to generate a target image corresponding to the first viewpoint information and a target mask image corresponding to the target image.

[0153] Optional, image prediction subunit, specifically used for:

[0154] Use the first modulation latent code corresponding to the first model as the second modulation latent code corresponding to the third model; or...

[0155] The second modulation latent code corresponding to the third model is determined based on the first modulation latent code corresponding to the first model and the fifth model; or...

[0156] The second modulation latent code corresponding to the third model is determined based on the noise vector sampled from the standard Gaussian distribution and the sixth model.

[0157] Optionally, the model parameters of the plurality of first models are adjusted based on the discrimination results of the first discriminator and the second discriminator, wherein the first discriminator is used for global parameter correction of the first model and the second discriminator is used for local parameter correction of the first model.

[0158] Optionally, the first discriminator is used to output a first discrimination result of the second two-dimensional color map and the second two-dimensional mask map output by the second model for multiple sample sampling points, as well as third-view information corresponding to the second two-dimensional color map and the second two-dimensional mask map; the second discriminator is used to output a second discrimination result of the semantic color map and a judgment semantic category corresponding to the semantic color map, wherein the semantic color map is obtained by multiplying the second two-dimensional color map and the second two-dimensional mask map output by the second model for multiple sample sampling points.

[0159] The image processing apparatus provided in this disclosure can execute the image processing method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the method.

[0160] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.

[0161] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Reference is made below. Figure 10 It illustrates an electronic device suitable for implementing embodiments of the present disclosure (e.g., Figure 10 The diagram below shows the structure of the terminal device or server 500. The terminal device in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and vehicle terminals (e.g., vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 10 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0162] like Figure 10 As shown, electronic device 500 may include a processing unit (e.g., central processing unit, graphics processor, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from storage device 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. An edit / output (I / O) interface 505 is also connected to bus 504.

[0163] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 10 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0164] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the methods of embodiments of this disclosure.

[0165] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0166] The electronic device provided in this embodiment and the image processing method provided in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0167] This disclosure provides a computer storage medium storing a computer program that, when executed by a processor, implements the image processing method provided in the above embodiments.

[0168] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0169] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0170] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0171] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to:

[0172] Obtain first position information corresponding to multiple sampling points to be synthesized and first view information corresponding to the multiple sampling points to be synthesized;

[0173] Based on the first location information, the first viewpoint information, and multiple first models, multiple local three-dimensional semantic features corresponding to the sampling points to be synthesized are obtained, wherein the local three-dimensional semantic features generated by the first model correspond to the local image information of the target image.

[0174] A first two-dimensional feature map corresponding to the multiple sampling points to be synthesized is determined based on multiple local three-dimensional semantic features;

[0175] The target image corresponding to the first viewpoint information is determined based on the first two-dimensional feature map.

[0176] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0177] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0178] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0179] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0180] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0181] According to one or more embodiments of this disclosure, [Example 1] provides an image processing method, including:

[0182] Obtain first position information corresponding to multiple sampling points to be synthesized and first view information corresponding to the multiple sampling points to be synthesized, and use the first position information and the first view information as first input information;

[0183] Obtain first position information corresponding to multiple sampling points to be synthesized and first view information corresponding to the multiple sampling points to be synthesized;

[0184] Based on the first location information, the first viewpoint information, and multiple first models, multiple local three-dimensional semantic features corresponding to the sampling points to be synthesized are obtained, wherein the local three-dimensional semantic features generated by the first model correspond to the local image information of the target image.

[0185] A first two-dimensional feature map corresponding to the multiple sampling points to be synthesized is determined based on multiple local three-dimensional semantic features;

[0186] The target image corresponding to the first viewpoint information is determined based on the first two-dimensional feature map.

[0187] According to one or more embodiments of this disclosure, [Example 2] provides an image processing method, further comprising:

[0188] Based on the first location information, the first viewpoint information, and multiple first models, multiple local three-dimensional semantic features corresponding to the sampling points to be synthesized are obtained, including:

[0189] A first modulation latent code corresponding to the first model is determined, wherein the first modulation latent code includes a shape modulation latent code for modulating the shape-aware network in the first model and a texture modulation latent code for modulating the texture-aware network in the first model;

[0190] The first position information, the first viewpoint information, and the first modulation latent code are input into multiple first models to obtain multiple local three-dimensional semantic features corresponding to the sampling points to be synthesized.

[0191] According to one or more embodiments of this disclosure, [Example 3] provides an image processing method, further comprising:

[0192] The step of determining the first two-dimensional feature map corresponding to the multiple sampling points to be synthesized based on the multiple local three-dimensional semantic features includes:

[0193] Based on multiple local three-dimensional semantic features and the second model, a first two-dimensional feature map corresponding to the multiple sampling points to be synthesized is determined.

[0194] According to one or more embodiments of this disclosure, [Example 4] provides an image processing method, further comprising:

[0195] The second model includes a weighted fusion layer and a volume aggregation layer, and the local three-dimensional semantic features include a first three-dimensional feature value, a first three-dimensional color value, a first three-dimensional semantic mask value, and a first residual directional distance field value;

[0196] The step of determining the first two-dimensional feature map corresponding to the multiple sampling points to be synthesized based on multiple local three-dimensional semantic features and the second model includes:

[0197] Multiple first three-dimensional feature values, first three-dimensional color values, and first three-dimensional semantic mask values ​​are input into the weighted fusion layer of the second model to obtain second three-dimensional feature values ​​and second three-dimensional color values ​​corresponding to the multiple sampling points to be synthesized.

[0198] Based on multiple first residual directional distance field values, determine the first image volume density value corresponding to the multiple sampling points to be synthesized;

[0199] The second three-dimensional feature value, the second three-dimensional color value, the first three-dimensional semantic mask value, and the first image volume density value are input into the volume aggregation layer of the second model to obtain the first two-dimensional feature map, the first two-dimensional color map, and the first two-dimensional mask map corresponding to the plurality of sampling points to be synthesized.

[0200] According to one or more embodiments of this disclosure, [Example 5] provides an image processing method, further comprising:

[0201] The step of inputting multiple first three-dimensional feature values, first three-dimensional color values, and first three-dimensional semantic mask values ​​into the weighted fusion layer of the second model to obtain second three-dimensional feature values ​​and second three-dimensional color values ​​corresponding to the multiple sampling points to be synthesized includes:

[0202] In the weighted fusion layer of the second model, multiple first three-dimensional feature values ​​and first three-dimensional color values ​​are weighted and fused using the first three-dimensional semantic mask value to obtain second three-dimensional feature values ​​and second three-dimensional color values ​​corresponding to the multiple sampling points to be synthesized.

[0203] According to one or more embodiments of this disclosure, [Example Six] provides an image processing method, further comprising:

[0204] The step of determining the first image volume density value corresponding to the plurality of sampling points to be synthesized based on a plurality of first residual directional distance field values ​​includes:

[0205] Determine the initial directional distance field value corresponding to the first residual directional distance field value as the first initial directional distance field value of the sphere;

[0206] Add the first residual directed distance field value to the first initial directed distance field value to obtain the first image directed distance field value corresponding to the plurality of sampling points to be synthesized;

[0207] Based on the directional distance field value of the first image and the preset volume density function, the volume density value of the first image corresponding to the plurality of sampling points to be synthesized is determined.

[0208] According to one or more embodiments of this disclosure, [Example Seven] provides an image processing method, further comprising:

[0209] Determining the target image corresponding to the first viewpoint information based on the first two-dimensional feature map includes:

[0210] Based on the first two-dimensional feature map and the third model, the target image corresponding to the first viewpoint information is determined.

[0211] According to one or more embodiments of this disclosure, [Example Eight] provides an image processing method, further comprising:

[0212] The step of determining the target image corresponding to the first viewpoint information based on the first two-dimensional feature map and the third model includes:

[0213] Determine the second modulation latent code corresponding to the third model;

[0214] The second modulation latent code, the first two-dimensional feature map, and the first two-dimensional mask map are input into the third model to generate a target image corresponding to the first viewpoint information and a target mask image corresponding to the target image.

[0215] According to one or more embodiments of this disclosure, [Example Nine] provides an image processing method, further comprising:

[0216] The model parameters of the plurality of first models are adjusted based on the discrimination results of the first discriminator and the second discriminator. The first discriminator is used for global parameter correction of the first model, and the second discriminator is used for local parameter correction of the first model.

[0217] According to one or more embodiments of this disclosure, [Example 10] provides an image processing method, further comprising:

[0218] The first discriminator is used to output a first discrimination result of the second two-dimensional color map and the second two-dimensional mask map output by the second model for multiple sample sampling points, as well as third view information corresponding to the second two-dimensional color map and the second two-dimensional mask map;

[0219] The second discriminator is used to output a second discrimination result of the semantic color map and a judgment semantic category corresponding to the semantic color map. The semantic color map is obtained by multiplying the second two-dimensional color map and the second two-dimensional mask map output by the second model for multiple sample sampling points.

[0220] According to one or more embodiments of this disclosure, [Example 11] provides an image processing apparatus, the apparatus comprising:

[0221] The information input module is used to acquire first position information corresponding to multiple sampling points to be synthesized and first view information corresponding to the multiple sampling points to be synthesized;

[0222] The semantic feature acquisition module is used to obtain multiple local three-dimensional semantic features corresponding to the sampling point to be synthesized based on the first location information, the first viewpoint information, and multiple first models, wherein the local three-dimensional semantic features generated by the first model correspond to the local image information of the target image.

[0223] The intermediate image determination module is used to determine a first two-dimensional feature map corresponding to the multiple sampling points to be synthesized based on multiple local three-dimensional semantic features;

[0224] The target image generation module is used to determine the target image corresponding to the first viewpoint information based on the first two-dimensional feature map.

[0225] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0226] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0227] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An image processing method, characterized in that, include: Obtain first position information corresponding to multiple sampling points to be synthesized and first view information corresponding to the multiple sampling points to be synthesized; A first modulation latent code corresponding to the first model is determined, wherein the first modulation latent code includes a shape modulation latent code for modulating the shape-aware network in the first model and a texture modulation latent code for modulating the texture-aware network in the first model; The first position information, the first viewpoint information, and the first modulation latent coding are input into multiple first models to obtain multiple local three-dimensional semantic features corresponding to the sampling points to be synthesized, wherein the local three-dimensional semantic features generated by the first model correspond to the local image information of the target image. A first two-dimensional feature map corresponding to the multiple sampling points to be synthesized is determined based on multiple local three-dimensional semantic features; Determine the second modulation latent code corresponding to the third model; wherein, the second modulation latent code refers to the coding information obtained by sampling the latent vector space of the image, and the second modulation latent code includes the shape features and texture features of the local region of the target object; The second modulation latent code, the first two-dimensional feature map, and the first two-dimensional mask map are input into the third model to generate a target image corresponding to the first viewpoint information and a target mask image corresponding to the target image.

2. The image processing method according to claim 1, characterized in that, The step of determining the first two-dimensional feature map corresponding to the multiple sampling points to be synthesized based on multiple local three-dimensional semantic features includes: Based on multiple local three-dimensional semantic features and the second model, a first two-dimensional feature map corresponding to the multiple sampling points to be synthesized is determined.

3. The image processing method according to claim 2, characterized in that, The second model includes a weighted fusion layer and a volume aggregation layer, and the local three-dimensional semantic features include a first three-dimensional feature value, a first three-dimensional color value, a first three-dimensional semantic mask value, and a first residual directional distance field value; The step of determining the first two-dimensional feature map corresponding to the multiple sampling points to be synthesized based on multiple local three-dimensional semantic features and the second model includes: Multiple first three-dimensional feature values, first three-dimensional color values, and first three-dimensional semantic mask values ​​are input into the weighted fusion layer of the second model to obtain second three-dimensional feature values ​​and second three-dimensional color values ​​corresponding to the multiple sampling points to be synthesized. Based on multiple first residual directional distance field values, determine the first image volume density value corresponding to the multiple sampling points to be synthesized; The second three-dimensional feature value, the second three-dimensional color value, the first three-dimensional semantic mask value, and the first image volume density value are input into the volume aggregation layer of the second model to obtain the first two-dimensional feature map, the first two-dimensional color map, and the first two-dimensional mask map corresponding to the plurality of sampling points to be synthesized.

4. The image processing method according to claim 3, characterized in that, The step of inputting multiple first three-dimensional feature values, first three-dimensional color values, and first three-dimensional semantic mask values ​​into the weighted fusion layer of the second model to obtain second three-dimensional feature values ​​and second three-dimensional color values ​​corresponding to the multiple sampling points to be synthesized includes: In the weighted fusion layer of the second model, multiple first three-dimensional feature values ​​and first three-dimensional color values ​​are weighted and fused using the first three-dimensional semantic mask value to obtain second three-dimensional feature values ​​and second three-dimensional color values ​​corresponding to the multiple sampling points to be synthesized.

5. The image processing method according to claim 3, characterized in that, The step of determining the first image volume density value corresponding to the plurality of sampling points to be synthesized based on a plurality of first residual directional distance field values ​​includes: Determine the initial directional distance field value corresponding to the first residual directional distance field value as the first initial directional distance field value of the sphere; Add the first residual directed distance field value to the first initial directed distance field value to obtain the first image directed distance field value corresponding to the plurality of sampling points to be synthesized; Based on the directional distance field value of the first image and the preset volume density function, the volume density value of the first image corresponding to the plurality of sampling points to be synthesized is determined.

6. The image processing method according to claim 1, characterized in that, The model parameters of the plurality of first models are adjusted based on the discrimination results of the first discriminator and the second discriminator. The first discriminator is used for global parameter correction of the first model, and the second discriminator is used for local parameter correction of the first model.

7. The image processing method according to claim 6, characterized in that, The first discriminator is used to output a first discrimination result of the second two-dimensional color map and the second two-dimensional mask map output by the second model for multiple sample sampling points, as well as third view information corresponding to the second two-dimensional color map and the second two-dimensional mask map; The second discriminator is used to output a second discrimination result of the semantic color map and a judgment semantic category corresponding to the semantic color map. The semantic color map is obtained by multiplying the second two-dimensional color map and the second two-dimensional mask map output by the second model for multiple sample sampling points.

8. An image processing apparatus, characterized in that, include: The information input module is used to acquire first position information corresponding to multiple sampling points to be synthesized and first view information corresponding to the multiple sampling points to be synthesized; A semantic feature acquisition module is used to determine a first modulation latent code corresponding to a first model, wherein the first modulation latent code includes a shape modulation latent code for modulating a shape-aware network in the first model and a texture modulation latent code for modulating a texture-aware network in the first model; the first position information, the first viewpoint information and the first modulation latent code are input into multiple first models to obtain multiple local three-dimensional semantic features corresponding to the sampling points to be synthesized, wherein the local three-dimensional semantic features generated by the first model correspond to the local image information of the target image; The intermediate image determination module is used to determine a first two-dimensional feature map corresponding to the multiple sampling points to be synthesized based on multiple local three-dimensional semantic features; The target image generation module is used to determine the second modulation latent code corresponding to the third model; wherein, the second modulation latent code refers to the coding information obtained by sampling the latent vector space of the image, and the second modulation latent code contains the shape features and texture features of the local region of the target object; the second modulation latent code, the first two-dimensional feature map and the first two-dimensional mask map are input into the third model to generate a target image corresponding to the first viewpoint information and a target mask image corresponding to the target image.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in any one of claims 1-7.

10. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the image processing method as described in any one of claims 1-7.

Images