Image reconstruction method, device, electronic device and computer readable storage medium

Abstract

The embodiments of the present application provide an image reconstruction method, device, electronic device and computer-readable storage medium, which relate to the fields of artificial intelligence, computer vision technology, smart transportation and cloud technology. The method includes: obtaining an image to be processed corresponding to a target object, and obtaining first texture information and first geometric information of the image to be processed. Performing style feature extraction on the image to be processed to obtain a first style feature of the image to be processed. The first texture information is repaired using the first style feature to obtain second texture information; image reconstruction is performed based on the second texture information and the first geometric information to obtain a reconstructed target image. The method repairs the first texture information based on the first style feature obtained by extracting the style feature of the image to be processed to obtain the second texture information. Image reconstruction is performed based on the second texture information and the first geometric information to obtain a target image with richer texture information.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence, computer vision technology, smart transportation and cloud technology. Specifically, the present application relates to an image reconstruction method, device, electronic device, computer-readable storage medium and computer program product. Background Art

[0002] In the related art, parametric methods or non-parametric methods are usually used when performing image reconstruction. When the parametric method is used to perform image reconstruction tasks, it takes a long time, the reconstruction results are usually too smooth, lack details and other significant structural features, and it is difficult to perform large-scale image reconstruction tasks. When the non-parametric method is used to perform image reconstruction tasks, due to the lack of a pre-defined topological structure, the resulting image is easily disturbed by noise and posture, and it is impossible to reconstruct a low-quality image into a high-quality image.

[0003] Based on the above, how to realize the reconstruction of low-quality images into high-quality images on a large scale and effectively improve the image quality of the images to be processed has become a technical problem that needs to be solved urgently in this field. Summary of the invention

[0004] The embodiments of the present application provide an image reconstruction method, device, electronic device, computer-readable storage medium and computer program product. The method can reconstruct a low-quality image corresponding to a target object into a corresponding high-quality image, thereby effectively improving the image quality of the image to be processed.

[0005] According to one aspect of an embodiment of the present application, there is provided an image reconstruction method, the method comprising:

[0006] Obtain the image to be processed corresponding to the target object;

[0007] Acquire initial image information of an image to be processed, where the initial image information includes first texture information and first geometric information;

[0008] Based on the image to be processed and the first texture information, the trained image reconstruction model performs the following operations to obtain the second texture information:

[0009] Extracting style features from the image to be processed to obtain a first style feature of the image to be processed; restoring the first texture information using the first style feature to obtain second texture information;

[0010] Image reconstruction is performed based on the second texture information and the first geometric information to obtain a target image corresponding to the image to be processed, wherein the image quality of the target image is higher than the image quality of the image to be processed.

[0011] According to another aspect of an embodiment of the present application, an image reconstruction device is provided, the device comprising an image acquisition module, an image information acquisition module, a texture restoration module and an image reconstruction module, wherein:

[0012] An image acquisition module is used to acquire an image to be processed corresponding to a target object;

[0013] An image information acquisition module, used to acquire initial image information of an image to be processed, wherein the initial image information includes first texture information and first geometric information;

[0014] The texture restoration module is used to obtain the second texture information by performing the following operations through the trained image reconstruction model based on the image to be processed and the first texture information:

[0015] Extracting style features from the image to be processed to obtain a first style feature of the image to be processed; restoring the first texture information using the first style feature to obtain second texture information;

[0016] The image reconstruction module is used to perform image reconstruction based on the second texture information and the first geometric information to obtain a target image corresponding to the image to be processed, wherein the image quality of the target image is higher than the image quality of the image to be processed.

[0017] Optionally, when the image reconstruction module performs image reconstruction based on the second texture information and the first geometric information to obtain a target image corresponding to the image to be processed, it is specifically used to:

[0018] Using the second texture information, repairing the first geometric information through an image reconstruction model to obtain second geometric information;

[0019] Image reconstruction is performed based on the second texture information and the second geometric information to obtain a target image.

[0020] Optionally, the first geometric information includes an initial surface normal vector map of the image to be processed, and the image reconstruction module, when using the second texture information to repair the first geometric information through the image reconstruction model to obtain the second geometric information, is specifically used to:

[0021] Reconstructing the image based on the second texture information and the first geometric information to obtain a first reconstructed image;

[0022] According to the first reconstructed image and the image to be processed, a first displacement feature map corresponding to the initial surface normal vector map is obtained through an image reconstruction model;

[0023] The initial surface normal vector map is corrected by using the first displacement feature map to obtain a target surface normal vector map, and the second geometric information is the target surface normal vector map.

[0024] Optionally, when the image reconstruction module acquires the first displacement feature map corresponding to the initial surface normal vector map through the image reconstruction model according to the first reconstructed image and the image to be processed, it is specifically used to:

[0025] Extracting displacement features from the first reconstructed image to obtain a second displacement feature map;

[0026] Extract displacement features from the image to be processed to obtain a third displacement feature map;

[0027] The second displacement feature map is merged with the third displacement feature map to obtain a fourth displacement feature map;

[0028] The fourth displacement characteristic map is reconstructed to obtain the first displacement characteristic map.

[0029] Optionally, the initial image information further includes image additional information, the image additional information includes at least one of acquisition environment information corresponding to the image to be processed or first posture information corresponding to the target object, and the image reconstruction module performs image reconstruction based on the second texture information and the first geometric information to obtain the first reconstructed image, specifically for:

[0030] Image reconstruction is performed based on the second texture information, the first geometric information and the image additional information to obtain a first reconstructed image.

[0031] Optionally, when the texture restoration module uses the first style feature to restore the first texture information to obtain the second texture information, it is specifically used to:

[0032] Performing feature extraction on the first texture information to obtain a first texture feature;

[0033] Modifying the first texture feature according to the first style feature to obtain a second texture feature;

[0034] Texture reconstruction is performed according to the second texture feature to obtain second texture information.

[0035] Optionally, the above device further includes a noise acquisition module,

[0036] The noise acquisition module is used to acquire random noise;

[0037] When the texture restoration module performs texture reconstruction according to the second texture feature to obtain the second texture information, it is specifically used to:

[0038] The random noise is fused with the first texture feature to obtain the first texture feature with noise;

[0039] The first style feature is fused with the first texture feature with noise to obtain a second texture feature.

[0040] Optionally, the initial image information further includes image additional information, the image additional information includes at least one of acquisition environment information corresponding to the image to be processed or first posture information corresponding to the target object, and the image reconstruction module performs image reconstruction based on the second texture information and the first geometric information to obtain a target image corresponding to the image to be processed, specifically for:

[0041] Image reconstruction is performed based on the second texture information, the first geometric information and the image additional information to obtain a target image.

[0042] Optionally, the image reconstruction model is trained by a model training device in the following manner:

[0043] Acquire a training data set, the training data set including a plurality of image pairs, each image pair including a first sample image and a second sample image having the same image content, and the image quality of the second sample image in an image pair is higher than the image quality of the first sample image;

[0044] For each image pair, first image information of a first sample image and second image information of a second sample image in the image pair are obtained, wherein the first image information includes third texture information and third geometric information, and the second image information includes fourth texture information and fourth geometric information;

[0045] Based on each image pair, and the corresponding third texture information and fourth texture information, repeatedly performing a first training operation on the initial texture restoration network until a first condition is satisfied to obtain a first texture restoration network, wherein the first training operation comprises: for each image pair, inputting a first sample image and the corresponding third texture information in the image pair into the initial texture restoration network to obtain fifth texture information; determining a first training loss based on the fifth texture information and the fourth texture information corresponding to each image pair, and adjusting a network parameter of the initial texture restoration network based on the first training loss;

[0046] For each first sample image, image reconstruction is performed based on third geometric information corresponding to the first sample image and sixth texture information of the first sample image obtained by the first texture restoration network to obtain a second reconstructed image;

[0047] Based on each image pair, the corresponding second reconstructed image and the fourth geometric information, repeatedly performing a second training operation on the initial geometric restoration network until a second condition is satisfied to obtain a first geometric restoration network, wherein the second training operation includes: for each image pair, inputting a first sample image and a corresponding second reconstructed image in the image pair into the initial geometric restoration network to obtain fifth geometric information; determining a second training loss based on the fifth geometric information and the fourth geometric information corresponding to each image pair, and adjusting a network parameter of the initial geometric restoration network based on the second training loss;

[0048] Based on each image pair, and the corresponding first image information and second image information, a third training operation is performed on the intermediate neural network until a third condition is met to obtain an image reconstruction model. The intermediate neural network includes a cascaded first texture restoration network and a first geometric restoration network.

[0049] According to another aspect of an embodiment of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the above method.

[0050] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0051] According to another aspect of the embodiments of the present application, a computer program product is provided, including a computer program, which implements the steps of the above method when executed by a processor.

[0052] The beneficial effects of the technical solution provided by the embodiment of the present application are:

[0053] The image reconstruction method provided by the present application, when reconstructing the image to be processed, extracts the style features of the image to be processed, and uses the style features to repair the first texture information (initial texture information) of the image to be processed, to obtain the repaired second texture information, and by integrating the style features of the image into the repair of the texture information, the quality of the image texture information can be improved, so that a high-quality target image can be reconstructed based on the repaired high-quality texture information and the geometric information of the image to be processed. The method provided by the embodiment of the present application effectively improves the texture information of the target object in the image, and realizes the reconstruction of the low-quality image corresponding to the target object into the corresponding high-quality image. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in describing the embodiments of the present application.

[0055] Figure 1 A flowchart of an image reconstruction method provided in an embodiment of the present application;

[0056] Figure 2 A structural diagram of an image reconstruction system provided in an embodiment of the present application is shown;

[0057] Figure 3 A schematic diagram showing the acquisition of second texture information through a texture restoration network in an embodiment of the present application is shown;

[0058] Figure 4A schematic diagram showing the acquisition of a target surface normal vector by a geometric restoration network in an embodiment of the present application is shown;

[0059] Figure 5 A schematic diagram of an image reconstruction system provided by an embodiment of the present application is shown;

[0060] Figure 6 A flowchart showing an information verification method in an application scenario applicable to an embodiment of the present application is shown;

[0061] Figure 7 A schematic diagram showing a display interface of a door access terminal in an application scenario applicable to an embodiment of the present application is shown;

[0062] Figure 8 Another schematic diagram showing a display interface of a door access terminal in an application scenario applicable to an embodiment of the present application;

[0063] Fig. 9 A flow chart showing a vehicle detection method in an application scenario applicable to an embodiment of the present application;

[0064] Fig.10 A schematic diagram of an image reconstruction device according to an embodiment of the present application is shown;

[0065] Fig.11 A schematic structural diagram of an electronic device applicable to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0066] The embodiments of the present application are described below in conjunction with the drawings in the present application. It should be understood that the implementation methods described below in conjunction with the drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0067] It will be understood by those skilled in the art that, unless specifically stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present application refer to that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude the implementation as other features, information, data, steps, operations, elements, components and / or combinations thereof supported by the technical field. It should be understood that when we say that an element is "connected" or "coupled" to another element, the one element may be directly connected or coupled to the other element, or it may refer to that the one element and the other element establish a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein indicates at least one of the items defined by the term, for example, "A and / or B" may be implemented as "A", or as "B", or as "A and B".

[0068] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0069] As described above, whether a parametric method or a non-parametric method is used for image reconstruction, it is impossible to reconstruct low-quality images into high-quality images on a large scale, and effectively improve the image quality of the image to be processed. Based on this, the embodiments of the present application provide an image reconstruction method, device, electronic device, computer-readable storage medium and computer program product. In this image reconstruction method, by incorporating the style features of the image into the restoration of texture information, the quality of the image texture information can be improved, so that a high-quality target image can be reconstructed based on the restored high-quality texture information and the geometric information of the image to be processed, effectively improving the texture information of the target object in the image, and realizing the reconstruction of the low-quality image corresponding to the target object into the corresponding high-quality image.

[0070] The execution subject of the image reconstruction method provided in the embodiment of the present application may be an image reconstruction device. The image reconstruction device may be any computer device, including but not limited to a terminal or a server. The server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal (also referred to as a user terminal or user equipment (UE)) may be a smart phone, a tablet computer, a laptop computer, a desktop computer, an intelligent voice interaction device (such as a smart speaker), a wearable electronic device (such as a smart watch), a vehicle terminal, a smart home appliance (such as a smart TV), an AR (Augmented Reality) / VR (Virtual Reality) device, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in this application. The image reconstruction method may be implemented by a processor calling computer-readable instructions stored in a memory.

[0071] The image reconstruction method provided in the embodiment of the present application can be implemented based on artificial intelligence (AI) technology. For example, an image reconstruction model is constructed based on artificial intelligence technology, and based on the constructed image reconstruction model, the image to be processed is reconstructed into a target image. AI is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0072] The above image reconstruction model can also be a neural network model based on computer vision technology (CV). Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing to make the computer processing into an image that is more suitable for human eye observation or transmission to instrument detection.

[0073] The image reconstruction method provided in the embodiment of the present application can be implemented based on cloud technology. For example, the data calculation involved in the training of the image reconstruction model can be implemented using cloud computing. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network in a wide area network or a local area network to achieve data calculation, storage, processing, and sharing.

[0074] The face recognition method provided in the embodiment of the present application can also be implemented in the field of intelligent transportation and applied to autonomous driving or traffic applications. For example, according to the image reconstruction method, blurred vehicle images, driver images and other images taken by driving photography equipment can be reconstructed to further determine whether the vehicle driven by the driver is an illegal vehicle based on the high-quality vehicle images and driver images.

[0075] Optionally, the image reconstruction method provided in the embodiment of the present application can also be implemented based on blockchain technology. Specifically, the data used in the image reconstruction process, such as training data sets, images to be processed, etc., can be stored on the blockchain.

[0076] The data processing method can be applied to various application scenarios, which can be any scenario that needs to obtain detailed information of the target object from a low-quality image, including but not limited to information verification, vehicle detection, etc. The application scenario can also include but not limited to the production process of face pendants, stickers, etc., so as to produce face pendants, stickers, etc. corresponding to the image information according to the image information. Among them, the target object can include but not limited to people, animals, vehicles, etc.

[0077] It is understandable that in the specific implementation of the present application, related data such as user images are involved. When the above embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0078] The following describes several exemplary embodiments to illustrate the technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application. It should be noted that the following embodiments can refer to, draw on or combine with each other, and the same terms, similar features and similar implementation steps in different embodiments will not be described repeatedly.

[0079] Figure 1 A flowchart of an image reconstruction method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes steps S110 to S140. Among them,

[0080] Step S110: Acquire the image to be processed corresponding to the target object.

[0081] The embodiment of the present application does not limit the image quality of the image to be processed. The image quality can be characterized by one or more image indicators, which is not limited by the present application. Specifically, the image indicators may include but are not limited to resolution, clarity, etc. For example, the acquired image to be processed of the target object may be an image with a resolution of 64×64, or an image with a resolution of 124×124, etc.

[0082] Step S120: Acquire initial image information of the image to be processed, where the initial image information includes first texture information and first geometric information.

[0083] In this step, the image to be processed can be reconstructed by any method in the related art, and the initial image information of the image to be processed can be extracted based on the reconstructed image, and the embodiment of the present application does not limit this. For example, the non-parametric method unsup3D (unsupervised 3D shape reconstruction from 3D imageGANs, unsupervised 3D shape reconstruction based on 3D image generative adversarial networks (GANs)) can be used to reconstruct the image, and the image information in the reconstructed image can be extracted, and the image information in the reconstructed image can be used as the initial image information of the image to be processed. This process can also be called 3D (3 dimensions) face initialization.

[0084] The initial image information of the image to be processed may include first texture information (i.e., initial texture information) and first geometric information (i.e., initial geometric information) corresponding to the image to be processed. Optionally, the initial image information of the image to be processed may also include additional image information, and the additional image information includes acquisition environment information corresponding to the image to be processed, posture information corresponding to the target object (i.e., first posture information or initial posture information), etc. Among them, the acquisition environment information corresponding to the image to be processed may include but is not limited to first illumination information (i.e., initial illumination information). Among them, the first texture information may be texture information based on UV-map (UV-displacement map, UV-space, also known as canonical view. U represents the horizontal axis of a 2D (2 dimensions) image, and V represents the vertical axis of a 2D image).

[0085] Step S130: Based on the image to be processed and the first texture information, the trained image reconstruction model performs the following operations to obtain the second texture information:

[0086] Style features are extracted from the image to be processed to obtain a first style feature of the image to be processed; and the first texture information is repaired using the first style feature to obtain second texture information.

[0087] Optionally, the first texture information in the UV-space may be set to texture information similar to the image structure and having the same size as the image to be processed, and then the image to be processed and the first texture information are input into the trained image reconstruction model to obtain the second texture information (i.e. the repaired texture information), thereby repairing the first texture information.

[0088] The trained image reconstruction model may include a trained texture restoration network, and the second texture information may be obtained through the texture restoration network based on the image to be processed and the first texture information.

[0089] Specifically, the texture restoration network may include a style encoding module. By inputting the image to be processed into the style encoding module, the style encoding module can be used to extract the style features of the image to be processed to obtain the first style information (i.e., style code) c1, c2, ..., c n , where c1, c2, ..., c n are all the same size, for example, c1, c2, ..., c n The size of may be 1×1. The embodiment of the present application does not limit the value of n, and it can be set according to actual conditions, for example, n is 4.

[0090] The style coding module may specifically include a downsampling feature extraction unit, a batch normalization unit, and an activation function unit. The embodiment of the present application does not limit the number of downsampling feature extraction units, the number of batch normalization units, and the number of activation function units in the style coding module, and can be set according to actual needs. The embodiment of the present application does not limit the specific implementation of the activation function unit. For example, the activation function unit may be a ReLU activation function (linear rectification function, also known as a corrected linear unit) unit. Among them, the specific process of extracting style features of the processed image using the style coding module may be: based on the result of downsampling feature extraction by the downsampling feature extraction unit in the style coding module (for example, 3×3×1024 style features), resize (adjust the size) to obtain n style features of the same size (for example, 9 1×1×1024 style features), and then convert the n style features of the same size into n first style information of the same size (for example, 9 1×1 first style information) through a fully connected layer, that is, c1, c2, ..., c n .

[0091] The style features of an image are the aesthetic features of the image. Taking the image to be processed as a face image as an example, the style information of the image to be processed can be comic cartoon style, portrait style, ancient style, etc.

[0092] The first texture information is repaired based on the first style feature of the image to be processed, so that the repaired second texture information is texture information with the style of the image to be processed, which is more consistent with the texture information of the real image.

[0093] Step S140: performing image reconstruction based on the second texture information and the first geometric information to obtain a target image corresponding to the image to be processed, wherein the image quality of the target image is higher than the image quality of the image to be processed.

[0094] The embodiment of the present application does not limit the image reconstruction method, and the specific image reconstruction method can be determined according to actual needs. Optionally, a differentiable rasterizer can be used to reconstruct the image based on the second texture information and the first geometric information to obtain a target image. The differentiable rasterizer can be a Neural 3d mesh renderer.

[0095] By reconstructing the image based on the second texture information and the first geometric information, the style features of the image can be integrated into the restoration of the texture information, which can improve the quality of the image texture information. Therefore, a high-quality target image can be reconstructed based on the restored high-quality texture information and the geometric information of the image to be processed, thereby effectively improving the texture information of the target object in the image and realizing the reconstruction of the low-quality image corresponding to the target object into the corresponding high-quality image.

[0096] Moreover, the method can be widely used in a variety of application scenarios. For example, based on the second texture information obtained by the method, the foundation can be laid for 3D image design, saving the creation time of art designers. For example, the second texture information obtained by the method can be used to design face pendants, stickers, etc. In applications related to image processing and video, the image quality can be optimized. Moreover, more reliable face image information can be provided for the information verification system, the quality requirements for the input image of the information verification system are loose, and it is suitable for a variety of scenarios, improving the information verification effect.

[0097] It should be noted that the image reconstruction method provided in the embodiment of the present application can be applied to the reconstruction of a single image as well as the reconstruction of multiple images of the same target object.

[0098] Figure 2 FIG. 2 shows a structural diagram of an image reconstruction system provided in an embodiment of the present application. Figure 2As shown, the image reconstruction model includes a texture restoration network and a geometric restoration network. In the specific implementation process: based on the image to be processed, the initial image information corresponding to the image to be processed can be obtained, and the initial image information includes first texture information, first geometric information and image additional information. The first texture information is repaired by the texture restoration network to obtain the second texture information. By reconstructing the image based on the second texture information, the first geometric information and the image additional information, a first reconstructed image can be obtained. By inputting the image to be processed and the first reconstructed image into the geometric restoration network, a first displacement feature map can be obtained, and the first geometric information is repaired based on the first displacement feature map to obtain the second geometric information. By reconstructing the image based on the second texture information, the second geometric information and the image additional information, a target image can be obtained. The image reconstruction system will be described below in conjunction with specific embodiments:

[0099] Optionally, the initial image information further includes image additional information, the image additional information includes at least one of acquisition environment information corresponding to the image to be processed or first posture information corresponding to the target object, and image reconstruction is performed based on the second texture information and the first geometric information to obtain the target image, including:

[0100] Image reconstruction is performed based on the second texture information, the first geometric information and the image additional information to obtain a target image.

[0101] As described above, the acquisition environment information corresponding to the image to be processed may include but is not limited to the first illumination information. Optionally, a differentiable rasterizer may be used to reconstruct the image based on the second texture information, the first geometric information, the first illumination information and the first posture information to obtain a target image.

[0102] By performing image reconstruction based on the second texture information, the first geometric information and the image additional information, the geometric information in the target image can be made the first geometric information in the image to be processed, the image additional information in the target image can be made the image additional information in the image to be processed, and the texture information in the target image can be made the second texture information that incorporates the style features of the image to be processed, thereby improving the simulation degree of the constructed target image and making the texture information in the target image clearer than the texture information in the image to be processed, effectively improving the texture information of the target object in the image, and realizing the reconstruction of the low-quality image corresponding to the target object into the corresponding high-quality image.

[0103] Optionally, using the first style feature to repair the first texture information to obtain the second texture information includes:

[0104] Performing feature extraction on the first texture information to obtain a first texture feature;

[0105] Modifying the first texture feature according to the first style feature to obtain a second texture feature;

[0106] The second texture feature is texture reconstructed to obtain second texture information.

[0107] As described above, the second texture information can be obtained through a texture restoration network based on the image to be processed and the first texture information. Specifically, the texture restoration network can also include a texture encoding module, a texture correction module and a texture decoding module.

[0108] Optionally, the texture coding module may include multiple cascaded downsampling feature extraction units, batch normalization units, and activation function units. The embodiment of the present application does not limit the number of downsampling feature extraction units, batch normalization units, and activation function units in the texture coding module, and can be set according to actual needs. The embodiment of the present application does not limit the specific implementation of the activation function unit. For example, the activation function unit can be a ReLU activation function unit.

[0109] The first texture information may be downsampled and feature extracted based on the texture encoding module to obtain texture features of at least one level (i.e., scale or size). For example, the obtained texture features may include g1, g2, ..., g k , where g1, g2, ..., g k The embodiment of the present application does not limit the value of k, and can be set according to actual conditions, for example, k is 5. In practical applications, some or all of the k texture features can be selected as the first texture feature.

[0110] Optionally, the texture correction module can be implemented by GANs. In the embodiment of the present application, the texture correction module can be StyleGANs (Style Generative Adversarial Network). Taking the texture correction module as StyleGANs as an example, the texture correction module can include multiple cascaded sub-modules, each of which includes a cascaded upsampling feature extraction unit and an AdaIN (adaptive instance normalization) unit. In the AdaIN unit, the fusion of the first style feature and the first texture feature can be achieved.

[0111] The texture correction module can be used to correct the first texture feature according to the first style feature to obtain at least one level of texture features. For example, the obtained texture features may include p1, p2, ..., p m , where p1, p2, ..., p mThe size of is reduced successively. The embodiment of the present application does not limit the value of m, and can be set according to actual conditions, for example, m is 9. In practical applications, some or all of the m texture features can be selected as the second texture feature. Preferably, the number of features in the first style feature is the sum of the number of features in the first texture feature and the number of features in the second texture feature.

[0112] Optionally, the texture decoding module may include multiple cascaded upsampling feature extraction units, batch normalization units, and activation function units. The embodiment of the present application does not limit the number of upsampling feature extraction units, batch normalization units, and activation function units in the texture decoding module, and can be set according to actual needs. The embodiment of the present application does not limit the specific implementation of the activation function unit.

[0113] The texture decoding module can be used to reconstruct the second texture feature to obtain the second texture information. The second texture information may include at least one level of texture features. For example, the obtained texture features may include f1, f2, ..., f w , where f1, f2, ..., f w The embodiment of the present application does not limit the value of w, and can be set according to actual conditions, for example, w is 4. In practical applications, f can be selected w As the second texture information.

[0114] Since the first texture information is texture information in the UV-space, and the first texture information does not include posture information, correspondingly, after the first texture information is extracted using the texture encoding module, the obtained first texture feature does not include posture information. Therefore, in the process of correcting the first texture feature according to the first style feature, the influence of the posture information on the correction process can be removed to obtain the second texture feature that does not include posture information. Therefore, the first texture feature can also become spatial guidance information.

[0115] Since the corresponding second texture information can be obtained by reconstructing the texture according to the second texture feature, the second texture feature can also be called texture prior information, and correspondingly, the texture decoding module can be called a priori decoding network.

[0116] By further extracting features from the first texture information to obtain a first texture feature, and correcting the first texture feature according to the first style feature to obtain a second texture feature, and performing texture reconstruction on the second texture feature, the first texture information can be repaired. The repaired second texture information can be texture information with the style of the image to be processed, which is more consistent with the texture information of the real image.

[0117] Optionally, the method further comprises: obtaining random noise;

[0118] The first texture feature is modified according to the first style feature to obtain a second texture feature, including:

[0119] The random noise is fused with the first texture feature to obtain the first texture feature with noise;

[0120] The first style feature is fused with the first texture feature with noise to obtain a second texture feature.

[0121] Optionally, random noise can be obtained based on any random processing method, and down-sampled feature extraction can be performed on the random noise to obtain a noise feature noise (noise) of the same size as the first texture feature. The noise feature noise of the same size as the first texture feature and the first texture feature are then fused (for example, added or averaged) to obtain the first texture feature with noise.

[0122] By fusing the first noise feature with the first texture feature, the obtained first texture feature with noise can be made more detailed, more natural, and consistent with the real image.

[0123] As described above, the first texture feature may include multiple texture features, such as g1, g2, ..., g k , and g1, g2, ..., g k The size of each texture feature in is different. Assume that the acquired multiple texture features g1, g2, ..., g k As the first texture feature, the obtained multiple style features c1, c2, ..., c n As the first style feature, correspondingly, after downsampling the random noise to extract the feature, the noise feature noise with the same size as the first texture feature can be made to have the size of the noise feature noise equal to the texture feature g k The style feature c1 and the noise feature noise are first input into the AdaIN unit, and adaptive instance normalization is performed based on the style feature c1 and the noise feature noise to obtain the feature p1. The feature p1 is combined with the texture feature g k By fusion (for example, addition or averaging), we can get the texture feature g with noise k ', style feature c2 and texture feature g k 'Input to AdaIN unit, based on style feature c2 and texture feature g k 'Perform adaptive instance normalization to obtain texture feature p2. Perform upsampling feature extraction on texture feature p2, and compare the texture feature p2' after feature extraction with texture feature g k-1 Fusion is performed to obtain the fused texture feature gk-1 ', style feature c3 and texture feature g k-1 'Input to AdaIN unit, based on style feature c3 and texture feature g k-1 'Perform adaptive instance normalization to obtain texture feature p3. Similarly, we obtain features p4 to p m Alternatively, the texture features p2, p3, ..., p m Part or all of is used as the second texture feature, and this embodiment of the present application does not limit this.

[0124] Based on the above processing, the second texture feature can be better expressed in details, and the texture information of the target object in the image to be processed can be more restored.

[0125] As a specific example, the size of the input image may be set to 64×64. Correspondingly, after the initial image information of the image to be processed is extracted, first texture information of a size of 64×64 may be obtained. Figure 3 FIG. 1 is a schematic diagram showing a second texture information obtained by a texture restoration network in an embodiment of the present application. It should be understood that Figure 3 The texture inpainting network shown is Figure 2 A specific implementation of the texture restoration network in the image reconstruction system. Figure 3 As shown, the texture encoding module (i.e. Figure 3 The texture coding network in ( ) can include 5 cascaded down-sampling feature extraction units, the style coding module (i.e. Figure 3 The style encoding network in ( ) can include 6 cascaded downsampling feature extraction units, texture correction module (i.e. Figure 3 StyleGANs in ) can include 9 cascaded upsampling feature extraction units, texture decoding modules (i.e. Figure 3 The prior encoding network in ) can include 3 cascaded upsampling feature extraction units.

[0126] After the first texture information of size 64×64 is input into the texture encoding module, each downsampling feature extraction unit in the texture encoding module can sequentially perform downsampling feature extraction on the first texture information to obtain the first texture feature (i.e. Figure 3In this example, all texture features of the first texture information that are downsampled and extracted are taken as the first texture features for illustration. The first texture information can be downsampled and extracted by the first downsampled feature extraction unit in the texture encoding module to obtain spatial guidance information g1 of size 32×32; the spatial guidance information g1 can be downsampled and extracted by the second downsampled feature extraction unit in the texture encoding module to obtain spatial guidance information g2 of size 16×16; and so on, spatial guidance information g3 of size 8×8, spatial guidance information g4 of size 4×4, and spatial guidance information g5 of size 2×2 can be obtained.

[0127] After the input image of size 64×64 is input into the style encoding module, each downsampling feature extraction unit in the style encoding module can sequentially perform downsampling feature extraction on the input image to obtain a style feature of size 3×3×1024, and then resize the style feature of size 3×3×1024 to obtain 9 style features of size 1×1×1024, and then convert 8 style features of size 1×1×1024 into 8 style features of size 1×1 through the fully connected layer (i.e. Figure 3 In this example, the style features in c1, c2, ..., c8 are taken as the first style features for explanation.

[0128] In the texture correction module, the first texture feature can be corrected according to the first style feature to obtain the second texture feature. Among them, the noise feature noise of size 2×2 can be obtained by downsampling the random noise and extracting the feature. The style feature c1 and the noise feature noise are first input into the AdaIN unit, and adaptive instance normalization is performed based on the style feature c1 and the noise feature noise to obtain a feature p1 of size 2×2. The feature p1 of size 2×2 is fused with the spatial guidance information g5 of size 2×2 to obtain a texture feature g5' of size 2×2 with noise, and the style feature c2 and the texture feature g5' of size 2×2 are input into the AdaIN unit, and adaptive instance normalization is performed based on the style feature c2 and the texture feature g5' of size 2×2 to obtain a texture feature p2 of size 2×2. The texture feature p2 of size 2×2 is upsampled and feature extracted to obtain the texture feature p2' of size 4×4, and the texture feature p2' of size 4×4 is fused with the spatial guidance information g4 of size 4×4 to obtain the texture feature g4' of size 4×4. The style feature c3 and the texture feature g4' of size 4×4 are input into the AdaIN unit, and adaptive instance normalization is performed based on the style feature c3 and the texture feature g4' of size 4×4 to obtain the texture feature p3 of size 4×4. Similarly, the texture feature p4 of size 8×8, the texture feature p5 of size 16×16, and the texture feature p6 of size 32×32 are obtained. The texture feature p6 of size 32×32 is further upsampled and extracted to obtain a texture feature p6' of size 64×64, and the texture feature p6' of size 64×64 and the style feature c7 are input into the AdaIN unit, and adaptive instance normalization is performed based on the texture feature p6' of size 64×64 and the style feature c7 to obtain a texture feature p7 of size 128×128. The texture feature p7 of size 128×128 is further upsampled and extracted to obtain a texture feature p7' of size 256×256, and the texture feature p7' of size 128×128 and the style feature c8 are input into the AdaIN unit, and adaptive instance normalization is performed based on the texture feature p7' of size 128×128 and the style feature c8 to obtain a texture feature p8 of size 256×256. In this example, the texture feature p 5~ p8 is taken as the second texture feature as an example. 5~ p8 is input to the texture decoding module.

[0129] In the texture decoding module, each upsampling feature extraction unit in the texture decoding module can sequentially perform upsampling feature extraction on the second texture feature to obtain the second texture information (i.e. Figure 3The repaired texture in a r ). Among them, the texture feature p5 can be upsampled and extracted by the first upsampling feature extraction unit in the texture decoding module to obtain a texture feature f1 of size 32×32, and the texture feature f1 of size 32×32 is fused with the texture feature p6 to obtain a texture feature f1' of size 32×32. The texture feature f1' of size 32×32 is upsampled and extracted by the second upsampling feature extraction unit in the texture encoding module to obtain a texture feature f2 of size 64×64, and the texture feature f2 of size 64×64 is fused with the texture feature p7 to obtain a texture feature f2' of size 64×64. And so on, a texture feature f4' of size 256×256 is obtained. And the texture feature f4' of size 256×256 is used as the second texture information.

[0130] Optionally, performing image reconstruction based on the second texture information and the first geometric information to obtain a target image includes:

[0131] Using the second texture information, repairing the first geometric information through an image reconstruction model to obtain second geometric information;

[0132] Image reconstruction is performed based on the second texture information and the second geometric information to obtain a target image.

[0133] Optionally, a differentiable rasterizer may be used to perform image reconstruction based on the second texture information, the second geometric information, the first illumination information and the first posture information to obtain a target image.

[0134] Optionally, the trained image reconstruction model may also include a trained geometric restoration network, which may use the second texture information to repair the first geometric information to obtain the second geometric information (i.e., the repaired geometric information), thereby repairing the first geometric information and enriching the geometric information.

[0135] By reconstructing the image based on the second texture information and the second geometric information, the texture information in the target image can be the second texture information, and the geometric information in the target image can be the second geometric information, thereby improving the simulation degree of the constructed target image, and making the texture information in the target image clearer than the texture information in the image to be processed, and the geometric information in the target image richer than the geometric information in the image to be processed, thereby obtaining a target image with richer colors and richer details, and realizing the reconstruction of the low-quality image corresponding to the target object into the corresponding high-quality image. When the image to be processed is a face image, detailed information such as eyelids, eyebrows, beards, wrinkles, etc. can also be obtained.

[0136] Optionally, the first geometric information includes an initial surface normal vector map of the image to be processed, and the second geometric information is repaired by using the image reconstruction model using the second texture information to obtain the second geometric information, including:

[0137] Reconstructing the image based on the second texture information and the first geometric information to obtain a first reconstructed image;

[0138] According to the first reconstructed image and the image to be processed, a first displacement feature map corresponding to the initial surface normal vector map is obtained through an image reconstruction model;

[0139] The initial surface normal vector map is corrected by using the first displacement feature map to obtain a target surface normal vector map, and the second geometric information is the target surface normal vector map.

[0140] Optionally, the initial image information further includes image additional information, the image additional information includes at least one of acquisition environment information corresponding to the image to be processed or first posture information corresponding to the target object, and image reconstruction is performed based on the second texture information and the first geometric information to obtain a first reconstructed image, including:

[0141] Image reconstruction is performed based on the second texture information, the first geometric information and the image additional information to obtain a first reconstructed image.

[0142] Optionally, based on the second texture information, the first geometric information and the image additional information, a differentiable rasterizer may be used to perform image reconstruction to obtain a first reconstructed image.

[0143] By performing image reconstruction based on the second texture information, the first geometric information and the image additional information, the geometric information in the first reconstructed image can be made the first geometric information in the image to be processed, the illumination information in the first reconstructed image can be made the first illumination information in the image to be processed, and the image additional information in the first reconstructed image can be made the image additional information in the image to be processed, thereby improving the simulation degree of the constructed first reconstructed image, and making the texture information in the first reconstructed image clearer than the texture information in the image to be processed, thereby realizing the reconstruction of a low-quality image corresponding to the target object into a corresponding high-quality image.

[0144] In an embodiment of the present application, when obtaining the initial image information of the image to be processed, the initial depth map d0 of the image to be processed can be obtained first. Each element in the initial depth map d0 corresponds one-to-one to each pixel in the image to be processed, and is used to represent the depth value of each pixel in the image to be processed in the UV-space. And based on further processing of the initial depth map d0 of the image to be processed, the initial surface normal vector map of the image to be processed is obtained. The embodiment of the present application does not limit the specific method of obtaining the initial surface normal vector map. Among them, each element in the initial surface normal vector map corresponds one-to-one to each pixel in the image to be processed, and is used to represent the three-dimensional vector coordinate value corresponding to each pixel in the image to be processed, such as x, y and z values, and the surface normal vector corresponding to each pixel in the image to be processed can be determined by the three-dimensional coordinate vector value.

[0145] Each element in the first displacement feature map corresponds one-to-one to each element in the initial surface normal vector map of the image to be processed, and is used to represent the displacement of each element in the initial surface normal vector map.

[0146] As described above, the texture information in the first reconstructed image is the second texture information with the style of the image to be processed. Compared with the image to be processed, the texture information in the first reconstructed image is clearer and of higher quality. By obtaining the first displacement feature map corresponding to the initial surface normal vector map using the image reconstruction model based on the first reconstructed image and the image to be processed, the details in the obtained first displacement feature map can be made clearer.

[0147] For an element in the initial surface normal vector map, the element can be corrected based on the displacement corresponding to the element in the first displacement feature map to obtain a corrected element, wherein the displacement can be a positive number, a negative number, or 0. When the displacement is a positive number, the element in the initial surface normal vector map is moved along the corresponding surface normal vector by a value corresponding to the displacement; when the displacement is a negative number, the element in the initial surface normal vector map is moved in the opposite direction of the corresponding surface normal vector by a value corresponding to the displacement. After correcting all elements in the initial surface normal vector map, a target normal vector map can be obtained. For example, for an element in the initial surface normal vector map, if the displacement corresponding to the element in the first displacement feature map is +1, each element in the initial surface normal vector map is moved by 1 along the corresponding normal vector direction.

[0148] By using the first displacement feature map to correct the initial surface normal vector map, the target surface normal vector map obtained can have more detailed information. When the image to be processed is a face image, the target normal vector map repaired according to the first displacement feature map can obtain the detailed information of the eyelids, eyebrows, beards, wrinkles, etc. corresponding to the person.

[0149] Optionally, according to the first reconstructed image and the image to be processed, obtaining a first displacement feature map corresponding to the initial surface normal vector map through an image reconstruction model includes:

[0150] Extracting displacement features from the first reconstructed image to obtain a second displacement feature map;

[0151] Extract displacement features from the image to be processed to obtain a third displacement feature map;

[0152] The second displacement feature map is merged with the third displacement feature map to obtain a fourth displacement feature map;

[0153] The fourth displacement characteristic map is reconstructed to obtain the first displacement characteristic map.

[0154] In this implementation, the geometric restoration network may include a first displacement feature extraction module, a second displacement feature extraction module, and a displacement feature map reconstruction module. The first displacement feature extraction module may be used to perform down-sampling displacement feature extraction on the first reconstructed image to obtain a second displacement feature map t. The second displacement feature extraction module may be used to perform down-sampling displacement feature extraction on the input image to obtain a third displacement feature map t', and the second displacement feature map is fused with the third displacement feature map to obtain a fourth displacement feature map T. The displacement reconstruction module may be used to reconstruct the displacement feature map on the fourth displacement feature map to obtain the first displacement feature map.

[0155] Optionally, downsampling feature extraction can be performed through the first displacement feature extraction module to obtain a second displacement feature map t of a different scale, and downsampling feature extraction can be performed through the second displacement feature extraction module to obtain a third displacement feature map t' of a different scale, so that the second displacement feature map t and the third displacement feature map t' of the corresponding scale are fused to obtain a fused displacement feature map t' of the corresponding scale. Specifically, the displacement feature map t1 of the same scale obtained by downsampling feature extraction through the first displacement feature extraction module and the displacement feature map t1' of the corresponding scale obtained by downsampling feature extraction through the second displacement feature extraction module can be fused to obtain a fused displacement feature map t1', and then downsampling feature extraction can be performed on the fused displacement feature map t1' through the second displacement feature extraction module to obtain a displacement feature map t2'. The displacement feature map t2' is fused with the displacement feature map t2 of the corresponding scale to obtain a fused displacement feature map t2'. And so on, until the fused displacement feature map t1 is obtained. s ". And the final fusion displacement feature map t s" is used as the fourth displacement feature map T. In practical applications, part or all of the feature maps extracted by the first displacement feature extraction module can be selected as the first displacement feature map, and part or all of the feature maps extracted by the second displacement feature extraction module can be selected as the second displacement feature map.

[0156] After obtaining and utilizing the displacement reconstruction module to perform displacement reconstruction on the fourth displacement feature map T, a first displacement feature map is obtained. Specifically, the upsampling feature extraction units in the displacement reconstruction module may be used to perform upsampling feature extraction on the fourth displacement feature map T in sequence until the first displacement feature map is obtained. Among them, the first upsampling feature extraction unit in the displacement reconstruction module may be used to perform upsampling feature extraction on the fourth displacement feature map to obtain a displacement feature map T1', and the second upsampling feature extraction unit in the displacement reconstruction module may be used to perform upsampling feature extraction on the displacement feature map T1' to obtain a displacement feature map T2'. And so on, until the first displacement feature map m is obtained.

[0157] By extracting the displacement feature of the first reconstructed image to obtain the second displacement feature map, extracting the displacement feature of the image to be processed to obtain the third displacement feature map, and fusing the second displacement feature map with the third displacement feature map to obtain the fourth displacement feature map. The fourth displacement feature map is subjected to displacement reconstruction to obtain the first displacement feature map including more detailed information.

[0158] As a specific example, Figure 4 FIG. 1 is a schematic diagram showing a method of obtaining a target surface normal vector by using a geometric restoration network in an embodiment of the present application. It should be understood that Figure 4 The geometric restoration network shown is Figure 2 A specific implementation of the geometric restoration network in the image reconstruction system. Figure 4 As shown, the size of the input image I can be set to 64×64. Based on the texture restoration of the input image I, the second texture information can be obtained, and the image is reconstructed based on the second texture information and the first geometric information of the input image I. The obtained first reconstructed image I r (Right now Figure 4 As shown, the reconstructed image I r ) is 256×256. In this example, the first displacement feature extraction module may include 5 downsampling feature extraction units, the second displacement feature extraction module may include 6 downsampling feature extraction units, and the displacement reconstruction module may include 8 upsampling feature extraction units.

[0159] After the first reconstructed image of size 256×256 is input into the first displacement feature extraction module, each downsampling feature extraction unit in the first displacement feature extraction module can sequentially perform downsampling feature extraction on the first reconstructed image to obtain a second displacement feature map. Among them, the first downsampling feature extraction unit in the first displacement feature extraction module can perform downsampling feature extraction on the first reconstructed image to obtain a displacement feature map t1 with texture information of size 128×128; the second downsampling feature extraction unit in the first displacement feature extraction module can perform downsampling feature extraction on the first displacement feature t1 with texture information of size 128×128 to obtain a displacement feature map t2 with texture information of size 64×64; and so on, displacement feature maps t3 with texture information of sizes 32×32, t4 with texture information of sizes 16×16, and t5 with texture information of sizes 8×8 can be obtained. Specifically, each down-sampling feature extraction unit in the first displacement feature extraction module can perform interval feature extraction on the first reconstructed image, that is, when performing down-sampling feature extraction on each pixel point in the first reconstructed image, feature extraction is performed every other pixel point.

[0160] Optionally, all or part of the result of downsampling feature extraction of the first reconstructed image may be selected as the second displacement feature map. In this example, the displacement feature maps t3 to t5 may be used as the second displacement feature map.

[0161] The first displacement features of different sizes are input into the second displacement feature extraction module, so that when the second displacement feature extraction module performs feature extraction, the texture information in the first reconstructed image can be taken into account, so that the obtained second displacement features contain more detail information.

[0162] After the image to be processed of size 64×64 is input to the second displacement feature extraction module, each downsampling feature extraction unit in the second displacement feature extraction module can sequentially perform downsampling feature extraction on the image to be processed to obtain a third displacement feature map. In the second displacement feature extraction module, the third displacement feature map obtained by performing downsampling feature extraction on the image to be processed and the second displacement feature map of the corresponding size can be fused, and then feature extraction can be performed on the fused displacement feature map to obtain a fourth displacement feature map.

[0163] The first downsampling feature extraction unit in the second displacement feature extraction module can perform downsampling feature extraction on the processed image to obtain a displacement feature map t1' of size 32×32; by fusing the displacement feature t1' of size 32×32 with the above-mentioned displacement feature map t3 of size 32×32 with texture information, a fused displacement feature map t1" of size 32×32 is obtained. The second downsampling feature extraction unit in the second displacement feature extraction module performs downsampling feature extraction on the fused displacement feature t1" of size 32×32 to obtain a displacement feature map t2' of size 16×16. By fusing the displacement feature map t2' of size 16×16 with the displacement feature map t4 of size 16×16 with texture information, a fused displacement feature map t2" of size 16×16 is obtained. The third downsampling feature extraction unit in the second displacement feature extraction module performs downsampling feature extraction on the fused displacement feature map t2" of size 16×16 to obtain a displacement feature map t3' of size 8×8. By analogy, displacement feature maps t4” of sizes 4×4, t5” of sizes 2×2, and t6” of sizes 1×1 can be obtained, respectively. Each downsampling feature extraction unit in the second displacement feature extraction module can perform interval downsampling feature extraction on the processed image. It can be understood that after obtaining the fused displacement feature map t4” of size 8×8, the fused displacement feature map t4” can be directly extracted by the sampling feature extraction unit in the second displacement feature extraction module, without the need to fuse it with the displacement feature map obtained in the second displacement feature extraction module. In this example, the displacement feature map t6” of size 1×1 can be used as the fourth displacement feature map T.

[0164] After the fourth displacement feature map of size 1×1 is input into the displacement reconstruction module, the second displacement feature of size 1×1 can be decoded in turn by each upsampling feature extraction unit in the displacement reconstruction module to obtain the first displacement feature map (i.e. Figure 4The displacement feature map m shown in the figure). Among them, the first upsampling feature extraction unit in the displacement reconstruction module can perform feature extraction on the fourth displacement feature map of size 1×1 to obtain a displacement feature map T1' of size 2×2. The second upsampling feature extraction unit in the displacement reconstruction module performs upsampling feature extraction on the displacement feature map T' of size 2×2 to obtain a displacement feature map T2' of size 4×4; and so on, displacement feature maps T3' of sizes 8×8, T4' of 16×16, T5' of 32×32, T6' of 64×64, T7' of 128×128, and T8' of 256×256 can be obtained. The displacement feature map T8' of size 256×256 is used as the first displacement feature map m. When upsampling feature extraction is performed through each upsampling feature extraction unit in the displacement reconstruction module, bilinear interpolation may be performed on the corresponding displacement feature map first, and then feature extraction may be performed to achieve feature extraction with intervals.

[0165] like Figure 4 As shown, the initial image information of the image to be processed can be extracted to obtain the initial depth map d0 of the image to be processed. The initial depth map d0 of the image to be processed is further processed to obtain the initial surface normal vector map n0. After obtaining the first displacement feature map m, the corresponding elements in the initial surface normal vector map n0 can be corrected based on each element in the first displacement feature map m. After correcting all elements in the initial surface normal vector map, the target normal vector map (i.e. Figure 4 The repaired normal vector n is shown r ).

[0166] Optionally, the image reconstruction model is trained by a model training device in the following manner:

[0167] Acquire a training data set, the training data set including a plurality of image pairs, each image pair including a first sample image and a second sample image having the same image content, and the image quality of the second sample image in an image pair is higher than the image quality of the first sample image;

[0168] For each image pair, first image information of a first sample image and second image information of a second sample image in the image pair are obtained, wherein the first image information includes third texture information and third geometric information, and the second image information includes fourth texture information and fourth geometric information;

[0169] Based on each image pair, and the corresponding third texture information and fourth texture information, repeatedly performing a first training operation on the initial texture restoration network until a first condition is satisfied to obtain a first texture restoration network, wherein the first training operation comprises: for each image pair, inputting a first sample image and the corresponding third texture information in the image pair into the initial texture restoration network to obtain fifth texture information; determining a first training loss based on the fifth texture information and the fourth texture information corresponding to each image pair, and adjusting a network parameter of the initial texture restoration network based on the first training loss;

[0170] For each first sample image, image reconstruction is performed based on third geometric information corresponding to the first sample image and sixth texture information of the first sample image obtained by the first texture restoration network to obtain a second reconstructed image;

[0171] Based on each image pair, the corresponding second reconstructed image and the fourth geometric information, repeatedly performing a second training operation on the initial geometric restoration network until a second condition is satisfied to obtain a first geometric restoration network, wherein the second training operation includes: for each image pair, inputting a first sample image and a corresponding second reconstructed image in the image pair into the initial geometric restoration network to obtain fifth geometric information; determining a second training loss based on the fifth geometric information and the fourth geometric information corresponding to each image pair, and adjusting a network parameter of the initial geometric restoration network based on the second training loss;

[0172] Based on each image pair, and the corresponding first image information and second image information, a third training operation is performed on the intermediate neural network until a third condition is met to obtain an image reconstruction model. The intermediate neural network includes a cascaded first texture restoration network and a first geometric restoration network.

[0173] Among them, the model training device can be the current execution entity, or it can be an electronic device different from the current execution entity, such as a cloud server.

[0174] Optionally, before executing the above training method, an initial neural network model may be constructed, the initial neural network model comprising a cascaded initial texture restoration network and an initial geometry restoration network. The cascade of the initial texture restoration network and the initial geometry restoration network may be implemented by a differentiable rasterizer renderer.

[0175] In this embodiment, the network structure of the initial texture restoration network is the same as the above texture restoration network, and the network structure of the initial geometry restoration network is the same as the above geometry restoration network, which will not be described in detail here.

[0176] Based on what has been described above, in this embodiment, the first image information also includes first image additional information, and the first image additional information includes acquisition environment information corresponding to the first sample image, posture information corresponding to the object in the first sample image, and the like. Among them, the acquisition environment information corresponding to the first sample image may include, but is not limited to, illumination information. Correspondingly, the second image information also includes second image additional information, and the second image additional information includes acquisition environment information corresponding to the second sample image, posture information corresponding to the object in the second sample image, and the like. Among them, the acquisition environment information corresponding to the second sample image may include, but is not limited to, illumination information. It can be understood that the object in the first sample image is the same object as the object in the second sample image. Of course, when obtaining the second reconstructed image, image reconstruction can be performed based on the sixth texture information, the third geometric information, and the first image additional information corresponding to the first sample image to obtain the second reconstructed image.

[0177] The third training operation may specifically include: determining a third training loss based on the sixth texture information corresponding to the first sample image, the sixth geometric information of the first sample image obtained through the first geometric restoration network, and the fourth texture information and fourth geometric information of the second sample image corresponding to the first sample image, and adjusting the network parameters of the intermediate neural network based on the third training loss.

[0178] Taking the image pairs in the training data set as face images as an example, the training data set can be collected based on at least one face database. The embodiments of the present application do not impose any restrictions on this. In addition, the embodiments of the present application do not impose any restrictions on the number of image pairs collected.

[0179] For example, multiple image pairs in the training dataset can be obtained based on the following steps 1 to 3:

[0180] Step 1: Based on the training data set collected from the CelebAMask-HQ (Celeb Attributes-Mask-HQ, a high-definition face data set, HQ, High Quality) database and the FFHQ (Flickr-Faces-HQ, a high-definition face data set) database, 30k (i.e. 30,000) high-definition face images with a resolution of 1024×1024 can be selected from the CelebAMask-HQ database, and 70k (i.e. 70,000) high-definition face images with a resolution of 1024×1024 can be selected from the FFHQ database, so as to obtain 100k (i.e. 100,000) high-definition face images with a resolution of 1024×1024.

[0181] Step 2: Randomly select 60k (i.e. 60,000) 1024×1024 resolution high-definition face images from these 100k 1024×1024 resolution high-definition face images in a ratio of 3:7 (i.e. the number of images in the CelebAMask-HQ database: the number of images in the FFHQ database = 3:7).

[0182] Step 3: After downsampling and blurring each of the 60k 1024×1024 resolution high-definition face images, we obtain the image pair corresponding to each image. For a 1024×1024 resolution high-definition face image, we can obtain a first sample image with a resolution of 64×64 and a second sample image with a resolution of 256×256 (i.e., the true value image) corresponding to the image.

[0183] After obtaining the image pair corresponding to each high-definition face image with a resolution of 1024×1024, image information is extracted from each image in each image pair to obtain the third texture information, the third geometric information and the first image additional information of each first sample image, and the fourth texture information and the fourth geometric information of each second sample image.

[0184] Optionally, the first training loss can be determined based on the first loss function, and the first training loss characterizes the accuracy of the first texture restoration network. The first loss function may include any of the following: softmax (normalized) loss function, center loss, etc., which is not limited in this application. The first condition may be that the training cycle for executing the first training operation reaches the first preset value, the first loss reaches the second preset value, the first loss function converges, etc. Among them, the first training operation is performed once based on all image pairs in the training data set, which can be regarded as a training cycle for executing the first training operation. For example, the first condition may be that the training cycle for executing the first training operation is 30 cycles. Among them, during the execution of the first training operation, the learning rate may be 0.0001, the batch (that is, the number of first sample images input into the initial texture complex network at one time is 16, and correspondingly, the number of third texture information of each first sample image is also 16) may be 16, and the optimizer may be any optimizer, for example, it may be Adam (an optimizer), a stochastic gradient descent optimizer, etc.

[0185] Optionally, the second training loss can be determined based on the second loss function, and the second training loss characterizes the accuracy of the first geometric restoration network. The second loss function may include any of the following: softmax loss function, centerloss, etc., which is not limited in this application. The second condition may be that the training cycle for executing the second training operation reaches a third preset value, the first loss reaches a fourth preset value, the second loss function converges, etc. Among them, the second training operation is performed once based on all image pairs in the training data set, which can be regarded as a training cycle for executing the second training operation. For example, the second condition may be that the training cycle for executing the second training operation is 30 cycles. Among them, during the execution of the second training operation, the learning rate may be 0.0001, the batch (that is, the number of first sample images input into the initial geometric restoration network at one time is 16, and correspondingly, the number of second reconstructed images corresponding to each first sample image is also 16) may be 16, and the optimizer is Adam.

[0186] In the process of performing the second training operation, the parameters of the first texture restoration network can be frozen. Optionally, the parameters of the first texture restoration network can be frozen by setting the learning rate of the convolution unit, the learning rate of the normalization unit, and the learning rate of the activation function unit in the first texture restoration network to zero, so that the weight of the convolution unit, the weight of the normalization unit, and the weight of the activation function unit in the first texture restoration network remain fixed.

[0187] Optionally, the third training loss can be determined based on the third loss function, and the third training loss characterizes the accuracy of the image reconstruction network. The third loss function may include any of the following: softmax loss function, center loss, etc., which is not limited in this application. The third condition may be that the training cycle for executing the third training operation reaches the fifth preset value, the first loss reaches the sixth preset value, the third loss function converges, etc. Among them, the third training operation is performed once based on all image pairs in the training data set, which can be regarded as a training cycle for executing the third training operation. For example, the third condition may be that the training cycle for executing the third training operation is 20 cycles. Among them, during the execution of the third training operation, the learning rate may be 0.0001, the batch (that is, the number of first sample images input into the intermediate neural network at one time is 16, and correspondingly, the number of third texture information corresponding to each first sample image is also 16, and the number of third reconstructed images corresponding to each first sample image is also 16) may be 16, and the optimizer is Adam.

[0188] It should be noted that the training cycle of executing the first training operation should be greater than or equal to the training cycle of executing the third training operation, and the training cycle of executing the second training operation should also be greater than or equal to the training cycle of executing the third training operation.

[0189] It should be noted that if only texture restoration is required for the processed image, the initial neural network may only include the initial texture restoration network. Correspondingly, after repeatedly performing the first training operation on the initial texture restoration network to obtain the first texture restoration network, the training may be stopped to obtain a trained image reconstruction model.

[0190] If only geometric restoration is required for the processed image, the initial neural network may only include the initial geometric restoration network. Correspondingly, the training process of the initial geometric restoration network may be:

[0191] Based on each image pair and the fourth geometric information, the fourth training operation is repeatedly performed on the initial geometric restoration network until the fourth condition is met and the second texture restoration network is obtained, and then the training can be stopped to obtain a trained image reconstruction model. The fourth training operation includes: for each image pair, the first sample image in the image pair is input into the initial geometric restoration network to obtain the seventh geometric information; based on the seventh geometric information and the fourth geometric information corresponding to each image pair, a fourth training loss is determined, and the network parameters of the initial geometric restoration network are adjusted based on the fourth training loss.

[0192] Through the above-mentioned training method, an image reconstruction model with a more accurate restoration degree can be obtained, so that when texture restoration is performed based on the trained image reconstruction model, clearer second texture information can be obtained; when geometric restoration is performed based on the trained image reconstruction model, more detailed detail information can be obtained.

[0193] Optionally, after obtaining the training data set, the method for training the image reconstruction model further includes:

[0194] Split the training data set into training set, validation set and test set according to the preset ratio;

[0195] Perform the first training operation, the second training operation and the third training operation on the initial neural network model according to the training set until the fifth condition is met, thereby obtaining a second neural network model;

[0196] According to the preset verification evaluation index, the second neural network model is evaluated based on the verification set to obtain a first evaluation result;

[0197] According to the preset test evaluation index, the second neural network model is evaluated based on the test set to obtain a second evaluation result;

[0198] When the index evaluation result satisfies the sixth condition, determining the second neural network model as the image reconstruction network model, wherein the index evaluation result includes the first evaluation result and the second evaluation result;

[0199] When the indicator evaluation result does not meet the sixth condition, the model parameters of the second neural network model are adjusted, and the adjusted model is continuously trained based on the training set;

[0200] The third condition mentioned above includes the fifth and sixth conditions;

[0201] Among them, the sixth condition includes at least one of the first evaluation condition or the second evaluation condition, and the indicator evaluation result satisfies the sixth condition includes: the first evaluation result satisfies the first evaluation condition or the second evaluation result satisfies at least one of the second evaluation condition.

[0202] In this implementation, the training data set can be divided into a training set, a validation set, and a test set according to a preset ratio, wherein the preset ratio can be configured according to actual needs (for example, it can be an empirical value or an experimental value), and this application does not impose any restrictions on this. For example, the preset ratio can be set to 4:1:1, that is, after segmentation, the ratio of the number of image pairs in the training set, the number of image pairs in the validation set, and the number of image pairs in the test set is 4:1:1. Taking the above-mentioned randomly selected 60k 1024×1024 resolution high-definition face images as an example, the corresponding number of image pairs in the training set is 40k, the number of image pairs in the validation set is 10k, and the number of image pairs in the test set is 10k.

[0203] In this implementation, the verification evaluation index may include but is not limited to the hyperparameters of the second neural network model. Among them, the first evaluation condition can be configured according to actual needs (for example, it can be an empirical value or an experimental value), and this application does not limit this. Among them, when the second neural network model does not meet the first evaluation condition, the hyperparameters of the second deep learning model can be adjusted, the second deep learning model can be optimized, and the second deep learning model can continue to be trained according to the training set and the verification set until the second neural network model meets the first evaluation condition.

[0204] Optionally, the test evaluation index may include but is not limited to the generalization ability of the second neural network model. Among them, the second evaluation condition can be configured according to actual needs (for example, it can be an empirical value or an experimental value), and the present application does not limit this. For example, the second evaluation condition may be that the generalization ability of the second neural network model reaches 90%, that is, the generalization ability of the second neural network model that meets the second evaluation condition is greater than or equal to 90%. When the second evaluation result does not meet the second evaluation condition, continue to train the second neural network model that meets the second evaluation condition according to the training set and the verification set until a second neural network model that meets the second evaluation condition is obtained, so that the final performance of the obtained image reconstruction model is good.

[0205] It can be understood that the final performance of the image reconstruction model obtained when the sixth condition includes the first evaluation condition and the second evaluation condition is better than the final performance of the image reconstruction model obtained when the sixth condition includes only the first evaluation condition or only the second evaluation condition.

[0206] Through the above method, the image reconstruction model can be made more accurate, have stronger generalization ability, and have higher coverage and precision.

[0207] Figure 5 A schematic diagram of an image reconstruction system provided by an embodiment of the present application is shown. Figure 5 As shown, the system includes a user terminal 81, a network 82, an application server 83 and a model training server 84. The user terminal 81 communicates with the application server 83 through the network 82. Communication interaction can be achieved between the application server 83 and the model training server 84. For example, the application server 83 can receive the trained image reconstruction model sent by the model training server 84. Among them, an application for information verification can be installed in the user terminal 81, or a plug-in for information verification is set in a certain application in the user terminal 81. By opening the application for information verification or the application with the above-mentioned plug-in for information verification, the terminal is started to perform information verification. Among them, the user terminal 81 can be a mobile phone, iPad, computer, etc. For example, in the process of payment, logging in / registering a new application, the information verification can be completed through the application for information verification, thereby completing the corresponding business.

[0208] Among them, the model training server 84 can be used to train the initial neural network model based on the training data set to obtain a trained image reconstruction model. And the trained image reconstruction model is sent to the application server 83, so that the application server 83 deploys the trained image reconstruction model. The application server 83 can be used to execute the image reconstruction method provided by the embodiment of the present application based on the deployed trained image reconstruction model. For example, the user terminal can be an access control system, and the image to be processed can be collected by the image acquisition device of the access control system, and the image to be processed is reconstructed to obtain a high-quality target image after reconstruction, and the identity information of the object corresponding to the target image is matched with the identity information of the preset object to determine whether the identity of the target object in the target image matches the identity information of the preset object.

[0209] The image reconstruction system provided in the embodiment of the present application can be applied to the information verification process. Figure 5The image reconstruction system shown in the figure illustrates the process of the information verification method in the application scenario. In the information verification method, the application server 83 in the image reconstruction system can be a door access server, and the user terminal 81 in the image reconstruction system can be a door access terminal. Figure 6 The flowchart of the information verification method in the application scenario applicable to the embodiment of the present application is shown. Figure 6 As shown, the method may include the following steps S11 to S15.

[0210] Step S11: obtaining a training data set (a plurality of sample pairs, each sample pair comprising a first sample image and a second sample image having the same content, wherein the image quality of the second sample image in a sample pair is higher than the image quality of the first sample image).

[0211] Step S12: model training, that is, training the initial neural network model based on the training data set to obtain a trained image reconstruction model, and sending the trained image reconstruction model to the access control server 83.

[0212] Step S13: The image acquisition device based on the access control system obtains the image of the user close to the access control terminal, and sends the user image to the access control server 83.

[0213] Figure 7 A schematic diagram of a display interface of a door access control terminal in an application scenario applicable to an embodiment of the present application is shown. Figure 8 Another schematic diagram of the access control terminal display interface in the application scenario applicable to the embodiment of the present application is shown. Figure 7 As shown, when the access control terminal is not used, the access control terminal display interface displays an image acquisition control 1, an image verification control 2, and a fingerprint verification control 3. When the fingerprint verification control 3 is detected to be triggered, the fingerprint of the user who currently triggers the fingerprint verification control 3 is collected, and the collected fingerprint is verified to determine whether the collected fingerprint is the fingerprint of the preset user. After the face verification control 2 is detected to be triggered, the image acquisition control 1 can be enabled to collect the image of the user currently close to the access control terminal, and the collected user image is verified to determine whether the collected user image is the image of the preset user. Of course, user fingerprint collection and user image collection can also be performed at the same time.

[0214] After enabling the image acquisition control 1, you can also jump from the current display interface to Figure 8 The access control terminal display interface shown in FIG. Figure 8 As shown, the user image captured by the image capture control 1 can be displayed on the current display interface.

[0215] Step S14: reconstructing the user image based on the trained image reconstruction model to obtain a target user image with higher quality corresponding to the image to be processed;

[0216] The user identity information corresponding to the target user image is matched with the preset user identity information to determine whether the user identity in the target user image matches the preset user identity information, and the matching result is sent to the access control terminal 81.

[0217] Step S15: If the user identity information corresponding to the target user image matches the preset user identity information, it is determined that the object to which the user image belongs is the preset object, and the access control is released;

[0218] If the user identity information corresponding to the target user image does not match the preset user identity information, it is determined that the object to which the user image belongs is not the preset object, and the access control is still closed.

[0219] like Figure 8 As shown, if it is detected that the user identity information corresponding to the target user image does not match the preset user identity information, prompt information such as "verification failed" can also be displayed on the access control terminal display interface.

[0220] Among them, step S11 and step S12 can be implemented by the model training server 84, step S13 can be implemented by the access control terminal 81, step S14 can be implemented by the access control server 83, and step S15 can be implemented by the access control terminal 81. Of course, step S14 can also be implemented by the access control terminal 81. Figure 5 The implementation of step S14 by the access control server 83 is used as an example for explanation. When step S14 is implemented by the access control terminal 81, in step S12, the trained image reconstruction model can be sent to the access control terminal 81; in step S13, the user image does not need to be sent to the access control server 83; in step S14, the matching result does not need to be sent to the access control terminal 81.

[0221] During vehicle driving, the road image acquisition device is usually unable to capture vehicle photos that can accurately identify the vehicles on the current road due to the vehicle speed being too fast or the vehicle being crowded. Based on this, the image reconstruction system provided by this application can be applied to the process of smart transportation, combined with Figure 5 The image reconstruction system shown in the figure illustrates the process of the vehicle detection method in the application scenario. In the vehicle detection method, the application server 83 in the image reconstruction system can be a vehicle detection server, and the user terminal 81 in the image reconstruction system can be a vehicle detection terminal. Fig. 9 The flowchart of the vehicle detection method in the application scenario applicable to the embodiment of the present application is shown. Fig. 9As shown, the method may include the following steps S21 to S23.

[0222] Step S21: triggering the road image acquisition device in the vehicle detection terminal to acquire a vehicle image, and sending the vehicle image to the vehicle detection server 83 .

[0223] Step S22: reconstructing the collected vehicle image based on the trained image reconstruction model to obtain a target vehicle image with higher quality;

[0224] The vehicle information corresponding to the target vehicle image is matched with the preset vehicle information to determine whether the vehicle information in the target vehicle image matches the preset vehicle information, and the matching result is sent to the vehicle detection terminal 81 .

[0225] Step S23: If the vehicle information corresponding to the target vehicle image matches the preset vehicle information, the captured vehicle is determined to be the preset vehicle, and subsequent processing is performed according to the specific information of the vehicle, for example, a "speeding" warning is issued to the driver corresponding to the vehicle;

[0226] If the vehicle information corresponding to the target vehicle image does not match the preset vehicle information, no subsequent processing is performed.

[0227] Among them, step S21 and step S23 can be implemented by the vehicle detection terminal 81, and step S22 can be implemented by the vehicle detection server 83. Of course, step S22 and step S23 can also be implemented by the vehicle detection terminal 81. Fig. 9 The example of executing step S22 by the vehicle detection server 83 is used for explanation. When step S22 is implemented by the vehicle detection terminal 81, in step S21, the vehicle image does not need to be sent to the vehicle detection server 83; in step S22, the matching result does not need to be sent to the vehicle detection terminal 81.

[0228] Based on the same principle as the image reconstruction method provided in the embodiment of the present application, the embodiment of the present application also provides an image reconstruction device. Fig.10 FIG. 1 is a schematic diagram of an image reconstruction device according to an embodiment of the present application. Fig.10 As shown, the device 30 includes an image acquisition module 301, an image information acquisition module 302, a texture restoration module 303 and an image reconstruction module 304, wherein:

[0229] An image acquisition module 301 is used to acquire an image to be processed corresponding to a target object;

[0230] An image information acquisition module 302 is used to acquire initial image information of an image to be processed, where the initial image information includes first texture information and first geometric information;

[0231] The texture restoration module 303 is used to obtain second texture information by performing the following operations through a trained image reconstruction model based on the image to be processed and the first texture information:

[0232] Extracting style features from the image to be processed to obtain a first style feature of the image to be processed; restoring the first texture information using the first style feature to obtain second texture information;

[0233] The image reconstruction module 304 is used to perform image reconstruction based on the second texture information and the first geometric information to obtain a target image corresponding to the image to be processed, wherein the image quality of the target image is higher than the image quality of the image to be processed.

[0234] Optionally, when the image reconstruction module 304 performs image reconstruction based on the second texture information and the first geometric information to obtain a target image corresponding to the image to be processed, it is specifically used to:

[0235] Using the second texture information, repairing the first geometric information through an image reconstruction model to obtain second geometric information;

[0236] Image reconstruction is performed based on the second texture information and the second geometric information to obtain a target image.

[0237] Optionally, the first geometric information includes an initial surface normal vector map of the image to be processed, and the image reconstruction module 304, when using the second texture information to repair the first geometric information through the image reconstruction model to obtain the second geometric information, is specifically used to:

[0238] Reconstructing the image based on the second texture information and the first geometric information to obtain a first reconstructed image;

[0239] According to the first reconstructed image and the image to be processed, a first displacement feature map corresponding to the initial surface normal vector map is obtained through an image reconstruction model;

[0240] The initial surface normal vector map is corrected by using the first displacement feature map to obtain a target surface normal vector map, and the second geometric information is the target surface normal vector map.

[0241] Optionally, when the image reconstruction module 304 acquires the first displacement feature map corresponding to the initial surface normal vector map through the image reconstruction model according to the first reconstructed image and the image to be processed, it is specifically used to:

[0242] Extracting displacement features from the first reconstructed image to obtain a second displacement feature map;

[0243] Extract displacement features from the image to be processed to obtain a third displacement feature map;

[0244] The second displacement feature map is merged with the third displacement feature map to obtain a fourth displacement feature map;

[0245] The fourth displacement characteristic map is reconstructed to obtain the first displacement characteristic map.

[0246] Optionally, the initial image information further includes image additional information, and the image additional information includes at least one of acquisition environment information corresponding to the image to be processed or first posture information corresponding to the target object. When the image reconstruction module 304 performs image reconstruction based on the second texture information and the first geometric information to obtain the first reconstructed image, it is specifically used to:

[0247] Image reconstruction is performed based on the second texture information, the first geometric information and the image additional information to obtain a first reconstructed image.

[0248] Optionally, when the texture restoration module 303 uses the first style feature to restore the first texture information to obtain the second texture information, it is specifically used to:

[0249] Performing feature extraction on the first texture information to obtain a first texture feature;

[0250] Modifying the first texture feature according to the first style feature to obtain a second texture feature;

[0251] Texture reconstruction is performed according to the second texture feature to obtain second texture information.

[0252] Optionally, the above device further includes a noise acquisition module, and the noise acquisition module is used to acquire random noise;

[0253] When the texture restoration module 303 performs texture reconstruction according to the second texture feature to obtain the second texture information, it is specifically used to:

[0254] The random noise is fused with the first texture feature to obtain the first texture feature with noise;

[0255] The first style feature is fused with the first texture feature with noise to obtain a second texture feature.

[0256] Optionally, the initial image information further includes image additional information, and the image additional information includes at least one of acquisition environment information corresponding to the image to be processed or first posture information corresponding to the target object. When the image reconstruction module 304 performs image reconstruction based on the second texture information and the first geometric information to obtain a target image corresponding to the image to be processed, it is specifically used to:

[0257] Image reconstruction is performed based on the second texture information, the first geometric information and the image additional information to obtain a target image.

[0258] Optionally, the image reconstruction model is trained by a model training device in the following manner:

[0259] Acquire a training data set, the training data set including a plurality of image pairs, each image pair including a first sample image and a second sample image having the same image content, and the image quality of the second sample image in an image pair is higher than the image quality of the first sample image;

[0260] For each image pair, first image information of a first sample image and second image information of a second sample image in the image pair are obtained, wherein the first image information includes third texture information and third geometric information, and the second image information includes fourth texture information and fourth geometric information;

[0261] Based on each image pair, and the corresponding third texture information and fourth texture information, repeatedly performing a first training operation on the initial texture restoration network until a first condition is satisfied to obtain a first texture restoration network, wherein the first training operation comprises: for each image pair, inputting a first sample image and the corresponding third texture information in the image pair into the initial texture restoration network to obtain fifth texture information; determining a first training loss based on the fifth texture information and the fourth texture information corresponding to each image pair, and adjusting a network parameter of the initial texture restoration network based on the first training loss;

[0262] For each first sample image, image reconstruction is performed based on third geometric information corresponding to the first sample image and sixth texture information of the first sample image obtained by the first texture restoration network to obtain a second reconstructed image;

[0263] Based on each image pair, the corresponding second reconstructed image and the fourth geometric information, repeatedly performing a second training operation on the initial geometric restoration network until a second condition is satisfied to obtain a first geometric restoration network, wherein the second training operation includes: for each image pair, inputting a first sample image and a corresponding second reconstructed image in the image pair into the initial geometric restoration network to obtain fifth geometric information; determining a second training loss based on the fifth geometric information and the fourth geometric information corresponding to each image pair, and adjusting a network parameter of the initial geometric restoration network based on the second training loss;

[0264] Based on each image pair, and the corresponding first image information and second image information, a third training operation is performed on the intermediate neural network until a third condition is met to obtain an image reconstruction model. The intermediate neural network includes a cascaded first texture restoration network and a first geometric restoration network.

[0265] The device of the embodiments of the present application can execute the method provided by the embodiments of the present application, and the implementation principles are similar. The actions performed by each module in the device of each embodiment of the present application correspond to the steps in the method of each embodiment of the present application. For the detailed functional description of each module of the device, please refer to the description in the corresponding method shown in the previous text, which will not be repeated here.

[0266] Based on the same principles as the image reconstruction method and device provided in the embodiments of the present application, an electronic device (such as a server) is also provided in the embodiments of the present application. The electronic device may include a memory, a processor, and a computer program stored in the memory. The processor executes the above-mentioned computer program to implement the steps of the method provided in any optional embodiment of the present application.

[0267] Optionally, Fig.11 A schematic diagram of the structure of an electronic device applicable to the embodiment of the present application is shown. Fig.11 As shown, Fig.11 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0268] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0269] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.11 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0270] The memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation herein.

[0271] The memory 4003 is used to store the computer program for executing the embodiment of the present application, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the computer program stored in the memory 4003 to implement the steps shown in the above method embodiment.

[0272] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented.

[0273] The embodiment of the present application also provides a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiment when executed by a processor.

[0274] The terms "first", "second", "third", "fourth", "1", "2", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that shown or described in the drawings.

[0275] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the implementation order of these steps is not limited to the order indicated by the arrows. Unless clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages may be executed at the same time, and each sub-step or stage in these sub-steps or stages may also be executed at different times respectively. In different scenarios of execution time, the execution order of these sub-steps or stages may be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0276] The above is only an optional implementation method for some implementation scenarios of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the solution of the present application, other similar implementation methods based on the technical ideas of the present application are also within the protection scope of the embodiments of the present application.

Claims

1. An image reconstruction method, characterized in that: include: Obtain the image to be processed corresponding to the target object; Acquire initial image information of the image to be processed, wherein the initial image information includes first texture information and first geometric information; Based on the image to be processed and the first texture information, the trained image reconstruction model is used to perform the following operations to obtain second texture information: Extracting style features of the image to be processed to obtain a first style feature of the image to be processed; restoring the first texture information using the first style feature to obtain second texture information; Image reconstruction is performed based on the second texture information and the first geometric information to obtain a target image corresponding to the image to be processed, wherein the image quality of the target image is higher than the image quality of the image to be processed.

2. The method according to claim 1, wherein the image reconstruction based on the second texture information and the first geometric information to obtain the target image comprises: Using the second texture information, and repairing the first geometric information through the image reconstruction model, to obtain second geometric information; Image reconstruction is performed based on the second texture information and the second geometric information to obtain the target image.

3. The method according to claim 2, characterized in that The first geometric information includes an initial surface normal vector map of the image to be processed, and the second texture information is used to repair the first geometric information through the image reconstruction model to obtain the second geometric information, including: Perform image reconstruction based on the second texture information and the first geometric information to obtain a first reconstructed image; Acquire a first displacement feature map corresponding to the initial surface normal vector map according to the first reconstructed image and the image to be processed; The initial surface normal vector map is corrected by using the first displacement feature map to obtain a target surface normal vector map, and the second geometric information is the target surface normal vector map.

4. The method according to claim 3, characterized in that The step of acquiring a first displacement feature map corresponding to the initial surface normal vector map according to the first reconstructed image and the image to be processed includes: Extracting displacement features from the first reconstructed image to obtain a second displacement feature map; Extracting displacement features of the image to be processed to obtain a third displacement feature map; The second displacement feature map is merged with the third displacement feature map to obtain a fourth displacement feature map; The fourth displacement characteristic map is reconstructed to obtain the first displacement characteristic map.

5. The method according to claim 3, characterized in that: The initial image information also includes image additional information, and the image additional information includes at least one of acquisition environment information corresponding to the image to be processed or first posture information corresponding to the target object. The image reconstruction based on the second texture information and the first geometric information to obtain a first reconstructed image includes: Image reconstruction is performed based on the second texture information, the first geometric information and the image additional information to obtain the first reconstructed image.

6. The method according to claim 1, characterized in that The step of repairing the first texture information by using the first style feature to obtain second texture information includes: Performing feature extraction on the first texture information to obtain a first texture feature; Correcting the first texture feature according to the first style feature to obtain a second texture feature; The second texture feature is texture reconstructed to obtain the second texture information.

7. The method according to claim 6, characterized in that The method further comprises: Get random noise; The step of correcting the first texture feature according to the first style feature to obtain a second texture feature includes: Fusion the random noise with the first texture feature to obtain a first texture feature with noise; The first style feature is fused with the first texture feature with noise to obtain the second texture feature.

8. The method according to claim 1, characterized in that The initial image information also includes image additional information, and the image additional information includes at least one of acquisition environment information corresponding to the image to be processed or first posture information corresponding to the target object. The image reconstruction based on the second texture information and the first geometric information to obtain the target image includes: Image reconstruction is performed based on the second texture information, the first geometric information and the image additional information to obtain the target image.

9. The method according to claim 2, characterized in that: The image reconstruction model is trained by a model training device in the following manner: Acquire a training data set, wherein the training data set includes a plurality of image pairs, each of the image pairs includes a first sample image and a second sample image having the same image content, and the image quality of the second sample image in one of the image pairs is higher than the image quality of the first sample image; For each of the image pairs, first image information of a first sample image and second image information of a second sample image in the image pair are obtained, wherein the first image information includes third texture information and third geometric information, and the second image information includes fourth texture information and fourth geometric information; Based on each of the image pairs, and the corresponding third texture information and fourth texture information, repeatedly performing a first training operation on the initial texture restoration network until a first condition is satisfied to obtain a first texture restoration network, wherein the first training operation comprises: for each of the image pairs, inputting a first sample image and the corresponding third texture information in the image pair into the initial texture restoration network to obtain fifth texture information; based on the fifth texture information and the fourth texture information corresponding to each of the image pairs, determining a first training loss, and adjusting a network parameter of the initial texture restoration network based on the first training loss; For each of the first sample images, image reconstruction is performed based on the third geometric information corresponding to the first sample image and the sixth texture information of the first sample image obtained by the first texture restoration network to obtain a second reconstructed image; Based on each of the image pairs, the corresponding second reconstructed image and the fourth geometric information, repeatedly performing a second training operation on the initial geometric restoration network until a second condition is satisfied to obtain a first geometric restoration network, wherein the second training operation comprises: for each of the image pairs, inputting a first sample image and a corresponding second reconstructed image in the image pair into the initial geometric restoration network to obtain fifth geometric information; determining a second training loss based on the fifth geometric information and the fourth geometric information corresponding to each of the image pairs, and adjusting network parameters of the initial geometric restoration network based on the second training loss; Based on each of the image pairs, and the corresponding first image information and second image information, a third training operation is performed on the intermediate neural network until a third condition is met to obtain the image reconstruction model, wherein the intermediate neural network includes the cascaded first texture restoration network and the first geometry restoration network.

10. An image reconstruction device, characterized in that: The device comprises an image acquisition module, an image information acquisition module, a texture restoration module and an image reconstruction module, wherein: An image acquisition module is used to acquire an image to be processed corresponding to a target object; An image information acquisition module, used to acquire initial image information of the image to be processed, wherein the initial image information includes first texture information and first geometric information; A texture restoration module is used to obtain second texture information by performing the following operations through a trained image reconstruction model based on the image to be processed and the first texture information: Extracting style features of the image to be processed to obtain a first style feature of the image to be processed; restoring the first texture information using the first style feature to obtain second texture information; An image reconstruction module is used to perform image reconstruction based on the second texture information and the first geometric information to obtain a target image corresponding to the image to be processed, wherein the image quality of the target image is higher than the image quality of the image to be processed.

11. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images