Image restoration method, character image restoration method, electronic device and storage medium

Abstract

The present application discloses an image restoration method, a character image restoration method, an electronic device and a storage medium, and relates to the field of image processing. The method comprises: obtaining a first region image from an image to be processed; performing three-dimensional reconstruction of the element to be restored based on the first region image to obtain a target reconstruction model; performing image segmentation of the element to be restored based on the first region image to obtain a segmentation mask, wherein the segmentation mask is used to characterize the pixel-level area of ​​the element to be restored in the first region image; redrawing the first region image based on the segmentation mask and the target reconstruction model to obtain a second region image; fusing the second region image and the image to be processed to obtain a target image. The present application solves the technical problem that the hand structure of a character generated by an image generation model in the related art is deformed, resulting in poor image perception.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image restoration method, a character image restoration method, an electronic device, and a storage medium. Background Art

[0002] In order to present the visual effect of different users wearing different clothes, image generation technology can be used to generate exquisite model clothing images for users based on mannequins or real-life images, and at the same time control the model's identity, features, and background. However, the characters generated by image generation technology have problems with deformed hand structures, such as incorrect number of fingers or irregular shapes, which affects the user experience. Summary of the invention

[0003] The embodiments of the present application provide an image restoration method, a character image restoration method, an electronic device and a storage medium, so as to at least solve the technical problem in the related art that the hand structure of the character generated by the image generation model is deformed, resulting in poor image perception.

[0004] According to one aspect of an embodiment of the present application, an image restoration method is provided, comprising: obtaining a first area image from an image to be processed, wherein the image to be processed includes a target object, and the first area image includes an element to be restored on the target object that requires image restoration; performing three-dimensional reconstruction on the element to be restored based on the first area image to obtain a target reconstruction model, wherein the target reconstruction model is used to characterize the three-dimensional model corresponding to the element to be restored when the structure is correct; performing image segmentation on the element to be restored based on the first area image to obtain a segmentation mask, wherein the segmentation mask is used to characterize the pixel-level area of ​​the element to be restored in the first area image; redrawing the first area image based on the segmentation mask and the target reconstruction model to obtain a second area image; and fusing the second area image with the image to be processed to obtain a target image.

[0005] According to another aspect of an embodiment of the present application, a method for repairing a person image is also provided, including: in response to detecting a change in a person image, performing an abnormality detection on the person image to obtain a detection result, wherein the detection result is used to characterize whether different person parts in the person image are normal; in response to detecting that a target part among the person parts is abnormal, capturing a first area image where the target part is located from the person image; performing three-dimensional reconstruction of the target part based on the first area image to obtain a part reconstruction model, and segmenting the first area image based on the target part to obtain a part segmentation mask, wherein the part reconstruction model is used to characterize the three-dimensional model corresponding to the target part when the structure is correct, and the part segmentation mask is used to characterize the pixel-level area of ​​the target part in the first area image; redrawing the first area image based on the part segmentation mask and the part reconstruction model to obtain a second area image; and fusing the second area image with the person image to obtain a target repaired image.

[0006] According to another aspect of an embodiment of the present application, an image restoration method is also provided, including: responding to an input instruction on an operation interface, displaying an image to be processed on the operation interface, wherein the image to be processed includes a target object, and there are elements to be restored on the target object that need to be restored; responding to an image restoration instruction on the operation interface, displaying a target image on the operation interface, wherein the target image is used to represent an image obtained by fusing a second region image and the image to be processed, the second region image is used to represent an image redrawn of the first region image based on a segmentation mask and a target reconstruction model, the first region image includes the elements to be restored, the segmentation mask is used to represent an image mask obtained by performing image segmentation on the elements to be restored based on the first region image, and the target reconstruction model is used to represent a three-dimensional model corresponding to the elements to be restored when the structure is correct.

[0007] According to another aspect of an embodiment of the present application, an image restoration method is also provided, including: obtaining an image to be processed by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter includes the image to be processed; obtaining a first area image from the image to be processed, wherein the image to be processed includes a target object, and the first area image includes an element to be restored on the target object that needs to be restored; performing three-dimensional reconstruction on the element to be restored based on the first area image to obtain a target reconstruction model, wherein the target reconstruction model is used to characterize the three-dimensional model corresponding to the element to be restored when the structure is correct; performing image segmentation on the element to be restored based on the first area image to obtain a segmentation mask, wherein the segmentation mask is used to characterize the pixel-level area of ​​the element to be restored in the first area image; redrawing the first area image based on the segmentation mask and the target reconstruction model to obtain a second area image; fusing the second area image and the image to be processed to obtain a target image; outputting the target image by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the target image.

[0008] According to another aspect of an embodiment of the present application, an image restoration device is also provided, including: an image capture module, used to capture a first area image from an image to be processed, wherein the image to be processed includes a target object, and the first area image includes an element to be restored on the target object that needs to be restored; a model reconstruction module, used to perform three-dimensional reconstruction of the element to be restored based on the first area image to obtain a target reconstruction model, wherein the target reconstruction model is used to characterize the three-dimensional model corresponding to the element to be restored when the structure is correct; an image segmentation module, used to perform image segmentation of the element to be restored based on the first area image to obtain a segmentation mask, wherein the segmentation mask is used to characterize the pixel-level area of ​​the element to be restored in the first area image; an image drawing module, used to redraw the first area image based on the segmentation mask and the target reconstruction model to obtain a second area image; an image fusion module, used to fuse the second area image with the image to be processed to obtain a target image.

[0009] According to another aspect of an embodiment of the present application, a person image restoration device is also provided, including: an image detection module, for performing abnormality detection on the person image in response to detecting a change in the person image, and obtaining a detection result, wherein the detection result is used to characterize whether different person parts in the person image are normal; a first interception module, for intercepting a first area image where the target part is located from the person image in response to detecting that a target part among the person parts is abnormal; an image processing module, for performing three-dimensional reconstruction of the target part based on the first area image to obtain a part reconstruction model, and for segmenting the first area image based on the target part to obtain a part segmentation mask, wherein the part reconstruction model is used to characterize the three-dimensional model corresponding to the target part when the structure is correct, and the part segmentation mask is used to characterize the position of the target part in the first area image; a first drawing module, for redrawing the first area image based on the part segmentation mask and the part reconstruction model to obtain a second area image; a first fusion module, for repairing the person image based on the part target reconstruction model and the part segmentation mask to obtain a target repaired image.

[0010] According to another aspect of an embodiment of the present application, an image restoration device is also provided, including: a first display module, used to respond to an input instruction acting on an operation interface, and display an image to be processed on the operation interface, wherein the image to be processed includes a target object, and there are elements to be restored on the target object that need to be restored; a second display module, used to respond to an image restoration instruction acting on the operation interface, and display a target image on the operation interface, wherein the target image is used to represent an image obtained by fusing the second area image and the image to be processed, the second area image is used to represent an image redrawn of the first area image based on a segmentation mask and a target reconstruction model, the first area image includes the elements to be restored, the segmentation mask is used to represent an image mask obtained by performing image segmentation of the elements to be restored based on the first area image, and the target reconstruction model is used to represent a three-dimensional model corresponding to the elements to be restored when the structure is correct.

[0011] According to another aspect of an embodiment of the present application, an image restoration device is also provided, including: a first calling module, used to obtain an image to be processed by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter includes the image to be processed; a second interception module, used to intercept a first region image from the image to be processed, wherein the image to be processed includes a target object, and the first region image includes an element to be restored on the target object that needs to be restored; a three-dimensional reconstruction module, used to perform three-dimensional reconstruction of the element to be restored based on the first region image to obtain a target reconstruction model, wherein the target reconstruction model is used to characterize the three-dimensional model corresponding to the element to be restored when the structure is correct; a first segmentation module, used to perform image segmentation of the element to be restored based on the first region image to obtain a segmentation mask, wherein the segmentation mask is used to characterize the pixel-level area of ​​the element to be restored in the first region image; a second drawing module, used to redraw the first region image based on the segmentation mask and the target reconstruction model to obtain a second region image; a second fusion module, used to fuse the second region image and the image to be processed to obtain a target image; a second calling module, used to output the target image by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the target image.

[0012] According to another aspect of the embodiments of the present application, an electronic device is further provided, including: a memory storing an executable program; and a processor for running the program, wherein the method of each embodiment of the present application is executed when the program is running.

[0013] According to another aspect of an embodiment of the present application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the methods in each embodiment of the present application.

[0014] According to another aspect of the embodiments of the present application, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present application is implemented.

[0015] According to another aspect of an embodiment of the present application, a computer program product is also provided, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present application is implemented.

[0016] According to another aspect of the embodiments of the present application, a computer program is also provided, and when the computer program is executed by a processor, the methods in the various embodiments of the present application are implemented.

[0017] In an embodiment of the present application, a first area image is obtained by cutting out from an image to be processed; based on the first area image, three-dimensional reconstruction is performed on the element to be repaired to obtain a target reconstruction model; based on the first area image, image segmentation is performed on the element to be repaired to obtain a segmentation mask; based on the segmentation mask and the target reconstruction model, the first area image is redrawn to obtain a second area image; the second area image and the image to be processed are fused to obtain a target image. After three-dimensional modeling of the element to be repaired, the first area image is redrawn according to the reconstructed model and the segmentation mask corresponding to the element to be repaired. This ensures that the redrawn second area image can meet the visual effect presented by the element to be repaired in the real scene, thereby ensuring the realism and completeness of the target image obtained by fusing the second area image and the image to be processed, and ensuring the visual effect of the target image, thereby solving the technical problem in the related art that the hand structure of the character generated by the image generation model is deformed, resulting in poor image perception.

[0018] It is easy to notice that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present application, and do not constitute a limitation of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 It is a hardware structure block diagram of an electronic device (or mobile device) for implementing an image restoration method according to the present application;

[0021] Figure 2 is a structural block diagram of a computing environment according to the present application;

[0022] Figure 3 is a structural block diagram of a service grid according to the present application;

[0023] Figure 4 is a flow chart of an image restoration method according to the present application;

[0024] Figure 5 is a flow chart of a method for restoring a human image according to the present application;

[0025] Figure 6 is a schematic diagram of a hand region restoration process of a person image according to the present application;

[0026] Figure 7 is a flow chart of an image restoration method according to the present application;

[0027] Figure 8 is a flow chart of an image restoration method according to the present application;

[0028] Fig. 9 is a schematic diagram of an image restoration device according to the present application;

[0029] Fig.10 is a schematic diagram of a person image restoration device according to the present application;

[0030] Fig.11 is a schematic diagram of another image restoration device according to the present application;

[0031] Fig.12 is a schematic diagram of another image restoration device according to the present application;

[0032] Fig.13 It is a structural block diagram of an electronic device according to the present application. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0036] Hand key points: 21 points used to represent the hand structure, with a fixed topological relationship.

[0037] Diffusion Model: Diffusion Model, a model for image generation.

[0038] ControlNet network: A control network used to introduce specific information into the diffusion model, which can assist the diffusion model in generating image structures, etc.

[0039] 3D hand reconstruction: Able to reconstruct a three-dimensional hand model from a two-dimensional RGB image.

[0040] Local Redraw: Regenerates a specified area in an image while keeping other areas of the image unchanged, while maintaining harmony between the regenerated area and other areas.

[0041] Hand deformities: The hand in the image has an incorrect number of fingers or irregular shapes.

[0042] Poisson blending: An image fusion technique that can seamlessly blend part of the source image into the target image.

[0043] According to an embodiment of the present application, an image restoration method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0044] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, an electronic device or a similar computing device. Figure 1 1 is a hardware structure block diagram of an electronic device (or mobile device) for implementing an image restoration method according to the present application. Figure 1 As shown, the electronic device 10 (or mobile device) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. Those skilled in the art will understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0045] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the electronic device 10 (or mobile device). As involved in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0046] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the methods in the embodiments of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the methods in the above embodiments. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the electronic device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0047] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the electronic device 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet wirelessly.

[0048] The display may be, for example, a touch screen liquid crystal display (LCD), which may enable a user to interact with a user interface of the electronic device 10 (or mobile device).

[0049] Figure 1 The hardware structure block diagram shown can be used not only as an exemplary block diagram of the electronic device 10 (or mobile device), but also as an exemplary block diagram of the server. In an optional embodiment, Figure 2 The block diagram shows the use of the above Figure 1The electronic device 10 (or mobile device) is shown as an embodiment of a computing node in the computing environment 201 . Figure 2 is a structural block diagram of a computing environment shown in the present application, such as Figure 2 As shown, computing environment 201 includes multiple computing nodes (such as servers) running on a distributed network (shown as 210-1, 210-2, ..., in the figure). The computing nodes all contain local processing and memory resources, and end users 202 can remotely run applications or store data in computing environment 201. Applications can be provided as multiple services 220-1, 220-2, 220-3 and 220-4 in computing environment 201, representing services "A", "D", "E" and "H" respectively.

[0050] The end user 202 can provide and access services through a web browser or other software application on the client, and in some embodiments, the end user 202's provision and / or request can be provided to the entry gateway 230. The entry gateway 230 can include a corresponding agent to handle the provision and / or request for the service (one or more services provided in the computing environment 201).

[0051] Services are provided or deployed based on various virtualization technologies supported by the computing environment 201. In some embodiments, services can be provided based on virtual machine (VM)-based virtualization, container-based virtualization, and / or similar methods. Virtual machine-based virtualization can be to simulate a real computer by initializing a virtual machine, and execute programs and applications without directly contacting any actual hardware resources. While the virtual machine virtualizes the machine, according to container-based virtualization, a container can be started to virtualize the entire operating system (OS) so that multiple workloads can run on a single operating system instance.

[0052] In an embodiment based on container virtualization, several containers of a service can be assembled into a Pod (e.g., a Kubernetes Pod). Figure 2 As shown, service 220-2 can be equipped with one or more Pods 240-1, 240-2, ..., 240-N (collectively referred to as Pods). Pods can include a proxy 245 and one or more containers 242-1, 242-2, ..., 242-M (collectively referred to as containers). One or more containers in a Pod process requests related to one or more corresponding functions of the service, and the proxy 245 generally controls network functions related to the service, such as routing, load balancing, etc. Other services can also be equipped with similar Pods.

[0053] During operation, executing a user request from end user 202 may require invoking one or more services in computing environment 201, and executing one or more functions of a service may require invoking one or more functions of another service. Figure 2 As shown, service "A" 220-1 receives a user request from end user 202 from ingress gateway 230, service "A" 220-1 may call service "D" 220-2, and service "D" 220-2 may request service "E" 220-3 to perform one or more functions.

[0054] The computing environment described above can be a cloud computing environment, where the allocation of resources is managed by the cloud service provider, allowing the development of functions without considering the implementation, adjustment or expansion of servers. The computing environment allows developers to execute code in response to events without building or maintaining complex infrastructure. Services can be divided into a set of functions that can be automatically and independently scaled, rather than expanding a single hardware device to handle potential loads.

[0055] In another optional embodiment, Figure 3 The block diagram shows the use of the above Figure 1 The electronic device 10 (or mobile device) is shown as an embodiment of a service mesh. Figure 3 is a structural block diagram of a service grid according to the present application, such as Figure 3 As shown, the service grid 300 is mainly used to facilitate secure and reliable communication between multiple microservices. Microservices refer to decomposing an application into multiple smaller services or instances and distributing them on different clusters / machines.

[0056] like Figure 3 As shown, the microservice may include an application service instance A and an application service instance B, which form a functional application layer of the service grid 300. In one embodiment, the application service instance A runs in a machine / workload container group 314 (Pod) in the form of a container / process 308, and the application service instance B runs in a machine / workload container group 316 (Pod) in the form of a container / process 310.

[0057] In one implementation, application service instance A may be a product query service, and application service instance B may be a product ordering service.

[0058] like Figure 3As shown, application service instance A and grid proxy (sidecar) 303 coexist in machine workload container group 314, and application service instance B and grid proxy 305 coexist in machine workload container 316. Grid proxy 303 and grid proxy 305 form the data plane layer (dataplane) of service grid 300. Grid proxy 303 and grid proxy 305 run in the form of container / process 304 and container / process 306 respectively, and can receive request 312 for commodity query service, and grid proxy 303 and application service instance A can communicate bidirectionally, and grid proxy 305 and application service instance B can communicate bidirectionally. In addition, grid proxy 303 and grid proxy 305 can also communicate bidirectionally.

[0059] In one embodiment, the traffic of application service instance A is routed to a suitable destination through grid proxy 303, and the network traffic of application service instance B is routed to a suitable destination through grid proxy 305. It should be noted that the network traffic mentioned here includes but is not limited to Hyper Text Transfer Protocol (HTTP), Representational State Transfer (REST), high-performance, general open source framework (google Remote Procedure Call, gRPC), open source in-memory data structure storage system (Redis), etc.

[0060] In one embodiment, the function of extending the data plane layer can be realized by writing a custom filter for the proxy (Envoy) in the service mesh 300. The service mesh proxy configuration can be to enable the service mesh to correctly proxy service traffic and realize service intercommunication and service governance. Mesh proxy 303 and mesh proxy 305 can be configured to perform at least one of the following functions: service discovery, health checking, routing, load balancing, authentication and authorization, and observability.

[0061] like Figure 3 As shown, the service grid 300 also includes a control plane layer. The control plane layer may be a group of services running in a dedicated namespace, and these services are hosted by a hosting control plane component 301 in a machine / workload container group (machine / Pod) 302. Figure 3As shown, the managed control plane component 301 performs bidirectional communication with the grid agent 303 and the grid agent 305. The managed control plane component 301 is configured to perform some control management functions. For example, the managed control plane component 301 receives telemetry data transmitted by the grid agent 303 and the grid agent 305, and can further aggregate the telemetry data. For these services, the managed control plane component 301 can also provide a user-oriented application programming interface (Application Programming Interface, API) to more easily manipulate network behavior, and provide configuration data to the grid agent 303 and the grid agent 305.

[0062] Under the above operating environment, this application provides Figure 4 The image restoration method shown. Figure 4 is a flow chart of an image restoration method according to the present application, such as Figure 4 As shown, the method may include the following steps:

[0063] Step S402: obtaining a first region image from the image to be processed.

[0064] The image to be processed includes a target object, and the first region image includes elements to be repaired on the target object that require image repair.

[0065] The above-mentioned image to be processed may refer to an image that currently needs to be repaired. The image to be processed may include at least one object that needs to be repaired, that is, the above-mentioned target object. The target object may include one or more areas that need to be repaired, and the elements corresponding to the areas may refer to the above-mentioned elements to be repaired that need to be repaired. For ease of understanding, assuming that the user currently uses a character image generation model to generate a character image, and the hand of the character generated in this character image is distorted, resulting in an incomplete hand, and the hand of the character in the image has a more obvious sense of disharmony with the overall image, then at this time, it can be considered that this generated character image is an image that needs to be repaired, that is, the above-mentioned image to be processed, and the character contained in this corresponding image can be the above-mentioned target object, the hand of the character can be the above-mentioned element to be repaired that needs to be partially redrawn, and the area where the hand of the character is located can be the above-mentioned first area image. It should be noted that the character image generated by the character image generation model described here, and the distorted hand in the image, are all exemplary displays for describing the above-mentioned image to be processed, target object, element to be repaired, first area image, etc., and are not limited here.

[0066] In an optional scheme of the present embodiment, when performing image restoration, the image restoration system may first acquire the image to be processed that currently needs to be restored, and then perform image recognition on the acquired image to be processed, determine the target object and the elements to be restored on the target object from the image to be processed, and in order to facilitate the processing of the elements to be restored, such as removing other elements in the elements to be restored that are not related to the elements to be restored, replacing and filling the elements to be restored, etc., the image restoration system may, after identifying the elements to be restored, intercept the area where the elements to be restored are located in the image to be restored, so as to obtain the above-mentioned first area image, and the image restoration system may directly repair the elements to be restored in the intercepted first area image without considering the entire image to be restored, thereby reducing the consumption of restoration resources and improving the efficiency of image restoration. Among them, the above-mentioned image to be processed can be an image actively obtained by the image restoration system from the image library to be restored, or it can be an image actively sent by the user to the image restoration system. The specific method of obtaining the image to be processed is not limited here. In order to ensure that the user's expected requirements can be met when the image to be processed is restored, the target object, the element to be restored, the first area image and other contents determined from the image to be processed can also be actively determined by the user, which is not limited here.

[0067] Step S404: Perform three-dimensional reconstruction on the element to be repaired based on the first region image to obtain a target reconstructed model.

[0068] Among them, the target reconstruction model is used to represent the three-dimensional model corresponding to the element to be repaired when the structure is correct.

[0069] The target reconstruction model mentioned above may refer to a three-dimensional model obtained by virtual modeling of the element to be repaired in a virtual environment.

[0070] In an optional scheme of the present embodiment, after the first area image is captured from the image to be processed, the image restoration system can repair the elements to be restored contained in the first area image. For example, the elements to be restored can be identified first to determine a series of features of the elements to be restored, such as the type, posture, texture, etc., and then complete elements matching these features can be obtained from a preset element image library to replace the elements to be restored contained in the first area image, so as to realize the operation of repairing the elements to be restored. Alternatively, changes in features such as texture and color of the first area image can be extracted, and textures and colors that match the changes can be selected to fill the first area image, so as to repair the elements to be restored contained in the first area image. Although simple element replacement or image filling can initially achieve the effect of element repair, the elements repaired in this way usually have a great sense of disharmony with the image to be processed, and the degree of fit between the two will be very low. Considering that the process of users viewing the image to be processed can be regarded as the process of viewing the target object in a real scene according to a specific visual angle, therefore, in order to ensure the accuracy of repairing the image to be processed and to ensure the degree of fit between the repaired elements and the elements to be processed, the image repair system can simulate the scene of the user viewing the elements to be repaired on the target object in a real scene through three-dimensional modeling, and then use the established three-dimensional model to repair the elements to be repaired in the first area image, so that the repaired elements can match the image to be processed. After intercepting the first area image, the image repair system can perform three-dimensional reconstruction on the elements to be repaired based on the first area image to obtain the corresponding three-dimensional model of the elements to be repaired when the structure is correct, that is, the above-mentioned target reconstruction model, wherein the target reconstruction model can represent the correct structure of the elements to be repaired and have a good degree of fit.

[0071] Step S406: performing image segmentation on the element to be repaired based on the first region image to obtain a segmentation mask.

[0072] The segmentation mask is used to represent the pixel-level region of the element to be repaired in the first region image.

[0073] In an optional scheme of the present embodiment, considering that the target reconstructed model obtained by three-dimensional reconstruction of the element to be repaired usually does not contain other elements except the element to be repaired, if the constructed target reconstruction model is directly used to repair the element to be repaired based on the first area image, errors in data such as the position and size of the repaired element may occur. Therefore, while constructing the target reconstruction model, the image repair system can also use the element to be repaired to segment the first area image to obtain the above-mentioned segmentation mask, that is, the pixel-level area of ​​the element to be repaired in the first area image, so as to determine the specific position, size and other information of the element to be repaired in the first area image, thereby ensuring that after the element to be repaired is repaired using the target reconstruction model, the element to be repaired can be accurately integrated into the image to be processed according to these position and size information, thereby improving the time effect presented by the image after the processed image is repaired.

[0074] Step S408: redraw the first region image based on the segmentation mask and the target reconstruction model to obtain a second region image.

[0075] In an optional solution of the present embodiment, after determining the target reconstruction model and the segmentation mask, the image restoration system may use the target reconstruction model and the segmentation mask to redraw the first region image to obtain an image that can reflect the visual effect presented by the element to be restored when the structure is correct, that is, the second region image. For example, the image restoration system may first determine the area that can be drawn from the first region image according to the segmentation mask, that is, the area where the element to be restored is located in the first region image when the structure is correct, and then adjust the target reconstruction model according to the visual parameters presented by the element to be restored in the first region image, such as visual angle, size and other information, and then draw the model details and detail positions presented by the adjusted target reconstruction model in the determined area that can be drawn according to the segmentation mask to obtain the second region image.

[0076] Step S410: Fusing the second region image and the image to be processed to obtain a target image.

[0077] In an optional scheme of the present embodiment, after drawing the second area image, the image restoration system may fuse the second area image with the image to be processed to obtain a target image with better visual effects. For ease of understanding, for example, the image restoration system may first adjust the target reconstruction model according to the visual parameters presented by the element to be restored in the first area image, such as visual angle, size and other information, and then draw the model details presented by the adjusted target reconstruction model into the segmentation mask according to the corresponding detail positions to obtain the restored element image, and finally fuse the element image into the target image, thereby realizing the operation of repairing the image to be processed, wherein, in order to ensure the degree of fit of the new image obtained by fusing different images, Poisson fusion technology can be used as the main method of image fusion in this application.

[0078] In an embodiment of the present application, a first area image is obtained by cutting out from an image to be processed; based on the first area image, three-dimensional reconstruction is performed on the element to be repaired to obtain a target reconstruction model; based on the first area image, image segmentation is performed on the element to be repaired to obtain a segmentation mask; based on the segmentation mask and the target reconstruction model, the first area image is redrawn to obtain a second area image; the second area image and the image to be processed are fused to obtain a target image. After three-dimensional modeling of the element to be repaired, the first area image is redrawn according to the reconstructed model and the segmentation mask corresponding to the element to be repaired. This ensures that the redrawn second area image can meet the visual effect presented by the element to be repaired in the real scene, thereby ensuring the realism and completeness of the target image obtained by fusing the second area image and the image to be processed, and ensuring the visual effect of the target image, thereby solving the technical problem in the related art that the hand structure of the character generated by the image generation model is deformed, resulting in poor image perception.

[0079] In an embodiment of the present application, the first area image is redrawn based on the segmentation mask and the target reconstruction model to obtain the second area image, including: based on the target reconstruction model, obtaining the depth image and edge image corresponding to the element to be repaired, wherein the depth image and the edge image are used to characterize the correct structure of the element to be repaired; binarizing the depth image to obtain a reconstructed area mask; merging the reconstructed area mask and the segmentation mask to obtain a repair area mask, wherein the repair area mask is used to characterize the pixel-level area of ​​the repaired element to be repaired in the first area image; based on the depth image, the edge image and the repair area mask, the first area image is redrawn through the image repair model to obtain the second area image.

[0080] In an optional scheme implemented in the present invention, in order to ensure the accuracy of the drawn second area image, the image restoration system can determine the depth image and edge image corresponding to the element to be restored when the structure is correct based on the constructed target reconstruction model, and binarize the depth image to obtain the area mask corresponding to the element to be restored when the structure is correct in the two-dimensional plane, that is, the above-mentioned reconstructed area mask, and then determine the intersection of the reconstructed area mask and the segmentation mask. For example, the reconstructed area mask and the segmentation mask can be merged to determine the pixel-level area of ​​the restored element to be restored in the first area image to obtain the above-mentioned restoration area mask. Finally, the image restoration system can use the image restoration model to redraw the first area image based on the depth image, edge image and restoration area mask to obtain the above-mentioned second area image.

[0081] In an embodiment of the present application, based on the target reconstruction model, a depth image and an edge image corresponding to the element to be repaired are obtained, including: aligning the target reconstruction model with the element to be repaired to obtain a reconstructed alignment model; extracting the depth value from the reconstructed alignment model to obtain a depth image; performing edge detection on the depth image to obtain an edge image.

[0082] In an optional scheme of the present embodiment, considering that if the target reconstructed model is not aligned with the element to be repaired, for example, the visual angle presented by the target reconstructed model is different from the visual angle presented by the element to be repaired in the first area image, or the size of the target reconstructed model is different from the size of the element to be repaired, then when depth processing and edge detection are performed on the target reconstructed model, the depth image and edge image obtained may not match the element to be repaired, resulting in errors in the second area image drawn using the depth image and edge image, and not matching the element to be repaired. Therefore, in order to ensure the accuracy of the determined depth image and edge image, the image restoration system can first align the target reconstructed model and the element to be repaired to obtain the above-mentioned reconstructed alignment model, and then perform depth processing on the reconstructed alignment model, for example, extracting the depth value from the reconstructed alignment model to obtain the above-mentioned depth image, and then performing edge detection on the depth image to obtain the above-mentioned edge image.

[0083] In an embodiment of the present application, the target reconstruction model is aligned with the element to be repaired to obtain a reconstruction alignment model, including: identifying the segmentation mask to obtain the element observation angle, wherein the element observation angle is used to characterize the visual angle presented when observing the element to be repaired in the scene corresponding to the segmentation mask; based on the element observation angle, the target reconstruction model and the element to be repaired are aligned to obtain a reconstruction alignment model.

[0084] In an optional solution of this embodiment, in order to ensure the accuracy of aligning the target reconstruction model and the element to be repaired, the image restoration system can first identify the segmentation mask to determine the visual angle of the user when observing the element to be repaired in the scene corresponding to the segmentation mask, thereby obtaining the above-mentioned element observation angle, and then use the element observation angle to adjust the visual angle presented by the target reconstruction model to align the target reconstruction model and the element to be repaired to obtain the above-mentioned reconstruction alignment model. In addition, when aligning, parameters such as the size and position of the element to be repaired can also be considered, which will not be repeated here.

[0085] In an embodiment of the present application, the image restoration model includes at least: an image restoration module and an image diffusion module. Based on the depth image, the edge image and the restoration area mask, the first area image is redrawn through the image restoration model to obtain the second area image, including: based on the depth image and the edge image, the structure of the element to be restored is drawn in the restoration area mask through the image restoration module to obtain an initial drawing image; feature extraction of the element to be restored in the first area image to obtain at least one element feature; based on the element feature, the initial drawing image is adjusted through the image diffusion module to obtain the second area image.

[0086] The image diffusion module may refer to a diffusion model, and a ControlNet network may be configured on the diffusion model.

[0087] The above-mentioned element features may refer to features that can reflect the visual effects presented by the element to be repaired, and may include but are not limited to: texture features, color features, style features and other features of the element to be repaired.

[0088] In an optional scheme of the present embodiment, in order to ensure the accuracy of the obtained second area image, the image restoration model may include at least two parts: an image restoration module and an image diffusion module. When the image restoration model is used to redraw the element to be restored, the image restoration system may first redraw the structure of the element to be restored in the determined restoration area mask through the image restoration module based on the depth image and the edge image to obtain an initial drawing image of the element to be restored with a correct structure. At the same time, the element features of the element to be restored can be extracted in the first area image to obtain the above-mentioned at least one element feature, and then the initial drawing image can be adjusted through the image diffusion model based on the extracted element features to obtain the above-mentioned second area image.

[0089] In an embodiment of the present application, the element features include at least: texture features, color features and style features. Based on the element features, the initial drawn image is adjusted through the image diffusion module to obtain the second area image, including: adjusting the texture of the initial drawn image based on the texture features to obtain a first adjusted image; adjusting the color of the initial drawn image based on the color features to obtain a second adjusted image; adjusting the style of the initial drawn image based on the style features to obtain a third adjusted image; and fusing the first adjusted image, the second adjusted image and the third adjusted image to obtain the second area image.

[0090] In an optional scheme of the present embodiment, as shown above, the above-mentioned element features may include but are not limited to: texture features, color features, style features and other features. Correspondingly, when the element features of the element to be repaired are used to adjust the initial drawn image, the image repair system may first use the texture features to adjust the texture of the initial drawn image to obtain the above-mentioned first adjusted image, and use the color features to adjust the color of the initial drawn image to obtain the above-mentioned second adjusted image, and at the same time use the style features to adjust the style of the initial drawn image to obtain the above-mentioned third adjusted image, and then merge the above-mentioned first adjusted image, the second adjusted image, and the third adjusted image to obtain the above-mentioned second area image.

[0091] In an embodiment of the present application, a three-dimensional reconstruction is performed on the element to be repaired based on the first area image to obtain a target reconstructed model, including: obtaining a model reconstruction tool and three-dimensional model parameters that match the element type of the element to be repaired; extracting feature points of the element to be repaired to obtain element feature points; based on the element feature points, using the model reconstruction tool to perform three-dimensional modeling of the element to be repaired to obtain an initial reconstructed model; adjusting the initial reconstructed model based on the three-dimensional reconstruction parameters to obtain a target reconstructed model.

[0092] In an optional solution of the present embodiment, considering that the detail distribution corresponding to different elements to be repaired may be different, different model reconstruction tools can be configured for different types of elements to be repaired. Correspondingly, when performing three-dimensional reconstruction on the elements to be repaired, the image repair system can first determine the element type of the elements to be repaired, and obtain model reconstruction and three-dimensional model parameters that match the element type of the elements to be repaired, wherein the three-dimensional model parameters can be used to supplement the constructed three-dimensional model in detail to ensure the accuracy of the model. While obtaining the model reconstruction tool and three-dimensional reconstruction parameters, the image repair system can also extract the feature points of the elements to be repaired. For example, when performing 3D hand reconstruction on the hands of a person, the hand key points contained in the hand image can be extracted to obtain the above-mentioned element feature points, and then the model reconstruction tool is used to perform three-dimensional reconstruction on the elements to be repaired based on the extracted element feature points to obtain an initial reconstruction model that matches the elements to be repaired, and then the initial reconstruction model is adjusted using the three-dimensional reconstruction model to obtain a target reconstruction model with higher accuracy.

[0093] In an embodiment of the present application, a first area image is captured from an image to be processed, including: identifying the image to be processed and determining an initial bounding box corresponding to the boundary of the element to be repaired; expanding the bounding box according to a preset graphic to obtain a target bounding box; capturing the image to be processed according to the target bounding box to obtain an initial area image; adjusting the initial area image according to a preset resolution to obtain a first area image.

[0094] In an optional scheme of the present embodiment, in order to avoid the situation where the intercepted first area image cannot contain the complete element to be repaired and to facilitate the processing of the first area image, when intercepting the first area image, the image restoration system can first identify the image to be processed to determine the initial bounding box corresponding to the boundary of the element to be repaired, and then expand the boundary image according to a preset graphic, such as a rectangle, to obtain the above-mentioned target bounding box, and then intercept the image to be processed according to the target bounding box to obtain the initial area image, and finally adjust the initial area image according to the preset resolution to obtain an easy-to-operate first area image.

[0095] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0096] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0097] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0098] According to an embodiment of the present application, a method for repairing a character image is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0099] Figure 5 is a flowchart of a method for restoring a human image according to the present application. Figure 5 As shown, the method may include the following steps:

[0100] Step S502, in response to detecting that a change has occurred in the person image, performing abnormality detection on the person image to obtain a detection result.

[0101] The detection result is used to indicate whether different parts of a person in a person image are normal.

[0102] The above-mentioned character image may refer to an image generated by a character image generation model, or may be a character image directly obtained from an image library, which is not limited here.

[0103] In an optional scheme of the present embodiment, considering that in a character image generated by a character image generation model, for example, a character fitting image generated for different clothing, the details of the character in the image may need to be supplemented due to the differences between the clothing, resulting in a situation where the supplemented details do not match the character as a whole, that is, deformed parts appear in the generated character image, and the corresponding user may feel a great sense of disharmony when viewing the character image. Therefore, in order to improve the appearance of the generated character image, the image restoration system can first obtain the character image generated by the character image model, and then, when it is detected that the generated character image has changed compared with the image before generation, further abnormality detection is performed on the generated character image to determine whether the different character parts contained in the generated character image are normal and whether there are any deformed target parts.

[0104] Step S504 , in response to detecting that there is an abnormal target part in the human body parts, capturing a first area image where the target part is located from the human body image.

[0105] In an optional scheme of the present embodiment, if an abnormal target part is detected in the character part, in order to facilitate the repair of the target part, such as removing other elements unrelated to the elements to be repaired, replacing and filling the elements to be repaired, etc., the image repair system can first determine the position of the target part in the character image, and then intercept the area where the target part is located from the character image to obtain the above-mentioned first area image. The image repair system can repair the target part in the intercepted first area image to reduce the consumption of repair resources while improving the efficiency of image repair.

[0106] Step S506 , three-dimensionally reconstructing the target part based on the first region image to obtain a part reconstruction model, and segmenting the first region image based on the target part to obtain a part segmentation mask.

[0107] The part reconstruction model is used to represent the three-dimensional model corresponding to the target part when the structure is correct, and the part segmentation mask is used to represent the position of the target part in the first region image.

[0108] Step S508 : redrawing the first region image based on the part segmentation mask and the part reconstruction model to obtain a second region image.

[0109] In an optional solution of the present embodiment, after determining the part reconstruction model and the part segmentation mask, the image restoration system may use the part reconstruction model and the part segmentation mask to redraw the above-mentioned first region image to obtain an image that can reflect the visual effect presented by the element to be repaired when the structure is correct, that is, the above-mentioned second region image. For example, the image restoration system may first determine the area that can be drawn from the first region image according to the part segmentation mask, that is, the area where the element to be repaired is located in the first region image when the structure is correct, and then adjust the part reconstruction model according to the visual parameters presented by the element to be repaired in the first region image, such as visual angle, size and other information, and then draw the model details and detail positions presented by the adjusted part reconstruction model in the determined area that can be drawn according to the part segmentation mask to obtain the above-mentioned second region image.

[0110] Step S510: Fusing the second region image and the person image to obtain a target restored image.

[0111] In an optional solution of this embodiment, after drawing the second region image, the image restoration system may fuse the second region image with the character image to obtain a target restoration image with better visual effect.

[0112] In an optional scheme of this embodiment, after obtaining the target reconstruction model through three-dimensional reconstruction and segmenting the first area image to obtain a segmentation mask, the image restoration system can use the target reconstruction model and the segmentation mask to repair the character image, thereby obtaining a target restoration image with better visual effect.

[0113] In an embodiment of the present application, a first region image is redrawn based on a part segmentation mask and a part reconstruction model to obtain a second region image, including: based on the part reconstruction model, obtaining a depth image and an edge image corresponding to the target part, wherein the depth image and the edge image are used to characterize the correct structure of the target part; binarizing the depth image to obtain a reconstructed region mask; merging the reconstructed region mask and the part segmentation mask to obtain a repaired region mask, wherein the repaired region mask is used to characterize the pixel-level region of the repaired target part in the first region image; based on the depth image, the edge image and the repaired region mask, the first region image is redrawn through an image repair model to obtain a second region image.

[0114] In an optional scheme implemented in the present invention, in order to ensure the accuracy of the drawn second area image, the image restoration system can determine the depth image and edge image corresponding to the target part when the structure is correct based on the constructed part reconstruction model, and binarize the depth image to obtain the area mask corresponding to the target part when the structure is correct in the two-dimensional plane, that is, the above-mentioned reconstructed area mask, and then determine the intersection of the reconstructed area mask and the part segmentation mask. For example, the reconstructed area mask and the part segmentation mask can be merged to determine the pixel-level area of ​​the repaired target part in the first area image to obtain the above-mentioned repaired area mask. Finally, the image restoration system can use the image restoration model to redraw the first area image based on the depth image, edge image and repaired area mask to obtain the above-mentioned second area image.

[0115] In an embodiment of the present application, the image restoration model includes at least: an image restoration module and an image diffusion module. Based on the depth image, the edge image and the restoration area mask, the first area image is redrawn through the image restoration model to obtain the second area image, including: based on the depth image and the edge image, the structure of the target part is drawn in the restoration area mask through the image restoration module to obtain an initial drawing image; feature extraction of the target part in the first area image to obtain at least one part feature; based on the part feature, the initial drawing image is adjusted through the image diffusion module to obtain the second area image.

[0116] The image diffusion module may refer to a diffusion model, and a ControlNet network may be configured on the diffusion model.

[0117] The above-mentioned part features may refer to features that can reflect the visual effects presented by the target part, and may include but are not limited to: texture features, color features, style features and other features of the target part.

[0118] In an optional scheme of the present embodiment, in order to ensure the accuracy of the obtained second area image, the image restoration model may include at least two parts, namely, an image restoration module and an image diffusion module. When the target part is redrawn using the image restoration model, the image restoration system may first redraw the structure of the target part in the determined restoration area mask through the image restoration module based on the depth image and the edge image to obtain an initial drawing image of the target part with a correct structure. At the same time, the part features of the target part may be extracted in the first area image to obtain the above-mentioned at least one part feature, and then the initial drawing image may be adjusted through the image diffusion model based on the extracted part features to obtain the above-mentioned second area image.

[0119] In an optional solution of this embodiment, in order to facilitate understanding of the image restoration or character image restoration process mentioned in the above embodiment, taking the restoration of the hand area of ​​the character image as an example, Figure 6 This is a schematic diagram of a process of repairing the hand area of ​​a person image according to the present application, wherein 601 represents the person image before repair, 602 represents the person image after repair, the part to be repaired in the person image is the user's hand, and the corresponding 603 represents the first area image corresponding to the target part with deformity, that is, the rectangular area where the person's hand is located, 604 represents the initial reconstruction model obtained by three-dimensional modeling of the person's hand in the first area image, 605 represents the target reconstruction model obtained after completing the details of the target reconstruction model, 606 represents the depth image obtained by depth processing the target model, or the edge image obtained by edge detection of the target model, wherein the edge image can also be obtained by detecting the edge of the depth image, 607 represents the mapping image obtained by two-dimensional mapping the target reconstruction model, that is, the reconstruction area mask, 608 represents the segmentation mask obtained by segmenting the user's hand in the first area image, 609 represents the repair area mask obtained by adjusting the segmentation mask using the mapping image, and 610 represents the second area image redrawn for the person's hand. As shown in FIG. Figure 6 As shown, when repairing a person image, the person image to be repaired can be obtained first, and then the person image can be detected, and the area where the target part to be repaired is located can be cut out from the person image, that is, the first area image where the user's hand is located, and then the user's hand is three-dimensionally modeled to obtain an initial reconstruction model, and the first area image is segmented to obtain a segmentation mask, and then the initial reconstruction model is adjusted according to the observation angle of the user's hand in the segmentation mask, and the adjusted initial reconstruction model is mapped to a two-dimensional plane to obtain a reconstruction area mask, wherein the reconstruction area mask can also be obtained by binarizing the depth image. The segmentation mask is used to adjust the reconstruction area mask, for example, the segmentation mask and the reconstruction area mask are merged to obtain a repair area mask. While mapping the initial reconstructed model to a two-dimensional plane, the initial reconstructed model can also be completed according to the user's hand details in the first area image to obtain a target reconstructed model. The image processing system can perform depth processing and edge detection on the target reconstructed model to obtain a depth image and an edge image of the target reconstructed model. Then, in the repair area mask, the first area image containing the user's hand is redrawn using the depth image and the edge image, as well as the user's hand features, to obtain a second area image. Finally, the second area image is fused with the character image before repair to obtain a repaired character image.

[0120] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the provided scheme, application scenario, and implementation process, but is not limited to the provided scheme.

[0121] According to an embodiment of the present application, an image restoration method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0122] Figure 7 is a flow chart of an image restoration method according to the present application, such as Figure 7 As shown, the method may include the following steps:

[0123] Step S702, in response to an input instruction on the operation interface, displaying the image to be processed on the operation interface.

[0124] The image to be processed includes a target object, and there are elements to be repaired on the target object that need to be repaired.

[0125] The image to be processed may refer to an image that currently needs to be repaired. The target object may refer to at least one object that needs to be repaired included in the image to be processed. The element to be repaired may refer to an element corresponding to one or more areas that need to be repaired on the target object.

[0126] In an optional solution of this embodiment, when an input instruction from a user on the operation interface is detected, the image restoration system may first obtain the above-mentioned image to be processed, and display the image to be processed in the above-mentioned operation interface for the convenience of user checking.

[0127] Step S704, responding to the image restoration instruction applied to the operation interface, and displaying the target image on the operation interface.

[0128] Among them, the target image is used to represent the image obtained by fusing the second area image and the image to be processed, the second area image is used to represent the image redrawn by the first area image based on the segmentation mask and the target reconstruction model, the first area image contains the element to be repaired, the segmentation mask is used to represent the image mask obtained by image segmentation of the element to be repaired based on the first area image, and the target reconstruction model is used to represent the three-dimensional model corresponding to the element to be repaired when the structure is correct.

[0129] In an optional solution of this embodiment, after receiving the image repair instruction input by the user on the operation interface, the image processing system can perform image repair on the image to be processed, and display the repaired target image on the operation interface for the convenience of the user to view. Wherein, when repairing the image to be processed, the image repair system can first perform image recognition on the acquired image to be processed, determine the target object and the elements to be repaired on the target object that need to be repaired from the image to be processed, and then after identifying the elements to be repaired, intercept the area where the elements to be repaired are located from the image to be processed to obtain the above-mentioned first area image. In order to ensure the accuracy of the repair of the image to be processed and to ensure the degree of fit between the repaired elements and the elements to be processed, the image repair system can simulate the scene of the user viewing the elements to be repaired on the target object in a real scene through three-dimensional modeling, and then use the details on the established three-dimensional model to repair the elements to be repaired in the first area image, so that the repaired elements can match the image to be processed. Based on this, after the first region image is intercepted, the image restoration system can perform three-dimensional reconstruction on the element to be restored based on the first region image to obtain a corresponding three-dimensional model, namely the above-mentioned target reconstruction model. Considering that the target reconstruction model obtained by three-dimensional reconstruction of the element to be restored usually does not contain other elements except the element to be restored, if the constructed target reconstruction model is directly used to repair the element to be restored based on the first region image, errors in the position, size and other data of the restored element may occur. Therefore, while constructing the target reconstruction model, the image restoration system can also use the element to be restored to segment the first region image to obtain the above-mentioned segmentation mask, thereby determining the specific position, size and other information of the element to be restored in the first region image. Finally, after obtaining the target reconstruction model through three-dimensional reconstruction and segmenting the first region image to obtain the segmentation mask, the image restoration system can use the segmentation mask and the target reconstruction model to redraw the above-mentioned first region image to obtain the second region image, and fuse the second region image with the image to be processed to obtain the target image.

[0130] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the provided scheme, application scenario, and implementation process, but is not limited to the provided scheme.

[0131] According to an embodiment of the present application, an image restoration method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0132] Figure 8is a flow chart of an image restoration method according to the present application, such as Figure 8 As shown, the method may include the following steps:

[0133] Step S802: Acquire the image to be processed by calling the first interface.

[0134] The first interface includes a first parameter, and a parameter value of the first parameter includes an image to be processed.

[0135] The first interface mentioned above may refer to an interface for obtaining a parameter value of a first parameter, that is, an interface for obtaining an image to be processed.

[0136] In an optional solution of this embodiment, when performing image restoration on the image to be processed, the image restoration system may first acquire the parameter value of the first parameter by calling the above-mentioned first interface, that is, acquire the image to be processed.

[0137] Step S804: obtaining a first region image from the image to be processed.

[0138] The image to be processed includes a target object, and the first region image includes elements to be repaired on the target object that require image repair.

[0139] In an optional scheme implemented in the present invention, after identifying the element to be repaired, the image repair system can capture the area where the element to be repaired is located in the image to be processed to obtain the above-mentioned first area image. The image repair system can directly repair the element to be repaired in the captured first area image without considering the entire image to be processed, thereby reducing the consumption of repair resources and improving the efficiency of image repair.

[0140] Step S806: Perform three-dimensional reconstruction on the element to be repaired based on the first region image to obtain a target reconstructed model.

[0141] Among them, the target reconstruction model is used to represent the three-dimensional model corresponding to the element to be repaired when the structure is correct.

[0142] In an optional solution implemented in this embodiment, in order to ensure the accuracy of repairing the image to be processed and to ensure the fit between the repaired elements and the elements to be processed, the image repair system can simulate the scene of the user viewing the elements to be repaired on the target object in a real scene through three-dimensional modeling, and then use the details on the established three-dimensional model to repair the elements to be repaired in the first area image, so that the repaired elements can match the image to be processed. Based on this, after intercepting the first area image, the image repair system can perform three-dimensional reconstruction of the elements to be repaired based on the first area image to obtain the three-dimensional model corresponding to the elements to be repaired when the structure is correct, that is, the above-mentioned target reconstruction model.

[0143] Step S808: performing image segmentation on the element to be repaired based on the first region image to obtain a segmentation mask.

[0144] The segmentation mask is used to represent the pixel-level region of the element to be repaired in the first region image.

[0145] In an optional scheme of the present embodiment, considering that the target reconstructed model obtained by three-dimensional reconstruction of the element to be repaired usually does not contain other elements except the element to be repaired, if the constructed target reconstruction model is directly used to repair the element to be repaired based on the first area image, errors in data such as the position and size of the repaired element may occur. Therefore, while constructing the target reconstruction model, the image repair system can also use the element to be repaired to segment the first area image to obtain the above-mentioned segmentation mask, that is, the pixel-level area of ​​the element to be repaired in the first area image, so as to determine the specific position, size and other information of the element to be repaired in the first area image, thereby ensuring that after the element to be repaired is repaired using the target reconstruction model, the element to be repaired can be accurately integrated into the image to be processed according to these position and size information, thereby improving the time effect presented by the image after the processed image is repaired.

[0146] Step S810: redraw the first region image based on the segmentation mask and the target reconstruction model to obtain a second region image.

[0147] In an optional solution of the present embodiment, after determining the target reconstruction model and the segmentation mask, the image restoration system may use the target reconstruction model and the segmentation mask to redraw the first region image to obtain an image that can reflect the visual effect presented by the element to be restored when the structure is correct, that is, the second region image. For example, the image restoration system may first determine the area that can be drawn from the first region image according to the segmentation mask, that is, the area where the element to be restored is located in the first region image when the structure is correct, and then adjust the target reconstruction model according to the visual parameters presented by the element to be restored in the first region image, such as visual angle, size and other information, and then draw the model details and detail positions presented by the adjusted target reconstruction model in the determined area that can be drawn according to the segmentation mask to obtain the second region image.

[0148] Step S812: Fusing the second region image and the image to be processed to obtain a target image.

[0149] In an optional solution of this embodiment, after drawing the second region image, the image restoration system may fuse the second region image with the image to be processed to obtain a target image with better visual effect.

[0150] Step S814: output the target image by calling the second interface.

[0151] The second interface includes a second parameter, and a parameter value of the second parameter includes a target image.

[0152] The second interface mentioned above may refer to an interface for outputting a parameter value of a second parameter, that is, a target image.

[0153] In an optional solution of this embodiment, after the target image is restored, the image restoration system can output the parameter value of the second parameter by calling the above-mentioned second interface, that is, output the target image to a preset operation interface for user viewing.

[0154] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the provided scheme, application scenario, and implementation process, but is not limited to the provided scheme.

[0155] According to an embodiment of the present application, an image restoration device for implementing the above-mentioned image restoration method is also provided, and the device can be deployed in a target client. Fig. 9 is a schematic diagram of an image restoration device according to the present application, such as Fig. 9 As shown, the device 900 includes: an image capture module 902 , a model reconstruction module 904 , an image segmentation module 906 , an image drawing module 908 and an image fusion module 910 .

[0156] Among them, the image capture module 902 is used to capture a first area image from the image to be processed, wherein the image to be processed includes a target object, and the first area image includes an element to be repaired on the target object that needs to be repaired; the model reconstruction module 904 is used to perform three-dimensional reconstruction of the element to be repaired based on the first area image to obtain a target reconstruction model, wherein the target reconstruction model is used to characterize the three-dimensional model corresponding to the element to be repaired when the structure is correct; the image segmentation module 906 is used to perform image segmentation of the element to be repaired based on the first area image to obtain a segmentation mask, wherein the segmentation mask is used to characterize the pixel-level area of ​​the element to be repaired in the first area image; the image drawing module 908 is used to redraw the first area image based on the segmentation mask and the target reconstruction model to obtain a second area image; the image fusion module 910 is used to fuse the second area image and the image to be processed to obtain a target image.

[0157] In an embodiment of the present application, the image drawing module 908 includes: an image acquisition unit, which is used to acquire a depth image and an edge image corresponding to the element to be repaired based on a target reconstruction model, wherein the depth image and the edge image are used to characterize the correct structure of the element to be repaired; a mask acquisition unit, which is used to binarize the depth image to obtain a reconstructed area mask; a mask merging unit, which is used to merge the reconstructed area mask and the segmentation mask to obtain a repair area mask, wherein the repair area mask is used to characterize the pixel-level area of ​​the repaired element to be repaired in the first area image; an image drawing unit, which is used to redraw the first area image through an image repair model based on the depth image, the edge image and the repair area mask to obtain a second area image.

[0158] In an embodiment of the present application, the image acquisition unit is also used to: align the target reconstruction model with the element to be repaired to obtain a reconstructed alignment model; extract the depth value from the reconstructed alignment model to obtain a depth image; perform edge detection on the depth image to obtain an edge image.

[0159] In an embodiment of the present application, the image acquisition unit is also used to: identify the segmentation mask to obtain the element observation angle, wherein the element observation angle is used to characterize the visual angle presented when observing the element to be repaired in the scene corresponding to the segmentation mask; align the target reconstruction model and the element to be repaired based on the element observation angle to obtain a reconstructed alignment model.

[0160] In an embodiment of the present application, the image restoration model includes at least: an image restoration module and an image diffusion module, and the image drawing unit is also used to: based on the depth image and the edge image, draw the structure of the element to be restored in the restoration area mask through the image restoration module to obtain an initial drawing image; extract features of the element to be restored in the first area image to obtain at least one element feature; based on the element feature, adjust the initial drawing image through the image diffusion module to obtain a second area image.

[0161] In an embodiment of the present application, the element features include at least: texture features, color features and style features, and the image drawing unit is further used to: adjust the texture of the initial drawing image based on the texture features to obtain a first adjusted image; adjust the color of the initial drawing image based on the color features to obtain a second adjusted image; adjust the style of the initial drawing image based on the style features to obtain a third adjusted image; and fuse the first adjusted image, the second adjusted image and the third adjusted image to obtain a second area image.

[0162] In the embodiment of the present application, the image segmentation module 906 includes: a tool acquisition unit, used to acquire a model reconstruction tool and three-dimensional model parameters that match the element type of the element to be repaired; a feature point extraction unit, used to extract feature points of the element to be repaired to obtain element feature points; a model reconstruction unit, used to perform three-dimensional modeling of the element to be repaired based on the element feature points using the model reconstruction tool to obtain an initial reconstructed model; and a model adjustment unit, used to adjust the initial reconstruction model based on the three-dimensional reconstruction parameters to obtain a target reconstruction model.

[0163] In an embodiment of the present application, the image capture module 902 includes: a bounding box recognition unit, which is used to recognize the image to be processed and determine the initial bounding box corresponding to the boundary of the element to be repaired; a bounding box expansion unit, which is used to expand the bounding box according to a preset graphic to obtain a target bounding box; an image capture unit, which is used to capture the image to be processed according to the target bounding box to obtain an initial area image; and an image adjustment unit, which is used to adjust the initial area image according to a preset resolution to obtain a first area image.

[0164] It should be noted that the image capture module 902, the model reconstruction module 904, the image segmentation module 906, the image drawing module 908 and the image fusion module 910 correspond to the above steps S402 to S410, and the five modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiment 1. It should be noted that the above modules or units may be hardware components or software components stored in a memory and processed by one or more processors, and the above modules may also be run in the provided server 10 as part of the device.

[0165] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the provided scheme, application scenario, and implementation process, but is not limited to the provided scheme.

[0166] According to an embodiment of the present application, a character image restoration device for implementing the above-mentioned character image restoration method is also provided, and the device can be deployed in a target client. Fig.10 is a schematic diagram of a person image restoration device according to the present application, such as Fig.10 As shown, the device 1000 includes: an image detection module 1002 , a first interception module 1004 , an image processing module 1006 , a first drawing module 1008 and a first fusion module 1010 .

[0167] Among them, the image detection module 1002 is used to perform abnormality detection on the person image in response to detecting a change in the person image, and obtain a detection result, wherein the detection result is used to characterize whether different person parts in the person image are normal; the first interception module 1004 is used to intercept a first area image where the target part is located from the person image in response to detecting that there is an abnormal target part among the person parts; the image processing module 1006 is used to perform three-dimensional reconstruction of the target part based on the first area image to obtain a part reconstruction model, and segment the first area image based on the target part to obtain a part segmentation mask, wherein the part reconstruction model is used to characterize the three-dimensional model corresponding to the target part when the structure is correct, and the part segmentation mask is used to characterize the position of the target part in the first area image; the first drawing module 1008 is used to redraw the first area image based on the part segmentation mask and the part reconstruction model to obtain a second area image; the first fusion module 1010 is used to repair the person image based on the part target reconstruction model and the part segmentation mask to obtain a target repaired image.

[0168] In an embodiment of the present application, the first drawing module 1008 includes: a first acquisition unit, which is used to acquire a depth image and an edge image corresponding to the target part based on a part reconstruction model, wherein the depth image and the edge image are used to characterize the correct structure of the target part; a binarization processing unit, which is used to binarize the depth image to obtain a reconstructed area mask; a mask determination unit, which is used to merge the reconstruction area mask and the segmentation mask to obtain a repair area mask, wherein the repair area mask is used to characterize the pixel-level area of ​​the repaired target part in the first area image; a redrawing unit, which is used to redraw the first area image through the image repair model based on the depth image, the edge image and the repair area mask to obtain a second area image.

[0169] In an embodiment of the present application, the image restoration model includes at least: an image restoration module and an image diffusion module. The redrawing unit is also used to: based on the depth image and the edge image, draw the structure of the target part in the restoration area mask through the image restoration module to obtain an initial drawing image; extract features of the target part in the first area image to obtain at least one part feature; based on the part feature, adjust the initial drawing image through the image diffusion module to obtain a second area image.

[0170] It should be noted that the image detection module 1002, the first interception module 1004, the image processing module 1006, the first drawing module 1008 and the first fusion module 1010 correspond to the above steps S502 to S510, and the five modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiment 1. It should be noted that the above modules or units may be hardware components or software components stored in a memory and processed by one or more processors, and the above modules may also be part of the device and run in the provided server 10.

[0171] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the provided scheme, application scenario, and implementation process, but is not limited to the provided scheme.

[0172] According to an embodiment of the present application, an image restoration device for implementing the above-mentioned image restoration method is also provided, and the device can be deployed in a target client. Fig.11 is a schematic diagram of another image restoration device according to the present application, such as Fig.11 As shown, the device 1100 includes: a first display module 1102 and a second display module 1104 .

[0173] Among them, the first display module 1102 is used to respond to the input instructions acting on the operation interface, and display the image to be processed on the operation interface, wherein the image to be processed includes a target object, and there are elements to be repaired on the target object that need to be repaired; the second display module 1104 is used to respond to the image repair instructions acting on the operation interface, and display the target image on the operation interface, wherein the target image is used to represent the image obtained by fusing the second area image and the image to be processed, the second area image is used to represent the image redrawn based on the segmentation mask and the target reconstruction model of the first area image, the first area image includes the elements to be repaired, the segmentation mask is used to represent the image mask obtained by image segmentation of the elements to be repaired based on the first area image, and the target reconstruction model is used to represent the three-dimensional model corresponding to the elements to be repaired when the structure is correct.

[0174] It should be noted that the first display module 1102 and the second display module 1104 correspond to the above steps S702 to S704, and the two modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiment 1. It should be noted that the above modules or units may be hardware components or software components stored in a memory and processed by one or more processors, and the above modules may also be part of the device and run in the provided server 10.

[0175] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the provided scheme, application scenario, and implementation process, but is not limited to the provided scheme.

[0176] According to an embodiment of the present application, an image restoration device for implementing the above-mentioned image restoration method is also provided, and the device can be deployed in a target client. Fig.12 is a schematic diagram of another image restoration device according to the present application, such as Fig.12 As shown, the device 1200 includes: a first calling module 1202 , a second intercepting module 1204 , a three-dimensional reconstruction module 1206 , a first segmentation module 1208 , a second drawing module 1210 , a second fusion module 1212 and a second calling module 1214 .

[0177] The first calling module 1202 is used to obtain the image to be processed by calling the first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter includes the image to be processed; the second interception module 1204 is used to intercept a first region image from the image to be processed, wherein the image to be processed includes a target object, and the first region image includes an element to be repaired on the target object that needs to be image repaired; the three-dimensional reconstruction module 1206 is used to perform three-dimensional reconstruction of the element to be repaired based on the first region image to obtain a target reconstruction model, wherein the target reconstruction model is used to characterize the three-dimensional model corresponding to the element to be repaired when the structure is correct; The first segmentation module 1208 is used to perform image segmentation on the element to be repaired based on the first area image to obtain a segmentation mask, wherein the segmentation mask is used to characterize the pixel-level area of ​​the element to be repaired in the first area image; the second drawing module 1210 is used to redraw the first area image based on the segmentation mask and the target reconstruction model to obtain the second area image; the second fusion module 1212 is used to fuse the second area image and the image to be processed to obtain the target image; the second calling module 1214 is used to output the target image by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the target image.

[0178] It should be noted that the first calling module 1202, the second intercepting module 1204, the 3D reconstruction module 1206, the first segmentation module 1208, the second drawing module 1210, the second fusion module 1212 and the second calling module 1214 correspond to the above steps S802 to S814, and the seven modules are the same as the examples and application scenarios implemented by the corresponding steps, but are not limited to the contents disclosed in the above embodiment 1. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be run in the provided server 10 as part of the device.

[0179] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the provided scheme, application scenario, and implementation process, but is not limited to the provided scheme.

[0180] The embodiment of the present application may provide an electronic device, which may be any electronic device in a group of electronic devices. Optionally, in this embodiment, the electronic device may also be replaced by a terminal device such as a mobile terminal.

[0181] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0182] In this embodiment, the electronic device can execute the program code in the method.

[0183] Optionally, Fig.13 is a structural block diagram of an electronic device according to the present application. Fig.13 As shown, the electronic device A may include: one or more (only one is shown in the figure) processors 1302, a memory 1304, a storage controller, and a peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display.

[0184] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implementing the methods in the above embodiments. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0185] Those skilled in the art will understand that Fig.13 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, and other terminal devices. The figure does not limit the structure of the above electronic devices. For example, the electronic device A may also include more or fewer components (such as a network interface, a display device, etc.) than those shown in the figure, or have a configuration different from that shown in the figure.

[0186] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0187] The embodiment of the present application further provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the method provided in the above embodiment.

[0188] Optionally, in this embodiment, the above storage medium may be located in any electronic device in a group of electronic devices in a computer network, or in any mobile terminal in a group of mobile terminals.

[0189] The embodiment of the present application further provides a computer program product. Optionally, in this embodiment, the computer program product may include a computer program, and the computer program implements the method provided in the embodiment when executed by a processor.

[0190] The embodiments of the present application also provide a computer program product. Optionally, the computer program product may include a non-volatile computer-readable storage medium, which may be used to store a computer program, and when the computer program is executed by a processor, the method provided in the embodiments is implemented.

[0191] The embodiment of the present application further provides a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, the method provided in the above embodiment is implemented.

[0192] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0193] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0194] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0195] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0196] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0197] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0198] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An image restoration method, characterized in that: include: A first region image is intercepted from an image to be processed, wherein the image to be processed includes a target object, and the first region image includes an element to be repaired on the target object that needs to be repaired; Performing three-dimensional reconstruction on the element to be repaired based on the first region image to obtain a target reconstruction model, wherein the target reconstruction model is used to represent a three-dimensional model corresponding to the element to be repaired when the structure is correct; Performing image segmentation on the element to be repaired based on the first region image to obtain a segmentation mask, wherein the segmentation mask is used to characterize a pixel-level region of the element to be repaired in the first region image; Redrawing the first region image based on the segmentation mask and the target reconstruction model to obtain a second region image; The second region image and the image to be processed are fused to obtain a target image.

2. The method according to claim 1, characterized in that: The step of redrawing the first region image based on the segmentation mask and the target reconstruction model to obtain a second region image includes: Based on the target reconstruction model, a depth image and an edge image corresponding to the element to be repaired are acquired, wherein the depth image and the edge image are used to characterize the correct structure of the element to be repaired; Binarizing the depth image to obtain a reconstructed region mask; Merging the reconstruction region mask and the segmentation mask to obtain a repair region mask, wherein the repair region mask is used to represent the pixel-level region of the element to be repaired in the first region image after repair; Based on the depth image, the edge image and the repair area mask, the first area image is redrawn by an image repair model to obtain the second area image.

3. The method according to claim 2, characterized in that Based on the target reconstruction model, a depth image and an edge image corresponding to the element to be repaired are obtained, including: Aligning the target reconstruction model with the element to be repaired to obtain a reconstructed alignment model; Extracting a depth value from the reconstructed alignment model to obtain the depth image; Perform edge detection on the depth image to obtain the edge image.

4. The method according to claim 3, characterized in that Aligning the target reconstruction model with the element to be repaired to obtain a reconstructed alignment model, including: Identifying the segmentation mask to obtain an element observation angle, wherein the element observation angle is used to represent a visual angle presented when observing the element to be repaired in a scene corresponding to the segmentation mask; The target reconstruction model and the element to be repaired are aligned based on the element observation angle to obtain the reconstructed alignment model.

5. The method according to claim 2, characterized in that: The image restoration model at least includes: an image restoration module and an image diffusion module. Based on the depth image, the edge image and the restoration area mask, the first area image is redrawn by the image restoration model to obtain the second area image, including: Based on the depth image and the edge image, the structure of the element to be repaired is drawn in the repair area mask by the image repair module to obtain an initial drawn image; Extracting features of the element to be repaired in the first region image to obtain at least one element feature; Based on the element features, the initial drawn image is adjusted by the image diffusion module to obtain the second region image.

6. The method according to claim 5, characterized in that The element features at least include: texture features, color features and style features. Based on the element features, the initial drawn image is adjusted by the image diffusion module to obtain the second region image, including: Adjusting the texture of the initial drawn image based on the texture feature to obtain a first adjusted image; Adjusting the color of the initial drawn image based on the color feature to obtain a second adjusted image; adjusting the style of the initial drawn image based on the style feature to obtain a third adjusted image; The first adjusted image, the second adjusted image and the third adjusted image are fused to obtain the second region image.

7. The method according to claim 1, characterized in that The three-dimensional reconstruction of the element to be repaired based on the first region image to obtain a target reconstruction model includes: Acquire a model reconstruction tool and three-dimensional model parameters that match the element type of the element to be repaired; Extracting feature points of the element to be repaired to obtain element feature points; Based on the element feature points, using the model reconstruction tool to perform three-dimensional modeling on the element to be repaired to obtain an initial reconstruction model; The initial reconstruction model is adjusted based on the three-dimensional reconstruction parameters to obtain the target reconstruction model.

8. The method according to claim 1, characterized in that The method of intercepting a first region image from the image to be processed comprises: Identify the image to be processed and determine an initial bounding box corresponding to the boundary of the element to be repaired; Expanding the bounding box according to a preset graphic to obtain a target bounding box; Intercepting the image to be processed according to the target bounding box to obtain an initial region image; The initial region image is adjusted according to a preset resolution to obtain the first region image.

9. A method for restoring a character image, characterized in that: include: In response to detecting a change in the person image, performing abnormality detection on the person image to obtain a detection result, wherein the detection result is used to indicate whether different person parts in the person image are normal; In response to detecting that a target part among the parts of the person is abnormal, intercepting a first area image where the target part is located from the person image; Based on the first region image, the target part is three-dimensionally reconstructed to obtain a part reconstruction model, and the first region image is segmented based on the target part to obtain a part segmentation mask, wherein the part reconstruction model is used to represent the three-dimensional model corresponding to the target part when the structure is correct, and the part segmentation mask is used to represent the pixel-level area of ​​the target part in the first region image; Redrawing the first region image based on the part segmentation mask and the part reconstruction model to obtain a second region image; The second region image and the person image are fused to obtain a target repaired image.

10. The method according to claim 9, characterized in that The step of redrawing the first region image based on the part segmentation mask and the part reconstruction model to obtain a second region image includes: Based on the part reconstruction model, a depth image and an edge image corresponding to the target part are acquired, wherein the depth image and the edge image are used to characterize the correct structure of the target part; Binarizing the depth image to obtain a reconstructed region mask; Merging the reconstruction region mask and the segmentation mask to obtain a repair region mask, wherein the repair region mask is used to represent a pixel-level region of the repaired target part in the first region image; Based on the depth image, the edge image and the repair area mask, the first area image is redrawn by an image repair model to obtain the second area image.

11. The method according to claim 10, characterized in that The image restoration model at least includes: an image restoration module and an image diffusion module. Based on the depth image, the edge image and the restoration area mask, the first area image is redrawn by the image restoration model to obtain the second area image, including: Based on the depth image and the edge image, the structure of the target part is drawn in the repair area mask by the image repair module to obtain an initial drawn image; Extracting features of the target part in the first region image to obtain at least one part feature; Based on the part features, the initial drawn image is adjusted by the image diffusion module to obtain the second region image.

12. An image restoration method, characterized in that: include: In response to an input instruction acting on an operation interface, an image to be processed is displayed on the operation interface, wherein the image to be processed includes a target object, and there are elements to be repaired on the target object that need to be repaired; In response to an image restoration instruction applied on the operation interface, a target image is displayed on the operation interface, wherein the target image is used to represent an image obtained by fusing a second region image and an image to be processed, the second region image is used to represent an image obtained by redrawing the first region image based on a segmentation mask and a target reconstruction model, the first region image includes the element to be restored, the segmentation mask is used to represent an image mask obtained by performing image segmentation on the element to be restored based on the first region image, and the target reconstruction model is used to represent a three-dimensional model corresponding to the element to be restored when the structure is correct.

13. An image restoration method, characterized in that: include: Acquire the image to be processed by calling a first interface, wherein the first interface includes a first parameter, and a parameter value of the first parameter includes the image to be processed; A first region image is intercepted from the image to be processed, wherein the image to be processed includes a target object, and the first region image includes an element to be repaired on the target object that needs to be repaired; Performing three-dimensional reconstruction on the element to be repaired based on the first region image to obtain a target reconstruction model, wherein the target reconstruction model is used to represent a three-dimensional model corresponding to the element to be repaired when the structure is correct; Performing image segmentation on the element to be repaired based on the first region image to obtain a segmentation mask, wherein the segmentation mask is used to characterize a pixel-level region of the element to be repaired in the first region image; Redrawing the first region image based on the segmentation mask and the target reconstruction model to obtain a second region image; fusing the second region image and the image to be processed to obtain a target image; The target image is output by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter includes the target image.

14. An electronic device, characterized in that: include: A memory storing an executable program; A processor, configured to run the program, wherein the program executes the method according to any one of claims 1 to 13 when running.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is run, the device where the storage medium is located is controlled to execute the method according to any one of claims 1 to 13.

16. A computer program product, characterized in that It comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 13.

Images