Mask image generation method, device, electronic device and computer readable medium

Abstract

The embodiments of the present disclosure disclose a mask image generation method, device, electronic device and computer-readable medium. A specific implementation of the method includes: extracting edge information from a damaged image to obtain an edge image of the damaged image; based on the damaged image and the edge image, extracting features of the damaged image to obtain a feature image of the damaged image; transforming the edge image to obtain a reduced edge image that matches the size of the feature image; based on the feature image and the reduced edge image, generating a mask image of the damaged area in the damaged image. This implementation provides a method for generating a mask image of a damaged area in a damaged image. The mask image can locate the damaged area, providing a basis for subsequent repair of the damaged area in the damaged image.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a mask image generation method, device, electronic device, and computer-readable medium. Background Art

[0002] With the development of the Internet and the popularization of artificial intelligence technology with deep learning as the core, technologies such as computer vision have been applied to various fields of people's lives. Image restoration technology has become an important research topic in the field of computer vision. Image restoration technology repairs the damaged area to obtain a repaired image. However, image restoration technology cannot directly determine the damaged area in the damaged image to be repaired from the damaged image. In order to complete the repair of the damaged image, there is a need to determine the damaged area in the damaged image in advance. Summary of the invention

[0003] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0004] Some embodiments of the present disclosure propose a mask image generation method, an apparatus, an electronic device, and a computer-readable medium to solve the technical problems mentioned in the above background technology section.

[0005] In a first aspect, some embodiments of the present disclosure provide a mask image generation method, the method comprising: extracting edge information in a damaged image to obtain an edge image of the damaged image; based on the damaged image and the edge image, performing feature extraction on the damaged image to obtain a feature image of the damaged image; transforming the edge image to obtain a reduced edge image that matches the size of the feature image; and generating a mask image of the damaged area in the damaged image based on the feature image and the reduced edge image.

[0006] In a second aspect, some embodiments of the present disclosure provide a mask image generating device, comprising: an edge extraction unit, configured to extract edge information from a damaged image to obtain an edge image of the damaged image; a feature extraction unit, configured to perform feature extraction on the damaged image based on the damaged image and the edge image to obtain a feature image of the damaged image; a transformation unit, configured to transform the edge image to obtain a reduced edge image that matches the size of the feature image; and a generation unit, configured to generate a mask image of the damaged area in the damaged image based on the feature image and the reduced edge image.

[0007] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any method as described in the first aspect.

[0008] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, any method in the first aspect is implemented.

[0009] The above-mentioned embodiments of the present disclosure have the following beneficial effects: the mask image generation method of some embodiments of the present disclosure generates a mask image of the damaged area to determine the damaged area in the damaged image. The mask image can determine the damaged area in the damaged image, providing a basis for subsequent image restoration. It is convenient for subsequent image restoration. Specifically, the inventor found that the reason why the related art cannot determine the damaged area in the damaged image is that the related art does not provide a method for determining the damaged area in the damaged image. Based on this, the mask image generation method of some embodiments of the present disclosure obtains the correlation between pixels by extracting a large amount of edge information in the damaged image. Then the main body information of the damaged image is obtained, so that the damaged area in the damaged image can be determined. Further, a mask image of the damaged area in the damaged image is generated. On this basis, the mask image of the damaged area determines the damaged area in the damaged image. Therefore, the edge information in the damaged image can be used to obtain the correlation between pixels. The mask image of the damaged area in the damaged image is generated by using the correlation between pixels. Then, the damaged area in the damaged image is determined. It provides a basis for repairing damaged areas in damaged images. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0011] Figure 1 is a schematic diagram of an application scenario of a mask image generation method according to some embodiments of the present disclosure;

[0012] Figure 2 is a flow chart of some embodiments of the mask image generation method according to the present disclosure;

[0013] Figure 3 are flowcharts of other embodiments of the mask image generation method according to the present disclosure;

[0014] Figure 4is a schematic structural diagram of some embodiments of the mask image generating device according to the present disclosure;

[0015] Figure 5 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0016] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0017] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0018] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0019] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0020] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0021] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0022] Figure 1 100 is a schematic diagram of an application scenario of a mask image generation method according to some embodiments of the present disclosure.

[0023] like Figure 1As shown, the computing device 101 extracts edge information of the damaged image 102 to obtain an edge image 103. As an example, the computing device 101 can use the Sobel edge detection algorithm to extract edge information in the damaged image. Afterwards, the damaged image 102 and the edge image 103 are combined to further perform feature extraction on the damaged image to obtain a feature image 104. As an example, the matrix of the three color channels of R, G, and B representing the damaged image 102 and the single channel matrix representing the edge image 103 can be firstly spliced ​​to obtain a spliced ​​image. Afterwards, the feature extraction algorithm is used to further perform feature extraction on the above spliced ​​image. As an example, the above feature extraction algorithm can be an LBP (Local Binary Pattern) algorithm. Afterwards, the computing device 101 transforms the edge image 103 to obtain a reduced edge image 105 that matches the size of the feature image 104. Finally, the computing device 101 generates a mask image 106 of the damaged area in the damaged image 102 based on the feature image 104 and the reduced edge image 105. As an example, the feature image 104 and the reduced edge image 105 may be channel-joined, and the mask image 106 may be generated using the joined images.

[0024] It should be noted that the computing device 101 can be hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device. When the computing device is embodied as software, it can be installed in the hardware devices listed above. It can be implemented as multiple software or software modules for providing distributed services, or as a single software or software module. No specific limitation is made here.

[0025] It should be understood that Figure 1 The number of computing devices in the embodiment is only illustrative. Any number of computing devices may be provided according to implementation requirements.

[0026] Continue to refer Figure 2 , shows a process 200 of some embodiments of the mask image generation method according to the present disclosure. The mask image generation method comprises the following steps:

[0027] Step 201: extract edge information from a damaged image to obtain an edge image of the damaged image.

[0028] In some embodiments, the damaged image may include but is not limited to at least one of the following: an image including a scratched area or a crease area and an image including a faded area. The damaged image may be obtained by scanning a paper image with scratches, creases or faded areas.

[0029] In some embodiments, the execution subject of the mask image generation method (eg Figure 1 The computing device shown in the figure can extract edge information from the damaged image by various edge detection algorithms to obtain an edge image of the damaged image. As an example, the execution subject can use a canny operator to extract edge information of the damaged image, or can use a sobel operator to extract edge information of the damaged image.

[0030] Step 202: Based on the damaged image and the edge image, feature extraction is performed on the damaged image to obtain a feature image of the damaged image.

[0031] In some embodiments, based on the damaged image and the edge image, the execution entity may perform feature extraction on the damaged image in various ways to obtain a feature image. As an example, the execution entity may grayscale the damaged image, and use an image normalization algorithm to convert the pixel values ​​of the pixels of the grayscale image to between 0 and 1. Afterwards, the normalized image is added to the pixel values ​​of the corresponding pixels of the edge image to obtain the added image. Afterwards, feature extraction is performed on the added image based on the feature extraction algorithm to obtain a feature image of the damaged image. As an example, the feature extraction algorithm may be a SIFT (Scale-Invariant Feature Transform) feature extraction algorithm. As another example, the feature extraction algorithm may be an LBP (Local Binary Pattern) feature extraction algorithm.

[0032] Step 203: transform the edge image to obtain a reduced edge image that matches the size of the feature image.

[0033] In some embodiments, as an example, the execution subject may input the edge image into a downsampling layer of a convolutional network to obtain a reduced edge image that matches the size of the feature image. As another example, the execution subject may transform the image based on equal-interval sampling to obtain a reduced edge image that matches the size of the feature image.

[0034] Step 204: Generate a mask image of the damaged area in the damaged image based on the feature image and the reduced edge image.

[0035] In some embodiments, the execution subject of the above method generates a mask image of the damaged area in the damaged image based on the feature image and the reduced edge image. The mask image divides the target area by setting the target area and the background area to different pixel values. For example, the mask image of the damaged image can be an image in which the pixel values ​​of the pixels in the damaged area are set to 1 and the pixel values ​​of other pixels are set to 0.

[0036] In some embodiments, as an example, the execution subject may grayscale the feature image. Afterwards, the grayscale feature image is fused with the reduced edge image. As an example, the grayscale feature image and the pixel values ​​of the corresponding pixels of the reduced edge image may be added. As another example, the grayscale feature image and the reduced edge image may be channel-merged. Finally, a mask image is generated based on the fused image. As an example, the fused image is input into a decoding network, such as a generator network of a GAN network.

[0037] The above-mentioned embodiments of the present disclosure have the following beneficial effects: the mask image generation method of some embodiments of the present disclosure generates a mask image of the damaged area to determine the damaged area in the damaged image. The mask image can determine the damaged area in the damaged image, providing a basis for subsequent image restoration. It is convenient for subsequent image restoration. Specifically, the inventor found that the reason why the related art cannot determine the damaged area in the damaged image is that the related art does not provide a method for determining the damaged area in the damaged image. Based on this, the mask image generation method of some embodiments of the present disclosure obtains the correlation between pixels by extracting a large amount of edge information in the damaged image. Then the main body information of the damaged image is obtained, so that the damaged area in the damaged image can be determined. Further, a mask image of the damaged area in the damaged image is generated. On this basis, the mask image of the damaged area determines the damaged area in the damaged image. Therefore, the edge information in the damaged image can be used to obtain the correlation between pixels. The mask image of the damaged area in the damaged image is generated by using the correlation between pixels. Then, the damaged area in the damaged image is determined. This provides a basis for repairing damaged areas in damaged images.

[0038] Further references Figure 3 , which shows a process 300 of another embodiment of a mask image generation method. The process 300 of the mask image generation method includes the following steps:

[0039] Step 301: extract edge information from a damaged image to obtain an edge image of the damaged image.

[0040] Step 302: Input the damaged image and the edge image into a feature extraction network in a pre-trained mask generation model to obtain a feature image of the damaged image.

[0041] In some embodiments, the mask generation model includes a feature extraction network and an image generation network. As an example, the mask generation model can be a Unet network or a FCN network.

[0042] In some embodiments, the execution subject inputs the damaged image and the edge image into a feature extraction network in a pre-trained mask generation model to obtain a feature image of the damaged image. As an example, the execution subject performs convolution and downsampling operations on the damaged image and the edge image through the feature extraction network to obtain the feature image of the damaged image.

[0043] This process can provide richer texture information for the feature extraction network by inputting the edge information of the damaged image, thereby obtaining the main information of the damaged image. At the same time, it can also enable the feature extraction network to extract the correlation information between pixels.

[0044] Step 303: transform the edge image to obtain a reduced edge image that matches the size of the feature image.

[0045] In some embodiments, the specific implementation of steps 301 and 303 and the technical effects brought about can be referred to Figure 2 The corresponding steps 201 and 203 in the embodiments are not described in detail here.

[0046] Step 304 , input the feature image and the reduced edge image into the image generation network in the pre-trained mask generation model to obtain a mask image of the damaged area in the damaged image.

[0047] In some optional implementations of some embodiments, the image generation network includes a convolution layer and an upsampling layer, and step 304 can be implemented by the following steps:

[0048] Step 3041, input the feature image and the reduced edge image into a convolutional layer to obtain a first type of feature image.

[0049] Step 3042: input the first type feature image into at least one upsampling layer to obtain a mask image.

[0050] Optionally, the first type of feature image is input into at least one upsampling layer, including: for each upsampling layer in the at least one upsampling layer, the first type of feature image output by the network before upsampling and the edge image corresponding to the upsampling layer are input into the upsampling layer to obtain a second type of feature image, wherein the edge image corresponding to the upsampling layer is obtained by transforming the edge image to the input size of the upsampling layer.

[0051] from Figure 3 It can be seen that Figure 2 Compared with the description of some corresponding embodiments, Figure 3The process 300 of the mask image generation method in some corresponding embodiments highlights the process of generating a mask image using a pre-trained mask generation model. The process first uses the feature extraction network of the mask generation model to extract image features to obtain a feature image. Afterwards, the feature information of the feature image and the reduced edge image is input into the image generation network to generate a mask image of the damaged area. By inputting the reduced edge image into the image generation network, the image generation network is provided with richer content information of the damaged area in the damaged image. It is convenient for the image generation network to extract the correlation between the pixels of the damaged image and to generate a mask image of the damaged area along the edge information.

[0052] Further references Figure 4 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a mask image generating device. These device embodiments are similar to Figure 2 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0053] like Figure 4 As shown, the mask image generating device 400 of some embodiments includes: an edge extraction unit 401, a feature extraction unit 402, a transformation unit 403 and a generation unit 404. The edge extraction unit 401 is configured to extract edge information in the damaged image to obtain an edge image of the damaged image; the feature extraction unit 402 is configured to perform feature extraction on the damaged image based on the damaged image and the edge image to obtain a feature image of the damaged image; the transformation unit 403 is configured to transform the edge image to obtain a reduced edge image matching the size of the feature image; the generation unit 404 is configured to generate a mask image of the damaged area in the damaged image based on the feature image and the reduced edge image.

[0054] In an optional implementation of some embodiments, the feature extraction unit 402 of the mask image generating device 400 is further configured to: input the damaged image and the edge image into a feature extraction network in a pre-trained mask generation model to obtain a feature image of the damaged image, wherein the mask generation model includes a feature extraction network and an image generation network.

[0055] In an optional implementation of some embodiments, the generation unit 404 of the mask image generation device 400 is further configured to: input the feature image and the reduced edge image into an image generation network in a pre-trained mask generation model to obtain a mask image of the damaged area in the damaged image.

[0056] In an optional implementation of some embodiments, the image generation network of the mask image generation device 400 includes a convolution layer and an upsampling layer; and the generation unit 404 is further configured to: input the feature image and the reduced edge image into the convolution layer to obtain a first type of feature image; input the first type of feature image into at least one upsampling layer to obtain a mask image.

[0057] In an optional implementation of some embodiments, the generation unit 404 of the mask image generation device 400 is further configured to: for each upsampling layer in at least one upsampling layer, input the first type of feature image output by the network before upsampling and the edge image corresponding to the upsampling layer into the upsampling layer to obtain a second type of feature image, wherein the edge image corresponding to the upsampling layer is obtained by transforming the edge image to the input size of the upsampling layer.

[0058] It is understood that the units described in the device 400 are similar to those described in the reference Figure 2 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 400 and the units included therein, and will not be described in detail here.

[0059] Reference below Figure 5 , which shows an electronic device (eg, Figure 1 The electronic devices in some embodiments of the present disclosure may include but are not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0060] like Figure 5 As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0061] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 5 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0062] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0063] It should be noted that the computer-readable medium in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0064] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0065] The computer-readable medium may be included in the computing device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: extracts edge information from the damaged image to obtain an edge image of the damaged image; extracts features of the damaged image based on the damaged image and the edge image to obtain a feature image of the damaged image; transforms the edge image to obtain a reduced edge image that matches the size of the feature image; generates a mask image of the damaged area in the damaged image based on the feature image and the reduced edge image.

[0066] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0067] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0068] The units described in some embodiments of the present disclosure may be implemented by software or hardware. The described units may also be provided in a processor, for example, may be described as: a processor including an edge extraction unit, a feature extraction unit, a transformation unit, and a generation unit. The names of these units do not, in some cases, constitute limitations on the units themselves, for example, the edge extraction unit may also be described as "a unit for extracting edge information from a damaged image and obtaining an edge image of the damaged image".

[0069] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0070] According to one or more embodiments of the present disclosure, a mask image generation method is provided, including: extracting edge information in a damaged image to obtain an edge image of the damaged image; based on the damaged image and the edge image, extracting features of the damaged image to obtain a feature image of the damaged image; transforming the edge image to obtain a reduced edge image that matches the size of the feature image; and generating a mask image of a damaged area in the damaged image based on the feature image and the reduced edge image.

[0071] According to one or more embodiments of the present disclosure, the above-mentioned feature extraction of the damaged image based on the damaged image and the edge image to obtain the feature image of the damaged image includes: inputting the damaged image and the edge image into a feature extraction network in a pre-trained mask generation model to obtain the feature image of the damaged image, wherein the mask generation model includes a feature extraction network and an image generation network.

[0072] According to one or more embodiments of the present disclosure, the above-mentioned generation of a mask image of a damaged area in a damaged image based on a feature image and a reduced edge image includes: inputting the feature image and the reduced edge image into an image generation network in a pre-trained mask generation model to obtain a mask image of the damaged area in the damaged image.

[0073] According to one or more embodiments of the present disclosure, the above-mentioned image generation network includes a convolution layer and an upsampling layer; and the feature image and the reduced edge image are input into the image generation network in the pre-trained mask generation model to obtain a mask image of the damaged area in the damaged image, including: inputting the feature image and the reduced edge image into the convolution layer to obtain a first type of feature image; inputting the first type of feature image into at least one upsampling layer to obtain a mask image.

[0074] According to one or more embodiments of the present disclosure, the above-mentioned inputting the first type of feature image into at least one upsampling layer includes: for each upsampling layer in the at least one upsampling layer, inputting the first type of feature image output by the network before upsampling and the edge image corresponding to the upsampling layer into the upsampling layer to obtain a second type of feature image, wherein the edge image corresponding to the upsampling layer is obtained by transforming the edge image to the input size of the upsampling layer.

[0075] According to one or more embodiments of the present disclosure, a mask image generating device is provided, comprising: an edge extraction unit, configured to extract edge information in a damaged image to obtain an edge image of the damaged image; a feature extraction unit, configured to perform feature extraction on the damaged image based on the damaged image and the edge image to obtain a feature image of the damaged image; a transformation unit, configured to transform the edge image to obtain a reduced edge image matching the size of the feature image; and a generation unit, configured to generate a mask image of a damaged area in the damaged image based on the feature image and the reduced edge image.

[0076] According to one or more embodiments of the present disclosure, the feature extraction unit of the mask image generating device is further configured to: input the damaged image and the edge image into a feature extraction network in a pre-trained mask generation model to obtain a feature image of the damaged image, wherein the mask generation model includes a feature extraction network and an image generation network.

[0077] According to one or more embodiments of the present disclosure, the generation unit of the mask image generation device is further configured to: input the feature image and the reduced edge image into the image generation network in the pre-trained mask generation model to obtain the mask image of the damaged area in the damaged image.

[0078] According to one or more embodiments of the present disclosure, the image generation network of the mask image generation device includes a convolution layer and an upsampling layer; and the generation unit is further configured to: input the feature image and the reduced edge image into the convolution layer to obtain a first type of feature image; input the first type of feature image into at least one upsampling layer to obtain a mask image.

[0079] According to one or more embodiments of the present disclosure, the generation unit of the mask image generating device is further configured to: for each upsampling layer in at least one upsampling layer, input the first type of feature image output by the network before upsampling and the edge image corresponding to the upsampling layer into the upsampling layer to obtain a second type of feature image, wherein the edge image corresponding to the upsampling layer is obtained by transforming the edge image to the input size of the upsampling layer.

[0080] According to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any of the above methods.

[0081] According to one or more embodiments of the present disclosure, a computer-readable medium is provided, on which a computer program is stored, wherein the program implements any of the above methods when executed by a processor.

[0082] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A mask image generation method, comprising: Extracting edge information from the damaged image to obtain an edge image of the damaged image; Based on the damaged image and the edge image, feature extraction is performed on the damaged image to obtain a feature image of the damaged image; comprising: Inputting the damaged image and the edge image into a feature extraction network in a pre-trained mask generation model to obtain a feature image of the damaged image, wherein the mask generation model includes the feature extraction network and an image generation network; wherein the image generation network includes a convolutional layer and at least one upsampling layer; Transforming the edge image to obtain a reduced edge image matching the size of the feature image; Inputting the feature image and the reduced edge image into the convolution layer to obtain a first type of feature image; The first type feature image is input into the at least one upsampling layer to obtain the mask image.

2. The method according to claim 1, wherein: The step of inputting the first type of feature image into the at least one upsampling layer comprises: For each upsampling layer of the at least one upsampling layer, a first type of feature image output by the network before upsampling and an edge image corresponding to the upsampling layer are input into the upsampling layer to obtain a second type of feature image, wherein the edge image corresponding to the upsampling layer is obtained by transforming the edge image to an input size of the upsampling layer.

3. A mask image generating device, comprising: an edge extraction unit, configured to extract edge information from the damaged image to obtain an edge image of the damaged image; A feature extraction unit is configured to extract features of the damaged image based on the damaged image and the edge image to obtain a feature image of the damaged image; comprising: Inputting the damaged image and the edge image into a feature extraction network in a pre-trained mask generation model to obtain a feature image of the damaged image, wherein the mask generation model includes the feature extraction network and an image generation network; wherein the image generation network includes a convolution layer and an upsampling layer; a transform unit configured to transform the edge image to obtain a reduced edge image matching the size of the feature image; The generating unit is configured to input the feature image and the reduced edge image into the convolution layer to obtain a first type of feature image; and input the first type of feature image into the at least one upsampling layer to obtain the mask image.

4. The device according to claim 3, wherein: The generating unit is further configured to: For each upsampling layer of the at least one upsampling layer, a first type of feature image output by the network before upsampling and an edge image corresponding to the upsampling layer are input into the upsampling layer to obtain a second type of feature image, wherein the edge image corresponding to the upsampling layer is obtained by transforming the edge image to an input size of the upsampling layer.

5. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1-2.

6. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 2 is implemented.

Images