An image inpainting method, device, equipment and computer readable storage medium

Abstract

The application provides an image repairing method, device and equipment and a computer readable storage medium. The method is applied to an image repairing model, the image repairing model comprises a feature encoding network, a feature repairing network and a feature decoding network, and the method comprises the following steps: feature encoding is performed on an image to be repaired containing mark information by using the feature encoding network, and corresponding encoded features are obtained; the mark information is used for indicating a region to be repaired of the image to be repaired; position transformation is performed on the encoded features by using the feature repairing network, position transformation features are obtained, feature repairing is performed on the encoded features based on the position transformation features, and corresponding repaired features are obtained; and feature decoding is performed on the repaired features by using the feature decoding network, and corresponding repaired images are obtained. According to the application, the efficiency of image repairing can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the image processing technology of artificial intelligence, and in particular to an image repairing method and device, equipment and a computer readable storage medium. BACKGROUND

[0002] Artificial intelligence (AI) is a comprehensive technology of computer science, which makes machines have the functions of perception, reasoning and decision-making by studying the design principles and implementation methods of various intelligent machines. Artificial intelligence technology is a comprehensive discipline, which involves a wide range of fields, such as natural language processing technology and machine learning / deep learning. With the development of technology, artificial intelligence technology will be applied in more and more fields and play an increasingly important role.

[0003] Image repairing is one of the important applications in the field of artificial intelligence. Image repairing refers to the process of repairing and reconstructing the missing or damaged to-be-repaired region in an image or video. Most mainstream image repairing algorithms are based on convolutional neural networks to repair the to-be-repaired region. However, in actual applications, the to-be-repaired region may be relatively large. In order to achieve a good repairing effect, related technologies usually use multiple independent image repairing models to perform multi-stage image repairing, which consumes a long time and has low image repairing efficiency. SUMMARY

[0004] The embodiments of the present application provide an image repairing method, device, equipment and computer readable storage medium, which can improve the efficiency of image repairing.

[0005] The technical solutions of the embodiments of the present application are as follows:

[0006] The embodiments of the present application provide an image repairing method applied to an image repairing model, wherein the image repairing model comprises a feature encoding network, a feature repairing network and a feature decoding network, and the method comprises the following steps:

[0007] The feature encoding network is used to perform feature encoding on a to-be-repaired image containing mark information, to obtain corresponding encoded features;

[0008] The mark information is used to indicate a to-be-repaired region of the to-be-repaired image.

[0009] The feature repairing network is used to perform position transformation on the encoded features, to obtain position transformation features, and perform feature repairing on the encoded features based on the position transformation features, to obtain corresponding repaired features;

[0010] The feature decoding network is used to perform feature decoding on the repaired features, to obtain a corresponding repaired image.

[0011] The embodiment of the present application provides a kind of image repairing device, applied to image repairing model, the image repairing model includes feature encoding network, feature repairing network and feature decoding network, the device includes:

[0012] Feature encoding module, for carrying out feature encoding to the to-be-repaired image containing mark information by the feature encoding network, obtains corresponding encoding feature;

[0013] Wherein, the mark information is used to indicate the to-be-repaired area of the to-be-repaired image;

[0014] Feature repairing module, for carrying out position transformation to the encoding feature by the feature repairing network, obtains position transformation feature, and carries out feature repairing to the encoding feature based on the position transformation feature, obtains corresponding repair feature;

[0015] Feature decoding module, for carrying out feature decoding to the repair feature by the feature decoding network, obtains corresponding repair image.

[0016] In the above scheme, the feature repairing module includes position transformation unit, and the position transformation unit is used to carry out feature segmentation to the encoding feature according to the segmentation factor of pre-set size, obtains the target number of sub-encoding features;

[0017] The target number of sub-encoding features is rearranged in position, and position transformation feature is obtained.

[0018] In the above scheme, the position transformation unit is used to obtain the feature map corresponding to the encoding feature of to-be-repaired image;

[0019] According to the segmentation factor of pre-set size, the feature map is segmented, and the target number of sub-feature maps is obtained, and the sub-feature map is determined as the feature map corresponding to the sub-encoding feature.

[0020] In the above scheme, the feature repairing module further includes matrix construction unit and feature updating unit, and the matrix construction unit is used to construct the adjacency matrix corresponding to the to-be-repaired image based on the position transformation feature;

[0021] The feature updating unit is used to carry out feature repairing to the encoding feature corresponding to the to-be-repaired area based on the adjacency matrix, and obtains corresponding repair feature.

[0022] In the above scheme, the matrix construction unit is also used

[0023] For each position in the position transformation feature, the similarity value between the position transformation feature corresponding to each position and the position transformation feature corresponding to other positions is obtained respectively;

[0024] An adjacency matrix corresponding to the to-be-repaired image is constructed, taking each position as a node and a connection relationship between nodes as an edge.

[0025] The matrix element of the adjacency matrix is a corresponding similarity value.

[0026] In the above scheme, the feature repairing module further includes a position inverse transformation unit,

[0027] The feature updating unit is further configured to perform normalization processing on the matrix elements in the adjacency matrix to obtain a normalized adjacency matrix.

[0028] The normalized adjacency matrix is dot-multiplied with the position transformation feature to obtain an updated position transformation feature.

[0029] The position inverse transformation unit is configured to perform position inverse transformation on the updated position transformation feature to obtain a corresponding repairing feature.

[0030] In the above scheme, the feature repairing module further includes a feature mapping unit,

[0031] The feature mapping unit is configured to perform feature mapping on the updated position transformation feature to obtain a mapping feature after the updated position transformation feature is obtained.

[0032] Correspondingly, the position inverse transformation unit is further configured to perform position inverse transformation on the mapping feature to obtain a corresponding repairing feature.

[0033] In the above scheme, after the updated position transformation feature is obtained, the feature mapping unit is further configured to perform feature mapping on the updated position transformation feature to obtain a first mapping feature.

[0034] The feature updating unit is further configured to dot-multiply the first mapping feature with the normalized adjacency matrix to obtain an updated mapping feature.

[0035] The feature mapping unit is further configured to perform feature mapping on the updated mapping feature to obtain a second mapping feature.

[0036] The position inverse transformation unit is further configured to perform position inverse transformation on the second mapping feature to obtain a corresponding repairing feature.

[0037] In the above scheme, the feature repairing module includes a position transformation unit, a feature sampling unit, a feature updating unit, and a position inverse transformation unit,

[0038] The feature sampling unit is configured to sample position transformation features corresponding to other regions except the region to be repaired in the position transformation features, to obtain sampled position transformation features.

[0039] The feature updating unit is configured to update position transformation features corresponding to the region to be repaired in the position transformation features by using the sampled position transformation features.

[0040] The position inverse transformation unit is configured to perform position inverse transformation on the updated position transformation features, to obtain corresponding repair features.

[0041] In the foregoing scheme, the feature encoding network is composed of at least two cascaded sub-encoding networks, the feature encoding module is configured to perform encoding processing on the image to be repaired containing the mark information by using a first sub-encoding network in the at least two sub-encoding networks, to obtain corresponding sub-encoding features.

[0042] The sub-encoding features are input into a subsequent cascaded sub-encoding network, so that the sub-encoding features are encoded by using the subsequent cascaded sub-encoding network to obtain target encoding features, and the target encoding features are output by using a last sub-encoding network, as the encoding features corresponding to the image to be repaired.

[0043] In the foregoing scheme, the feature decoding network includes sub-decoding networks with the same number as the sub-encoding networks, and the feature decoding module is configured to perform decoding processing on the repair features by using a first sub-decoding network in the at least two sub-decoding networks, to obtain corresponding initial repair images.

[0044] The repair features corresponding to the initial repair images are output to a subsequent cascaded sub-decoding network, so that the repair features corresponding to the initial repair images are decoded by using the subsequent cascaded sub-decoding network to obtain target repair images, and the target repair images are output by using a last sub-decoding network, as the repair images corresponding to the image to be repaired.

[0045] In the foregoing scheme, the device further includes a model training module, the model training module is configured to

[0046] The feature encoding network is used to extract features from the image to be repaired containing the mark information in the image sample, to obtain corresponding encoding features.

[0047] The image sample includes an original image and an image to be repaired corresponding to the original image, and the mark information is used to indicate a region to be repaired in the image to be repaired.

[0048] The encoded features are transformed using the feature repair network to obtain position-transformed features, and the encoded features are repaired based on the position-transformed features to obtain corresponding repaired features.

[0049] The feature decoding network is used to perform feature decoding on the repair features to obtain the corresponding repaired image;

[0050] The differences between the restored image and the original image are obtained, and the model parameters of the image restoration model are updated based on the obtained differences.

[0051] This application provides an electronic device, including:

[0052] Memory, used to store executable instructions;

[0053] The processor, when executing executable instructions stored in the memory, implements the image restoration method provided in the embodiments of this application.

[0054] This application provides a computer-readable storage medium storing executable instructions for inducing a processor to execute and implement the image restoration method provided in this application.

[0055] The embodiments of this application have the following beneficial effects:

[0056] By performing position transformation on the coded features corresponding to the image to be repaired, position transformation features are obtained. Based on the position transformation features, feature repair is performed on the coded features of the image to be repaired. Here, the coded features correspond to the original features of the image to be repaired. By performing position transformation on the coded features, the positions of the original features are rearranged, so that when the coded features at a certain position are updated, multiple coded features corresponding to that position will be updated simultaneously, thus improving the efficiency of image repair. Attached Figure Description

[0057] Figure 1 A schematic diagram of an optional architecture for an image restoration system provided in an embodiment of this application;

[0058] Figure 2 An optional structural schematic diagram of the electronic device provided in the embodiments of this application;

[0059] Figure 3 A schematic diagram of an optional architecture for the image restoration model provided in an embodiment of this application;

[0060] Figures 4A-4D A schematic diagram of an optional structural composition of the feature repair network provided in the embodiments of this application;

[0061] Figure 5An optional flowchart of the image repairing method provided by the embodiment of the present application is shown in FIG. 1.

[0062] Figure 6 An optional position conversion diagram provided by the embodiment of the present application is shown in FIG. 2.

[0063] Figure 7 An optional flowchart of the method for determining the repairing feature provided by the embodiment of the present application is shown in FIG. 3.

[0064] Figure 8 An optional flowchart of the method for constructing the adjacency matrix provided by the embodiment of the present application is shown in FIG. 4.

[0065] Figure 9 An optional flowchart of the method for determining the repairing feature provided by the embodiment of the present application is shown in FIG. 5.

[0066] Figure 10 An optional flowchart of the method for determining the repairing feature provided by the embodiment of the present application is shown in FIG. 6.

[0067] Figure 11 An optional data flow diagram of the feature repairing network provided by the embodiment of the present application is shown in FIG. 7.

[0068] Figure 12 An optional flowchart of the image repairing method provided by the embodiment of the present application is shown in FIG. 8.

[0069] Figure 13 An optional structure diagram of the image repairing device provided by the embodiment of the present application is shown in FIG. 9. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application will be described in further detail below with reference to the drawings, and the described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by those skilled in the art without making creative efforts belong to the protection scope of the present application.

[0071] In the following description, “some embodiments” are related to a subset of all possible embodiments, but it can be understood that “some embodiments” can be the same subset or different subset of all possible embodiments, and can be combined with each other without conflict.

[0072] In the following description, the terms “first\second…” are only to distinguish similar objects, and do not represent a specific order of the objects, and it can be understood that “first\second…” can be interchanged with a specific order or sequence as allowed, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0073] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to be limiting of this application.

[0074] The inventors have found that, when the to-be-repaired region of the to-be-repaired image is relatively large, in order to achieve a better repair effect, the related art usually uses multiple independent repair models to perform image repair, for example, two stages are used to perform image repair, specifically, in the first stage, an image repair model is used to repair the to-be-repaired image to generate a relatively rough initial repair image, and then in the second stage, another image repair model independent of the image repair model in the first stage is used to repair the initial repair image obtained in the first stage to further generate a fine final repair image. In the repair process, a context attention method is generally used to find features related to the to-be-repaired region and perform repair, specifically, by obtaining a similarity value between the features corresponding to the to-be-repaired region and the features corresponding to other regions, the most similar features are found for repair, and generally, only one repair is performed, and then the repaired features are input into a subsequent network for reconstruction. However, due to the above-mentioned two-stage image repair method, two repair models are needed to perform image repair, which consumes a long time; when the context attention repair is used, one repair cannot achieve a good repair effect.

[0075] In view of this, the embodiments of the present application provide an image repair method, device, equipment and computer readable storage medium, which improve the efficiency and effect of image repair.

[0076] The image repair method provided by the embodiments of the present application can be realized by a terminal or a server alone; or can be realized by a terminal and a server in cooperation, for example, the terminal alone bears the image repair method described below, or the terminal sends a to-be-repaired image to the server, and the server performs the image repair method on the received to-be-repaired image.

[0077] The electronic device for image repair provided by the embodiments of the present application can be various types of terminal devices or servers, wherein the server can be a stand-alone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services; the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server can be connected directly or indirectly through wired or wireless communication, which is not limited in the present application.

[0078] Taking a server as an example, for example, it can be a server cluster deployed in the cloud, and an artificial intelligence cloud service (AI as a Service, AIaaS) is opened to users. The AIaaS platform will split several common AI services and provide independent or packaged services in the cloud. This service mode is similar to an AI theme mall, and all users can access and use one or more artificial intelligence services provided by the AIaaS platform through an application programming interface.

[0079] For example, one of the artificial intelligence cloud services can be an image repair service, that is, the server in the cloud encapsulates the image repair program provided by the embodiment of the application. The user calls the image repair service in the cloud service through the terminal (running the client, such as the browser client, the video playing client, etc.) to make the server deployed in the cloud call the encapsulated image repair program to repair the image, thereby improving the efficiency and effect of image repair. For example, the date and watermark of the video image can be removed to greatly improve the processing speed; for the movie with a flawed picture, the image repair method is used to repair the flawed picture, thereby improving the speed and effect of online image repair.

[0080] Referring to Figure 1 , Figure 1 An optional architecture schematic diagram of the image repair system 100 provided by the embodiment of the application is shown, in order to realize an example application. The terminal (exemplarily showing the terminal 400-1 and the terminal 400-2) is connected to the server 200 through the network 300, the network 300 can be a wide area network or a local area network, or a combination of the two, and a wireless link is used to realize data transmission.

[0081] The terminal runs a client, such as a browser client, a video playing client, etc. The terminal is used to obtain a to-be-repaired image, and the to-be-repaired image contains marking information used to indicate a to-be-repaired region of the to-be-repaired image.

[0082] In some embodiments, an image repair plug-in can be implanted in the client running in the terminal, so as to realize an image repair method locally in the client. The image repair method is applied to an image repair model, the image repair model includes a feature encoding network, a feature repair network and a feature decoding network. The terminal encodes features of the to-be-repaired image through the feature encoding network to obtain corresponding encoded features. The encoded features are positionally transformed through the feature repair network to obtain positionally transformed features, and the encoded features are repaired based on the positionally transformed features to obtain corresponding repaired features. The repaired features are decoded through the feature decoding network to obtain a corresponding repaired image.

[0083] In some embodiments, after the terminal obtains the image to be repaired, the terminal calls an image repairing interface of the server 200, the server 200 encodes features of the image to be repaired through a feature encoding network to obtain corresponding encoded features, performs position transformation on the encoded features through a feature repairing network to obtain position-transformed features, and performs feature repairing on the encoded features based on the position-transformed features to obtain corresponding repaired features, and decodes the repaired features through a feature decoding network to obtain a corresponding repaired image and return the repaired image to the terminal for presentation.

[0084] Referring to Figure 2 , Figure 2 An optional structural schematic diagram of an electronic device 500 provided by the embodiments of the present application is shown in the figure. In actual application, the electronic device 500 can be a terminal or the server 200 in Figure 1 , and the electronic device is taken as the terminal shown in Figure 1 for example, to describe the electronic device implementing the image repairing method of the embodiments of the present application. Figure 2 The electronic device 500 shown in the figure includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together through a bus system 540. It can be understood that the bus system 540 is used to realize the connection and communication between the components. The bus system 540 includes a data bus, a power bus, a control bus, and a status signal bus in addition to the data bus. However, for the purpose of clear illustration, all the buses are marked as the bus system 540 in Figure 2 .

[0085] The processor 510 can be an integrated circuit chip with a signal processing capability, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0086] The user interface 530 includes one or more output devices 531 enabling presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, a mouse, a microphone, a touch screen display, a camera, other input buttons and controls.

[0087] The memory 550 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 550 optionally includes one or more storage devices physically located in proximity to the processor 510.

[0088] The memory 550 includes volatile memory or nonvolatile memory, and can include both volatile and nonvolatile memory. The nonvolatile memory can be read only memory (ROM), and the volatile memory can be random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.

[0089] In some embodiments, the memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, which are exemplarily illustrated below.

[0090] The operating system 551 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0091] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including Bluetooth, wireless compatibility certification (WiFi), and universal serial bus (USB), etc.

[0092] The presentation module 553 is used to enable the presentation of information via one or more output devices 531 associated with the user interface 530 (e.g., a display screen, a speaker, etc.), such as a user interface for operating peripheral devices and displaying content and information.

[0093] The input processing module 554 is used to detect and interpret one or more user inputs or interactions from one or more input devices 532.

[0094] In some embodiments, the image repairing apparatus provided by the embodiments of the present application can be implemented in a software manner, Figure 2 An image repairing apparatus 555 stored in the memory 550 is shown, which can be software in the form of programs and plug-ins, etc., including the following software modules: feature encoding module 5551, feature repairing module 5552, and feature decoding module 5553, which are logical, and thus can be combined or further split according to the implemented functions.

[0095] The functions of the various modules will be described below.

[0096] In some embodiments, the image inpainting apparatus provided by the embodiments of the present application can be implemented in a hardware manner. For example, the image inpainting apparatus provided by the embodiments of the present application can be a hardware decoding processor programmed to perform the image inpainting method provided by the embodiments of the present application. For example, the hardware decoding processor can be implemented by one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs) or other electronic elements.

[0097] Next, the image inpainting method provided by the embodiments of the present application is described. The image inpainting method provided by the embodiments of the present application is applied to an image inpainting model. Referring to FIG. 2, the image inpainting method provided by the embodiments of the present application includes the following steps. Figure 3 Figure 3 An optional architecture diagram of the image inpainting model provided by the embodiments of the present application is shown in FIG. 3. The image inpainting model includes a feature encoding network, a feature inpainting network and a feature decoding network. The image to be inpainted is input into the image inpainting model, and a series of processes are performed on the region to be inpainted by the feature encoding network, the feature inpainting network and the feature decoding network to obtain the inpainted image corresponding to the region to be inpainted. The obtained inpainted image corresponding to the region to be inpainted is fused with other regions of the image to be inpainted except the region to be inpainted to obtain the final inpainted image. Figure 3

[0098] In actual applications, the image inpainting model needs to be trained before the image to be inpainted is inpainted by the image inpainting model. In some embodiments, the image inpainting model can be trained in the following manner.

[0099] The feature encoding network is used to extract features from the image to be inpainted containing the mark information in the image sample to obtain the corresponding encoded features. The image sample includes an original image and an image to be inpainted corresponding to the original image, and the mark information is used to indicate the region to be inpainted of the image to be inpainted. The feature inpainting network is used to perform position transformation on the encoded features to obtain position transformation features, and perform feature inpainting on the encoded features based on the position transformation features to obtain corresponding inpainting features. The feature decoding network is used to perform feature decoding on the inpainting features to obtain the corresponding inpainted image. The difference between the inpainted image and the original image is obtained, and the model parameters of the image inpainting model are updated based on the obtained difference.

[0100] ​​The feature encoding network consists of at least two cascaded sub-encoding networks, the feature decoding network consists of at least two cascaded sub-decoding networks, and the feature inpainting network is mainly used to perform a series of processes such as position transformation, feature update, and inverse position transformation on the encoded features of the image to be repaired in the corresponding image samples obtained by the feature encoding network, so as to obtain the final repaired features.

[0101] In some embodiments, see Figure 4A , Figure 4A A schematic diagram of an optional structural composition of the feature repair network provided in the embodiments of this application is shown below. Figure 4A As shown, the feature inpainting network includes a position transformation layer, a feature update layer, and a position inverse transformation layer. In actual implementation, the position transformation layer transforms the encoded features of the image to be repaired in the image sample to obtain the corresponding position transformation features. The feature update layer updates the encoded features of the region to be repaired based on the position transformation features. The position inverse transformation layer performs a position inverse transformation on the updated position transformation features to restore the updated position transformation features to repair features with the same spatial size as the encoded features, so that subsequent processing can be performed based on the obtained repair features.

[0102] In other embodiments, see Figure 4B , Figure 4B A schematic diagram of an optional structural composition of the feature repair network provided in the embodiments of this application is shown below. Figure 4B As shown, the feature repair network includes: a position transformation layer, a feature update layer, a feature mapping layer, and a position inverse transformation layer. The feature mapping layer is used to enhance the high-frequency information in the repaired features. Therefore, after updating the position transformation features, the updated position transformation features are input into the feature mapping layer (such as multiple convolutional layers) for nonlinear mapping, and the obtained mapped features are input into the position inverse transformation layer for position inverse transformation. The nonlinear transformation is the activation function (such as the ReLU function).

[0103] In other embodiments, see Figure 4C , Figure 4C A schematic diagram of an optional structural composition of the feature repair network provided in the embodiments of this application is shown below. Figure 4C As shown, the feature inpainting network includes: a position transformation layer, N feature update layers, a feature mapping layer, and a position inverse transformation layer. This is because when the region to be repaired in the image is large, in order to achieve a good repair effect, the position transformation features are updated multiple times. Here, N is an integer greater than or equal to 2, and the value of N is positively correlated with the size of the region to be repaired. Then, the position transformation features after multiple updates are input into the feature mapping layer (such as multiple convolutional layers) for nonlinear mapping, and the obtained mapped features are input into the position inverse transformation layer for position inverse transformation.

[0104] In some embodiments, referring to Figure 4D , Figure 4D An optional structural composition diagram of the feature repairing network provided by the embodiments of the present application is shown in Figure 4D The feature repairing network includes a position transformation layer, N feature updating layers, N feature mapping layers, and a position inverse transformation layer, where N is an integer greater than or equal to 2, and the value of N is in a positive correlation with the size of the region to be repaired. After the position transformation feature updated each time is nonlinearly mapped by the feature mapping layer, the mapping feature obtained is subjected to feature updating again. The mapping feature obtained by the last feature updating and feature mapping is input to the position inverse transformation layer for position inverse transformation.

[0105] It should be noted that the above is only an exemplary structural composition of the feature repairing network provided by the embodiments of the present application. In actual applications, each feature processing layer can be extended and combined based on the above implementation principle, and the present application does not specifically limit the structural composition of the feature repairing network.

[0106] When the image sample is processed by the image repairing model to obtain the corresponding repaired image, the value of the loss function of the image repairing model can be determined according to the difference between the repaired image and the original image. When the value of the loss function reaches a preset threshold, the corresponding error signal is determined based on the value of the loss function of the image repairing model. The error signal is back-propagated in the image repairing model, and the model parameters of each network of the image repairing model are updated in the process of propagation.

[0107] Here, the back-propagation is described. The image sample to be trained is input to the input layer of the neural network model, passes through the hidden layer, and finally reaches the output layer and outputs the result, which is the forward propagation process of the neural network model. Since there is an error between the output result of the neural network model and the actual result, the error between the output result and the actual value is calculated, and the error is back-propagated from the output layer to the hidden layer until it propagates to the input layer. In the process of back-propagation, the value of the model parameter is adjusted according to the error. The above process is iterated continuously until convergence.

[0108] Based on the composition and training process of the above image repairing model, the image repairing method provided by the embodiments of the present application is described. Referring to Figure 5 , Figure 5 An optional flowchart of the image repairing method provided by the embodiments of the present application is shown in Figure 5 The image repairing method provided by the embodiments of the present application is described with reference to the steps shown in

[0109] Step 101: The terminal encodes the features of the image to be repaired containing the label information by the feature encoding network to obtain the corresponding encoded features.

[0110] The mark information is used for indicating a to-be-repaired region of the to-be-repaired image.

[0111] In some embodiments, the feature encoding network is composed of at least two sub-encoding networks in cascade, and the feature encoding of the to-be-repaired image containing the mark information through the feature encoding network can be realized by the following manner to obtain the corresponding encoded features:

[0112] The first sub-encoding network in the at least two sub-encoding networks is used for encoding processing of the to-be-repaired image containing the mark information to obtain corresponding sub-encoding features; the sub-encoding features are input into the subsequent cascaded sub-encoding network to obtain target encoding features through the subsequent cascaded sub-encoding network, and the target encoding features are output through the last sub-encoding network as the corresponding encoded features of the to-be-repaired image.

[0113] Here, the network depth of the feature encoding network can be multi-layered, and in actual implementation, the to-be-repaired image is input into the feature encoding network, and the final encoded features are obtained through the encoding processing of the multi-layer sub-encoding network, for example, the to-be-repaired image is encoded in the first layer (i.e., the first sub-encoding network) of the feature encoding network to obtain the feature map (i.e., the sub-encoding features) corresponding to the first layer, and the feature map is input into the subsequent network layer for encoding processing, such as in the i-th layer (i>=2, i.e., the i-th sub-encoding network) of the feature encoding network, the feature map of the (i-1)-th layer (i.e., the (i-1)-th sub-encoding network) of the feature encoding network is encoded to obtain the feature map corresponding to the i-th layer, and then the obtained feature map corresponding to the i-th layer is input into the (i+1)-th layer (i.e., the (i+1)-th sub-encoding network) of the feature encoding network for continuous encoding processing, and the above step is repeated until the final encoded features corresponding to the to-be-repaired image are obtained and output through the last layer (i.e., the last sub-encoding network) of the feature encoding network.

[0114] Step 102: position transformation of the encoded features through the feature repair network to obtain position transformation features, and feature repair of the encoded features based on the position transformation features to obtain corresponding repair features.

[0115] In some embodiments, the position transformation of the encoded features to obtain the position transformation features can be realized by the following manner:

[0116] The encoded features are feature segmented according to a preset size of a segmentation factor to obtain a target number of sub-encoding features; and the target number of sub-encoding features are position rearranged to obtain the position transformation features.

[0117] In some embodiments, the feature segmentation of the encoded features according to the preset size of the segmentation factor to obtain the target number of sub-encoding features can be realized by the following manner:

[0118] obtain a feature map corresponding to the encoding feature of the image to be repaired; perform segmentation on the feature map according to a preset size of a segmentation factor to obtain a target number of sub-feature maps, and determine the sub-feature maps as feature maps corresponding to sub-encoding features.

[0119] Here, the form of the encoding feature obtained by the feature encoding network is actually the form of a feature map, such as representing the encoding feature in the form of a matrix. The feature map corresponding to the image to be repaired is segmented according to a sub-matrix (i.e., a segmentation factor) of a certain size to obtain a plurality of sub-feature maps. Then, the plurality of obtained sub-feature maps are rearranged to obtain a new position transformation feature. In this way, after the encoding feature corresponding to the image to be repaired is segmented according to a preset size of a segmentation factor and then rearranged, the position of the original encoding feature is rearranged, and the new position transformation feature obtained by the rearrangement is smaller than the original encoding feature in terms of length and width, and the number of channels is increased.

[0120] The size of the segmentation factor can be determined according to actual conditions. For example, when the feature map of the encoding feature corresponding to the image to be repaired is large, a larger segmentation factor can be set to reduce the complexity of the calculation, or a smaller segmentation factor can be set to pack adjacent features together.

[0121] For example, referring to Figure 6 , Figure 6 The position transformation schematic diagram provided by the embodiments of the present application is shown in Figure 6 , assuming that the size of the encoding feature (feature map) corresponding to the image to be repaired obtained by the feature encoding network is 4x4, and the size of the segmentation factor is 2x2, then after the position of the original encoding feature is rearranged according to the segmentation factor, 4 sub-regions (i.e., sub-feature maps) of 2x2 are obtained. For each sub-region, the sub-encoding features corresponding to the 4 positions inside are arranged in a column according to a certain order, so that a 4x2x2 body is formed, and the position transformation feature of the feature map corresponding to the image to be repaired is obtained. For each position, the purpose of the position rearrangement is to pack the features of adjacent positions together.

[0122] In some embodiments, referring to Figure 7 , Figure 7 An optional flowchart of a method for determining a repair feature provided by the embodiments of the present application is shown in Figure 5 The step 102 of performing feature repair on the encoding feature based on the position transformation feature to obtain the corresponding repair feature is shown in Figure 7 , which can be implemented by steps 201-202 as shown:

[0123] Step 201: constructing an adjacency matrix corresponding to the image to be repaired based on the position transformation feature.

[0124] Step 202: based on the adjacency matrix, performing feature repairing on the coding features corresponding to the to-be-repaired region to obtain corresponding repaired features.

[0125] In some embodiments, referring to Figure 8 , Figure 8 An optional flowchart of a method for constructing an adjacency matrix provided by an embodiment of the present application is shown in Figure 7 Step 201 shown in the figure can be implemented by steps 2011-2012 as shown in Figure 8 :

[0126] Step 2011: for each position in the position transformation feature, respectively obtaining a similarity value between the position transformation feature corresponding to each position and the position transformation features corresponding to other positions;

[0127] Step 2012: constructing an adjacency matrix corresponding to the to-be-repaired image with each position as a node and the connection relationship between nodes as edges; wherein the matrix elements of the adjacency matrix are corresponding similarity values.

[0128] Here, the similarity between the position transformation feature of each position and the position transformation features of other positions is calculated on the position transformation features after position rearrangement. When calculating the similarity, the calculation method of cosine distance or the calculation method of Euclidean distance can be used.

[0129] In some embodiments, referring to Figure 9 , Figure 9 An optional flowchart of a method for determining repaired features provided by an embodiment of the present application is shown in Figure 7 Step 202 shown in the figure can be implemented by steps 2021-2023 as shown in Figure 9 :

[0130] Step 2021: performing normalization processing on the matrix elements in the adjacency matrix to obtain a normalized adjacency matrix;

[0131] Step 2022: performing dot product summation on the normalized adjacency matrix and the position transformation feature to obtain updated position transformation features;

[0132] Step 2023: performing position inverse transformation on the updated position transformation features to obtain corresponding repaired features.

[0133] Here, all weights except itself of each row of the adjacency matrix are normalized and dot product summation is performed with the position transformation feature to obtain updated position transformation features corresponding to each position. Here, only the position transformation features corresponding to positions in the to-be-repaired region are updated. Position inverse transformation is performed on the updated position transformation features to obtain corresponding repaired features. The structure diagram of the feature repairing network in the above embodiment corresponds to Figure 4A .

[0134] In some embodiments, after obtaining the updated position transformation feature, the terminal can further perform feature mapping on the updated position transformation feature to obtain a mapped feature; correspondingly, the terminal can realize position inverse transformation on the updated position transformation feature to obtain the corresponding repair feature by performing position inverse transformation on the mapped feature to obtain the corresponding repair feature.

[0135] The structural diagram of the feature repair network in the above embodiments corresponds to Figure 4B In order to strengthen the high-frequency information in the repair feature, after performing feature updating on the position transformation feature, the updated position transformation feature is input into the feature mapping layer (such as a plurality of convolution layers) to perform nonlinear mapping, and the obtained mapped feature is input into the position inverse transformation layer to perform position inverse transformation to obtain the final repair feature.

[0136] In some embodiments, after obtaining the updated position transformation feature, the terminal can further perform feature mapping on the updated position transformation feature to obtain a first mapped feature; the first mapped feature is dot multiplied and summed with the normalized adjacency matrix to obtain an updated mapped feature; the updated mapped feature is further mapped to obtain a second mapped feature; correspondingly, the terminal can realize position inverse transformation on the updated position transformation feature to obtain the corresponding repair feature by performing position inverse transformation on the second mapped feature to obtain the corresponding repair feature.

[0137] The structural diagram of the feature repair network in the above embodiments corresponds to Figure 4D That is, the feature repair network can include a plurality of feature updating layers and feature mapping layers, the updated position transformation feature each time is nonlinearly mapped through the feature mapping layer, and then the obtained mapped feature is updated again, the mapped feature obtained by the last feature updating and feature mapping is input into the position inverse transformation layer to perform position inverse transformation to obtain the final repair feature.

[0138] In some embodiments, referring to Figure 10 , Figure 10 An optional flowchart for determining the repair feature provided by the embodiments of the present application is shown in Figure 5 The step of performing feature repair on the encoded feature based on the position transformation feature in step 102 to obtain the corresponding repair feature can be realized by steps 301-303 as shown in Figure 10

[0139] Step 301: sampling the position transformation features corresponding to other regions except the region to be repaired in the position transformation feature to obtain sampled position transformation features;

[0140] ​Step 302: update the position transformation feature corresponding to the region to be repaired in the position transformation feature by using the sampling position transformation feature;

[0141] Step 303: perform position inverse transformation on the updated position transformation feature to obtain the corresponding repair feature.

[0142] In some embodiments, after updating the position transformation feature corresponding to the region to be repaired in the position transformation feature by using the sampling position transformation feature, the updated position transformation feature can also be subjected to nonlinear mapping to obtain the corresponding mapping feature; and the mapping feature is subjected to position inverse transformation to obtain the corresponding repair feature.

[0143] Here, the feature sampled from the position transformation feature corresponding to other regions outside the region to be repaired is directly updated to the feature corresponding to the region to be repaired, which can greatly improve the efficiency of feature repair.

[0144] Step 103: perform feature decoding on the repair feature by the feature decoding network to obtain the corresponding repair image.

[0145] In some embodiments, the feature decoding network is composed of at least two cascaded sub-decoding networks, and the terminal can perform feature decoding on the repair feature by the feature decoding network to obtain the corresponding repair image.

[0146] The repair feature is decoded by the first sub-decoding network in the at least two sub-decoding networks to obtain the corresponding initial repair image; the repair feature corresponding to the initial repair image is output to the subsequent cascaded sub-decoding network to decode the repair feature corresponding to the initial repair image by the subsequent cascaded sub-decoding network to obtain the target repair image, and the target repair image is output by the last sub-decoding network as the repair image corresponding to the image to be repaired.

[0147] In actual application, the feature decoding network can include multiple layers of sub-decoding networks, and the number of layers of the sub-decoding networks included in the feature decoding network can be the same as or different from the number of layers of the sub-encoding networks included in the feature encoding network. The repair feature of the image to be repaired is decoded by the first sub-decoding network to obtain the first initial repair image; then the repair feature corresponding to the first initial repair image is input into the second sub-decoding network, and the repair feature corresponding to the first initial repair image is decoded by the second sub-decoding network to obtain the second initial repair image, and the above steps are repeated until the repair feature corresponding to the j-1 (where j is the number of layers of the feature decoding network) initial repair image is input into the jth sub-decoding network, and the repair feature corresponding to the j-1 initial repair image is decoded by the jth sub-decoding network to obtain the final repair image.

[0148] By the above manner, the position rearrangement of the original coding features is realized by the position transformation operation on the original coding features corresponding to the image to be repaired, that is, the coding features in multiple positions are packed in one position, so that after the position rearrangement, the multiple coding features of the to-be-repaired region are packed together, and since the original features of the image to be repaired are updated according to the position during the feature repair, when the coding features in a position are updated, the multiple coding features corresponding to the position are updated at the same time, thereby improving the efficiency of image repair.

[0149] In the following, an exemplary application of the embodiment of the present application in an actual application scenario will be described.

[0150] The embodiment of the present application provides a general image repair method, which is suitable for applications such as video image date removal, watermarking, etc. For a movie with a flawed screen during playback, the image repair is performed on the flawed screen to improve the screen quality, which can greatly improve the speed and effect of image repair.

[0151] Referring to Figure 3 The image repair model provided by the embodiment of the present application is a convolutional neural network, which includes a feature encoding network, a feature repair network, and a feature decoding network. In actual implementation, the image to be repaired containing the mark information indicating the to-be-repaired region is input into the image repair model, a series of processing is performed on the to-be-repaired region through the feature encoding network, the feature repair network, and the feature decoding network, the repair image corresponding to the to-be-repaired region is obtained, and the obtained repair image corresponding to the to-be-repaired region is fused with other regions in the image to be repaired except the to-be-repaired region to obtain the final repair image.

[0152] The feature repair network is mainly used for a series of processing such as position rearrangement (i.e., the above-mentioned position transformation), feature update, and position inverse transformation on the feature map of the image to be repaired obtained through the feature encoding network, to obtain the final repair feature map and output to the feature decoding network to obtain the final repair image. Before image repair, the image repair model needs to be trained, and during training, the size of the to-be-repaired region needs to be considered. Generally, when the to-be-repaired region is larger, the to-be-repaired region needs to be updated multiple times, such as twice or more than twice, to achieve better repair effect.

[0153] Referring to Figure 11 , Figure 11A data flow diagram of the feature repairing network provided by the embodiment of the present application is shown in FIG. 4. The input of the feature repairing network is the intermediate layer feature map of the convolutional neural network, i.e., the original feature map of the corresponding image to be repaired output by the feature encoding network (corresponding to the encoded feature described above). The original feature map is subjected to position rearrangement (i.e., position transformation) by the feature repairing network to obtain a new rearranged feature map F (corresponding to the position-transformed feature described above). The final feature map corresponding to the image to be repaired is obtained through two feature updates, so as to output the final feature map to the subsequent feature decoding network for feature decoding to obtain the final repaired image.

[0154] Based on the architecture of the image repairing model and the feature repairing network described above, the image repairing method provided by the embodiment of the present application is described with reference to FIG. 5. Figure 12 , Figure 12 An optional flow diagram of the image repairing method provided by the embodiment of the present application is shown in FIG. 5. The steps shown in FIG. 5 will be described below. Figure 12

[0155] Step 401: The feature encoding network is used to perform feature encoding on the image to be repaired to obtain the original feature map corresponding to the image to be repaired.

[0156] Here, the image to be repaired is input into the image repairing model. First, the feature encoding network is used to perform feature encoding on the image to be repaired to obtain the original feature map corresponding to the image to be repaired, and the obtained original feature map is input into the feature repairing network.

[0157] Step 402: The feature repairing network is used to segment the original feature map according to a preset size of a segmentation factor to obtain a target number of sub-feature maps, and the sub-feature maps are subjected to position rearrangement to obtain a rearranged feature map.

[0158] The size of the segmentation factor can be determined according to actual conditions, such as 2x2, 3x3 or other sizes. If the size of the original feature map corresponding to the image to be repaired is 4x4, the original feature map is subjected to position rearrangement according to a segmentation factor with a size of 2x2 to obtain four 2x2 sub-regions (i.e., sub-feature maps). The new rearranged feature map obtained through rearrangement is smaller in length and width relative to the original feature map, and the number of channels is increased. For each sub-region, the sub-encoding features corresponding to the four positions inside are arranged in a column in a certain order, so that a 4x2x2 volume is formed, and a new rearranged feature map corresponding to the image to be repaired is obtained. For each position, the purpose of position rearrangement is to pack the features of adjacent positions together.

[0159] Step 403: The similarity values between the features of each position and the features of other positions are obtained for each position in the rearranged feature map.

[0160] ​Step 404: constructing an adjacency matrix corresponding to the image to be repaired, taking each position as a node and the connection relationship between the nodes as an edge; wherein the matrix elements of the adjacency matrix are corresponding similarity values.

[0161] Here, the similarity between the features of each position and the features of other positions is calculated on the rearranged feature map after position rearrangement, and then an adjacency matrix is constructed. When calculating the similarity, the calculation method of cosine distance or the calculation method of Euclidean distance can be used.

[0162] Step 405: normalizing the matrix elements in the adjacency matrix to obtain a normalized adjacency matrix;

[0163] Step 406: pointwise multiplication and summation of the normalized adjacency matrix and the rearranged feature map to obtain an updated feature map;

[0164] Step 407: performing nonlinear mapping on the updated feature map to obtain a mapped feature map.

[0165] Here, in order to strengthen the high-frequency information in the features, after updating the feature map, the updated feature map is input into multiple convolution layers for nonlinear mapping to obtain a mapped feature map.

[0166] In actual application, steps 406-407 can be repeated multiple times according to actual conditions.

[0167] Step 408: performing position inverse transformation on the mapped feature map to obtain a corresponding final feature map.

[0168] Here, the mapped feature map (corresponding to the above-mentioned mapped feature) is subjected to position inverse transformation to obtain a final feature map (corresponding to the above-mentioned repair feature), so as to restore the mapped feature map to the same spatial size as the original feature map, and perform subsequent processing based on the obtained final feature map.

[0169] Step 409: performing feature encoding on the final feature map through a feature encoding network to obtain a corresponding repair image.

[0170] In the above manner, the features of multiple positions are packed in one position, so that after position rearrangement, multiple features of the repair area are packed together. Since the features of the image to be repaired are updated according to positions during feature repair, the features of multiple positions corresponding to the position will be updated simultaneously when updating the features of the position, thereby improving the efficiency of image repair. Meanwhile, after updating the feature map, the updated feature map is input into multiple convolution layers to introduce nonlinear transformation, thereby strengthening the high-frequency information in the features and improving the image repair effect.

[0171] The following continues to illustrate an exemplary structure of the image repairing apparatus 555 provided by the embodiments of the present application, which is implemented as a software module, and the image repairing apparatus provided by the embodiments of the present application is applied to an image repairing model, which includes a feature encoding network, a feature repairing network and a feature decoding network. In some embodiments, as shown in Figure 13 Figure 13 The structure diagram of the image repairing apparatus provided by the embodiments of the present application, the software module in the image repairing apparatus 555 stored in the memory 550 can include:

[0172] The feature encoding module 5551 is configured to perform feature encoding on the to-be-repaired image containing the mark information through the feature encoding network, to obtain corresponding encoded features.

[0173] The mark information is used to indicate a to-be-repaired region of the to-be-repaired image.

[0174] The feature repairing module 5552 is configured to perform position transformation on the encoded features through the feature repairing network, to obtain position transformation features, and perform feature repairing on the encoded features based on the position transformation features, to obtain corresponding repaired features.

[0175] The feature decoding module 5553 is configured to perform feature decoding on the repaired features through the feature decoding network, to obtain a corresponding repaired image.

[0176] In some embodiments, the feature repairing module includes a position transformation unit, which is configured to perform feature segmentation on the encoded features according to a segmentation factor of a preset size, to obtain a target number of sub-encoded features.

[0177] The target number of sub-encoded features are position rearranged to obtain position transformation features.

[0178] In some embodiments, the position transformation unit is configured to obtain a feature map corresponding to the encoded features of the to-be-repaired image.

[0179] The feature map is segmented according to a segmentation factor of a preset size, to obtain a target number of sub-feature maps, and the sub-feature maps are determined as the feature maps corresponding to the sub-encoded features.

[0180] In some embodiments, the feature repairing module further includes a matrix construction unit and a feature updating unit. The matrix construction unit is configured to construct an adjacency matrix corresponding to the to-be-repaired image based on the position transformation features.

[0181] The feature updating unit is configured to perform feature repairing on the encoded features corresponding to the to-be-repaired region based on the adjacency matrix, to obtain corresponding repaired features. ​

[0182] In some embodiments, the matrix construction unit is further configured to

[0183] For each position in the position transformation features, a similarity value between the position transformation feature corresponding to the position and the position transformation features corresponding to other positions is obtained respectively.

[0184] An adjacency matrix corresponding to the image to be repaired is constructed, taking each position as a node and a connection relationship between nodes as an edge.

[0185] The matrix elements of the adjacency matrix are the corresponding similarity values.

[0186] In some embodiments, the feature repair module further comprises a position inverse transformation unit,

[0187] The feature update unit is further configured to normalize the matrix elements in the adjacency matrix to obtain a normalized adjacency matrix.

[0188] The normalized adjacency matrix is dot-multiplied with the position transformation features to obtain updated position transformation features.

[0189] The position inverse transformation unit is configured to perform position inverse transformation on the updated position transformation features to obtain corresponding repair features.

[0190] In some embodiments, the feature repair module further comprises a feature mapping unit,

[0191] The feature mapping unit is configured to perform feature mapping on the updated position transformation features to obtain mapped features after the updated position transformation features are obtained.

[0192] Correspondingly, the position inverse transformation unit is further configured to perform position inverse transformation on the mapped features to obtain corresponding repair features.

[0193] In some embodiments, after the updated position transformation features are obtained, the feature mapping unit is further configured to perform feature mapping on the updated position transformation features to obtain first mapped features.

[0194] The feature update unit is further configured to dot-multiply the first mapped features with the normalized adjacency matrix to obtain updated mapped features.

[0195] The feature mapping unit is further configured to perform feature mapping on the updated mapped features to obtain second mapped features.

[0196] The position inverse transformation unit is further configured to perform position inverse transformation on the second mapped features to obtain corresponding repair features.

[0197] In some embodiments, the feature repairing module comprises a position transformation unit, a feature sampling unit, a feature updating unit and a position inverse transformation unit,

[0198] The feature sampling unit is configured to sample the position transformation features corresponding to other regions outside the region to be repaired in the position transformation features to obtain sampled position transformation features.

[0199] The feature updating unit is configured to update the position transformation features corresponding to the region to be repaired in the position transformation features by using the sampled position transformation features.

[0200] The position inverse transformation unit is configured to perform position inverse transformation on the updated position transformation features to obtain corresponding repairing features.

[0201] In some embodiments, the feature encoding network is composed of at least two cascaded sub-encoding networks, and the feature encoding module is configured to perform encoding processing on the image to be repaired containing the mark information by using a first sub-encoding network in the at least two cascaded sub-encoding networks to obtain corresponding sub-encoding features.

[0202] The sub-encoding features are input into a subsequent cascaded sub-encoding network to perform encoding processing on the sub-encoding features by using the subsequent cascaded sub-encoding network to obtain target encoding features, and the target encoding features are output by a last sub-encoding network as the encoding features corresponding to the image to be repaired.

[0203] In some embodiments, the feature decoding network is composed of at least two cascaded sub-decoding networks, and the feature decoding module is configured to perform decoding processing on the repairing features by using a first sub-decoding network in the at least two cascaded sub-decoding networks to obtain corresponding initial repairing images.

[0204] The repairing features corresponding to the initial repairing images are output to a subsequent cascaded sub-encoding network to perform decoding processing on the repairing features corresponding to the initial repairing images by using the subsequent cascaded sub-decoding network to obtain target repairing images, and the target repairing images are output by a last sub-decoding network as the repairing images corresponding to the image to be repaired.

[0205] In some embodiments, the device further comprises a model training module, and the model training module is configured to

[0206] The feature encoding network is used to perform feature extraction on the image to be repaired containing the mark information in the image sample to obtain corresponding encoding features.

[0207] The image sample comprises an original image and an image to be repaired corresponding to the original image, and the mark information is used to indicate a region to be repaired of the image to be repaired.

[0208] The feature repair network is used for position transformation on the encoded features to obtain position-transformed features, and feature repair is performed on the encoded features based on the position-transformed features to obtain corresponding repaired features.

[0209] The feature decoding network is used for feature decoding on the repaired features to obtain corresponding repaired images.

[0210] Differences between the repaired images and the original images are obtained, and model parameters of the image repair model are updated based on the obtained differences.

[0211] The embodiment of the present application provides a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the image repair method provided in the embodiment of the present application.

[0212] The embodiment of the present application provides a computer readable storage medium storing executable instructions, wherein the executable instructions are stored in the computer readable storage medium. When the executable instructions are executed by a processor, the processor will execute the method provided in the embodiment of the present application.

[0213] In some embodiments, the computer readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disc, or CD-ROM; or various devices comprising one or any combination of the above memories.

[0214] In some embodiments, the executable instructions can be in the form of programs, software, software modules, scripts or codes, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and can be deployed in any form, including being deployed as independent programs or being deployed as modules, components, subroutines or other units suitable for use in a computing environment.

[0215] As an example, the executable instructions can but not necessarily correspond to files in a file system, can be stored in part of a file storing other programs or data, for example, stored in one or more scripts in a Hyper Text Markup Language (HTML) document, stored in a single file dedicated to the program in question, or stored in multiple cooperative files (for example, files storing one or more modules, subroutines or code portions).

[0216] As an example, the executable instructions can be deployed to execute on one computing device, or on multiple computing devices located at one site, or distributed across multiple sites and interconnected by a communication network.

[0217] The above merely provides illustration of the embodiment of the present application, but is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, and improvement made within the spirit and range of the present application shall fall within the protection scope of the present application.

Claims

1. An image inpainting method characterized by, The method is applied to an image repairing model, and the image repairing model comprises a feature encoding network, a feature repairing network and a feature decoding network. The feature encoding network is used to perform feature encoding on a to-be-repaired image containing mark information, to obtain corresponding encoded features, wherein the mark information is used to indicate a to-be-repaired region of the to-be-repaired image. The feature repairing network is used to perform feature segmentation on the encoded features according to a preset size of a segmentation factor, to obtain a target number of sub-encoded features; the target number of sub-encoded features are subjected to position rearrangement, to obtain position transformation features; the feature repairing network is used to perform feature repairing on the encoded features based on the position transformation features, to obtain corresponding repairing features, wherein the length of the position transformation features is less than the length of the encoded features, the width of the position transformation features is less than the width of the encoded features, and the number of channels of the position transformation features is greater than the number of channels of the encoded features; the feature repairing based on the position transformation features to obtain the corresponding repairing features comprises: constructing an adjacency matrix corresponding to the to-be-repaired image based on the position transformation features; performing feature repairing on the encoded features corresponding to the to-be-repaired region based on the adjacency matrix, to obtain corresponding repairing features. The feature decoding network is used to perform feature decoding on the repairing features, to obtain a corresponding repaired image.

2. The method of claim 1, wherein, The feature segmentation according to the preset size of the segmentation factor to obtain the target number of sub-encoded features comprises: obtaining a feature map corresponding to the encoded features of the to-be-repaired image; segmenting the feature map according to the preset size of the segmentation factor to obtain a target number of sub-feature maps, and determining the sub-feature maps as feature maps corresponding to the sub-encoded features.

3. The method of claim 1, wherein, The construction of the adjacency matrix corresponding to the to-be-repaired image based on the position transformation features comprises: for each position in the position transformation features, obtaining a similarity value between the position transformation features corresponding to each position and the position transformation features corresponding to other positions; constructing an adjacency matrix corresponding to the to-be-repaired image by taking each position as a node and a connection relationship between the nodes as an edge; wherein, the matrix elements of the adjacency matrix are corresponding similarity values.

4. The method of claim 1, wherein, The feature repairing based on the adjacency matrix to obtain the corresponding repairing features comprises: performing normalization processing on the matrix elements in the adjacency matrix to obtain a normalized adjacency matrix; point-wise multiplication and summation of the normalized adjacency matrix and the position transformation features are performed to obtain updated position transformation features; performing position inverse transformation on the updated position transformation features to obtain corresponding repairing features.

5. The method of claim 4, wherein, After obtaining the updated position transformation features, the method further comprises: performing feature mapping on the updated position transformation features to obtain mapping features; correspondingly, the position inverse transformation of the updated position transformation features to obtain the corresponding repairing features comprises: performing position inverse transformation on the mapping features to obtain corresponding repairing features.

6. The method of claim 4, wherein, After the updated position transformation feature is obtained, the method further comprises: performing feature mapping on the updated position transformation feature to obtain a first mapping feature; performing point multiplication and summation on the first mapping feature and the normalized adjacency matrix to obtain an updated mapping feature; performing feature mapping on the updated mapping feature to obtain a second mapping feature; correspondingly, performing position inverse transformation on the updated position transformation feature to obtain a corresponding repair feature, comprising: performing position inverse transformation on the second mapping feature to obtain a corresponding repair feature.

7. The method of claim 1, wherein, The feature repair of the encoding feature based on the position transformation feature to obtain a corresponding repair feature, comprising: sampling the position transformation feature corresponding to other regions outside the repair region in the position transformation feature to obtain a sampling position transformation feature; updating the position transformation feature corresponding to the repair region in the position transformation feature by using the sampling position transformation feature; performing position inverse transformation on the updated position transformation feature to obtain a corresponding repair feature.

8. The method of claim 1, wherein, The feature encoding network is composed of at least two sub-encoding network cascades, and the feature encoding of the repair image containing the mark information by the feature encoding network to obtain the corresponding encoding feature, comprising: encoding processing of the repair image containing the mark information by the first sub-encoding network in the at least two sub-encoding networks to obtain the corresponding sub-encoding feature; inputting the sub-encoding feature into the subsequent cascaded sub-encoding network to obtain the target encoding feature by encoding processing of the sub-encoding feature by the subsequent cascaded sub-encoding network, and outputting the target encoding feature by the last sub-encoding network as the corresponding encoding feature of the repair image.

9. The method of claim 1, wherein, The feature decoding network is composed of at least two sub-decoding network cascades, and the feature decoding of the repair feature by the feature decoding network to obtain the corresponding repair image, comprising: decoding processing of the repair feature by the first sub-decoding network in the at least two sub-decoding networks to obtain the corresponding initial repair image; outputting the repair feature corresponding to the initial repair image to the subsequent cascaded sub-encoding network to obtain the target repair image by decoding processing of the repair feature corresponding to the initial repair image by the subsequent cascaded sub-decoding network, and outputting the target repair image by the last sub-decoding network as the corresponding repair image of the repair image.

10. The method of claim 1, wherein, The method further comprises: feature extraction of the repair image containing the mark information in the image sample by the feature encoding network to obtain the corresponding encoding feature; wherein the image sample comprises an original image and a repair image corresponding to the original image, and the mark information is used to indicate the repair region of the repair image; performing position transformation on the encoding feature by the feature repair network to obtain a position transformation feature, and performing feature repair on the encoding feature based on the position transformation feature to obtain a corresponding repair feature; performing feature decoding on the repair feature by the feature decoding network to obtain a corresponding repair image; Obtaining a difference between the repaired image and the original image, and updating model parameters of the image repairing model based on the obtained difference.

11. An image inpainting apparatus characterized by comprising: The image repairing model comprises a feature encoding network, a feature repairing network, and a feature decoding network. The feature encoding module is configured to perform feature encoding on the image to be repaired containing the mark information through the feature encoding network to obtain corresponding encoded features. The feature repairing module is configured to perform feature segmentation on the encoded features according to a preset size of a segmentation factor to obtain a target number of sub-encoded features, perform position rearrangement on the target number of sub-encoded features to obtain position-transformed features, and perform feature repairing on the encoded features based on the position-transformed features to obtain corresponding repaired features. The feature decoding module is configured to perform feature decoding on the repaired features through the feature decoding network to obtain a corresponding repaired image.

12. The apparatus of claim 11, wherein, The feature encoding network is composed of at least two sub-encoding networks in cascade. The feature encoding module is configured to perform encoding processing on the image to be repaired containing the mark information through a first sub-encoding network in the at least two sub-encoding networks to obtain corresponding sub-encoded features, and input the sub-encoded features into a subsequent cascaded sub-encoding network to perform encoding processing on the sub-encoded features through the subsequent cascaded sub-encoding network to obtain target encoded features, and output the target encoded features as the encoded features corresponding to the image to be repaired through a last sub-encoding network.

13. The apparatus of claim 11, wherein, The feature decoding network is composed of at least two sub-decoding networks in cascade. The feature decoding module is configured to perform decoding processing on the repaired features through a first sub-decoding network in the at least two sub-decoding networks to obtain a corresponding initial repaired image, and output the repaired features corresponding to the initial repaired image to a subsequent cascaded sub-encoding network to perform decoding processing on the repaired features corresponding to the initial repaired image through the subsequent cascaded sub-decoding network to obtain a target repaired image, and output the target repaired image as the repaired image corresponding to the image to be repaired through a last sub-decoding network.

14. An electronic device, comprising: The memory is configured to store executable instructions. The processor is configured to execute the executable instructions stored in the memory to implement the image repairing method in any one of claims 1 to 10. The memory stores executable instructions for being executed by the processor to implement the image repairing method in any one of claims 1 to 10.

15. A computer-readable storage medium, characterized in that, ​ 16. A computer program product comprising computer-executable instructions or a computer program, characterized in that, The computer executable instructions or computer programs, when executed by a processor, implement the image inpainting method of any one of claims 1 to 10.

Images