Watermark removal method, device, terminal equipment and readable storage medium

Abstract

The present application is applicable to the field of image processing technology, and in particular, relates to a watermark removal method, device, terminal device and readable storage medium. The method comprises: obtaining a file containing a watermark; determining the position information of the watermark in the file; obtaining a target area in the file according to the position information, the target area being the area containing the watermark in the file; performing watermark removal processing on the target area to obtain a watermark-free file. That is, the present application determines the position information of the watermark in the file; obtains the target area in the file according to the position information, and does not need to process the entire file. It only needs to perform watermark removal processing on the target area to obtain a watermark-free file, thereby increasing the speed of watermark removal and reducing the amount of calculation for watermark removal.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to a watermark removal method, apparatus, terminal device and readable storage medium. Background Art

[0002] The main methods for removing book watermarks on the current market are: using post-image restoration software to remove, or purchasing corresponding paid software to achieve batch removal of specific watermarks, or using traditional image processing solutions to remove watermarks.

[0003] However, when using the existing book watermark removal method to remove the watermark, it is necessary to process all the pixels of the entire image containing the watermark, resulting in a slow watermark removal speed and a large amount of calculation for watermark removal. Summary of the invention

[0004] The watermark removal method, apparatus, terminal device and readable storage medium provided in the embodiments of the present application improve the speed of watermark removal and reduce the amount of calculation for watermark removal.

[0005] In a first aspect, an embodiment of the present application provides a watermark removal method, the method comprising:

[0006] Get the file containing the watermark;

[0007] Determine the position information of the watermark in the file;

[0008] Acquire a target area in the file according to the position information, wherein the target area is an area in the file containing the watermark;

[0009] The target area is subjected to watermark removal processing to obtain a watermark-removed file.

[0010] In a possible implementation of the first aspect, determining the position information of the watermark in the file includes:

[0011] The file is input into a first neural network model for processing, and the position information of the watermark in the file output by the first neural network model is obtained.

[0012] Wherein, the first neural network model is a YOLO v3 model.

[0013] Wherein, acquiring the target area in the file according to the location information includes:

[0014] The file is cropped according to the position information to obtain the target area.

[0015] The step of performing watermark removal processing on the target area to obtain a watermark-free file includes:

[0016] Inputting the target area into a second neural network model for processing, obtaining a first area output by the second neural network model, wherein the first area is an area of ​​the target area after the watermark is removed;

[0017] The first area is merged with the second area in the file to obtain a watermark-free file, where the second area is an area in the file excluding the target area.

[0018] The process of processing the target area using the second neural network model includes:

[0019] Acquire a third area corresponding to the target area, where the third area is a background area corresponding to the target area, and the third area does not contain the watermark;

[0020] Determine the position information of the watermark in the target area;

[0021] Determine a fourth area corresponding to the watermark in the third area according to the position information of the watermark in the target area;

[0022] The watermark in the target area is replaced with the fourth area.

[0023] Wherein, determining the location information of the watermark in the target area includes:

[0024] The target area is subjected to mask processing to obtain mask information of the target area, and the position information of the watermark in the target area is determined according to the mask information.

[0025] In a second aspect, an embodiment of the present application provides a watermark removal device, characterized in that the device comprises:

[0026] A first acquisition module, used to acquire a file containing a watermark;

[0027] A determination module, used to determine the position information of the watermark in the file;

[0028] A second acquisition module, used for acquiring a target area in the file according to the position information, wherein the target area is an area in the file containing the watermark;

[0029] The processing module is used to perform watermark removal processing on the target area to obtain a watermark-free file.

[0030] In a third aspect, an embodiment of the present application provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the watermark removal method as described in the first aspect when executing the computer program.

[0031] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the watermark removal method as described in the first aspect is implemented.

[0032] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the present application obtains a file containing a watermark; determines the position information of the watermark in the file; obtains the target area in the file according to the position information, and the target area is the area containing the watermark in the file; performs watermark removal processing on the target area to obtain the watermark-free file. That is, the present application determines the position information of the watermark in the file; obtains the target area in the file according to the position information, and does not need to process the entire file, but only needs to perform watermark removal processing on the target area to obtain the watermark-free file, thereby increasing the speed of watermark removal and reducing the amount of calculation for watermark removal. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0034] Figure 1 It is a schematic diagram of a network architecture of a watermark removal method provided in an embodiment of the present application;

[0035] Figure 2 It is a flowchart of a watermark removal method provided by an embodiment of the present application;

[0036] Figure 3a is an example diagram of the network structure of the YOLO v3 model provided in an embodiment of the present application;

[0037] Figure 3b It is a flowchart of a method for training a target detection model provided in one embodiment of the present application;

[0038] Figure 3c is an example diagram of the position information of the rectangular frame provided in an embodiment of the present application;

[0039] Figure 4It is a flowchart of a method for performing watermark removal processing on a target area provided by an embodiment of the present application;

[0040] Figure 5a is an example diagram of the network structure of the second neural network model provided in one embodiment of the present application;

[0041] Figure 5b is an example diagram of the network structure of an encoder provided by an embodiment of the present application;

[0042] Figure 5c is an example diagram of a network structure of a decoder provided in an embodiment of the present application;

[0043] Figure 6 It is a flowchart of a training method for a second neural network model provided in one embodiment of the present application;

[0044] Figure 7 This is an example diagram of obtaining a first area after removing a watermark provided by an embodiment of the present application;

[0045] Figure 8 It is a flowchart of a method for obtaining a first area after removing a watermark by applying a second neural network model provided by an embodiment of the present application;

[0046] Fig. 9 It is a structural schematic diagram of a watermark removal device provided in an embodiment of the present application;

[0047] Fig.10 It is a structural diagram of a terminal device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0048] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application. In other cases, the specific technical details of each embodiment may be referenced to each other, and a specific system not described in one embodiment may be referenced to other embodiments.

[0049] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0050] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0051] References to "an embodiment of the present application" or "some embodiments" described in the specification of the present application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in other embodiments", "an embodiment of the present application", "other embodiments of the present application", etc. that appear in different places in the specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0052] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0053] When removing watermarks from books, existing methods need to process all pixels of the entire image containing the watermark, which results in a slow watermark removal speed and a large amount of calculation for removing watermarks.

[0054] In order to solve the above-mentioned defects, the inventive concept of the present application is as follows:

[0055] By determining the location information of the watermark in the file and obtaining the target area in the file according to the location information, it is not necessary to process the entire file. Only the watermark removal process needs to be performed on the target area to obtain the watermark-free file, thereby increasing the speed of watermark removal and reducing the amount of calculation for watermark removal.

[0056] In order to illustrate the technical solution of the present application, a specific embodiment is provided below for illustration.

[0057] See also Figure 1 , Figure 1 1 is a schematic diagram of a network architecture of a watermark removal method provided in an embodiment of the present application. For the convenience of explanation, only the part related to the present application is shown. The network architecture includes: a terminal device 100 and a server 200.

[0058] In the network architecture, the terminal device 100 may include but is not limited to a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. The terminal device 100 may be used to deploy the first neural network model and the second neural network model.

[0059] In the network architecture, the server 200 is essentially an electronic device with computing capabilities, which is deployed in the cloud and can also be used to deploy the first neural network model and the second neural network model. The server 200 mainly provides services for the terminal device 100.

[0060] The terminal device 100 accesses the Internet through a well-known network access method and establishes a data communication link with the cloud server 200 to start training the first neural network model and the second neural network model, perform watermark processing on files containing watermark areas, and other operations.

[0061] See also Figure 2 , Figure 2 : is a flow chart of a watermark removal method provided by an embodiment of the present application. As an implementation method, Figure 2 The execution subject of the method in can be Figure 1 The terminal device 100 in the embodiment of the present invention, as another implementation, Figure 2 The execution subject of the method in can also be Figure 1 The server 200 in Figure 2 As shown, the method includes: S201 to S204.

[0062] S201, obtaining a file containing a watermark.

[0063] Specifically, the file containing a watermark in the embodiment of the present application may be an image containing a watermark, a portable document format (PDF) containing a watermark, a web page containing a watermark, etc. The embodiment of the present application does not limit the type of file containing a watermark.

[0064] In an embodiment of the present application, when it is necessary to remove the watermark from a group of files containing watermarks, the group of files containing watermarks is input into a terminal device or a server, and the terminal device or the server can obtain the group of files containing watermarks.

[0065] In other embodiments of the present application, when it is necessary to remove watermarks from multiple groups of files containing watermarks, the multiple groups of files containing watermarks are input into a terminal device or a server, and the terminal device or the server can obtain the multiple groups of files containing watermarks. That is, the terminal device or the server of the embodiment of the present application can batch process multiple groups of files containing watermarks to increase the speed of removing watermarks.

[0066] S202: Determine the location information of the watermark in the file.

[0067] Specifically, in an embodiment of the present application, after obtaining a file containing a watermark, the position information of the watermark in the file is determined by a first neural network model.

[0068] In the embodiment of the present application, the first neural network model is a target detection model, which includes a faster region-based convolutional neural network model (Faster R-CNN model), a single deep neural network detection model (Single Shot MultiBox Detector, SSD model) model and an advanced real-time target detection model (Real-Time Object Detetion, YOLO model), etc. Among them, the YOLO model includes the YOLO v1 model, the YOLO v2 model and the YOLO v3 model. The embodiment of the present application uses the YOLOv3 model for example. Please refer to Figure 3a , Figure 3a is an example diagram of the network structure of the YOLO v3 model provided in one embodiment of the present application.

[0069] The basic network of the YOLO v3 model is the Darknet53 network, which includes 52 convolutional layers, 1 average pooling layer, 1 fully connected layer, and 1 activation function layer (softmax).

[0070] Among them, the 52 convolutional layers include: 1 convolution kernel with 32 filters, 5 downsampling layers, and 5 repeated residual units resblock_body (these 5 groups of residual units use the skip-layer connection method of the residual network (Residual Neural Network, ResNet). Each unit consists of a separate convolutional layer and a group of repeated convolutional layers. The repeated convolutional layers are repeated 1 times, 2 times, 8 times, 8 times and 4 times respectively; in each repeated convolutional layer, a 1x1 convolution operation is performed first, and then a 3x3 convolution operation is performed), a total of 52 layers.

[0071] The convolution layer calculation method is:

[0072] 52=1+5+(1*2)+(2*2)+(8*2)+(8*2)+(4*2).

[0073] The Darknet53 network set in the YOLO v3 model is to obtain features of different sizes of the input file. For example, the input file is an image file, and the size of the image file is 416*416*3, where 416*416 represents the resolution of the image and 3 represents the number of channels of the image. The image file is processed by 5 downsampling layers and 5 sets of repeated residual units, that is, the image is downsampled by 2 times (2) to obtain a feature image of 208*208*64 size, and by 4 times (2 2 ) to obtain a feature image of size 104*104*128, 8 times downsampling (2 3 ) to obtain a feature image of size 52*52*256, downsampled by 16 times (2 4 ) to obtain a feature image of size 26*26*512, downsampled by 32 times (2 5 ) to obtain a feature image of size 13*13*1024.

[0074] The network structure of the YOLO v3 model also includes three prediction layers, which are connected to the last three layers of residual units in the Darknet53 network through multiple convolutional layers, multiple upsampling layers, and multiple tensor splicing layers.

[0075] The three prediction layers in the YOLO v3 model are used to detect features of multiple sizes of the input file. For example, the input file is tested three times using the three prediction layers in the YOLO v3 model, at 32 times downsampling, 16 times downsampling, and 8 times downsampling, respectively, to detect features of different sizes of the input file and output the detection results.

[0076] The upsampling layer is set in the YOLO v3 model to expand the features obtained by low-multiple downsampling processing, so that the features obtained by low-multiple downsampling can be better expressed. For example, the features obtained by low-multiple downsampling processing are 13*13, and after the upsampling layer expansion operation, the features obtained are 26*26.

[0077] In the embodiment of the present application, the tensor splicing layer is set in the YOLO v3 model to splice the feature image output by the Darknet53 network with the feature image obtained by upsampling.

[0078] Please refer to Figure 3b , Figure 3b FIG. 1 is a flow chart of a method for training a target detection model provided in an embodiment of the present application. Figure 3a As shown, the method includes: S301 to S303.

[0079] S301, obtaining multiple groups of sample files.

[0080] Specifically, the sample file is a file containing a watermark. In some embodiments, a batch of files that need to remove watermarks can be collected in advance. For example, 1000 to 3000 (e.g., 1500) images that need to remove watermarks are collected in advance. The embodiment of the present application does not limit the number of files that need to remove watermarks.

[0081] Acquiring multiple groups of sample files means acquiring 1000 to 3000 pre-collected images that need to remove watermarks.

[0082] S302: Mark the watermark in each sample file to obtain target area information of each sample file.

[0083] Specifically, the target area is the area containing the watermark in the sample file. After obtaining multiple groups of sample files, each sample file needs to be labeled. Exemplarily, the embodiment of the present application can use a general text annotation tool (Labelme) to mark watermarks in 1500 images respectively (for example, the watermark can be marked with a labeling box), and the annotation results can be stored in a json file format to obtain 1500 groups of json files, and the json files include the location information of the watermark labeling box. In the embodiment of the present application, the information contained in these 1500 groups of json files is referred to as the target area information of each sample file.

[0084] In the embodiment of the present application, the form of the marking box includes: polygon, rectangle, circle, etc. The embodiment of the present application takes the marking box as a rectangular box as an example.

[0085] In the embodiment of the present application, the position information of the rectangular box is the position information of the watermark in the sample file, and the position information of the watermark in the sample file can be represented by the coordinates of the four vertices of the rectangular box. Figure 3c , Figure 3c is an example diagram of the location information of the rectangular frame provided by an embodiment of the present application. Figure 3c In the figure, w represents a rectangular box. H represents a sample file. The coordinate system is established with the upper left corner of the image as the origin, the width of the image as the positive direction of the x-axis, and the height of the image as the positive direction of the y-axis. In this coordinate system, the coordinates of the four vertices are expressed as (x_top_left, y_top_left), (x_top_right, y_top_right), (x_bottom_left, y_bottom_left) and (x_bottom_right, y_bottom_right).

[0086] In the embodiment of the present application, the target area information of each sample file obtained in S301 and each sample file obtained in S302 is used as a data set, a certain proportion of data is randomly selected from the data set as a training set of the first neural network model, and the remaining proportion of data is used as a validation set of the first neural network model. Exemplarily, 70% to 90%, for example, 80% of the data is randomly selected from the data set as the training set of the first neural network model, and 30% to 10%, for example, 20% of the data is used as the validation set of the first neural network model.

[0087] S303: Input the training set and the validation set into the first neural network model for training, and save the training parameters.

[0088] Specifically, each sample file in the training set and the target area information of each sample file are input into the YOLO v3 model. The Darknet53 network in the YOLO v3 model generates feature images of different sizes from each sample file and the target area information of each sample file. When detecting feature images of different sizes, the feature image is divided into S*S grid cells (for example, a feature image of 16*16 size is divided into 16*16 grid cells). If the target area in the feature image falls into any grid cell, the target area is detected by the grid cell.

[0089] In the embodiment of the present application, three bounding boxes are set for each grid unit. When the YOLO v3 model detects the target area, the bounding boxes of each feature image are calculated with the rectangular box of the target area marked in advance to obtain the measurement standard (Intersection Over Union, IOU). Only the bounding box with the largest IOU can be used to predict the target area.

[0090] IOU is a standard for measuring the accuracy of detecting corresponding objects in a specific data set. This standard is used to measure the correlation between the actual and predicted objects. The larger the IOU value, the higher the correlation.

[0091] The calculation formula of IOU is:

[0092]

[0093] The embodiment of the present application can calculate the IOU of the bounding box and the target area respectively through the above IOU calculation formula.

[0094] The YOLO v3 model in the embodiment of the present application performs detection on image features of multiple sizes. The predicted output features have two dimensions, which are the dimensions of the extracted feature image (for example, 13*13), and one dimension (depth) is B*(5+C), where B represents the number of bounding boxes predicted for each grid unit, C represents the number of categories of the bounding boxes, and 5 represents the coordinate information of the four bounding boxes and the confidence of a target area.

[0095] In the embodiment of the present application, each sample file and the target area information of each sample file in the training set and the validation set are input into the model for training. When the number of training epochs reaches 100 times, or the accuracy of the training validation set reaches a certain threshold (such as 90%), the model training is considered complete. The optimal model weight parameters are saved to extract the location information of the watermark in the sample file.

[0096] In an embodiment of the present application, a file is input into a first neural network model for processing to obtain position information of a watermark output by the first neural network model in the file.

[0097] Specifically, the file to be watermarked is input into the trained YOLO v3 model for processing, so that the target area containing the watermark in the file can be identified and the information of the target area can be obtained. The information of the target area includes the position information of the watermark annotation box, that is, the vertex coordinate information of the annotation box.

[0098] S203: Acquire the target area in the file according to the location information.

[0099] Specifically, the file is cropped according to the position information to obtain the target area.

[0100] In some embodiments, the target area of ​​the file can be obtained by cropping the file according to the coordinate area information of the four vertices of the rectangular frame obtained in S202.

[0101] In some embodiments, after the file is cropped according to the position information to obtain the target area, mask processing is performed on the target area to obtain mask information of the target area.

[0102] In the mask information, the pixel value of the watermark area is not 0, and the pixel values ​​of other areas are 0.

[0103] S204: Perform watermark removal processing on the target area to obtain a file after watermark removal.

[0104] For details, please refer to Figure 4 , Figure 4 1 is a flow chart of a method for removing watermarks from a target area provided by an embodiment of the present application. As an implementation method, Figure 4 The execution subject of the method in can be Figure 1The terminal device 100 in the embodiment of the present invention, as another implementation, Figure 4 The execution subject of the method in can also be Figure 1 Server 200 in. Figure 4 As shown, the method includes: S401 to S402.

[0105] S401, inputting the target area into the second neural network model for processing to obtain the first area output by the second neural network model.

[0106] Specifically, in the embodiment of the present application, the first area is the area after the watermark is removed from the target area.

[0107] In the embodiment of the present application, the second neural network model is a neural network model with an encoding and decoding structure. Figure 5a , Figure 5a This is an example diagram of the network structure of the second neural network model provided in one embodiment of the present application.

[0108] The second neural network model includes a contraction path composed of multiple encoders, an expansion path composed of multiple decoders, and a replacement module. The embodiment of the present application has 6 encoders. For the convenience of description, these 6 encoders are denoted as A, B, C, D, E, and F. The embodiment of the present application has 5 decoders. For the convenience of description, these 5 decoders are denoted as a, b, c, d, and e. For the convenience of description, the replacement module is denoted as G.

[0109] The contraction path mainly extracts features by downsampling at each level of encoder, while the expansion path gradually restores feature images with higher and higher resolutions by upsampling at each level of decoder. Contextual information is selected during the step-by-step downsampling of the expansion path. Therefore, in order to compensate for feature loss, each decoder takes as input the concatenation of the upsampled features of its previous level and the upsampled features of the encoder at the same level to compensate for contextual information, so as to ensure the quality of the restored image. The image restored by the last level of decoder can be restored to a color closer to the original image after a post-processing step.

[0110] In the embodiment of the present application, please refer to the network structure of each encoder. Figure 5b , Figure 5b This is an example diagram of the network structure of an encoder provided in an embodiment of the present application.

[0111] Each encoder contains a convolutional layer, an activation function layer (Relu), a batch normalization layer (BatchNorm) and a maximum pooling layer. In each encoder, the number of channels of the feature image becomes twice the original number of channels and increases layer by layer, but the spatial size decreases layer by layer and becomes 1 / 2 of the spatial size of the original feature image.

[0112] In the embodiment of the present application, please refer to the network structure of each decoder. Figure 5c , Figure 5c This is an example diagram of the network structure of a decoder provided in one embodiment of the present application.

[0113] Each decoder contains a tensor concatenation layer, a transposed convolution layer, a convolution layer, an activation function layer (Relu) and a batch normalization layer (BatchNorm). The tensor concatenation layer is used to connect the encoder at the same level as the decoder, and the transposed convolution layer is used to expand the size of the feature image. In each decoder, the number of channels of the feature image becomes 1 / 2 of the original number of channels and decreases layer by layer, but the spatial size increases layer by layer and becomes twice the spatial size of the original feature image.

[0114] For the second neural network model, its residual connection is reflected in the network structure of each encoder and decoder, such as Figure 5b and Figure 5c As shown in the figure, repeat x 3 means that the residual connection is repeated 3 times. The advantage of this is that it can expand the receptive field and help improve the quality of the restored image.

[0115] In the embodiment of the present application, the replacement module G is connected to the decoder, and the replacement module includes a 1*1 convolution layer and an activation function layer (sigmoid). The replacement module processes the feature image output by the encoder and the decoder.

[0116] Please refer to Figure 6 , Figure 6 FIG. 1 is a flow chart of a training method for a second neural network model provided in one embodiment of the present application. Figure 6 As shown, the method includes: S601 to S604.

[0117] S601: Obtain training samples.

[0118] Specifically, when removing the watermark from a file in the embodiment of the present application, in order to increase the speed of removing the watermark, it is not necessary to process the entire file, but to determine the location information of the watermark in the file through the output result of the first neural network model. The target area in the file is obtained according to the location information, and then the target area only needs to be processed for watermark removal in the second neural network model. Therefore, in the embodiment of the present application, the multiple groups of sample files in S301 can be processed by the first neural network model, and the target areas in the multiple groups of sample files obtained can be used as training samples for the second neural network model.

[0119] S602: Input the training samples into multiple encoders in the contraction path of the second neural network model, and gradually downsample to extract multi-size feature images of the training samples.

[0120] For example, please refer to Figure 5a , the size features of each target area in the training sample are 128*128*3 (length*width*height, where height can be understood as the number of channels). After downsampling and feature extraction by encoder A, they become 64*64*32 feature images and are output to encoder B. Similarly, after passing through encoder C, a 32*32*64 feature image is output, and then after passing through encoder D, it becomes a 16*16*128 feature image, after passing through encoder E, it becomes a 8*8*256 feature image, and after passing through encoder F, it becomes a 4*4*512 feature image. The obtained multi-size feature images are passed into the expansion path.

[0121] Step S603: input the multi-size feature maps into multiple decoders in the extension path of the second neural network model, and gradually upsample and restore them to feature images with higher resolution.

[0122] Specifically, each decoder performs restoration with reference to a feature image obtained by a corresponding encoder.

[0123] Please refer to Figure 5a In the second neural network model, except for the middle encoder F, the encoders and decoders on both sides are symmetrical in structure, where each decoder obtains two inputs, one is the upsampled image features of the decoder at the previous level, and the other is the cascade of the upsampled image features of the encoder at its symmetric level.

[0124] In the embodiment of the present application, the 4*4*512 feature image is restored to an 8*8*256 feature image after passing through decoder a, and then restored to a 16*16*128 feature image through decoder b, and then restored to a 32*32*64 feature image through decoder c, and then restored to 64*64*32 through encoder d, and finally restored to 128*128*3 through encoder e, which is the same size as the feature image when the encoder is input.

[0125] S604: Process the output of the decoder and the output of the last stage of the encoder to obtain the first area after the watermark is removed.

[0126] Specifically, in the embodiment of the present application, the feature image output by the decoder and the feature image output by the encoder are both transmitted to a replacement module G for replacement. In the replacement module, the feature image output by the decoder and the feature image output by the encoder are processed using the sigmoid function and regularization processing according to the inherent principles of the second neural network model to obtain the first area after removing the watermark.

[0127] Please refer to Figure 7 , Figure 7 is an example diagram of obtaining a first area after removing a watermark provided by an embodiment of the present application. Figure 7Target area C r , after being processed by the decoder, the output feature map contains a third region corresponding to the target region Feature image of the target area wait.

[0128] In some embodiments, the third area is a background area corresponding to the target area, and the third area does not contain a watermark. In some embodiments, the characteristic image of the target area is a mask image, in which the pixel value of the watermark area is not 0, and the pixel value of other areas is 0.

[0129] In the embodiment of the present application, in the replacement module G, the position information of the watermark in the target area is determined.

[0130] Specifically, mask processing is performed on the target area to obtain mask information of the target area, and the position information of the watermark in the target area is determined according to the mask information.

[0131] In some embodiments, a mask image is obtained by performing mask processing on the target area, and the position information of the watermark in the target area can be determined according to the mask of the watermark area in the mask image.

[0132] In the embodiment of the present application, a fourth area in the third area corresponding to the watermark is determined according to the position information of the watermark in the target area.

[0133] Specifically, by performing XOR processing on the third area and the watermark image in the target area, the fourth area corresponding to the watermark in the third area can be determined. The specific formula is as follows:

[0134]

[0135] In the embodiment of the present application, the fourth area is used to replace the watermark in the target area.

[0136] Specifically, the following formula can be used to replace the target area with the fourth area to obtain the first area after the watermark is removed.

[0137]

[0138] in, Indicates the first area.

[0139] In an embodiment of the present application, when the number of iterations of the second neural network model training reaches a certain number (for example, 100 times), the neural network model completes the training, and the saved model weight parameters are used to obtain the first area after removing the watermark.

[0140] In some embodiments, in order to supervise the training effect of the second neural network model, the third area is used as the target value, and the first area after removing the watermark is used as the prediction value. The total loss function formed by the linear superposition of multiple loss functions is used to supervise the sample training process, so that the prediction value obtained by training is maximized and approaches the target value as the number of iterations increases.

[0141] Please refer to Figure 8 , Figure 8 1 is a flowchart of a method for obtaining a first area after removing a watermark by applying a second neural network model provided by an embodiment of the present application. As an implementation method, Figure 8 The execution subject of the method in can be Figure 1 The terminal device 100 in the embodiment of the present invention, as another implementation, Figure 8 The execution subject of the method in can also be Figure 1 Server 200 in. Figure 8 As shown, the method includes: S801 to S804.

[0142] S801: Acquire a third area corresponding to the target area.

[0143] In an embodiment of the present application, the target area is input into a trained second neural network model. After the second neural network model is trained, its relevant parameters are optimized, and the feature image output by its decoder is the third area corresponding to the target area.

[0144] S802: Determine the location information of the watermark in the target area.

[0145] In the embodiment of the present application, the method for determining the position information of the watermark in the target area is the same as the method for determining the position information of the watermark in the target area in S604, and will not be repeated here.

[0146] S803: Determine a fourth area in the third area corresponding to the watermark according to the position information of the watermark in the target area.

[0147] In the embodiment of the present application, the method for determining the fourth area is the same as the method for determining the fourth area in S604, and will not be repeated here.

[0148] S804: Use the fourth area to replace the watermark in the target area.

[0149] In the embodiment of the present application, the fourth area is used to replace the watermark in the target area to obtain the first area after the watermark is removed.

[0150] The method of replacing the watermark in the target area with the fourth area is the same as the method of replacing the watermark in the target area with the fourth area in S604, and will not be described in detail here.

[0151] S402: Merge the first region with the second region in the file to obtain a watermark-free file.

[0152] Specifically, the first area is the area after the watermark is removed from the target area, and the second area is the area other than the target area in the file.

[0153] In the embodiment of the present application, the information of the target area includes the position information of the watermarked box, that is, the vertex coordinate information of the marked box. The first area and the second area are merged according to the vertex coordinate information of the marked box to obtain a file without watermark.

[0154] In the embodiment of the present application, the first region and the second region may be fused by using methods such as weighted average, wavelet transform, fuzzy neural network, pyramid decomposition, etc. The embodiment of the present application does not limit the fusion method of the first region and the second region.

[0155] In summary, the embodiment of the present application obtains a file containing a watermark; determines the position information of the watermark in the file using the first neural network model; obtains the target area in the file according to the position information, and the target area is the area containing the watermark in the file; and performs watermark removal processing on the target area using the second neural network model to obtain the watermark-free file. That is, the present application determines the position information of the watermark in the file; obtains the target area in the file according to the position information, and does not need to process the entire file. It only needs to perform watermark removal processing on the target area to obtain the watermark-free file, thereby improving the speed of removing the watermark.

[0156] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0157] See also Fig. 9 , Fig. 9 is a schematic diagram of the structure of a watermark removal device provided in an embodiment of the present application. As an implementation method, the device can be applied to Figure 1 The terminal device 100, as another implementation, the device can also be applied to Figure 1 The server 200. The device comprises:

[0158] The first acquisition module 91 is used to acquire a file containing a watermark.

[0159] The determination module 92 is used to determine the position information of the watermark in the file.

[0160] The second acquisition module 93 is used to acquire a target area in the file according to the position information, where the target area is an area in the file containing a watermark.

[0161] The processing module 94 is used to perform watermark removal processing on the target area to obtain a file after watermark removal.

[0162] The determination module 92 includes:

[0163] The first processing unit 921 is used to input the file into the first neural network model for processing, and obtain the position information of the watermark output by the first neural network model in the file.

[0164] The second acquisition module 93 includes:

[0165] The cropping unit 931 is used to crop the file according to the position information to obtain a target area.

[0166] The processing module 94 includes:

[0167] The second processing unit 941 is used to input the target area into the second neural network model for processing to obtain a first area output by the second neural network model, where the first area is the area after the watermark is removed from the target area.

[0168] The fusion unit 942 is used to fuse the first area with the second area in the file to obtain a watermark-free file, where the second area is an area in the file other than the target area.

[0169] The second processing unit 941 includes:

[0170] The first acquisition subunit 9411 is used to acquire a third area corresponding to the target area, where the third area is a background area corresponding to the target area, and the third area does not contain a watermark.

[0171] The second acquisition subunit 9412 is used to determine the position information of the watermark in the target area.

[0172] The determination subunit 9413 is used to determine a fourth area in the third area corresponding to the watermark according to the position information of the watermark in the target area.

[0173] The replacement subunit 9414 is used to replace the watermark in the target area with the fourth area.

[0174] The second acquisition subunit 9412 includes:

[0175] The mask processing subunit 9415 is used to perform mask processing on the target area to obtain mask information of the target area, and determine the position information of the watermark in the target area according to the mask information.

[0176] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0177] like Fig.10 As shown, an embodiment of the present application also provides a terminal device 20, including a memory 21, a processor 22, and a computer program 23 stored in the memory 21 and executable on the processor 22. When the processor 22 executes the computer program 23, the display screen defect locating method of the above-mentioned embodiments is implemented.

[0178] The processor 22 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0179] The memory 21 may be an internal storage unit of the terminal device 100. The memory 21 may also be an external storage device of the terminal device 100, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 100. Further, the memory 21 may also include both an internal storage unit of the terminal device 100 and an external storage device. The memory 21 is used to store computer programs and other programs and data required by the terminal device 100. The memory 21 may also be used to temporarily store data that has been output or is to be output.

[0180] The embodiments of the present application further provide a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the watermark removal methods of the above embodiments are implemented.

[0181] An embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device implements the watermark removal method of each of the above embodiments when executing the computer program product.

[0182] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may at least include: any entity or device that can carry the computer program code to the camera / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, a computer-readable storage medium cannot be an electric carrier signal and a telecommunication signal.

[0183] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0184] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0185] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0186] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A watermark removal method, characterized in that: The method comprises: Get the file containing the watermark; Determine the position information of the watermark in the file, wherein the position information is the vertex coordinate information of the marked box including the watermark; Cropping the file according to the position information to obtain a target area, where the target area is an area in the file containing the watermark; Performing watermark removal processing on the target area to obtain a watermark-free file includes: Inputting the target area into a second neural network model for processing, obtaining a first area output by the second neural network model, wherein the first area is an area of ​​the target area after the watermark is removed; Merging the first region with a second region in the file to obtain a watermark-free file, wherein the second region is a region in the file excluding the target region; The second neural network includes a contraction path composed of multiple encoders, an expansion path composed of multiple decoders and a replacement module. The replacement module uses a sigmoid function and regularization processing to process the feature images output by the encoder and the decoder to obtain the first area after the watermark is removed.

2. The method according to claim 1, characterized in that Determining the location information of the watermark in the file includes: The file is input into a first neural network model for processing, and the position information of the watermark in the file output by the first neural network model is obtained.

3. The method according to claim 2, characterized in that The first neural network model is a YOLO v3 model.

4. The method according to any one of claims 1 to 3, characterized in that: The process of the second neural network model processing the target area includes: Acquire a third area corresponding to the target area, where the third area is a background area corresponding to the target area, and the third area does not contain the watermark; Determine the position information of the watermark in the target area; Determine a fourth area corresponding to the watermark in the third area according to the position information of the watermark in the target area; The watermark in the target area is replaced with the fourth area.

5. The method according to claim 4, characterized in that The determining the location information of the watermark in the target area includes: The target area is subjected to mask processing to obtain mask information of the target area, and the position information of the watermark in the target area is determined according to the mask information.

6. A watermark removal device, characterized in that: The device comprises: A first acquisition module, used to acquire a file containing a watermark; A determination module, used to determine the position information of the watermark in the file, wherein the position information is the vertex coordinate information of the marked box including the watermark; A second acquisition module, used for acquiring a target area in the file according to the position information, wherein the target area is an area in the file containing the watermark; A processing module, used for performing watermark removal processing on the target area to obtain a watermark-free file; The second acquisition module includes a clipping unit, which is used to clip the file according to the position information to obtain the target area; The processing module comprises: A second processing unit is used to input the target area into the second neural network model for processing, and obtain a first area output by the second neural network model, where the first area is the area after the watermark is removed from the target area; A fusion unit, used for fusing the first area with the second area in the file to obtain a watermark-free file, wherein the second area is an area in the file other than the target area; The second neural network includes a contraction path composed of multiple encoders, an expansion path composed of multiple decoders and a replacement module. The replacement module uses a sigmoid function and regularization processing to process the feature images output by the encoder and the decoder to obtain the first area after the watermark is removed.

7. A terminal device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the watermark removal method according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the watermark removal method according to any one of claims 1 to 5 is implemented.

Images