Image watermark removing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an image watermark removing method and device, electronic equipment and a storage medium, belongs to the technical field of image processing, and is suitable for the field of financial science and technology. The method comprises the following steps: acquiring a sample watermark image and a sample original image; carrying out recovery area identification on the sample watermark image to obtain a watermark area frame selection image; the watermark area frame selection image comprises various types of image elements; performing element category identification on the image elements to obtain element categories; performing image restoration on the sample watermark image based on a preset image restoration model, the element category and the watermark region frame selection image to obtain a predicted original image; training a preset image restoration model according to the sample original image and the predicted original image to obtain a target image restoration model; and obtaining a target watermark image, and performing watermark removal on the target watermark image based on the target image restoration model. According to the embodiment of the invention, watermark removal can be accurately realized.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, is applicable to the field of financial technology, and in particular to an image watermark removal method and device, an electronic device and a storage medium. Background Art

[0002] Watermark removal refers to removing the watermark from an image and restoring the image to its original state without the watermark. For example, when making a car insurance claim, the photos submitted by the user may have watermarks automatically added by third-party applications, such as date or shooting device information, which may cause the loss of image details. In ordinary watermark removal, it is generally based on image patching, but this method relies on the information of the pixels around the watermark to generate the fill area, and cannot accurately patch the image for complex textures or high-detail areas. Therefore, how to accurately achieve watermark removal has become a problem that needs to be solved. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose an image watermark removal method and device, an electronic device and a storage medium, aiming to accurately achieve watermark removal.

[0004] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a method for removing an image watermark, the method comprising:

[0005] Acquire a sample watermark image and a sample original image; wherein the sample watermark image is an image formed by overlaying a watermark on the sample original image;

[0006] Performing restoration area recognition on the sample watermark image to obtain a watermark area frame selection image; wherein the watermark area frame selection image includes at least one category of image elements;

[0007] Performing element category recognition on the image elements to obtain element categories;

[0008] Performing image restoration on the sample watermark image based on a preset image restoration model, the element category and the watermark area frame selection image to obtain a predicted original image;

[0009] Training the preset image restoration model according to the sample original image and the predicted original image to obtain a target image restoration model;

[0010] A target watermark image is acquired, and watermark removal is performed on the target watermark image based on the target image restoration model.

[0011] In some embodiments, the restoring area recognition of the sample watermark image to obtain a watermark area frame selection image includes:

[0012] Performing foreground segmentation on the sample watermark image to obtain a sample foreground image;

[0013] Performing periodic texture recognition on the sample foreground image to obtain watermark coordinate data;

[0014] The sample foreground image is framed according to the watermark coordinate data to obtain the watermark area framed image.

[0015] In some embodiments, performing periodic texture recognition on the sample foreground image to obtain watermark coordinate data includes:

[0016] Performing Fourier transform on the sample foreground image to obtain sample frequency domain data;

[0017] Performing intermediate frequency screening on the sample frequency domain data to obtain intermediate frequency domain data;

[0018] Performing period screening on the intermediate frequency domain data to obtain periodic frequency domain data;

[0019] Watermark extraction is performed on the periodic frequency domain data to obtain the watermark coordinate data.

[0020] In some embodiments, the step of performing watermark extraction on the periodic frequency domain data to obtain the watermark coordinate data includes:

[0021] Performing inverse Fourier transform on the periodic frequency domain data to obtain an initial watermark image;

[0022] Adaptively binarizing the initial watermark image to obtain a target watermark image;

[0023] The target watermark image is subjected to watermark extraction to obtain the watermark coordinate data.

[0024] In some embodiments, performing image selection on the sample foreground image according to the watermark coordinate data to obtain the watermark area selection map includes:

[0025] Zeroing pixels of the sample foreground image according to the watermark coordinate data to obtain an element set image;

[0026] Performing element segmentation on the element set image to obtain at least one image element;

[0027] A frame selection image is performed on each of the image elements to obtain the watermark area frame selection image.

[0028] In some embodiments, the element categories include formula elements, image elements, and text elements; and restoring the sample watermark image based on a preset image restoration model, the element categories, and the watermark region frame selection image to obtain a predicted original image includes:

[0029] Performing formula image restoration on the watermark area frame selection image of the formula element based on the preset image restoration model to obtain a formula element restoration image;

[0030] Performing image restoration on the watermark area frame selection image of the image element based on the preset image restoration model to obtain an image element restoration image;

[0031] Performing text restoration on the watermark area frame selection image of the text element based on the preset image restoration model to obtain a text element restoration image;

[0032] The sample watermark image is replaced according to the formula element restored image, the image element restored image, and the text element restored image to obtain the predicted original image.

[0033] In some embodiments, performing formula image restoration on the watermark area frame selection image of the formula element based on the preset image restoration model to obtain a formula element restoration image includes:

[0034] Performing character recognition on the watermark area frame selection image of the formula element to obtain initial formula characters;

[0035] Constructing graph structure data for the initial formula characters to obtain formula graph structure data;

[0036] Perform formula character restoration on the formula graph structure data based on the preset image restoration model to obtain target formula characters;

[0037] A formula image is constructed according to the target formula characters to obtain the formula element restoration image.

[0038] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides an image watermark removal device, the device comprising:

[0039] A data acquisition module is used to acquire a sample watermark image and a sample original image; wherein the sample watermark image is an image formed by overlaying a watermark on the sample original image;

[0040] A region identification module is used to perform restoration region identification on the sample watermark image to obtain a watermark region frame selection image; wherein the watermark region frame selection image includes at least one category of image elements;

[0041] A category recognition module, used to perform element category recognition on the image elements to obtain element categories;

[0042] An image restoration module, used to restore the sample watermark image based on a preset image restoration model, the element category and the watermark area frame selection map to obtain a predicted original image;

[0043] A model training module, used for training the preset image restoration model according to the sample original image and the predicted original image to obtain a target image restoration model;

[0044] The watermark removal module is used to obtain a target watermark image and remove the watermark from the target watermark image based on the target image restoration model.

[0045] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0046] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0047] The present application proposes an image watermark removal method and device, electronic device and storage medium, which obtain a sample original image and a sample watermark image formed by covering the sample original image with a watermark, then perform restoration area identification on the sample watermark image to obtain a watermark area frame selection map, then perform element category identification on the image elements in the sample watermark image to obtain the element category, then perform image restoration on the sample watermark image based on a preset image restoration model, element category and watermark area frame selection map to obtain a predicted original image, and finally train the preset image restoration model based on the sample original image and the predicted original image to obtain a target image restoration model that can accurately remove the watermark image, and solve the problem of difficult repair in complex textures or high-detail areas by refining the watermark removal of the element category of the watermark area frame selection map; finally, obtain the target watermark image, remove the watermark of the target watermark image based on the target image restoration model, and achieve accurate repair of the target watermark image. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flow chart of an image watermark removal method provided by an embodiment of the present application;

[0049] Figure 2 yes Figure 1 Flow chart of step S102 in FIG.

[0050] Figure 3 yes Figure 2 Flow chart of step S202 in FIG.

[0051] Figure 4 yes Figure 3 Flow chart of step S304 in FIG.

[0052] Figure 5 yes Figure 2 Flow chart of step S203 in FIG.

[0053] Figure 6 yes Figure 1 Flow chart of step S104 in FIG.

[0054] Figure 7 yes Figure 6 Flow chart of step S601 in FIG.

[0055] Figure 8 It is a structural schematic diagram of an image watermark removal device provided in an embodiment of the present application;

[0056] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0060] First, some nouns involved in this application are analyzed:

[0061] Watermark: In the field of image processing and digital media, a watermark is an implicit or explicit mark added to an image, video or document to indicate copyright, source or identity authentication. Watermarks can be visible, such as station logos in television broadcasts, or invisible, implemented by embedding information in the digital code of the file. Watermarking technology is an interdisciplinary subject of computer science and digital rights management, involving fields such as image processing, cryptography and information security. The main purpose of watermarking is to protect copyright and prevent unauthorized copying and distribution, while ensuring the integrity and ownership verification of the content. In the field of artificial intelligence, watermarking technology is also used to track and manage generated content, such as deep fakes and source tracking of digital artworks.

[0062] Fourier transform: is a mathematical transformation used to analyze the frequency components of various functions or signals. Convert a function or signal from the time domain or space domain to the frequency domain and decompose it into a combination of a series of frequencies, each frequency representing a component of a sine wave. Fourier transform has a wide range of applications in engineering, physics, applied mathematics, and computer science, especially playing a core role in signal processing and image analysis. In the field of artificial intelligence, Fourier transform is used to analyze and process signal data, which can effectively identify and filter the noise components in the signal and enhance the useful frequency components, thereby improving the accuracy and efficiency of data processing.

[0063] Inverse Fourier Transform (IFT): It is the inverse operation of Fourier Transform, which is used to convert a function or signal from the frequency domain back to the time domain (or space domain). IFT is crucial in many fields such as signal processing, image processing and communication, allowing data to be reconstructed into its original time domain form after being analyzed and processed in the frequency domain. In the field of artificial intelligence, IFT is often used to reconstruct signals from modified frequency domain data, which is very useful in image restoration, audio enhancement and other signal restoration applications. For example, through IFT, a near-original image or sound signal can be recovered from the frequency components processed during compression or denoising. This technology not only supports improving data quality, but also helps improve the ability of machine learning and deep learning models to process complex data. The application of IFT in mathematics and engineering shows its fundamental role in understanding and operating complex systems.

[0064] Periodic texture: In the field of image processing and computer vision, periodic texture refers to recurring patterns or structures in an image that appear periodically at a certain spatial frequency and direction. Periodic textures are commonly found in nature and man-made environments, such as brick walls, floor tiles, fabrics, and other material surfaces. In the field of artificial intelligence, identifying and analyzing periodic textures is essential for tasks such as image classification, material recognition, and 3D reconstruction. By applying mathematical tools such as Fourier transform, the periodic features of textures can be effectively extracted from images, which can then be used to enhance the visual effects of the image or perform further image analysis. Periodic texture analysis not only helps improve image compression algorithms and image editing techniques, but also plays an important role in many technical fields such as pattern recognition, art design, and machine vision.

[0065] Watermark removal refers to removing the watermark from an image and restoring the image to its original state without the watermark. For example, when making a car insurance claim, the photos submitted by the user may have watermarks automatically added by third-party applications, such as date or shooting device information, which may cause the loss of image details. In ordinary watermark removal, it is generally based on image patching, but this method relies on the information of the pixels around the watermark to generate the fill area, and cannot accurately patch the image for complex textures or high-detail areas. Therefore, how to accurately achieve watermark removal has become a problem that needs to be solved.

[0066] Based on this, the embodiments of the present application provide an image watermark removal method and device, an electronic device and a storage medium, aiming to accurately achieve watermark removal.

[0067] An image watermark removal method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the image watermark removal method in the embodiments of the present application is described.

[0068] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0069] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0070] The image watermark removal method provided in the embodiment of the present application relates to the field of image processing technology and is applicable to the field of financial technology. The image watermark removal method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the image watermark removal method, etc., but is not limited to the above forms.

[0071] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0072] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0073] Figure 1 is an optional flow chart of the image watermark removal method provided in the embodiment of the present application. Figure 1The method may include but is not limited to steps S101 to S106.

[0074] Step S101, obtaining a sample watermark image and a sample original image; wherein the sample watermark image is an image formed by overlaying a watermark on the sample original image;

[0075] Step S102, performing restoration area recognition on the sample watermark image to obtain a watermark area frame selection image; wherein the watermark area frame selection image includes at least one category of image elements;

[0076] Step S103, performing element category recognition on the image elements to obtain element categories;

[0077] Step S104, performing image restoration on the sample watermark image based on a preset image restoration model, element category and watermark region frame selection map to obtain a predicted original image;

[0078] Step S105, training a preset image restoration model according to the sample original image and the predicted original image to obtain a target image restoration model;

[0079] Step S106, obtaining a target watermark image, and removing the watermark from the target watermark image based on a target image restoration model.

[0080] Steps S101 to S106 shown in the embodiment of the present application are as follows: by acquiring a sample original image and a sample watermark image formed by covering the sample original image with a watermark, restoring the sample watermark image, identifying the restored area, obtaining a watermark area frame selection map, identifying the element categories of the image elements in the sample watermark image, obtaining the element categories, and then restoring the sample watermark image based on a preset image restoration model, the element categories and the watermark area frame selection map to obtain a predicted original image; finally, training the preset image restoration model based on the sample original image and the predicted original image to obtain a target image restoration model capable of accurately removing the watermark image; and refining the watermark removal based on the element categories of the watermark area frame selection map to solve the problem of difficulty in repairing complex textures or high-detail areas; finally, acquiring the target watermark image, removing the watermark from the target watermark image based on the target image restoration model to achieve accurate repair of the target watermark image.

[0081] In step S101 of some embodiments, the sample original image refers to the original image that has not been subjected to additional modification processing such as adding a watermark, and is the basic presentation of the image content, carrying the original picture information, color information, and the subject content expressed by the image, etc. The sample watermark image is a new image formed by overlaying the watermark on the sample original image based on the sample original image.

[0082] See also Figure 2In some embodiments, step S102 may include but is not limited to steps S201 to S203:

[0083] Step S201, performing foreground segmentation on the sample watermark image to obtain a sample foreground image;

[0084] Step S202, performing periodic texture recognition on the sample foreground image to obtain watermark coordinate data;

[0085] Step S203, performing image selection on the sample foreground image according to the watermark coordinate data to obtain a watermark area selection map.

[0086] In steps S201 to S203 shown in the embodiment of the present application, by performing foreground segmentation on the sample watermark image, the foreground part containing the watermark information in the sample watermark image is effectively separated from the background, thereby preliminarily extracting the watermark and obtaining a sample foreground image. Then, periodic texture recognition is performed on the sample foreground image to accurately determine the coordinate data of the watermark in the image, thereby accurately selecting the watermark area based on these coordinate data and obtaining a watermark area selection map, thereby screening out the image with the watermark from the sample watermark image in the form of a framed line map.

[0087] In step S201 of some embodiments, foreground segmentation is to separate the foreground area and the background area in the image, and the segmented foreground area is the image with the watermark, that is, the sample foreground image. In one embodiment, the OTSU segmentation method is used to perform foreground segmentation on the sample watermark image based on the bimodal distribution of the watermark pixels and the image background pixels to obtain the sample foreground image.

[0088] See also Figure 3 In some embodiments, step S202 may include but is not limited to steps S301 to S304:

[0089] Step S301, performing Fourier transform on the sample foreground image to obtain sample frequency domain data;

[0090] Step S302, performing intermediate frequency screening on the sample frequency domain data to obtain intermediate frequency domain data;

[0091] Step S303, performing period screening on the intermediate frequency domain data to obtain periodic frequency domain data;

[0092] Step S304: extract watermark from periodic frequency domain data to obtain watermark coordinate data.

[0093] In steps S301 to S304 shown in the embodiment of the present application, sample frequency domain data is obtained by performing Fourier transform on the sample foreground image, and then intermediate frequency filtering is performed on the sample frequency domain data to obtain intermediate frequency frequency domain data, and then periodic filtering is performed on the intermediate frequency frequency domain data to obtain periodic frequency domain data, and finally watermark extraction is performed on the periodic frequency domain data to obtain watermark coordinate data, thereby accurately extracting the watermark from the frequency domain space.

[0094] In step S301 of some embodiments, the Fourier transform is to convert the spatial domain into a superposition of a series of sine and cosine functions of different frequencies to obtain a frequency domain space representation. In one embodiment, the sample foreground image is Fourier transformed based on a preset Fourier function to obtain sample frequency domain data.

[0095] In step S302 of some embodiments, after the image is converted into the frequency domain space, the watermark is usually concentrated in the intermediate frequency domain. The sample frequency domain is screened by a preset low-frequency threshold and a preset high-frequency threshold to obtain data in the sample frequency domain that is higher than the low-frequency threshold and the low-frequency threshold, and obtain intermediate frequency domain data.

[0096] It should be noted that the present application does not make any specific limitations on the low-frequency threshold and the high-frequency threshold, and they may be set to other values ​​in other embodiments.

[0097] In step S303 of some embodiments, the watermark information is distributed in a symmetrical structure in the spatial domain, i.e., the sample foreground image. After being converted into the frequency domain, the intermediate frequency domain data is screened by a preset periodicity threshold range. First, the periodicity of the intermediate frequency domain data is calculated, and then the intermediate frequency domain data with a periodicity within the periodicity threshold range is obtained to obtain the periodic frequency domain data.

[0098] See also Figure 4 In some embodiments, step S304 may include but is not limited to steps S401 to S403:

[0099] Step S401, performing inverse Fourier transform on the periodic frequency domain data to obtain an initial watermark image;

[0100] Step S402, adaptively binarizing the initial watermark image to obtain a target watermark image;

[0101] Step S403: extract the watermark from the target watermark image to obtain watermark coordinate data.

[0102] In steps S401 to S403 shown in the embodiment of the present application, the target frequency domain data is obtained by amplitude amplification of the periodic frequency domain data, and then the target frequency domain data is inverse Fourier transformed to obtain an initial watermark image, thereby enhancing the performance of the watermark in the spatial domain, that is, enhancing the pixel value of the watermark information on the picture, and then the initial watermark image is adaptively binarized to obtain a target watermark image, and then the target watermark image in the spatial domain is further watermarked. Binarization is performed to obtain an accurate watermark image, that is, the target watermark image, and finally the target watermark image is watermarked to obtain watermark coordinate data, thereby realizing accurate extraction of the coordinate data of the watermark in the spatial domain, that is, in the picture.

[0103] In step S401 of some embodiments, the inverse Fourier transform is to convert the frequency domain data into the spatial domain data. In one embodiment, the target frequency domain data is inversely transformed by a preset inverse Fourier function, thereby converting the frequency domain data into an image in the spatial domain, i.e., the initial watermark image.

[0104] In step S402 of some embodiments, the initial watermark image includes a watermark image and a small amount of watermark images. Since the pixel values ​​of the watermark part and the non-watermark part in the initial watermark image are bimodal, adaptive binarization is performed through OTSU, and the image is further segmented once, so as to filter out the watermark image and obtain the target watermark image.

[0105] In step S403 of some embodiments, after adaptive binarization, the target watermark image includes a watermark image and a small amount of non-watermark images, and in general, the grayscale value of the watermark image is higher than the grayscale value of the non-watermark image. After adaptive binarization, the watermark image appears as a black image and the non-watermark image appears as a white image. Therefore, the target watermark image is screened based on the preset pixel value, and the coordinate values ​​of the pixel points where the pixel values ​​are lower than the preset pixel values ​​are obtained to obtain the watermark coordinate data.

[0106] See also Figure 5 In some embodiments, step S203 includes but is not limited to steps S501 to S503:

[0107] Step S501, setting pixels of the sample foreground image to zero according to the watermark coordinate data to obtain an element set image;

[0108] Step S502, performing element segmentation on the element set image to obtain at least one image element;

[0109] Step S503, performing a frame selection on each image element to obtain a watermark area frame selection.

[0110] In steps S501 to S503 shown in the embodiment of the present application, pixels of the sample foreground image are set to zero according to the watermark coordinate data to obtain an element set image, and then the element set image is segmented to obtain at least one image element. Finally, each image element is framed to select the watermark area accurately to obtain a watermark area frame selection map.

[0111] In step S501 of some embodiments, pixel zeroing is to set the value of the pixel point corresponding to the watermark coordinate data in the sample foreground image to 0, thereby converting the sample foreground image into an image including only image elements, that is, an element set image.

[0112] In step S502 of some embodiments, element segmentation refers to segmenting each element in the element set image separately to form an element image, specifically, firstly calculating the connected regions of the element set image to obtain at least one connected region, and each connected region is an image element.

[0113] In step S503 of some embodiments, the frame selection is to perform a selection operation of enclosing the image according to a preset pattern, so as to enclose the image element in a corresponding frame to form a visual frame selection of the image element. In one embodiment, the minimum enclosing rectangle is calculated for each image element to obtain a watermark area frame selection.

[0114] In step S103 of some embodiments, element category recognition is to classify each image element contained in the image into a specific category, that is, to perform element recognition on each image element to obtain the element category of each element. In one embodiment, the image element is firstly calculated in terms of area, edge density and aspect ratio. Then, the image element is classified based on area, edge density and aspect ratio by using edge density threshold, as shown in formula (1):

[0115]

[0116] Among them, C(C k ) is the element category, F is the formula category, I is the picture category, T is the text category, E k is the edge density, R k is the aspect ratio, A k is the area, A min is the preset minimum area threshold of the text category, E th1 is the edge density threshold of the preset formula category, E th3 The preset edge density threshold of the image category, E th5 is the edge density threshold of the preset text category, A th2 is the area threshold of the preset image category, A th4is the preset area threshold of the text category, δ1 is the preset formula aspect ratio threshold, and δ2 is the preset text aspect ratio threshold.

[0117] Among them, for the classification of formulas, the edge density of formula elements is high and the aspect ratio is close to 1. For the classification of pictures, the area of ​​picture elements is large and the edge density is low. For the classification of texts, the area of ​​text elements is moderate, the edge density is high, and the aspect ratio is large.

[0118] It should be noted that this application does not make any specific limitations on the specific threshold value, and other threshold values ​​may be set in other scenarios.

[0119] See also Figure 6 In some embodiments, the element categories include formula elements, image elements, and text elements. Step S104 includes but is not limited to steps S601 to S604:

[0120] Step S601, performing formula image restoration on the watermark area frame selection image of the formula element based on a preset image restoration model to obtain a formula element restoration image;

[0121] Step S602, performing image restoration on the watermark region frame selection image of the image element based on a preset image restoration model to obtain an image element restoration image;

[0122] Step S603, performing text restoration on the watermark area frame selection image of the text element based on a preset image restoration model to obtain a text element restoration image;

[0123] Step S604, performing image replacement on the sample watermark image according to the formula element restored image, the image element restored image, and the text element restored image to obtain a predicted original image.

[0124] In steps S601 to S604 shown in the embodiment of the present application, based on a preset image restoration model, image restoration is performed on formula elements, image elements and text elements respectively, and formula element restoration images, image element restoration images and text element restoration images are obtained respectively. Finally, the sample watermark image is replaced according to the formula element restoration image, image element restoration image and text element restoration image to obtain a predicted original image, thereby achieving effective restoration of the three main elements of formula, image and text respectively. The sample watermark image is replaced based on these restored element images to restore a predicted original image close to the sample original image. Compared with the method of performing an overall rough restoration, the watermark is eliminated more accurately.

[0125] See also Figure 7 In some embodiments, step S601 may include but is not limited to steps S701 to S704:

[0126] Step S701, performing character recognition on the watermark area frame selection image of the formula element to obtain initial formula characters;

[0127] Step S702, constructing graph structure data for the initial formula characters to obtain formula graph structure data;

[0128] Step S703, restoring the formula characters of the formula graph structure data based on a preset image restoration model to obtain target formula characters;

[0129] Step S704, constructing a formula image according to the target formula characters to obtain a formula element restoration image.

[0130] In steps S701 to S704 shown in the embodiment of the present application, character recognition is performed on the watermark area frame selection image of the formula element to obtain the initial formula characters, and then graph structure data is constructed for the initial formula characters to obtain the formula graph structure data, and then formula characters are restored on the formula graph structure data based on a preset image recovery model to obtain the target formula characters, and finally a formula image is constructed based on the target formula characters to obtain a formula element recovery image, thereby realizing graph structure data construction after the initial formula characters are extracted, fully considering the logical relationship between the characters of the formula, the typesetting structure and other factors, avoiding treating the formula as a simple list of isolated characters, and instead sorting it out according to its original mathematical or logical structure, thereby ensuring the structural integrity of the formula during the recovery process and achieving accurate formula recovery.

[0131] In step S701 of some embodiments, character recognition is to convert text characters in an image into an editable text format. In one embodiment, a pre-trained OCR character recognition model is used to perform character recognition on a watermark area frame selection image of a formula element to obtain initial formula characters.

[0132] In step S702 of some embodiments, a graph data structure is a data structure including nodes and edges. When constructing graph structure data for initial formula characters, the initial formula characters are first combined to form multiple variables. For example, in "Area_1=Area_2+Area_3", the characters "Area_1" are used as a variable, "Area_2" is used as a variable, and "Area_3" is used as a variable. The variables are then abstracted into nodes, and the mathematical symbols in the initial formula characters are abstracted into edges between nodes to form formula graph structure data.

[0133] In step S703 of some embodiments, formula character recovery is a method of repairing and restoring character information in the formula, which is used to restore defective or partially defective formula characters to an accurate, complete and clearly legible state. In one embodiment, based on a pre-trained GNN network as a preset image recovery model, the formula graph structure data is input into the pre-trained GNN network to obtain the target formula characters.

[0134] In step S704 of some embodiments, the formula image is constructed by converting the formula image into a mathematical formula, and then forming the mathematical formula into a mathematical image of the same size as the watermark area frame selection image of the formula element to obtain a formula element restored image.

[0135] In step S602 of some embodiments, image restoration refers to the process of restoring the watermark region frame image of the image element to obtain the restored image of the image element. In one embodiment, the watermark region frame image of the image element is restored based on a pre-trained conditional diffusion model, the conditional diffusion model includes a combination of an image mapping network and a Unet network, the watermark region frame image of the image element is input into the image mapping network to obtain the image conditional diffusion feature, and then the image conditional diffusion feature is input into the Unet network to obtain the image element restored image.

[0136] In step S603 of some embodiments, text recovery refers to the process of restoring the watermark area frame image of the text element to obtain a restored image of the text element. In one embodiment, text recovery is performed on the watermark area frame image of the text element based on a pre-trained BERT network. First, OCR character recognition is performed on the watermark area frame image of the text element through a pre-trained OCR model to obtain text characters, and then the text characters are input into the pre-trained BERT network to obtain restored text, and then the restored text is formed into a picture with a size consistent with the size of the watermark area frame image of the text element to obtain a restored image of the text element.

[0137] In step S604 of some embodiments, the pixel points of the watermark area frame selection image of the corresponding formula element are replaced according to the formula element recovery image, and the pixel points of the watermark area frame selection image of the corresponding image element are replaced according to the image element recovery image, and the pixel points of the watermark area frame selection image of the corresponding text element are replaced according to the text element recovery text, so as to obtain the final predicted original image.

[0138] In step S105 of some embodiments, a training loss value is obtained by calculating the mean square error between the sample original image and the predicted original image, and then the parameters of the preset image restoration model are optimized based on the training loss value to obtain a target image restoration model.

[0139] In step S106 of some embodiments, the target watermark image is an image that needs to have the watermark removed. The target watermark image is composed of an image formed by covering a watermark on an image that has not been added. After obtaining the target watermark image, the target watermark image is firstly restored in an area to obtain a target watermark frame image. Then, the image elements of the target watermark frame image are identified in element categories to obtain a target element category. Then, the target watermark image, the target element category, and the image elements are input into a target image restoration model to obtain a target restored image.

[0140] See also Figure 8 The embodiment of the present application also provides an image watermark removal device, which can implement the above-mentioned image watermark removal method, and the device includes:

[0141] The data acquisition module 801 is used to acquire a sample watermark image and a sample original image; wherein the sample watermark image is an image formed by overlaying a watermark on the sample original image.

[0142] The region identification module 802 is used to perform restoration region identification on the sample watermark image to obtain a watermark region frame selection image; wherein the watermark region frame selection image includes at least one type of image element;

[0143] The category recognition module 803 is used to perform element category recognition on the image elements to obtain element categories;

[0144] An image restoration module 804 is used to restore the sample watermark image based on a preset image restoration model, element category and watermark region frame selection map to obtain a predicted original image;

[0145] The model training module 805 is used to train the preset image restoration model according to the sample original image and the predicted original image to obtain the target image restoration model;

[0146] The watermark removal module 806 is used to obtain a target watermark image and remove the watermark from the target watermark image based on a target image restoration model.

[0147] The specific implementation of the image watermark removal device is basically the same as the specific implementation of the above-mentioned image watermark removal method, and will not be repeated here.

[0148] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned image watermark removal method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0149] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0150] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0151] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the image watermark removal method of the embodiment of this application;

[0152] Input / output interface 903, used to implement information input and output;

[0153] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0154] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0155] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0156] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned image watermark removal method is implemented.

[0157] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0158] The image watermark removal method, image watermark removal device, electronic device and storage medium provided in the embodiments of the present application obtain a sample original image and a sample watermark image formed by covering the sample original image with a watermark, then perform restoration area identification on the sample watermark image to obtain a watermark area frame selection image, then perform element category identification on the image elements in the sample watermark image to obtain the element category, then perform image restoration on the sample watermark image based on a preset image restoration model, element category and watermark area frame selection image to obtain a predicted original image, finally train the preset image restoration model based on the sample original image and the predicted original image, thereby obtaining a target image restoration model that can accurately remove the watermark image, and solve the problem of difficult repair in complex textures or high-detail areas by performing refined watermark removal on the element category of the watermark area frame selection image; finally, obtain the target watermark image, perform watermark removal on the target watermark image based on the target image restoration model, and achieve accurate repair of the target watermark image.

[0159] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0160] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0161] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0162] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0163] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0164] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0165] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0166] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0167] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0168] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0169] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for removing image watermark, characterized in that: The method comprises: Acquire a sample watermark image and a sample original image; wherein the sample watermark image is an image formed by overlaying a watermark on the sample original image; Performing restoration area recognition on the sample watermark image to obtain a watermark area frame selection image; wherein the watermark area frame selection image includes at least one category of image elements; Performing element category recognition on the image elements to obtain element categories; Performing image restoration on the sample watermark image based on a preset image restoration model, the element category and the watermark area frame selection image to obtain a predicted original image; Training the preset image restoration model according to the sample original image and the predicted original image to obtain a target image restoration model; A target watermark image is acquired, and watermark removal is performed on the target watermark image based on the target image restoration model.

2. The method according to claim 1, characterized in that The restoring area recognition of the sample watermark image to obtain a watermark area frame selection image includes: Performing foreground segmentation on the sample watermark image to obtain a sample foreground image; Performing periodic texture recognition on the sample foreground image to obtain watermark coordinate data; The sample foreground image is framed according to the watermark coordinate data to obtain the watermark area framed image.

3. The method according to claim 2, characterized in that The step of performing periodic texture recognition on the sample foreground image to obtain watermark coordinate data includes: Performing Fourier transform on the sample foreground image to obtain sample frequency domain data; Performing intermediate frequency screening on the sample frequency domain data to obtain intermediate frequency domain data; Performing period screening on the intermediate frequency domain data to obtain periodic frequency domain data; Watermark extraction is performed on the periodic frequency domain data to obtain the watermark coordinate data.

4. The method according to claim 3, characterized in that The step of extracting the watermark from the periodic frequency domain data to obtain the watermark coordinate data includes: Performing inverse Fourier transform on the periodic frequency domain data to obtain an initial watermark image; Adaptively binarizing the initial watermark image to obtain a target watermark image; The target watermark image is subjected to watermark extraction to obtain the watermark coordinate data.

5. The method according to claim 2, characterized in that: The step of performing image frame selection on the sample foreground image according to the watermark coordinate data to obtain the watermark area frame selection image includes: Zeroing pixels of the sample foreground image according to the watermark coordinate data to obtain an element set image; Performing element segmentation on the element set image to obtain at least one image element; A frame selection image is performed on each of the image elements to obtain the watermark area frame selection image.

6. The method according to any one of claims 1 to 5, characterized in that: The element categories include formula elements, image elements and text elements; the image restoration of the sample watermark image based on the preset image restoration model, the element categories and the watermark area frame selection image to obtain a predicted original image includes: Performing formula image restoration on the watermark area frame selection image of the formula element based on the preset image restoration model to obtain a formula element restoration image; Performing image restoration on the watermark area frame selection image of the image element based on the preset image restoration model to obtain an image element restoration image; Performing text restoration on the watermark area frame selection image of the text element based on the preset image restoration model to obtain a text element restoration image; The sample watermark image is replaced according to the formula element restored image, the image element restored image, and the text element restored image to obtain the predicted original image.

7. The method according to claim 6, characterized in that The performing formula image restoration on the watermark area frame selection image of the formula element based on the preset image restoration model to obtain a formula element restoration image includes: Performing character recognition on the watermark area frame selection image of the formula element to obtain initial formula characters; Constructing graph structure data for the initial formula characters to obtain formula graph structure data; Perform formula character restoration on the formula graph structure data based on the preset image restoration model to obtain target formula characters; A formula image is constructed according to the target formula characters to obtain the formula element restoration image.

8. An image watermark removal device, characterized in that: The device comprises: A data acquisition module is used to acquire a sample watermark image and a sample original image; wherein the sample watermark image is an image formed by overlaying a watermark on the sample original image; A region identification module is used to perform restoration region identification on the sample watermark image to obtain a watermark region frame selection image; wherein the watermark region frame selection image includes at least one category of image elements; A category recognition module, used to perform element category recognition on the image elements to obtain element categories; An image restoration module, used to restore the sample watermark image based on a preset image restoration model, the element category and the watermark area frame selection map to obtain a predicted original image; A model training module, used for training the preset image restoration model according to the sample original image and the predicted original image to obtain a target image restoration model; The watermark removal module is used to obtain a target watermark image and remove the watermark from the target watermark image based on the target image restoration model.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the image watermark removal method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the image watermark removal method according to any one of claims 1 to 7 is implemented.

Images