Watermark identification method and device, electronic equipment and readable storage medium

Abstract

The invention provides a watermark recognition method and device, electronic equipment and a readable storage medium. The method comprises the steps of obtaining a to-be-recognized image; and inputting the to-be-recognized image into a pre-trained watermark recognition model to enable the watermark recognition model to calculate the contribution probability of each image position in the to-be-recognized image to the watermark classification result, determining the output probability of each watermark type included in the to-be-recognized image, and outputting the watermark recognition result in the to-be-recognized image. Therefore, wrong recognition of the watermark in the complex to-be-recognized image can be avoided, and the watermark recognition accuracy of the watermark recognition model can be improved; meanwhile, a plurality of synthesized sample watermark images carrying automatically labeled watermark type labels and watermark mask labels are adopted to train the watermark classification model, so that the problems of training sample richness and few samples are solved, and the accuracy of the trained watermark recognition model on watermark recognition is further improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a watermark recognition method, device, electronic device and readable storage medium. Background Art

[0002] The purpose of visual watermark recognition technology is to accurately identify embedded visual watermarks from complex image backgrounds. In related technologies, classification networks are generally used for recognition. A large number of images containing visual watermarks are labeled as training data, and a classification model is trained based on the training data so that the classification model can learn to distinguish watermarks and non-watermark areas in the image through feature extraction. Finally, the trained classification model is used to identify the watermark in the image to be identified. However, the classification model obtained by the above training has poor effect and low accuracy in identifying watermarks in images when dealing with complex backgrounds, small or inconspicuous watermarks.

[0003] Furthermore, the use of classification networks for watermark recognition in related technologies also relies on large-scale and high-quality annotated data. However, obtaining sufficient annotated data is often time-consuming and difficult to cover all possible watermark types and background situations. Therefore, when the training data is insufficient or unevenly distributed, the performance of the model will be significantly reduced, that is, the model will have a poor effect on watermark recognition in images and low accuracy. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide a watermark recognition method, device, electronic device and readable storage medium. The watermark recognition model obtained by training with the watermark type label and watermark mask label carried by the sample watermark image can analyze the relationship between the image position and the watermark type, pay attention to the position where the watermark may exist when recognizing the image to be recognized, avoid the wrong recognition of the watermark in the complex image to be recognized, and help improve the accuracy of watermark recognition by the watermark recognition model; at the same time, the watermark classification model is trained using multiple sample watermark images that are synthesized and carry automatically annotated watermark type labels and watermark mask labels, which solves the problems of richness of training samples and small number of samples, and further improves the accuracy of watermark recognition by the trained watermark recognition model.

[0005] In a first aspect, an embodiment of the present application provides a watermark recognition method, the watermark recognition method comprising:

[0006] Obtain an image to be recognized;

[0007] Inputting the image to be identified into a pre-trained watermark recognition model, so that the watermark recognition model calculates the contribution probability of each image position in the image to be identified to the watermark classification result, determines the output probability of each watermark type contained in the image to be identified, and outputs the watermark recognition result in the image to be identified;

[0008] The watermark recognition result includes whether the image to be recognized contains a watermark and the type of watermark contained therein;

[0009] The watermark recognition model is obtained based on a sample watermark image set including a plurality of sample watermark images with synthesized automatically annotated watermark type labels and watermark mask labels combined with watermark type attention training.

[0010] In a second aspect, an embodiment of the present application further provides a watermark recognition device, the watermark recognition device comprising:

[0011] An image acquisition module, used for acquiring an image to be identified;

[0012] A watermark recognition module, used for inputting the image to be recognized into a pre-trained watermark recognition model, so that the watermark recognition model calculates the contribution probability of each image position in the image to be recognized to the watermark classification result, determines the output probability of each watermark type contained in the image to be recognized, and outputs the watermark recognition result in the image to be recognized;

[0013] The watermark recognition result includes whether the image to be recognized contains a watermark and the type of watermark contained therein;

[0014] The watermark recognition model is obtained based on a sample watermark image set including a plurality of sample watermark images with synthesized automatically annotated watermark type labels and watermark mask labels combined with watermark type attention training.

[0015] In the third aspect, an embodiment of the present disclosure further provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the storage medium communicate through the bus, and the processor executes the machine-readable instructions to perform the watermark recognition method as described in the first aspect.

[0016] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the watermark recognition method as described in the first aspect is executed.

[0017] The embodiment of the present application provides a watermark recognition method, device, electronic device and readable storage medium, which obtains an image to be recognized; inputs the image to be recognized into a pre-trained watermark recognition model, so that the watermark recognition model calculates the contribution probability of each image position in the image to be recognized to the watermark classification result, determines the output probability of each watermark type contained in the image to be recognized, and outputs the watermark recognition result in the image to be recognized. In this way, the watermark recognition model obtained by training the watermark type label and watermark mask label carried by the sample watermark image can analyze the relationship between the image position and the watermark type, pay attention to the position where the image to be recognized may exist when recognizing the image to be recognized, avoid the wrong recognition of the watermark in the complex image to be recognized, and help improve the accuracy of the watermark recognition model for watermark recognition; at the same time, the watermark classification model is trained using multiple sample watermark images with automatically annotated watermark type labels and watermark mask labels, which solves the problems of richness of training samples and small number of samples, and further improves the accuracy of watermark recognition by the trained watermark recognition model.

[0018] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 A schematic diagram of an image classification model in the related art provided in an embodiment of the present application;

[0021] Figure 2 A flow chart of a watermark recognition method provided by an embodiment of the present application;

[0022] Figure 3 A schematic diagram of the training process of the watermark recognition model provided in the embodiment of the present application;

[0023] Figure 4 One of the structural schematic diagrams of a watermark recognition device provided in an embodiment of the present application;

[0024] Figure 5 The second structural diagram of a watermark recognition device provided in an embodiment of the present application;

[0025] Figure 6A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work belongs to the scope of protection of the present application.

[0027] The background technology of the present disclosure is introduced as follows:

[0028] The purpose of visual watermark recognition technology is to accurately identify embedded visual watermarks from complex image backgrounds. In related technologies, classification networks are generally used for recognition. A large number of images containing visual watermarks are labeled as training data, and a classification model is trained based on the training data so that the classification model can learn to distinguish watermarks and non-watermark areas in the image through feature extraction. Finally, the trained classification model is used to identify the watermark in the image to be identified. However, the classification model obtained by the above training has poor effect and low accuracy in identifying watermarks in images when dealing with complex backgrounds, small or inconspicuous watermarks.

[0029] Furthermore, the use of classification networks for watermark recognition in related technologies also relies on large-scale and high-quality annotated data. However, obtaining sufficient annotated data is often time-consuming and difficult to cover all possible watermark types and background situations. Therefore, when the training data is insufficient or unevenly distributed, the performance of the model will be significantly reduced, that is, the model will have a poor effect on watermark recognition in images and low accuracy.

[0030] For example, see Figure 1 , Figure 1 A schematic diagram of an image classification model in the related art provided in the embodiment of the present application, such as Figure 1As shown in the figure, the image is divided into a series of image blocks (patch), and each image block is passed to the Transformer model as an input Token sequence. The Transformer model stacks multiple Transformer Blocks, uses the self-attention mechanism to capture the global context information in the image, and uses a multi-layer perceptron (MLP) for feature extraction and classification. When extracting features, the model adds a CLS Token to the image block sequence. The token will be used as a unit of sequence information for feature extraction. During the extraction process, the token will interact with other features and integrate the features of other image blocks. Finally, the CLS Token after feature extraction is used for full connection classification. The relevant technology has improved the accuracy of watermark recognition to a certain extent, but it still faces many challenges in practical applications, especially when facing complex backgrounds, small or inconspicuous watermarks, it is often difficult to achieve ideal recognition effects.

[0031] Based on this, an embodiment of the present application provides a watermark recognition method to improve the accuracy of watermark recognition model for watermark recognition.

[0032] See also Figure 2 , Figure 2 This is a flow chart of a watermark recognition method provided by an embodiment of the present application. Figure 2 As shown in, the watermark recognition method provided by the embodiment of the present application includes:

[0033] S201: Obtain an image to be recognized.

[0034] S202. Input the image to be identified into a pre-trained watermark recognition model so that the watermark recognition model calculates the contribution probability of each image position in the image to be identified to the watermark classification result, determines the output probability of each watermark type contained in the image to be identified, and outputs the watermark recognition result in the image to be identified.

[0035] A watermark recognition method provided by the embodiments of the present disclosure can analyze the relationship between the image position and the watermark type by training a watermark recognition model using the watermark type label and the watermark mask label carried by the sample watermark image. When recognizing the image to be recognized, attention is paid to the position where the image to be recognized may exist, thereby avoiding the wrong recognition of the watermark in the complex image to be recognized, and helping to improve the accuracy of the watermark recognition model in watermark recognition. At the same time, a watermark classification model is trained using multiple sample watermark images that are synthesized and carry automatically annotated watermark type labels and watermark mask labels, thereby solving the problems of richness of training samples and small number of samples, and further improving the accuracy of the trained watermark recognition model in watermark recognition.

[0036] The following is a description of the exemplary steps of the embodiment of the present disclosure:

[0037] S201: Obtain an image to be recognized.

[0038] In the embodiment of the present application, the image to be identified is an image that may contain a watermark, and the watermark recognition model provided in the embodiment of the present application is needed to identify whether there is a watermark in the image to be identified and the image type in the image to be identified.

[0039] Among them, the watermark that may be contained in the image to be identified may be a visual watermark. In the related technology, a classification network is generally used for identification. A large number of images containing visual watermarks are labeled as training data, and a classification model is trained based on the training data so that the classification model can learn to distinguish the watermark and non-watermark areas in the image through feature extraction. Finally, the trained classification model is used to identify the watermark in the image to be identified. However, the classification model obtained by the above training has a poor effect on watermark identification in images when dealing with complex backgrounds, small or inconspicuous watermarks, and has low accuracy.

[0040] In a possible implementation, after acquiring the image to be identified, the image to be identified can be input into a trained watermark recognition model. Since the watermark recognition model takes into account the relationship between the watermark type and the image position during training, during the identification process of the image to be identified, it is possible to focus on identifying the output contribution of different positions in the image to be identified to the watermark type classification result, so as to ensure that the watermark type existing in each image position can be accurately checked.

[0041] S202. Input the image to be identified into a pre-trained watermark recognition model so that the watermark recognition model calculates the contribution probability of each image position in the image to be identified to the watermark classification result, determines the output probability of each watermark type contained in the image to be identified, and outputs the watermark recognition result in the image to be identified.

[0042] The watermark recognition result includes whether the image to be recognized contains a watermark and the type of the watermark contained.

[0043] In a possible implementation, the watermark type of the image to be identified represents the watermark style type contained in the image to be identified. The watermark style type may include graphics, patterns, colors, etc. of the watermark. Different watermarks have different watermark style types.

[0044] In one possible implementation, there are multiple multi-head self-attention modules in the watermark recognition model, and the multi-head self-attention modules can be used to perform feature extraction and classification on the image to be recognized, determine the contribution probability of each image position in the image to be recognized to the watermark classification result, and then determine the watermark type contained in the image to be recognized and the output probability of each watermark type, and output the watermark recognition result in the image to be recognized.

[0045] Specifically, the step of "inputting the image to be identified into a pre-trained watermark recognition model so that the watermark recognition model calculates the contribution probability of each image position in the image to be identified to the watermark classification result, determines the output probability of each watermark type contained in the image to be identified, and outputs the watermark recognition result in the image to be identified" includes:

[0046] a1: Input the image to be identified into a pre-trained watermark recognition model so that each multi-head self-attention module uses the self-attention mechanism to extract features of the image to be identified and extract information from the image to be identified.

[0047] a2: classifying the image to be identified after feature extraction according to the watermark type, and calculating the output probability of each watermark type contained in the image to be identified.

[0048] a3: Based on the output probability of each watermark type, output the watermark recognition result in the image to be recognized.

[0049] In a possible implementation, the watermark recognition model can be a Transformer-based image classification network architecture; after the image to be recognized is input into the watermark recognition model, it will be segmented into multiple image blocks according to a preset segmentation ratio, and multiple image block sequences will be obtained. Here, the number of image blocks to be segmented is usually determined by the size of the input image and the preset segmentation ratio.

[0050] Furthermore, through the self-attention mechanism in multiple multi-head self-attention modules in the watermark recognition model, it is determined to capture the global context information in the image to be recognized, and a multi-layer perceptron (MLP) is used for feature extraction and classification.

[0051] In a possible implementation, when extracting features from an image to be identified, the watermark recognition model adds a CLS Token to the image block sequence, and the Token is used as a unit of sequence information for feature extraction. During the extraction process, the Token interacts with other features and integrates the features of other image blocks. Finally, the CLS Token after feature extraction is used for full connection classification to determine whether the image to be identified contains a watermark and at least one type of watermark contained in the image to be identified.

[0052] Furthermore, for the image to be identified, the image to be identified may contain a watermark or not. When the image to be identified contains a watermark, it may contain only one type of watermark, or it may contain watermarks of multiple types at the same time. After feature extraction of the image to be identified, the type of watermark contained in the image to be identified can be determined, and the output probability of each watermark type contained in the image to be identified can be determined through the contribution of each image position (which can be a pixel) in the image to be identified to the watermark classification result, thereby determining the type of watermark contained in the image to be identified.

[0053] In a possible implementation, each watermark type corresponds to an output probability threshold. If the output probability threshold of the watermark type included in the image to be identified is greater than the corresponding output probability threshold, it is determined that the image to be identified contains a watermark of this watermark type. By setting the output probability threshold, it is possible to avoid erroneous identification of the image to be identified containing a watermark of the corresponding watermark type based on the output probability of the existence of the watermark type (even if this output probability is particularly small), which helps to improve the accuracy of watermark identification in the image to be identified.

[0054] Specifically, the step of "outputting the watermark recognition result in the image to be recognized based on the output probability of each watermark type" includes:

[0055] b1: for each watermark type, determine whether the watermark type of the watermark type is greater than the output probability threshold corresponding to the watermark type; if so, determine that the image to be identified contains a watermark of the watermark type.

[0056] In an embodiment of the present application, after determining at least one watermark type contained in the image to be identified and the output probability of each watermark type, it is determined for each watermark type whether the output probability of the watermark type is greater than the output probability threshold corresponding to the watermark type, and the watermark type greater than the corresponding output probability threshold is determined as the watermark type contained in the image to be identified.

[0057] Among them, the output probability corresponding to each watermark type can be set according to the set probability of each watermark type. The output probabilities corresponding to different watermark types can be the same or different. Generally speaking, the output probabilities corresponding to different watermark types are generally different due to different watermark settings and different occurrence situations.

[0058] Exemplarily, the watermark recognition model determines that the image to be recognized contains a watermark, and may contain a watermark of type A watermark, a watermark of type B watermark, and a watermark of type C watermark by performing feature extraction on the image to be recognized, and determines the output probability A of type A watermark, the output probability B of type B watermark, and the output probability C of type C watermark; determines by calculation that the output probability A of type A watermark is greater than a preset output probability threshold A of type A watermark, and the output probability B of type B watermark is less than a preset output probability threshold B of type B watermark; and the output probability C of type C watermark is greater than a preset output probability threshold C of type C watermark. At this time, the output result of the watermark recognition model is that the image to be recognized contains watermarks of type A watermark and type C watermark.

[0059] In a possible implementation, in the related art, watermark recognition is usually completed based on closed training data and classification networks. The training of classification models generally relies on large-scale and high-quality annotated data. In practical applications, obtaining sufficient annotated data is often time-consuming and expensive, and it is difficult to cover all possible watermark types and background situations. Therefore, when the training data is insufficient or unevenly distributed, the performance of the model will be significantly reduced.

[0060] Based on this, in an embodiment of the present application, when training a watermark classification model, a basic watermark foreground image is used to synthesize it with a background image and automatically annotate it, thereby solving the problem of abundant watermark types but less existing annotation data.

[0061] Specifically, the sample watermark image set is determined by the following steps:

[0062] c1: Get multiple watermark foreground images and background images to be synthesized; each watermark foreground image corresponds to a watermark type label.

[0063] c2: Determine multiple watermark positions and the synthesis probability of each watermark position corresponding to each watermark type.

[0064] c3: For each background image, screen a preset number of watermark foreground images of watermark types, and synthesize them with the background image according to the watermark position synthesis probability corresponding to the watermark type to obtain a synthesized watermark image; wherein the synthesized watermark image carries a watermark type label and a watermark mask label.

[0065] c4: The synthesized multiple watermark images and the multiple manually annotated watermark images are collected to obtain the sample watermark image set.

[0066] In an embodiment of the present application, multiple watermark foreground images to be synthesized are obtained, wherein the watermark foreground image includes a watermark image and a transparent background image, and the watermark foreground image is a four-channel image, which includes mask information of the watermark before the three-channel data of the image, indicating the position of the watermark in the foreground image; at the same time, multiple background images are obtained, and the background images can be obtained by searching.

[0067] In a possible implementation, in order to ensure the accuracy of the composite image and the comprehensiveness of the information, after obtaining the watermark foreground image and background image characters, the image needs to be cleaned to ensure that the watermark foreground image and background image of the composite image are images with comprehensive information.

[0068] Specifically, after the step of "obtaining a plurality of watermark foreground images to be synthesized and a plurality of background images", the watermark recognition method further comprises:

[0069] d1: Delete images with non-preset number of channels and images that do not meet the requirements of the synthetic image in the watermark foreground image.

[0070] In the embodiment of the present application, the requirements for the composite image include an image with an unclear background image or an image that is completely black or completely white. The preset number of channels is four channels. The three-channel data of the image also includes watermark mask information. The watermark foreground image that is not a four-channel image and the watermark foreground image and background image that do not meet the requirements of the composite image are deleted. The watermark foreground image and background image that have four channels and meet the requirements of the composite image are retained, which can improve the accuracy of the composite image and the comprehensiveness of the information.

[0071] In a possible implementation, when the watermark foreground image is obtained, the watermark type label of the watermark foreground image is also obtained. For each watermark type, the synthesis probability of each watermark position of each watermark type can be determined by analyzing the distribution pattern of the watermark type in the image, so that the subsequent synthesized image can be synthesized according to the corresponding watermark position synthesis probability.

[0072] The watermark position includes but is not limited to a regular position at the corner or center of the image, a random position, and a tiled mode. Here, the tiled position refers to the same watermark appearing multiple times in one image, and may have a certain rotation angle.

[0073] Furthermore, after determining the watermark position synthesis probability of each watermark type, for each background image, watermark foreground images of a preset number of watermark types are screened and synthesized with the background image according to the watermark position synthesis probability corresponding to the watermark type to obtain a synthesized watermark image.

[0074] Specifically, the step of "selecting a preset number of watermark foreground images of watermark types for each background image, and synthesizing them with the background image according to the watermark position synthesis probability corresponding to the watermark type to obtain a synthesized watermark image" includes:

[0075] e1: For each background image, determine multiple watermark types to be synthesized for the background image.

[0076] e2: For each background image, multiple watermark foreground images containing various watermark types are obtained, and each watermark foreground image is superimposed on the background image according to the watermark position synthesis probability corresponding to the various watermark types. After merging the watermark masks of the same watermark type, the synthetic watermark image corresponding to the background image is obtained.

[0077] In an embodiment of the present application, for each background image, the watermark type to be synthesized into the background image can be screened out according to the watermark type distribution probability. At the same time, according to the watermark position information indicated by the mask label of the watermark foreground image corresponding to the watermark type, and according to the watermark position synthesis probability corresponding to various watermark types, each watermark foreground image is superimposed on the background image to obtain a synthesized watermark image.

[0078] Among them, the watermark type distribution probability can be the occurrence and distribution probability of each watermark type, and can be determined based on the distribution of multiple watermark types in the image. For example, multiple images with visible watermarks (for example, 100 images) can be obtained in advance, and the distribution probability of the watermark type can be determined according to the watermark type contained in each image, and then the watermark type synthesized into the background image is selected according to the distribution probability of the watermark type.

[0079] Specifically, taking 100 synthetic watermark images as an example, if the distribution probability of watermark type A is greater than the distribution probability of watermark type B, then the number of synthetic watermark images containing watermark type A among the 100 synthetic watermark images is greater than the number of synthetic watermark images containing watermark type B.

[0080] In a possible implementation, each watermark foreground image corresponds to a mask tag, which ensures the position of the watermark of this watermark type in the image. Specifically, in the field of image processing, a mask is a special image used to specify the area in the image that can be operated (i.e., the watermark position in the embodiment of the present application). The mask is usually a binary image (each pixel of the image takes two specific values) to better distinguish the areas that need to be paid attention to in the image.

[0081] Exemplarily, the pixel value at the watermark position in the watermark foreground image is set to 1, and the pixel value at the non-watermark position in the watermark foreground image is set to 0, and a mask label is generated to identify the position of the watermark in the image through the mask label.

[0082] In a possible implementation, the watermark foreground image is superimposed on the background image according to the set watermark position and the corresponding watermark position synthesis probability. At the same time, in order to ensure the richness of the watermark superposition and obtain a variety of sample watermark images, the watermark foreground image can be processed and then added to the background image.

[0083] Specifically, the step of “superimposing each watermark foreground image onto the background image” includes:

[0084] f1: After pre-processing each watermark foreground image, superimpose it on the background image.

[0085] The preset processing includes at least one of the following methods:

[0086] The watermark ratio in the watermark foreground image is scaled and the transparency of the watermark in the watermark foreground image is adjusted.

[0087] In a possible implementation, before superimposing the watermark foreground image on the background image, the watermark foreground image can be scaled according to the watermark scaling ratio, and / or the transparency of the watermark in the watermark foreground image can be adjusted to obtain a different watermark foreground image, and then the adjusted watermark foreground image can be superimposed on the background image to enhance the richness of the sample watermark image and increase the number of sample watermark atlases.

[0088] The watermark scaling and transparency can be set as required, or can be set differently based on different watermark types. The specific settings are not limited here.

[0089] In a possible implementation, a background image may be superimposed with watermarks of various types, and watermark foreground images of different watermark types may be superimposed on the background image according to the corresponding watermark position synthesis probability; at the same time, in order to enrich the diversity of sample watermark images, the watermark foreground image may not be superimposed on the background image, so as to serve as an image sample that does not contain a watermark.

[0090] Exemplarily, for a background image, 0-K watermark foreground images of different watermark types may be randomly superimposed on the background image at different watermark positions, and mask labels of the same type may be merged, wherein K may generally be set to 2.

[0091] Furthermore, after obtaining a synthetic watermark image synthesized by a watermark foreground image and a background image, when generating a sample watermark atlas, the manually annotated watermark image can be reintroduced to form a sample watermark atlas with rich coverage images, which helps to improve the accuracy and comprehensiveness of subsequent watermark recognition model training.

[0092] In a possible implementation, after obtaining the synthetic watermark image, the synthetic watermark image may be randomly enhanced to obtain more sample images and enrich the sample watermark image set.

[0093] Specifically, after the step of "selecting a preset number of watermark foreground images of watermark types for each background image, and synthesizing them with the background image according to the watermark position synthesis probability corresponding to the watermark type to obtain a synthesized watermark image", the watermark recognition method further includes:

[0094] g1: performing image enhancement processing on the synthetic watermark image to obtain an enhanced synthetic watermark image, and adding the enhanced synthetic watermark image to the sample watermark image set.

[0095] Wherein, the image enhancement processing includes at least one of the following enhancement methods;

[0096] The synthesized watermark image is scaled, the watermark image is transformed, the watermark image is rotated, and the synthesized watermark image is compressed according to a preset image format.

[0097] In an embodiment of the present application, when enhancing the synthetic watermark image, the mask (mask image) of the synthetic watermark image can also be enhanced. Specifically, the enhancement method for the synthetic watermark image includes scaling and / or transforming and / or rotating the synthetic watermark image and the mask image according to random scaling ratios, affine transformations and rotation matrices; or compressing the watermark image and the mask image in image format according to random compression quality.

[0098] In order to ensure the richness of the sample image set, the specific scaling, transformation and rotation parameters of the synthetic watermark image are not completely restricted, and can be randomly transformed according to needs; further, the compression format of the synthetic watermark image can be JPEG compression of the synthetic watermark image.

[0099] Furthermore, after obtaining a rich set of sample watermark images, the watermark recognition model can be trained through the sample watermark image set. The training process of the watermark recognition model is divided into two stages: pre-training and model fine-tuning. Through the two-stage training, a recognition model with high robustness, generalization and recognition accuracy can be obtained. The training process of the watermark recognition model will be specifically explained below.

[0100] Specifically, the watermark recognition model is trained through the following steps:

[0101] h1: Input the sample watermark images in the sample watermark image set and the watermark type label and watermark mask label corresponding to each sample watermark image into a pre-constructed image classification network architecture, so that the image classification network architecture learns the corresponding relationship between the watermark type and the position in the sample watermark image based on the watermark type label and the watermark mask label. When the first loss of the image classification network architecture is less than a first preset threshold, it is determined that the training of the image classification network architecture is completed, and an initial watermark classification model is obtained.

[0102] h2: Input the processed sample watermark images in the sample watermark image set and the watermark type label and watermark mask label corresponding to each sample watermark image into the trained initial watermark classification model, so that the initial watermark classification model can learn the difference between the image area with watermark in the processed sample watermark image and the watermark position in the generated attention image based on the introduced type attention mechanism. When the second loss of the initial watermark classification model is less than the second preset threshold, it is determined that the training of the initial watermark classification model is completed, and the watermark classification model is obtained.

[0103] In an embodiment of the present application, a Transformer-based image classification network architecture may be used for training. In the pre-training stage for the watermark recognition model, the image classification network architecture is supervised and trained by the watermark type label and the watermark mask label corresponding to each sample watermark image in the sample watermark image. When the first loss value of the image classification network architecture is less than the first preset threshold, it is determined that the training of the image classification network architecture is completed, and the initial watermark classification model is obtained.

[0104] In a possible implementation, the sample watermark image may contain multiple watermarks of different watermark types, so the training loss uses a multi-label classification loss, that is, a binary cross entropy loss to determine the first loss of the image classification network architecture.

[0105] Specifically, the Sigmoid function can be used to convert the model output into a probability value for each watermark type, and then the binary cross entropy loss is calculated based on the watermark type label.

[0106] Specifically, the first loss is determined by the following steps:

[0107] i1: Determine at least one watermark type contained in each sample watermark image and the predicted probability of each watermark type, and determine the image loss value corresponding to each sample watermark image.

[0108] i2: Determine the average of the image loss values ​​corresponding to the multiple sample watermark images as the first loss.

[0109] In an embodiment of the present application, the image loss value corresponding to each sample watermark image can be determined for at least one watermark type contained in each sample watermark image and the predicted probability of each watermark type, and the image loss values ​​corresponding to multiple sample watermark images can be averaged to determine the first loss value.

[0110] Specifically, the first loss can be calculated by the following formula:

[0111]

[0112] Where L1 is the first loss; N is the number of sample watermark images; C is the number of watermark types; y ic is the c-th watermark type label of sample watermark image i, with a value of 0 or 1; p ic is the predicted probability that the sample watermark image i belongs to the watermark type c.

[0113] Furthermore, after the pre-built image classification network architecture is trained with a sample watermark image set to obtain an initial watermark classification model, the initial watermark classification model can be fine-tuned. In the fine-tuning stage, the type attention of the watermark area is further introduced into the training process of the watermark recognition model. The level mask information of the synthetic watermark graphic is used to prompt the watermark recognition model to pay attention to the watermark area in the image, thereby improving the watermark recognition model's ability to discriminate watermarks.

[0114] Specifically, the step of "inputting the processed sample watermark images in the sample watermark image set and the watermark type label and the watermark mask label corresponding to each sample watermark image into the trained initial watermark classification model, so that the initial watermark classification model learns the difference between the image area with watermark in the processed sample watermark image and the watermark position in the generated attention image based on the introduced type attention mechanism, and when the second loss of the initial watermark classification model is less than the second preset threshold, determining that the training of the initial watermark classification model is completed, and obtaining the watermark classification model" includes:

[0115] j1: Input the processed sample watermark images in the sample watermark image set and the watermark type label and watermark mask label corresponding to each sample watermark image into the trained initial watermark classification model, so that each multi-head self-attention module in the initial watermark classification model processes each sample watermark image to obtain at least one attention matrix.

[0116] j2: Process the at least one attention matrix to obtain at least one control attention image; the control attention image is an image that is independent of the watermark type; each position in the control attention image represents the contribution rate of the position to the watermark classification result.

[0117] j3: Adjust the attention average of each multi-head self-attention module based on the control attention image.

[0118] j4: Perform gradient calculation on the at least one attention matrix based on each multi-head self-attention module that adjusts the attention average value to obtain at least one sample attention image.

[0119] j5: Fuse at least one sample attention image obtained to determine the sample fused attention image.

[0120] j6: Based on the degree of overlap between the target area in each sample fusion attention image and the watermark position area in the processed sample watermark image, the difference between the image area with watermark in the processed sample watermark image and the watermark position in the generated attention image is learned. When the second loss of the initial watermark classification model is less than the second preset threshold, it is determined that the training of the initial watermark classification model is completed, and the watermark classification model is obtained.

[0121] In the embodiment of the present application, the initial watermark recognition model is a Transformer network, which is composed of multiple continuous Transformer blocks, each of which is composed of a multi-head self-attention module and a feed-forward connection. First, for each multi-head self-attention module, A i ∈R (n+1)×(n+1)The attention matrix represents the attention coefficient between any two tokens encoded by each head, n is the number of input patches, where n = h × w; further, an additional learnable CLStoken is included; A i [0,1:] is represented by Reshape it into 2D shape A CLS ∈R h×w , we can get a type-independent control attention image, where each position on the image represents its contribution to the classification. Here i represents the i-th self-attention module. Then, use Represents the average of the attentions of all heads in the multi-head attention.

[0122] Furthermore, by calculating A i Gradient Get the sample attention image related to the type. Specifically, use the chain rule to backpropagate the corresponding type score y for each watermark type c c , for the i-th attention block, get the attention matrix A i The corresponding gradient map and extract Gradient map relative to watermark type c Among them, each image position represents the contribution of this token to the classification output of watermark type c; and After element-by-element multiplication and using the ReLU function, the output is the sample attention image related to the type. Sample attention images of all types for all multi-head self-attention modules transformer Perform cumulative fusion to obtain F C , and for F C The values ​​are normalized to the range of 0-1 and reshaped to the initial watermark recognition model input image size H×W to obtain the final sample fusion attention image of each watermark type

[0123] In one possible implementation, a region with higher activation in the image contributes more to the output predicted watermark classification result, and the spatial relationship (e.g., degree of overlap) between the highly activated region and the true watermark region can be used to measure the degree of attention paid by the model to the watermark region.

[0124] Each sample watermark image obtained by data synthesis includes an image-level watermark type label and a pixel-level mask annotation for each watermark type. Each watermark type mask in the sample needs to be processed to obtain a processed sample watermark image.

[0125] Specifically, the processed sample watermark image is obtained through the following steps:

[0126] k1: performing convolution calculation on the sample watermark image based on a preset convolution kernel, and performing calculation on the convolution kernel and the mask information in the sample watermark image to obtain the sample watermark image after dilation processing.

[0127] In the embodiment of the present application, it is necessary to perform image expansion processing on the sample watermark image. Specifically, the sample watermark image can be subjected to convolution calculation, and each pixel point in the sample watermark image is subjected to an "AND" operation using the convolution kernel element and the mask element. If both are 0, then the target pixel point is 0, otherwise it is 1. For each sample watermark image, the watermark mask M after the watermark mask of each watermark type is expanded is obtained. C .

[0128] Furthermore, the second loss when fine-tuning the initial watermark recognition model can be calculated by using the attention image of each watermark type and the expanded watermark mask (including the position information in the sample image), and when the second loss of the initial watermark classification model is less than a second preset threshold, it is determined that the training of the initial watermark classification model is completed, and a watermark classification model is obtained.

[0129] Specifically, the second loss is determined by the following steps:

[0130] l1: Determine the attention loss between the processed sample watermark image and the sample fused attention image.

[0131] l2: Perform weighted calculation on the attention loss and the first loss to determine the second loss.

[0132] The calculation method of the first loss is the same as the previous calculation method, which will not be repeated here.

[0133] Specifically, the second loss can be calculated by the following formula:

[0134] L2 = βL1 + (1-β)La;

[0135] Among them, L2 is the second loss, L1 is the first loss; L a is the attention loss; β is the preset weight coefficient.

[0136] In a possible implementation, when fine-tuning the initial watermark recognition model, an attention loss L is introduced a To calculate the difference between the high activation area in the type attention map and the watermark area after the corresponding sample watermark is pushed to the expansion, it helps to prompt the model to pay attention to the watermark area in the sample watermark image during the training process and improve the model's ability to discriminate watermarks.

[0137] Specifically, the attention loss is determined by the following steps:

[0138] m1: Determines the pixel-by-pixel binary cross entropy loss between the processed sample watermark image and the sample fused attention image.

[0139] m2: Determine the overlap loss between the processed sample watermark image and the sample fused attention image.

[0140] m3: Perform weighted calculation on the pixel-by-pixel binary cross entropy loss and the overlap loss to determine the attention loss.

[0141] Here, the pixel-by-pixel binary cross entropy loss requires calculating the binary cross entropy loss for each pixel and then averaging the losses of all pixels. The calculation formula is as follows:

[0142]

[0143] Among them, L pixel is the pixel-by-pixel binary cross entropy loss; N is the number of sample watermark images with mask annotations; M c (i, j) is the value of position (i, j) on the sample watermark image; is the value of position (i, j) on the sample fusion attention image, and S represents the number of pixels of each sample.

[0144] Specifically, the attention loss can be calculated by the following formula:

[0145] L a =L pixel +γL IOU ;

[0146] Among them, L a L is the loss of attention. pixel is the pixel-by-pixel binary cross entropy loss; L IOU is the overlap loss; γ is a preset weight system used to balance the pixel-by-pixel binary cross entropy loss and the overlap loss.

[0147] Furthermore, the overlap loss can be calculated by the following formula:

[0148]

[0149] Among them, L IOU is the overlap loss; N is the number of sample watermark images with mask annotation; is the value of position (i, j) on the sample fusion attention image; M c (i, j) is the value of position (i, j) on the sample watermark image; ε is a preset coefficient greater than zero, which is used to prevent the denominator in the formula from being 0.

[0150] In a possible implementation, the method for determining the completion of the training of the initial watermark classification model or the watermark recognition model may be a preset number of model training cycles in addition to comparing the first loss or the second loss with a preset loss threshold. If it is determined that the number of model training cycles of the initial watermark classification model or the watermark recognition model is greater than the preset cycle number threshold, it can also be determined that the training of the initial watermark classification model or the watermark recognition model is completed.

[0151] The following is an example to illustrate the training process of the watermark recognition model in the embodiment of the present disclosure:

[0152] See also Figure 3 , Figure 3 A schematic diagram of the training process of the watermark recognition model provided in the embodiment of the present application is shown in FIG. Figure 3 As shown in the figure, a large amount of data is automatically generated through data synthesis using the basic watermark image and its own mask information, including the synthesized image and its corresponding image-level type annotation and pixel-level mask annotation. These synthetic data are combined with other manually annotated data as the training set of the watermark classification model; in the pre-training stage of the watermark recognition model, the watermark classification model is trained on multi-label classification tasks using the automatically annotated data and manually annotated real data; in the fine-tuning stage of the watermark recognition model, the type attention of the watermark area is further introduced into the training process of the watermark recognition model, and the pixel-level mask information of the synthetic data is used to prompt the model to pay attention to the watermark area, thereby improving the watermark recognition model's ability to discriminate watermarks.

[0153] The embodiment of the present application provides a watermark recognition method, which includes obtaining an image to be recognized; inputting the image to be recognized into a pre-trained watermark recognition model, so that the watermark recognition model calculates the contribution probability of each image position in the image to be recognized to the watermark classification result, determines the output probability of each watermark type contained in the image to be recognized, and outputs the watermark recognition result in the image to be recognized. In this way, the watermark recognition model obtained by training with the watermark type label and watermark mask label carried by the sample watermark image can analyze the relationship between the image position and the watermark type, pay attention to the position where the image to be recognized may exist when recognizing the image to be recognized, avoid the wrong recognition of the watermark in the complex image to be recognized, and help improve the accuracy of the watermark recognition model for watermark recognition; at the same time, the watermark classification model is trained using multiple sample watermark images that are synthesized and carry automatically annotated watermark type labels and watermark mask labels, which solves the problem of richness of training samples and small number of samples, and further improves the accuracy of watermark recognition by the trained watermark recognition model.

[0154] Based on the same inventive concept, a watermark recognition device corresponding to the watermark recognition method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned watermark recognition method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0155] See also Figure 4 , Figure 5 , Figure 4 This is one of the structural schematic diagrams of a watermark recognition device provided in an embodiment of the present application. Figure 5 This is a second structural diagram of a watermark recognition device provided in an embodiment of the present application. Figure 4 As shown in , the watermark recognition device 400 includes:

[0156] An image acquisition module 410 is used to acquire an image to be recognized;

[0157] The watermark recognition module 420 is used to input the image to be recognized into a pre-trained watermark recognition model so that the watermark recognition model calculates the contribution probability of each image position in the image to be recognized to the watermark classification result, determines the output probability of each watermark type contained in the image to be recognized, and outputs the watermark recognition result in the image to be recognized;

[0158] The watermark recognition result includes whether the image to be recognized contains a watermark and the type of watermark contained therein;

[0159] The watermark recognition model is obtained based on a sample watermark image set including a plurality of sample watermark images with synthesized automatically annotated watermark type labels and watermark mask labels combined with watermark type attention training.

[0160] In a possible implementation, the watermark recognition model includes a plurality of multi-head self-attention modules; the watermark recognition module 420 is used to input the image to be recognized into a pre-trained watermark recognition model so that the watermark recognition model calculates the contribution probability of each image position in the image to be recognized to the watermark classification result, determines the output probability of each watermark type contained in the image to be recognized, and outputs the watermark recognition result in the image to be recognized. When the watermark recognition module 420 is used to:

[0161] Input the image to be identified into a pre-trained watermark recognition model, so that each multi-head self-attention module uses the self-attention mechanism to extract features of the image to be identified and extract information from the image to be identified;

[0162] Classifying the image to be identified after feature extraction according to the watermark type, and calculating the output probability of each watermark type contained in the image to be identified;

[0163] Based on the output probability of each watermark type, a watermark recognition result in the image to be recognized is output.

[0164] In a possible implementation manner, when the watermark recognition module 420 is used to output the watermark recognition result in the to-be-recognized image based on the output probability of each watermark type, the watermark recognition module 420 is used to:

[0165] For each watermark type, it is determined whether the watermark type of the watermark type is greater than the output probability threshold corresponding to the watermark type. If so, it is determined that the image to be identified contains a watermark of the watermark type.

[0166] In a possible implementation, Figure 5 As shown, the watermark recognition device 400 further includes a sample acquisition module 430, and the sample acquisition module 430 is used to determine the sample watermark image set through the following steps:

[0167] Obtain multiple watermark foreground images and multiple background images to be synthesized; wherein each watermark foreground image corresponds to a watermark type label;

[0168] Determine multiple watermark positions and the synthesis probability of each watermark position corresponding to each watermark type;

[0169] For each background image, a watermark foreground image of a preset number of watermark types is selected, and the foreground images are synthesized with the background image according to the watermark position synthesis probability corresponding to the watermark type to obtain a synthesized watermark image; wherein the synthesized watermark image carries a watermark type label and a watermark mask label;

[0170] The plurality of synthesized watermark images and the plurality of manually annotated watermark images are combined to obtain the sample watermark image set.

[0171] In a possible implementation manner, when the sample acquisition module 430 is used to screen a preset number of watermark foreground images of watermark types for each background image, and synthesize them with the background image according to the watermark position synthesis probability corresponding to the watermark type to obtain a synthesized watermark image, the sample acquisition module 430 is further used to:

[0172] For each background image, determining a plurality of watermark types to be synthesized for the background image;

[0173] For each background image, multiple watermark foreground images containing various watermark types are obtained, and each watermark foreground image is synthesized according to the watermark position synthesis probability corresponding to the various watermark types. Each watermark foreground image is superimposed on the background image, and after merging the watermark masks of the same watermark type, the synthetic watermark image corresponding to the background image is obtained.

[0174] In a possible implementation manner, when the sample acquisition module 430 is used to superimpose each watermark foreground image onto the background image, the sample acquisition module 430 is used to:

[0175] After each watermark foreground image is processed by a preset method, it is superimposed on the background image;

[0176] The preset processing includes at least one of the following methods:

[0177] The watermark ratio in the watermark foreground image is scaled and the transparency of the watermark in the watermark foreground image is adjusted.

[0178] In a possible implementation manner, the sample acquisition module 430 is further configured to:

[0179] Performing image enhancement processing on the synthetic watermark image to obtain an enhanced synthetic watermark image, and adding the enhanced synthetic watermark image to the sample watermark image set;

[0180] Wherein, the image enhancement processing includes at least one of the following enhancement methods;

[0181] The synthesized watermark image is scaled, the watermark image is transformed, the watermark image is rotated, and the synthesized watermark image is compressed according to a preset image format.

[0182] In a possible implementation manner, the sample acquisition module 430 is further configured to:

[0183] The images with non-preset number of channels and the images not meeting the requirements of the synthetic image are deleted from the watermark foreground image.

[0184] In a possible implementation, Figure 5 As shown, the watermark recognition device 400 further includes a model training module 440, and the model training module 440 is used to train the watermark recognition model through the following steps:

[0185] Inputting the sample watermark images in the sample watermark image set and the watermark type label and the watermark mask label corresponding to each sample watermark image into a pre-constructed image classification network architecture, so that the image classification network architecture learns the corresponding relationship between the watermark type and the position in the sample watermark image based on the watermark type label and the watermark mask label, and when the first loss of the image classification network architecture is less than a first preset threshold, determining that the image classification network architecture training is completed, and obtaining an initial watermark classification model;

[0186] The processed sample watermark images in the sample watermark image set and the watermark type label and the watermark mask label corresponding to each sample watermark image are input into the trained initial watermark classification model, so that the initial watermark classification model learns the difference between the image area with watermark in the processed sample watermark image and the watermark position in the generated attention image based on the introduced type attention mechanism. When the second loss of the initial watermark classification model is less than the second preset threshold, it is determined that the training of the initial watermark classification model is completed, and the watermark classification model is obtained.

[0187] In a possible implementation, the model training module 440 is used to obtain a processed sample watermark image through the following steps:

[0188] The sample watermark image is convolved based on a preset convolution kernel, and the convolution kernel is operated with the mask information in the sample watermark image to obtain the sample watermark image after expansion processing.

[0189] In a possible implementation manner, the model training module 440 is used to input the processed sample watermark images in the sample watermark image set and the watermark type label and the watermark mask label corresponding to each sample watermark image into the trained initial watermark classification model, so that the initial watermark classification model learns the difference between the image area with watermark in the processed sample watermark image and the watermark position in the generated attention image based on the introduced type attention mechanism. When the second loss of the initial watermark classification model is less than the second preset threshold, it is determined that the training of the initial watermark classification model is completed. When the watermark classification model is obtained, the model training module 440 is used to:

[0190] Input the processed sample watermark images in the sample watermark image set and the watermark type label and the watermark mask label corresponding to each sample watermark image into the trained initial watermark classification model, so that each multi-head self-attention module in the initial watermark classification model processes each sample watermark image to obtain at least one attention matrix;

[0191] Processing the at least one attention matrix to obtain at least one comparison attention image; the comparison attention image is an image that is irrelevant to the watermark type; each position in the comparison attention image represents a contribution rate of the position to the watermark classification result;

[0192] Adjusting the attention average of each multi-head self-attention module based on the control attention image;

[0193] Performing gradient calculation on the at least one attention matrix based on each multi-head self-attention module that adjusts the attention average value to obtain at least one sample attention image;

[0194] Fusing at least one obtained sample attention image to determine a sample fused attention image;

[0195] Based on the degree of overlap between the target area in each sample fused attention image and the watermark position area in the processed sample watermark image, the difference between the image area with watermark in the sample watermark image after learning and the watermark position in the generated attention image is learned. When the second loss of the initial watermark classification model is less than a second preset threshold, it is determined that the training of the initial watermark classification model is completed, and a watermark classification model is obtained.

[0196] In a possible implementation, the model training module 440 is used to determine the first loss through the following steps:

[0197] Determine at least one watermark type contained in each sample watermark image and the predicted probability of each watermark type, and determine the image loss value corresponding to each sample watermark image;

[0198] An average value of image loss values ​​corresponding to a plurality of sample watermark images is determined as the first loss.

[0199] In a possible implementation, the model training module 440 is used to determine the second loss through the following steps:

[0200] determining an attention loss between a processed sample watermark image and the sample fused attention image;

[0201] The attention loss and the first loss are weightedly calculated to determine the second loss.

[0202] In a possible implementation, the model training module 440 is used to determine the attention loss through the following steps:

[0203] Determining a pixel-by-pixel binary cross entropy loss between a processed sample watermark image and the sample fused attention image;

[0204] Determining the overlap loss between the processed sample watermark image and the sample fused attention image;

[0205] The pixel-by-pixel binary cross entropy loss and the overlap loss are weightedly calculated to determine the attention loss.

[0206] In a possible implementation manner, the watermark type of the image to be identified represents the type of watermark pattern contained in the image to be identified.

[0207] The embodiment of the present application provides a watermark recognition method, which includes obtaining an image to be recognized; inputting the image to be recognized into a pre-trained watermark recognition model, so that the watermark recognition model calculates the contribution probability of each image position in the image to be recognized to the watermark classification result, determines the output probability of each watermark type contained in the image to be recognized, and outputs the watermark recognition result in the image to be recognized. In this way, the watermark recognition model obtained by training with the watermark type label and watermark mask label carried by the sample watermark image can analyze the relationship between the image position and the watermark type, pay attention to the position where the image to be recognized may exist when recognizing the image to be recognized, avoid the wrong recognition of the watermark in the complex image to be recognized, and help improve the accuracy of the watermark recognition model for watermark recognition; at the same time, the watermark classification model is trained using multiple sample watermark images that are synthesized and carry automatically annotated watermark type labels and watermark mask labels, which solves the problem of richness of training samples and small number of samples, and further improves the accuracy of watermark recognition by the trained watermark recognition model.

[0208] See also Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 6 As shown in , the electronic device 600 includes a processor 610 , a memory 620 and a bus 630 .

[0209] The memory 620 stores machine-readable instructions executable by the processor 610. When the electronic device 600 is running, the processor 610 communicates with the memory 620 via the bus 630, so that the processor 610 executes the following instructions when running:

[0210] Obtain an image to be recognized;

[0211] Inputting the image to be identified into a pre-trained watermark recognition model, so that the watermark recognition model calculates the contribution probability of each image position in the image to be identified to the watermark classification result, determines the output probability of each watermark type contained in the image to be identified, and outputs the watermark recognition result in the image to be identified;

[0212] The watermark recognition result includes whether the image to be recognized contains a watermark and the type of watermark contained therein;

[0213] The watermark recognition model is obtained based on a sample watermark image set including a plurality of sample watermark images with synthesized automatically annotated watermark type labels and watermark mask labels combined with watermark type attention training.

[0214] In an optional implementation, in the instructions executed by the processor 610, the watermark recognition model includes a plurality of multi-head self-attention modules; the inputting the image to be recognized into the pre-trained watermark recognition model so that the watermark recognition model calculates the contribution probability of each image position in the image to be recognized to the watermark classification result, determines the output probability of each watermark type contained in the image to be recognized, and outputs the watermark recognition result in the image to be recognized, including:

[0215] Input the image to be identified into a pre-trained watermark recognition model, so that each multi-head self-attention module uses the self-attention mechanism to extract features of the image to be identified and extract information from the image to be identified;

[0216] Classifying the image to be identified after feature extraction according to the watermark type, and calculating the output probability of each watermark type contained in the image to be identified;

[0217] Based on the output probability of each watermark type, a watermark recognition result in the image to be recognized is output.

[0218] In an optional implementation manner, in the instructions executed by the processor 610, outputting the watermark recognition result in the image to be recognized based on the output probability of each watermark type includes:

[0219] For each watermark type, it is determined whether the watermark type of the watermark type is greater than the output probability threshold corresponding to the watermark type. If so, it is determined that the image to be identified contains a watermark of the watermark type.

[0220] In an optional implementation manner, in the instructions executed by the processor 610, the sample watermark image set is determined by the following steps:

[0221] Obtain multiple watermark foreground images and multiple background images to be synthesized; wherein each watermark foreground image corresponds to a watermark type label;

[0222] Determine multiple watermark positions and the synthesis probability of each watermark position corresponding to each watermark type;

[0223] For each background image, a watermark foreground image of a preset number of watermark types is selected, and the foreground images are synthesized with the background image according to the watermark position synthesis probability corresponding to the watermark type to obtain a synthesized watermark image; wherein the synthesized watermark image carries a watermark type label and a watermark mask label;

[0224] The plurality of synthesized watermark images and the plurality of manually annotated watermark images are combined to obtain the sample watermark image set.

[0225] In an optional implementation manner, in the instructions executed by the processor 610, for each background image, screening a preset number of watermark foreground images of watermark types, and synthesizing them with the background image according to the watermark position synthesis probability corresponding to the watermark type to obtain a synthesized watermark image includes:

[0226] For each background image, determining a plurality of watermark types to be synthesized for the background image;

[0227] For each background image, multiple watermark foreground images containing various watermark types are obtained, and each watermark foreground image is synthesized according to the watermark position synthesis probability corresponding to the various watermark types. Each watermark foreground image is superimposed on the background image, and after merging the watermark masks of the same watermark type, the synthetic watermark image corresponding to the background image is obtained.

[0228] In an optional implementation manner, in the instructions executed by the processor 610, superimposing each watermark foreground image onto the background image includes:

[0229] After each watermark foreground image is processed by a preset method, it is superimposed on the background image;

[0230] The preset processing includes at least one of the following methods:

[0231] The watermark ratio in the watermark foreground image is scaled and the transparency of the watermark in the watermark foreground image is adjusted.

[0232] In an optional implementation manner, the instructions executed by the processor 610 further include:

[0233] Performing image enhancement processing on the synthetic watermark image to obtain an enhanced synthetic watermark image, and adding the enhanced synthetic watermark image to the sample watermark image set;

[0234] Wherein, the image enhancement processing includes at least one of the following enhancement methods;

[0235] The synthesized watermark image is scaled, the watermark image is transformed, the watermark image is rotated, and the synthesized watermark image is compressed according to a preset image format.

[0236] In an optional implementation manner, the instructions executed by the processor 610 further include:

[0237] The images with non-preset number of channels and the images not meeting the requirements of the synthetic image are deleted from the watermark foreground image.

[0238] In an optional implementation manner, in the instructions executed by the processor 610, the watermark recognition model is trained through the following steps:

[0239] Inputting the sample watermark images in the sample watermark image set and the watermark type label and the watermark mask label corresponding to each sample watermark image into a pre-constructed image classification network architecture, so that the image classification network architecture learns the corresponding relationship between the watermark type and the position in the sample watermark image based on the watermark type label and the watermark mask label, and when the first loss of the image classification network architecture is less than a first preset threshold, determining that the image classification network architecture training is completed, and obtaining an initial watermark classification model;

[0240] The processed sample watermark images in the sample watermark image set and the watermark type label and the watermark mask label corresponding to each sample watermark image are input into the trained initial watermark classification model, so that the initial watermark classification model learns the difference between the image area with watermark in the processed sample watermark image and the watermark position in the generated attention image based on the introduced type attention mechanism. When the second loss of the initial watermark classification model is less than the second preset threshold, it is determined that the training of the initial watermark classification model is completed, and the watermark classification model is obtained.

[0241] In an optional implementation manner, in the instructions executed by the processor 610, the processed sample watermark image is obtained through the following steps:

[0242] The sample watermark image is convolved based on a preset convolution kernel, and the convolution kernel is operated with the mask information in the sample watermark image to obtain the sample watermark image after expansion processing.

[0243] In an optional implementation manner, in the instructions executed by the processor 610, the processed sample watermark images in the sample watermark image set and the watermark type label and the watermark mask label corresponding to each sample watermark image are input into the trained initial watermark classification model, so that the initial watermark classification model learns the difference between the image area with the watermark in the processed sample watermark image and the watermark position in the generated attention image based on the introduced type attention mechanism, and when the second loss of the initial watermark classification model is less than the second preset threshold, it is determined that the training of the initial watermark classification model is completed, and the watermark classification model is obtained, including:

[0244] Input the processed sample watermark images in the sample watermark image set and the watermark type label and the watermark mask label corresponding to each sample watermark image into the trained initial watermark classification model, so that each multi-head self-attention module in the initial watermark classification model processes each sample watermark image to obtain at least one attention matrix;

[0245] Processing the at least one attention matrix to obtain at least one comparison attention image; the comparison attention image is an image that is irrelevant to the watermark type; each position in the comparison attention image represents a contribution rate of the position to the watermark classification result;

[0246] Adjusting the attention average of each multi-head self-attention module based on the control attention image;

[0247] Performing gradient calculation on the at least one attention matrix based on each multi-head self-attention module that adjusts the attention average value to obtain at least one sample attention image;

[0248] Fusing at least one obtained sample attention image to determine a sample fused attention image;

[0249] Based on the degree of overlap between the target area in each sample fused attention image and the watermark position area in the processed sample watermark image, the difference between the image area with watermark in the sample watermark image after learning and the watermark position in the generated attention image is learned. When the second loss of the initial watermark classification model is less than a second preset threshold, it is determined that the training of the initial watermark classification model is completed, and a watermark classification model is obtained.

[0250] In an optional implementation manner, in the instruction executed by the processor 610, the first loss is determined by the following steps:

[0251] Determine at least one watermark type contained in each sample watermark image and the predicted probability of each watermark type, and determine the image loss value corresponding to each sample watermark image;

[0252] An average value of image loss values ​​corresponding to a plurality of sample watermark images is determined as the first loss.

[0253] In an optional implementation manner, in the instruction executed by the processor 610, the second loss is determined by the following steps:

[0254] determining an attention loss between a processed sample watermark image and the sample fused attention image;

[0255] The attention loss and the first loss are weightedly calculated to determine the second loss.

[0256] In an optional implementation manner, in the instructions executed by the processor 610, the attention loss is determined by the following steps:

[0257] Determining a pixel-by-pixel binary cross entropy loss between a processed sample watermark image and the sample fused attention image;

[0258] Determining the overlap loss between the processed sample watermark image and the sample fused attention image;

[0259] The pixel-by-pixel binary cross entropy loss and the overlap loss are weightedly calculated to determine the attention loss.

[0260] In an optional implementation manner, the watermark type of the image to be identified represents the type of watermark pattern contained in the image to be identified.

[0261] In the above manner, the watermark recognition model obtained by training the watermark type label and watermark mask label carried by the sample watermark image can analyze the relationship between the image position and the watermark type, pay attention to the position where the image to be recognized may exist when recognizing the image to be recognized, avoid the wrong recognition of the watermark in the complex image to be recognized, and help improve the accuracy of the watermark recognition model for watermark recognition; at the same time, the watermark classification model is trained by using multiple sample watermark images with synthetic watermark type labels and watermark mask labels that are automatically annotated, which solves the problem of richness of training samples and small number of samples, and further improves the accuracy of watermark recognition by the trained watermark recognition model. Further, by setting the output probability threshold, it is possible to avoid the wrong recognition of determining that the image to be recognized contains the watermark of the corresponding watermark type by determining the output probability of the existence of the watermark type (even if the output probability is particularly small), which helps to improve the accuracy of watermark recognition in the image to be recognized. Further, after obtaining the synthetic watermark image, the synthetic watermark image is randomly enhanced to obtain more sample images and enrich the sample watermark image set.

[0262] The present disclosure also provides a computer-readable storage medium, on which a computer program is stored, so that when the computer program is executed by a processor, the following instructions are executed:

[0263] Obtain an image to be recognized;

[0264] Inputting the image to be identified into a pre-trained watermark recognition model, so that the watermark recognition model calculates the contribution probability of each image position in the image to be identified to the watermark classification result, determines the output probability of each watermark type contained in the image to be identified, and outputs the watermark recognition result in the image to be identified;

[0265] The watermark recognition result includes whether the image to be recognized contains a watermark and the type of watermark contained therein;

[0266] The watermark recognition model is obtained based on a sample watermark image set including a plurality of sample watermark images with synthesized automatically annotated watermark type labels and watermark mask labels combined with watermark type attention training.

[0267] In an optional implementation, in the instructions executed by a computer-readable storage medium, the watermark recognition model includes a plurality of multi-head self-attention modules; the inputting the image to be recognized into a pre-trained watermark recognition model so that the watermark recognition model calculates the contribution probability of each image position in the image to be recognized to the watermark classification result, determines the output probability of each watermark type contained in the image to be recognized, and outputs the watermark recognition result in the image to be recognized, including:

[0268] Input the image to be identified into a pre-trained watermark recognition model, so that each multi-head self-attention module uses the self-attention mechanism to extract features of the image to be identified and extract information from the image to be identified;

[0269] Classifying the image to be identified after feature extraction according to the watermark type, and calculating the output probability of each watermark type contained in the image to be identified;

[0270] Based on the output probability of each watermark type, a watermark recognition result in the image to be recognized is output.

[0271] In an optional implementation manner, in the instructions executed by the computer-readable storage medium, outputting the watermark recognition result in the image to be recognized based on the output probability of each watermark type includes:

[0272] For each watermark type, it is determined whether the watermark type of the watermark type is greater than the output probability threshold corresponding to the watermark type. If so, it is determined that the image to be identified contains a watermark of the watermark type.

[0273] In an optional implementation manner, in the instructions executed by the computer-readable storage medium, the sample watermark image set is determined by the following steps:

[0274] Obtain multiple watermark foreground images and multiple background images to be synthesized; wherein each watermark foreground image corresponds to a watermark type label;

[0275] Determine multiple watermark positions and the synthesis probability of each watermark position corresponding to each watermark type;

[0276] For each background image, a watermark foreground image of a preset number of watermark types is selected, and the foreground images are synthesized with the background image according to the watermark position synthesis probability corresponding to the watermark type to obtain a synthesized watermark image; wherein the synthesized watermark image carries a watermark type label and a watermark mask label;

[0277] The plurality of synthesized watermark images and the plurality of manually annotated watermark images are combined to obtain the sample watermark image set.

[0278] In an optional implementation, in the instructions executed by a computer-readable storage medium, for each background image, screening a preset number of watermark foreground images of watermark types, and synthesizing them with the background image according to the watermark position synthesis probability corresponding to the watermark type to obtain a synthesized watermark image includes:

[0279] For each background image, determining a plurality of watermark types to be synthesized for the background image;

[0280] For each background image, multiple watermark foreground images containing various watermark types are obtained, and each watermark foreground image is synthesized according to the watermark position synthesis probability corresponding to the various watermark types. Each watermark foreground image is superimposed on the background image, and after merging the watermark masks of the same watermark type, the synthetic watermark image corresponding to the background image is obtained.

[0281] In an optional implementation manner, in the instructions executed by the computer-readable storage medium, superimposing each watermark foreground image onto the background image includes:

[0282] After each watermark foreground image is processed by a preset method, it is superimposed on the background image;

[0283] The preset processing includes at least one of the following methods:

[0284] The watermark ratio in the watermark foreground image is scaled and the transparency of the watermark in the watermark foreground image is adjusted.

[0285] In an optional implementation manner, the instructions executed by the computer-readable storage medium further include:

[0286] Performing image enhancement processing on the synthetic watermark image to obtain an enhanced synthetic watermark image, and adding the enhanced synthetic watermark image to the sample watermark image set;

[0287] Wherein, the image enhancement processing includes at least one of the following enhancement methods;

[0288] The synthesized watermark image is scaled, the watermark image is transformed, the watermark image is rotated, and the synthesized watermark image is compressed according to a preset image format.

[0289] In an optional implementation manner, the instructions executed by the computer-readable storage medium further include:

[0290] The images with non-preset number of channels and the images not meeting the requirements of the synthetic image are deleted from the watermark foreground image.

[0291] In an optional implementation manner, in the instructions executed by the computer-readable storage medium, the watermark recognition model is trained by the following steps:

[0292] Inputting the sample watermark images in the sample watermark image set and the watermark type label and the watermark mask label corresponding to each sample watermark image into a pre-constructed image classification network architecture, so that the image classification network architecture learns the corresponding relationship between the watermark type and the position in the sample watermark image based on the watermark type label and the watermark mask label, and when the first loss of the image classification network architecture is less than a first preset threshold, determining that the image classification network architecture training is completed, and obtaining an initial watermark classification model;

[0293] The processed sample watermark images in the sample watermark image set and the watermark type label and the watermark mask label corresponding to each sample watermark image are input into the trained initial watermark classification model, so that the initial watermark classification model learns the difference between the image area with watermark in the processed sample watermark image and the watermark position in the generated attention image based on the introduced type attention mechanism. When the second loss of the initial watermark classification model is less than the second preset threshold, it is determined that the training of the initial watermark classification model is completed, and the watermark classification model is obtained.

[0294] In an optional implementation manner, in the instructions executed by the computer-readable storage medium, the processed sample watermark image is obtained by the following steps:

[0295] The sample watermark image is convolved based on a preset convolution kernel, and the convolution kernel is operated with the mask information in the sample watermark image to obtain the sample watermark image after expansion processing.

[0296] In an optional implementation, in the instructions executed by a computer-readable storage medium, the processed sample watermark images in the sample watermark image set and the watermark type label and the watermark mask label corresponding to each sample watermark image are input into the trained initial watermark classification model, so that the initial watermark classification model learns the difference between the image area with the watermark in the processed sample watermark image and the watermark position in the generated attention image based on the introduced type attention mechanism, and when the second loss of the initial watermark classification model is less than the second preset threshold, it is determined that the training of the initial watermark classification model is completed, and the watermark classification model is obtained, including:

[0297] Input the processed sample watermark images in the sample watermark image set and the watermark type label and the watermark mask label corresponding to each sample watermark image into the trained initial watermark classification model, so that each multi-head self-attention module in the initial watermark classification model processes each sample watermark image to obtain at least one attention matrix;

[0298] Processing the at least one attention matrix to obtain at least one comparison attention image; the comparison attention image is an image that is irrelevant to the watermark type; each position in the comparison attention image represents a contribution rate of the position to the watermark classification result;

[0299] Adjusting the attention average of each multi-head self-attention module based on the control attention image;

[0300] Performing gradient calculation on the at least one attention matrix based on each multi-head self-attention module that adjusts the attention average value to obtain at least one sample attention image;

[0301] Fusing at least one obtained sample attention image to determine a sample fused attention image;

[0302] Based on the degree of overlap between the target area in each sample fused attention image and the watermark position area in the processed sample watermark image, the difference between the image area with watermark in the sample watermark image after learning and the watermark position in the generated attention image is learned. When the second loss of the initial watermark classification model is less than a second preset threshold, it is determined that the training of the initial watermark classification model is completed, and a watermark classification model is obtained.

[0303] In an optional implementation manner, in the instructions executed by the computer-readable storage medium, the first loss is determined by the following steps:

[0304] Determine at least one watermark type contained in each sample watermark image and the predicted probability of each watermark type, and determine the image loss value corresponding to each sample watermark image;

[0305] An average value of image loss values ​​corresponding to a plurality of sample watermark images is determined as the first loss.

[0306] In an optional implementation manner, in the instructions executed by the computer-readable storage medium, the second loss is determined by the following steps:

[0307] determining an attention loss between a processed sample watermark image and the sample fused attention image;

[0308] The attention loss and the first loss are weightedly calculated to determine the second loss.

[0309] In an optional implementation, in the instructions executed by a computer-readable storage medium, the attention loss is determined by the following steps:

[0310] Determining a pixel-by-pixel binary cross entropy loss between a processed sample watermark image and the sample fused attention image;

[0311] Determining the overlap loss between the processed sample watermark image and the sample fused attention image;

[0312] The pixel-by-pixel binary cross entropy loss and the overlap loss are weightedly calculated to determine the attention loss.

[0313] In an optional implementation manner, the watermark type of the image to be identified represents the type of watermark pattern contained in the image to be identified.

[0314] In the above manner, the watermark recognition model obtained by training the watermark type label and watermark mask label carried by the sample watermark image can analyze the relationship between the image position and the watermark type, pay attention to the position where the image to be recognized may exist when recognizing the image to be recognized, avoid the wrong recognition of the watermark in the complex image to be recognized, and help improve the accuracy of the watermark recognition model for watermark recognition; at the same time, the watermark classification model is trained by using multiple sample watermark images with synthetic watermark type labels and watermark mask labels that are automatically annotated, which solves the problem of richness of training samples and small number of samples, and further improves the accuracy of watermark recognition by the trained watermark recognition model. Further, by setting the output probability threshold, it is possible to avoid the wrong recognition of determining that the image to be recognized contains the watermark of the corresponding watermark type by determining the output probability of the existence of the watermark type (even if the output probability is particularly small), which helps to improve the accuracy of watermark recognition in the image to be recognized. Further, after obtaining the synthetic watermark image, the synthetic watermark image is randomly enhanced to obtain more sample images and enrich the sample watermark image set.

[0315] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0316] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0317] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0318] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0319] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application can essentially be embodied in the form of a software product, or in other words, the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0320] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A watermark recognition method, characterized in that: The watermark recognition method comprises: Obtain an image to be recognized; Inputting the image to be identified into a pre-trained watermark recognition model, so that the watermark recognition model calculates the contribution probability of each image position in the image to be identified to the watermark classification result, determines the output probability of each watermark type contained in the image to be identified, and outputs the watermark recognition result in the image to be identified; The watermark recognition result includes whether the image to be recognized contains a watermark and the type of watermark contained therein; The watermark recognition model is obtained based on a sample watermark image set including a plurality of sample watermark images with synthesized automatically annotated watermark type labels and watermark mask labels combined with watermark type attention training.

2. The watermark recognition method according to claim 1, characterized in that: The watermark recognition model includes a plurality of multi-head self-attention modules; the image to be recognized is input into a pre-trained watermark recognition model so that the watermark recognition model calculates the contribution probability of each image position in the image to be recognized to the watermark classification result, determines the output probability of each watermark type contained in the image to be recognized, and outputs the watermark recognition result in the image to be recognized, including: Input the image to be identified into a pre-trained watermark recognition model, so that each multi-head self-attention module uses the self-attention mechanism to extract features of the image to be identified and extract information from the image to be identified; Classifying the image to be identified after feature extraction according to the watermark type, and calculating the output probability of each watermark type contained in the image to be identified; Based on the output probability of each watermark type, a watermark recognition result in the image to be recognized is output.

3. The watermark recognition method according to claim 2, characterized in that: The step of outputting the watermark recognition result in the image to be recognized based on the output probability of each watermark type includes: For each watermark type, it is determined whether the watermark type of the watermark type is greater than the output probability threshold corresponding to the watermark type. If so, it is determined that the image to be identified contains a watermark of the watermark type.

4. The watermark recognition method according to claim 1, characterized in that: The sample watermark image set is determined by the following steps: Obtain multiple watermark foreground images and multiple background images to be synthesized; wherein each watermark foreground image corresponds to a watermark type label; Determine multiple watermark positions and the synthesis probability of each watermark position corresponding to each watermark type; For each background image, a watermark foreground image of a preset number of watermark types is selected, and the foreground images are synthesized with the background image according to the watermark position synthesis probability corresponding to the watermark type to obtain a synthesized watermark image; wherein the synthesized watermark image carries a watermark type label and a watermark mask label; The plurality of synthesized watermark images and the plurality of manually annotated watermark images are combined to obtain the sample watermark image set.

5. The watermark recognition method according to claim 4, characterized in that: The method of selecting a watermark foreground image of a preset number of watermark types for each background image and synthesizing the watermark foreground images with the background image according to the watermark position synthesis probability corresponding to the watermark type to obtain a synthesized watermark image includes: For each background image, determining a plurality of watermark types to be synthesized for the background image; For each background image, multiple watermark foreground images containing various watermark types are obtained, and each watermark foreground image is synthesized according to the watermark position synthesis probability corresponding to the various watermark types. Each watermark foreground image is superimposed on the background image, and after merging the watermark masks of the same watermark type, the synthetic watermark image corresponding to the background image is obtained.

6. The watermark recognition method according to claim 5, characterized in that: The step of superimposing each watermark foreground image onto the background image comprises: After each watermark foreground image is processed by a preset method, it is superimposed on the background image; The preset processing includes at least one of the following methods: The watermark ratio in the watermark foreground image is scaled and the transparency of the watermark in the watermark foreground image is adjusted.

7. The watermark recognition method according to claim 4, characterized in that: After selecting a watermark foreground image of a preset number of watermark types for each background image and synthesizing the foreground image with the background image according to the watermark position synthesis probability corresponding to the watermark type to obtain a synthesized watermark image, the watermark recognition method further includes: Performing image enhancement processing on the synthetic watermark image to obtain an enhanced synthetic watermark image, and adding the enhanced synthetic watermark image to the sample watermark image set; Wherein, the image enhancement processing includes at least one of the following enhancement methods; The synthesized watermark image is scaled, the watermark image is transformed, the watermark image is rotated, and the synthesized watermark image is compressed according to a preset image format.

8. The watermark recognition method according to claim 4, characterized in that: After obtaining a plurality of watermark foreground images and a plurality of background images to be synthesized, the watermark recognition method further includes: The images with non-preset number of channels and the images not meeting the requirements of the synthetic image are deleted from the watermark foreground image.

9. The watermark recognition method according to claim 1, characterized in that: The watermark recognition model is trained by the following steps: Inputting the sample watermark images in the sample watermark image set and the watermark type label and the watermark mask label corresponding to each sample watermark image into a pre-constructed image classification network architecture, so that the image classification network architecture learns the corresponding relationship between the watermark type and the position in the sample watermark image based on the watermark type label and the watermark mask label, and when the first loss of the image classification network architecture is less than a first preset threshold, determining that the image classification network architecture training is completed, and obtaining an initial watermark classification model; The processed sample watermark images in the sample watermark image set and the watermark type label and the watermark mask label corresponding to each sample watermark image are input into the trained initial watermark classification model, so that the initial watermark classification model learns the difference between the image area with watermark in the processed sample watermark image and the watermark position in the generated attention image based on the introduced type attention mechanism. When the second loss of the initial watermark classification model is less than the second preset threshold, it is determined that the training of the initial watermark classification model is completed, and the watermark classification model is obtained.

10. The watermark recognition method according to claim 9, characterized in that: The processed sample watermark image is obtained through the following steps: The sample watermark image is convolved based on a preset convolution kernel, and the convolution kernel is operated with the mask information in the sample watermark image to obtain the sample watermark image after expansion processing.

11. The watermark recognition method according to claim 9, characterized in that: The processed sample watermark images in the sample watermark image set and the watermark type label and the watermark mask label corresponding to each sample watermark image are input into the trained initial watermark classification model, so that the initial watermark classification model learns the difference between the image area with watermark in the processed sample watermark image and the watermark position in the generated attention image based on the introduced type attention mechanism, and when the second loss of the initial watermark classification model is less than the second preset threshold, it is determined that the training of the initial watermark classification model is completed, and the watermark classification model is obtained, including: Input the processed sample watermark images in the sample watermark image set and the watermark type label and the watermark mask label corresponding to each sample watermark image into the trained initial watermark classification model, so that each multi-head self-attention module in the initial watermark classification model processes each sample watermark image to obtain at least one attention matrix; Processing the at least one attention matrix to obtain at least one comparison attention image; the comparison attention image is an image that is irrelevant to the watermark type; each position in the comparison attention image represents a contribution rate of the position to the watermark classification result; Adjusting the attention average of each multi-head self-attention module based on the control attention image; Performing gradient calculation on the at least one attention matrix based on each multi-head self-attention module that adjusts the attention average value to obtain at least one sample attention image; Fusing at least one obtained sample attention image to determine a sample fused attention image; Based on the degree of overlap between the target area in each sample fused attention image and the watermark position area in the processed sample watermark image, the difference between the image area with watermark in the sample watermark image after learning and the watermark position in the generated attention image is learned. When the second loss of the initial watermark classification model is less than a second preset threshold, it is determined that the training of the initial watermark classification model is completed, and a watermark classification model is obtained.

12. The watermark recognition method according to claim 9, characterized in that: The first loss is determined by the following steps: Determine at least one watermark type contained in each sample watermark image and the predicted probability of each watermark type, and determine the image loss value corresponding to each sample watermark image; An average value of image loss values ​​corresponding to a plurality of sample watermark images is determined as the first loss.

13. The watermark recognition method according to claim 12, characterized in that: The second loss is determined by the following steps: determining an attention loss between a processed sample watermark image and the sample fused attention image; The attention loss and the first loss are weightedly calculated to determine the second loss.

14. The watermark recognition method according to claim 13, characterized in that: The attention loss is determined by the following steps: Determining a pixel-by-pixel binary cross entropy loss between a processed sample watermark image and the sample fused attention image; Determining the overlap loss between the processed sample watermark image and the sample fused attention image; The pixel-by-pixel binary cross entropy loss and the overlap loss are weightedly calculated to determine the attention loss.

15. The watermark recognition method according to claim 1, characterized in that: The watermark type of the image to be identified indicates the type of watermark pattern contained in the image to be identified.

16. A watermark recognition device, characterized in that: The watermark recognition device comprises: An image acquisition module, used for acquiring an image to be identified; A watermark recognition module, used for inputting the image to be recognized into a pre-trained watermark recognition model, so that the watermark recognition model calculates the contribution probability of each image position in the image to be recognized to the watermark classification result, determines the output probability of each watermark type contained in the image to be recognized, and outputs the watermark recognition result in the image to be recognized; The watermark recognition result includes whether the image to be recognized contains a watermark and the type of watermark contained therein; The watermark recognition model is obtained based on a sample watermark image set including a plurality of sample watermark images with synthesized automatically annotated watermark type labels and watermark mask labels combined with watermark type attention training.

17. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate via the bus, and the processor executes the machine-readable instructions to perform the steps of the watermark recognition method as described in any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the watermark recognition method according to any one of claims 1 to 15.

Images