Image watermark extraction method and device, equipment and storage medium

Abstract

The invention discloses an image watermark extraction method and device, equipment and a storage medium, and belongs to the technical field of image watermarks. The method comprises the steps that a watermark image is acquired, the watermark image is obtained by embedding watermark features corresponding to watermark information in an original image, and the watermark features comprise spatial domain features and frequency domain features; extracting fusion features of the spatial domain features and the frequency domain features based on a feature extraction network; and carrying out watermark extraction on the fusion feature based on a watermark prediction network to obtain the image watermark. According to the method, the matching degree of the feature extraction network and the watermark features including the spatial domain features and the frequency domain features can be improved, so that the accuracy of extracting the image watermark by the feature extraction network is improved.

Description

Technical Field

[0001] The present application relates to the field of image watermark technology, and in particular to an image watermark extraction method, device, equipment and storage medium. Background Art

[0002] Image watermarking technology includes watermark embedding technology and watermark extraction technology. Among them, watermark embedding technology uses the original image as a carrier to embed the watermark information into the original image to obtain a watermark image; watermark extraction technology extracts the watermark information embedded in the watermark image to obtain an image watermark.

[0003] In the related art, the image watermark is obtained by extracting the frequency domain features corresponding to the watermark information in the watermark image using a feature extraction network.

[0004] However, since the watermark embedding process requires conversion between the spatial domain and the frequency domain, that is, the watermark features corresponding to the image watermark include both frequency domain features and spatial domain features. Therefore, when the feature extraction network does not match the watermark features, the accuracy of extracting the image watermark using the feature extraction network is low. Summary of the invention

[0005] The present application provides a method, device, equipment and storage medium for extracting image watermarks. The technical solution is as follows:

[0006] According to one aspect of the present application, a method for extracting an image watermark is provided, the method comprising:

[0007] Acquire a watermark image, wherein the watermark image is obtained by embedding a watermark feature corresponding to the watermark information in the original image, wherein the watermark feature includes a spatial domain feature and a frequency domain feature;

[0008] Extracting fusion features of the spatial domain features and the frequency domain features based on a feature extraction network;

[0009] The watermark is extracted from the fused features based on a watermark prediction network to obtain the image watermark.

[0010] According to another aspect of the present application, a device for extracting an image watermark is provided, the device comprising:

[0011] An acquisition module, used for acquiring a watermark image, wherein the watermark image is obtained by embedding a watermark feature corresponding to the watermark information in the original image, wherein the watermark feature includes a spatial domain feature and a frequency domain feature;

[0012] An extraction module, used for extracting fusion features of the spatial domain features and the frequency domain features based on a feature extraction network;

[0013] The extraction module is also used to extract the watermark from the fusion feature based on a watermark prediction network to obtain the image watermark.

[0014] According to another aspect of the present application, a computer device is provided, comprising: a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the image watermark extraction method as described above.

[0015] According to another aspect of the present application, a computer-readable storage medium is provided, in which at least one program is stored. The at least one program is loaded and executed by a processor to implement the image watermark extraction method as described above.

[0016] According to another aspect of the present application, a computer program product is provided, which includes at least one program segment, and the at least one program segment is stored in a computer-readable storage medium; a processor of a computer device reads the at least one program segment from the computer-readable storage medium, and the processor executes the at least one program segment, so that the computer device executes the image watermark extraction method described in the above aspect.

[0017] The beneficial effects of the technical solution provided by this application include at least:

[0018] By extracting the fusion features of spatial domain features and frequency domain features, the feature extraction network is adapted to the watermark features in the watermark image during feature extraction, thereby improving the matching degree between the feature extraction network and the watermark features including spatial domain features and frequency domain features, thereby improving the accuracy of the feature extraction network in extracting image watermarks. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 is a structural diagram of a hardware device provided by an exemplary embodiment of the present application;

[0021] Figure 2 is a schematic diagram of an image watermark extraction method provided by an exemplary embodiment of the present application;

[0022] Figure 3 is a schematic diagram of an image watermark extraction method provided by an exemplary embodiment of the present application;

[0023] Figure 4 is a flow chart of a method for extracting an image watermark provided by an exemplary embodiment of the present application;

[0024] Figure 5 is a flow chart of a method for extracting an image watermark provided by an exemplary embodiment of the present application;

[0025] Figure 6 is a schematic diagram of an image watermark extraction method provided by an exemplary embodiment of the present application;

[0026] Figure 7 is a flow chart of a method for extracting an image watermark provided by an exemplary embodiment of the present application;

[0027] Figure 8 is a flow chart of a method for extracting an image watermark provided by an exemplary embodiment of the present application;

[0028] Fig. 9 is a flow chart of a method for extracting an image watermark provided by an exemplary embodiment of the present application;

[0029] Fig.10 is a schematic diagram of an image watermark extraction method provided by an exemplary embodiment of the present application;

[0030] Fig.11 is a schematic diagram of an image watermark extraction method provided by an exemplary embodiment of the present application;

[0031] Fig.12 is a flow chart of a method for extracting an image watermark provided by an exemplary embodiment of the present application;

[0032] Fig.13 is a schematic diagram of an image watermark extraction method provided by an exemplary embodiment of the present application;

[0033] Fig.14 is a flow chart of a method for extracting an image watermark provided by an exemplary embodiment of the present application;

[0034] Fig.15 is a schematic diagram of an image watermark extraction method provided by an exemplary embodiment of the present application;

[0035] Fig.16 is a flow chart of a method for extracting an image watermark provided by an exemplary embodiment of the present application;

[0036] Fig.17 is a schematic diagram of an image watermark extraction method provided by an exemplary embodiment of the present application;

[0037] Fig.18is a schematic diagram of an image watermark extraction method provided by an exemplary embodiment of the present application;

[0038] Fig.19 is a schematic diagram of an image watermark extraction method provided by an exemplary embodiment of the present application;

[0039] Fig. 20 is a flow chart of a method for extracting an image watermark provided by an exemplary embodiment of the present application;

[0040] Fig.21 is a schematic diagram of an image watermark extraction method provided by an exemplary embodiment of the present application;

[0041] Fig. 22 is a schematic diagram of an image watermark extraction method provided by an exemplary embodiment of the present application;

[0042] Fig.23 is a schematic diagram of an image watermark extraction method provided by an exemplary embodiment of the present application;

[0043] Fig.24 is a schematic diagram of an image watermark extraction method provided by an exemplary embodiment of the present application;

[0044] Fig.25 is a flow chart of a method for extracting an image watermark provided by an exemplary embodiment of the present application;

[0045] Fig.26 is a schematic diagram of an image watermark extraction method provided by an exemplary embodiment of the present application;

[0046] Fig. 27 is a schematic diagram of an image watermark extraction method provided by an exemplary embodiment of the present application;

[0047] Fig.28 is a schematic diagram of an image watermark extraction method provided by an exemplary embodiment of the present application;

[0048] Fig.29 is a block diagram of an image watermark extraction device provided by an exemplary embodiment of the present application;

[0049] Fig.30 is a block diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0051] In some embodiments, watermark information is embedded in the original image to obtain a watermark image, thereby protecting the copyright of the original image. Figure 1 As shown, watermark information is embedded into the original image by the watermark embedding device 100 to obtain a watermark image including the watermark information. In some embodiments, the watermark embedding device 100 is a terminal device or a server.

[0052] In some embodiments, the watermark information in the watermark image is extracted to obtain the image watermark, thereby determining the copyright of the original image. Figure 1 As shown, the watermark extraction device 200 obtains the watermark image generated by the watermark embedding device 100, and obtains the image watermark corresponding to the watermark information by extracting the watermark information from the watermark image. In some embodiments, the watermark extraction device 200 is a terminal device or a server.

[0053] In some embodiments, the image watermark embedding method and the image watermark extraction method are used in enterprise security protection work. For example, after an enterprise employee opens the original image through the enterprise's internal security terminal, the watermark embedding device 100 embeds the enterprise information in the original image to obtain a watermark image. When it is necessary to determine the copyright of the original image, the watermark image including the enterprise information is input into the watermark extraction device 200, and the image watermark corresponding to the enterprise information is extracted, thereby proving that the copyright of the original image belongs to the enterprise.

[0054] In some embodiments, the terminal device is connected to the server via a wireless network or a wired network.

[0055] The server includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Exemplarily, the server includes a processor and a memory, and the memory includes a receiving module, a control module, and a sending module. The receiving module is used to receive a request sent by a terminal device; the control module is used to control the rendering of the interface; and the sending module is used to send a response to the terminal device. The server is used to provide background services for the terminal device. Optionally, the server undertakes the main computing work and the terminal device undertakes the secondary computing work; or, the server undertakes the secondary computing work and the terminal device undertakes the main computing work; or, the server and the terminal device undertake the computing work in a collaborative manner.

[0056] Optionally, the terminal device includes multiple device types, including at least one of a smart phone, a smart watch, a vehicle-mounted terminal device, a wearable device, a smart TV, a tablet computer, an e-book reader, an MP3 player, an MP4 player, a laptop computer, and a desktop computer. The terminal device is installed with a client, such as a video client, an image production platform client, an information stream client, a browser client, etc.

[0057] Those skilled in the art will appreciate that the number of the terminal devices may be more or less. For example, the terminal device may be only one, or the terminal devices may be dozens or hundreds, or more. The embodiment of the present application does not limit the number and type of terminal devices.

[0058] In the related art, the watermark information is embedded in the original image by transforming the original image between the spatial domain and the frequency domain to obtain the watermark image. Since the watermark image is obtained after the original image is transformed between the spatial domain and the frequency domain, the watermark features corresponding to the watermark image include spatial domain features and frequency domain features.

[0059] In some embodiments, in order to determine the copyright of the original image, it is necessary to extract the watermark information in the watermark image to obtain the image watermark. The image watermark in the watermark image is extracted by using a deep neural network. For example, Figure 2 As shown, by using the feature extraction network 10 and the watermark prediction network 20, the image watermark 2 is extracted from the watermark image 1. The watermark image 1 is input into the feature extraction network 10 to obtain the watermark feature; the watermark feature output by the feature extraction network 10 is input into the watermark prediction network 20 to obtain the image watermark 2.

[0060] In the related art, the watermark features extracted by the feature extraction network 10 include frequency domain features. When the watermark features include spatial domain features and frequency domain features, the features in the watermark embedding process and the features in the watermark extraction process will not match, resulting in low accuracy in extracting the image watermark.

[0061] In order to solve the above problems, an image watermark embedding method is proposed in an embodiment of the present application, which can improve the accuracy of extracting image watermarks. The method is described below.

[0062] In some embodiments, the feature extraction network includes an n-level extraction network, each level of the extraction network in the n-level extraction network includes a convolution extraction unit and a fusion extraction unit, and the spatial domain features and frequency domain features in the watermark image are extracted by the fusion extraction unit in the n-level extraction network to obtain fusion features, and the value of n is a positive integer greater than 1.

[0063] In some embodiments, the watermark prediction network includes a sampling network and an n-level prediction network. The fusion feature is input into the sampling network in the watermark prediction network to obtain the prediction feature of the n-1th scale to the prediction feature of the first scale. The prediction feature of the nth scale is input into the n-level prediction network to obtain the prediction result of the n+1th scale. In some embodiments, the image watermark is predicted based on the n prediction results to obtain the image watermark.

[0064] For example, Figure 3 As shown, the feature extraction network 10 includes a first-level extraction network 23, a second-level extraction network 24 to an n-th-level extraction network 25, and each level of the extraction network includes a convolution extraction unit 11 and a fusion extraction unit 12. The watermark image is input into the feature extraction network 10, and the first-level fusion feature is obtained through the first-level extraction network 23, and the first-level fusion feature is input into the second-level extraction network 24 to obtain the second-level fusion feature, until the n-1-th-level fusion feature is input into the n-th-level extraction network 25 to obtain the n-th-level fusion feature.

[0065] The n-th level fusion feature is input into the watermark prediction network 20. The watermark prediction network 20 includes a sampling network 60 and a first-level prediction network 87, a second-level prediction network 88 to an n-th level prediction network 90. ​​The n-th level fusion feature is input into the sampling network 60 to obtain prediction features of the n-1th scale to the prediction features of the first scale. The prediction features of the first scale are input into the first-level prediction network 87 to obtain the deep and shallow fusion features of the second scale, and the watermark prediction result 83 of the second scale is obtained based on the deep and shallow fusion features of the second scale. The deep and shallow fusion features of the second scale obtained by the first-level prediction network 87 are used as the input of the second-level prediction network 88 to obtain the deep and shallow fusion features of the third scale, and the watermark prediction result 84 of the third scale is obtained based on the deep and shallow fusion features of the third scale. The deep and shallow fusion features of the n-1th level prediction network are used as the input of the n-1th level prediction network 90 to obtain the deep and shallow fusion features of the n+1th scale, and the watermark prediction result 86 of the n+1th scale is obtained based on the deep and shallow fusion features of the n+1th scale. Based on the watermark prediction result 83 of the second scale, the watermark prediction result 84 of the third scale, and the watermark prediction result 86 of the n+1th scale, the image watermark is predicted to obtain the image watermark.

[0066] Figure 4 1 is a flowchart of an image watermark extraction method provided by an exemplary embodiment of the present application. In some embodiments, the method is performed by a computer device. In some embodiments, the computer device is a terminal device. In some embodiments, the computer device is a server. The method includes:

[0067] Step 120: Obtain a watermark image;

[0068] In some embodiments, an original image is obtained, which is an image without watermark information embedded in it. Watermark information is embedded in the original image to obtain a watermark image, which is an image with watermark information embedded in it.

[0069] In some embodiments, the original image is transformed from the spatial domain to the frequency domain to obtain an original spectrum diagram corresponding to the original image. In some embodiments, the original image corresponds to a spatial domain feature in the spatial domain. Exemplarily, assuming that the original image includes a person, the spatial domain feature corresponding to the person includes at least one of an action feature, an expression feature, and a morphological feature. In some embodiments, the spatial domain feature corresponding to the original image is transformed into a frequency domain feature. The spatial domain feature is transformed into a frequency domain feature by a discrete Fourier transform (DFT) or a discrete cosine transform (DCT) or a discrete wavelet transform (DWT). In some embodiments, based on the frequency domain feature, the original spectrum diagram corresponding to the original image is determined.

[0070] In some embodiments, the original spectrum graph includes at least one pixel. The watermark image is obtained by modifying the brightness amplitude of the pixel on the original spectrum graph corresponding to the original image. In some embodiments, the brightness amplitude of the pixel located on the watermark pattern is modified in the original spectrum graph according to the watermark pattern to obtain the embedded spectrum graph. The watermark pattern is used to indicate the position of the pixel in the original spectrum graph whose brightness amplitude needs to be modified. In some embodiments, the embedded spectrum graph is transformed from the frequency domain to the spatial domain to obtain the watermark image.

[0071] Step 140: extracting fusion features of spatial domain features and frequency domain features based on a feature extraction network;

[0072] In some embodiments, the watermark image is input into a feature extraction network to obtain watermark features. The feature extraction network is used to extract watermark features in the watermark image, and the watermark features include fusion features of spatial domain features and frequency domain features, and the watermark features include at least one of features corresponding to the watermark pattern, features corresponding to the watermark position, and features corresponding to the brightness amplitude of the pixel point. The feature extraction network is a neural network used to extract and fuse spatial domain features and frequency domain features through at least one network layer. In some embodiments, spatial domain features refer to image features of the watermark image in the spatial domain. Frequency domain features refer to image features of the watermark image in the frequency domain. In some embodiments, frequency is related to amplitude and phase, and frequency domain features include amplitude features and phase features.

[0073] Step 160: Extract watermarks from the fused features based on the watermark prediction network to obtain image watermarks.

[0074] In some embodiments, the fused features are input into a watermark prediction network to obtain an image watermark, which is predicted based on spatial domain features and frequency domain features. The watermark prediction network is a neural network for predicting and reasoning about image watermarks using the fused features as input through at least one network layer. For example, Figure 2 As shown, the watermark image 1 is input into the feature extraction network 10 to obtain the watermark feature; the watermark feature output by the feature extraction network 10 is input into the watermark prediction network 20 to obtain the image watermark 2.

[0075] To sum up, the method provided in this embodiment, by extracting the fusion features of spatial domain features and frequency domain features, makes the feature extraction network adapt to the watermark features in the watermark image during feature extraction, improves the matching degree between the feature extraction network and the watermark features including spatial domain features and frequency domain features, thereby improving the accuracy of image watermark extraction by the feature extraction network.

[0076] In some embodiments, the feature extraction network includes a convolution extraction unit and a fusion extraction unit. Figure 5 As shown, the above step 140 includes the following sub-steps:

[0077] Step 141: extracting the initial image features of the watermark image through a convolution extraction unit to obtain the intermediate image features of the watermark image;

[0078] In some embodiments, the convolution extraction unit is used to extract the initial image features of the watermark image. In some embodiments, the initial image features of the watermark image are obtained based on the watermark image, the initial image features of the watermark image are input into the convolution extraction unit, and the intermediate image features of the watermark image are output.

[0079] For example, Figure 6 As shown, the feature extraction network 10 includes a convolution extraction unit 11. The watermark image 1 is input into the feature extraction network 10 to obtain the initial image features of the watermark image; the initial image features of the watermark image are input into the convolution extraction unit 11 to obtain the intermediate image features of the watermark image. In the embodiment of the present application, the convolution extraction unit is taken as the C3 module as an example for illustration.

[0080] In some embodiments, the convolution extraction unit includes a convolution extraction branch and a residual extraction branch, such as Figure 7 As shown, the above step 141 includes the following sub-steps:

[0081] Step 1411: performing convolution feature extraction on the initial image features of the watermark image through the convolution extraction branch to obtain a first feature map;

[0082] In some embodiments, the convolution extraction branch is used to perform convolution feature extraction on the initial image features of the watermark image, and the initial image features of the watermark image are input into the convolution extraction branch in the convolution extraction unit to obtain a first feature map. The first feature map is a feature map corresponding to the initial image features of the watermark image.

[0083] For example, Figure 8 As shown, the convolution extraction unit 11 includes a convolution extraction branch 51, and the initial image features 53 of the watermark image are input into the convolution extraction branch 51 to obtain a first feature map 54.

[0084] Step 1412: extracting the initial image features of the watermark image through the residual extraction branch to perform residual feature extraction to obtain a second feature map;

[0085] In some embodiments, the residual extraction branch is used to extract residual features from the initial image features of the watermark image, and the initial image features of the watermark image are input into the residual extraction branch in the convolution extraction unit to obtain a second feature map. The second feature map is a feature map corresponding to the initial image features of the watermark image.

[0086] For example, Figure 8 As shown, the convolution extraction unit 11 includes a residual extraction branch 52, and the initial image features 53 of the watermark image are input into the residual extraction branch 52 to obtain a second feature map 55.

[0087] It is worth noting that the above steps 1411 and 1412 are not sequentially sequenced, and steps 1411 and 1412 may be executed simultaneously, or step 1411 may be executed slightly later than step 1412, or step 1412 may be executed slightly later than step 1411.

[0088] Step 1413: Fuse the first feature map and the second feature map to obtain the intermediate image features of the watermark image.

[0089] In some embodiments, the convolution extraction unit includes a fusion unit. The fusion unit fuses the first feature map and the second feature map to obtain an intermediate image feature of the watermark image. In some embodiments, the intermediate image feature of the watermark image is an intermediate fusion feature corresponding to the initial image feature of the watermark image.

[0090] For example, Figure 8 As shown, the convolution extraction unit 11 includes a fusion unit 24, and the first feature map 54 and the second feature map 55 are input into the fusion unit 24 to obtain the intermediate image feature 13.

[0091] Step 142: The spatial domain features and the frequency domain features in the intermediate image features are fused and extracted by a fusion extraction unit to obtain fusion features.

[0092] In some embodiments, the fusion extraction unit is used to extract fusion features, which include spatial domain features and frequency domain features. In some embodiments, the intermediate image features output by the convolution extraction unit are input into the fusion extraction unit, and the spatial domain features and frequency domain features in the intermediate image features are output.

[0093] For example, Figure 6 As shown, the feature extraction network 10 includes a fusion extraction unit 12. The watermark image 1 is input into the feature extraction network 10 to obtain the initial image features of the watermark image; the initial image features of the watermark image are input into the convolution extraction unit 11 to obtain the intermediate image features of the watermark image; the intermediate image features of the watermark image are input into the fusion extraction unit 12, and the spatial domain features and frequency domain features in the intermediate image features are output to obtain fusion features. In the embodiment of the present application, the fusion extraction unit is taken as a frequency-mask-spatial-attention (FMSA) module as an example for illustration.

[0094] In some embodiments, Fig. 9 As shown, the above step 142 includes the following sub-steps:

[0095] Step 1421: converting the intermediate image features of the watermark image from the spatial domain to the frequency domain through the fusion extraction unit to obtain a first frequency domain feature;

[0096] In some embodiments, the fusion extraction unit is used to implement conversion between the spatial domain and the frequency domain, including conversion from the spatial domain to the frequency domain, and conversion from the frequency domain to the spatial domain.

[0097] In some embodiments, the intermediate image features of the watermark image are input into the fusion extraction unit, and the intermediate image features of the watermark image are converted from the spatial domain to the frequency domain to obtain the first frequency domain features. In some embodiments, the first frequency domain features include amplitude features and phase features.

[0098] In some embodiments, the intermediate image features of the watermark image are transformed from the spatial domain to the frequency domain by DFT or DCT or DWT.

[0099] For example, Fig.10 As shown, the intermediate image feature 13 is input into the fusion extraction unit, and after DFT, a first frequency domain feature 14 is obtained.

[0100] Step 1422: extracting the amplitude feature and the phase feature in the first frequency domain feature to obtain a second frequency domain feature;

[0101] In some embodiments, feature extraction is performed on the first frequency domain feature to obtain the second frequency domain feature. The first frequency domain feature is a shallow feature corresponding to the intermediate image feature of the watermark image, and the second frequency domain feature is a deep feature corresponding to the first frequency domain feature.

[0102] In some embodiments, the first frequency domain feature includes an amplitude feature and a phase feature. Or it can be understood that the first frequency domain feature is represented by the amplitude feature and the phase feature.

[0103] In some embodiments, a first product of an amplitude feature and an amplitude weight mask is calculated; and a second product of a phase feature and a phase weight mask is calculated; the first product and the second product are fused to obtain a second frequency domain feature. In some embodiments, the scale of the amplitude feature is the same as the scale of the phase feature. The scale of the amplitude feature is the same as the scale of the amplitude weight mask; the scale of the phase feature is the same as the scale of the phase weight mask. The amplitude weight mask is used to indicate the learning weight of the amplitude feature; the phase weight mask is used to indicate the learning weight of the phase feature.

[0104] For example, Fig.11 As shown, a first product of the amplitude feature 17 and the amplitude weight mask 18 is calculated; and a second product of the phase feature 19 and the phase weight mask 56 is calculated; the first product and the second product are fused to obtain a second frequency domain feature.

[0105] Step 1423: Convert the second frequency domain feature from the frequency domain to the spatial domain to obtain a fusion feature.

[0106] In some embodiments, the second frequency domain feature is converted from the frequency domain to the space domain by the fusion extraction unit to obtain a fusion feature including the space domain feature and the frequency domain feature. In some embodiments, the fusion feature is a space domain feature.

[0107] In some embodiments, corresponding to the method of converting the intermediate image features of the watermark image from the spatial domain to the frequency domain to obtain the first frequency domain features, the second frequency domain features are converted from the frequency domain to the spatial domain by inverse transformation to obtain a fusion feature including the spatial domain features and the frequency domain features. In some embodiments, when the intermediate image features of the watermark image are converted from the spatial domain to the frequency domain by DFT, the second frequency domain features are converted from the frequency domain to the spatial domain by Inverse Discrete Fourier Transform (IDFT). In some embodiments, when the intermediate image features of the watermark image are converted from the spatial domain to the frequency domain by DCT, the second frequency domain features are converted from the frequency domain to the spatial domain by Inverse Discrete Cosine Transform (IDCT). In some embodiments, when the intermediate image features of the watermark image are converted from the spatial domain to the frequency domain by DWT, the second frequency domain features are converted from the frequency domain to the spatial domain by Inverse Discrete Wavelet Transform (IDWT).

[0108] For example, Fig.10 As shown, after IDFT, the second frequency domain feature 15 is converted from the frequency domain to the space domain to obtain the fusion feature 16.

[0109] In summary, the method provided in this embodiment uses a convolution extraction unit to extract the intermediate image features of the watermark image from the initial image features of the watermark image, and then uses a fusion extraction unit to extract the spatial domain features and frequency domain features corresponding to the image features from the intermediate image features, so that the feature extraction network matches the embedded watermark features when extracting the watermark, thereby improving the accuracy of the feature extraction network in extracting the image watermark.

[0110] In some embodiments, Fig.12 As shown, the above step 1423 includes the following sub-steps:

[0111] Step 14231: convert the second frequency domain feature from the frequency domain to the space domain to obtain the first space domain feature;

[0112] In some embodiments, the second frequency domain feature is converted from the frequency domain to the space domain by a fusion extraction unit to obtain the first space domain feature.

[0113] In some embodiments, corresponding to the method of converting the intermediate image features of the watermark image from the spatial domain to the frequency domain to obtain the first frequency domain features, the second frequency domain features are converted from the frequency domain to the spatial domain by inverse transformation to obtain the first spatial domain features. In some embodiments, when the intermediate image features of the watermark image are converted from the spatial domain to the frequency domain by DFT, the second frequency domain features are converted from the frequency domain to the spatial domain by IDFT. In some embodiments, when the intermediate image features of the watermark image are converted from the spatial domain to the frequency domain by DCT, the second frequency domain features are converted from the frequency domain to the spatial domain by IDCT. In some embodiments, when the intermediate image features of the watermark image are converted from the spatial domain to the frequency domain by DWT, the second frequency domain features are converted from the frequency domain to the spatial domain by IDWT.

[0114] For example, Fig.13 As shown, after IDFT, the second frequency domain feature 15 is converted from the frequency domain to the space domain to obtain the first space feature 21.

[0115] Step 14232: Perform maximum sampling on the first spatial domain feature to obtain a first sampling result;

[0116] In some embodiments, maximum sampling is performed on the first spatial domain feature to obtain a first sampling result. Maximum sampling refers to taking the maximum value of the feature in the first spatial domain feature as the first sampling result. Maximum sampling belongs to local sampling, such as sampling t features to obtain a maximum value as the sampling result, and the value of t is a positive integer greater than 0. In some embodiments, maximum sampling is performed on the first spatial domain feature through a maximum sampling layer to obtain a first sampling result. In some embodiments, the scale of the first sampling result is smaller than the scale of the first spatial domain feature, such as the scale of the first spatial domain feature is 3, and the scale of the first sampling result is 1.

[0117] For example, Fig.13 As shown, the maximum value sampling is performed on the first spatial domain feature 21 through the maximum value sampling layer to obtain the first sampling result 22.

[0118] Step 14233: perform average sampling on the first spatial domain feature to obtain a second sampling result;

[0119] In some embodiments, the first spatial domain feature is averaged and sampled to obtain a second sampling result. Average sampling refers to taking the feature average in the first spatial domain feature as the second sampling result. Average sampling belongs to global sampling, such as sampling t features to obtain the average of t features as the sampling result, and the value of t is a positive integer greater than 0. In some embodiments, the first spatial domain feature is averaged and sampled through an average sampling layer to obtain a second sampling result. In some embodiments, the scale of the second sampling result is smaller than the scale of the first spatial domain feature, such as the scale of the first spatial domain feature is 5, and the scale of the second sampling result is 1.

[0120] For example, Fig.13 As shown, the first spatial domain feature 21 is sampled by the average value sampling layer to obtain the second sampling result 23.

[0121] It is worth noting that the above steps 14232 and 14233 do not distinguish between the order of priority, and step 14232 and step 14233 may be executed simultaneously, or step 14232 may be executed slightly later than step 14233, or step 14232 may be executed slightly later than step 14233.

[0122] Step 14234: Fusing the first sampling result and the second sampling result to obtain a second spatial domain feature;

[0123] In some embodiments, the scale of the first sampling result and the second sampling result is the same. The first sampling result and the second sampling result are fused to obtain a second spatial domain feature. In some embodiments, the scale of the second spatial domain feature is the same as the scale of the first sampling result and the second sampling result, such as the scale of the second spatial domain feature = the scale of the first sampling result = the scale of the second sampling result = 1.

[0124] In some embodiments, the fusion extraction unit includes a fusion unit. The first sampling result and the second sampling result are input into the fusion unit to obtain the second spatial domain feature. For example, Fig.13 As shown, the first sampling result 22 and the second sampling result 23 are input into the fusion unit 24 to obtain the second spatial domain feature 57.

[0125] Step 14235: Multiply the intermediate image feature and the second spatial domain feature to obtain a fused feature.

[0126] In some embodiments, the scale of the intermediate image feature is the same as the scale of the first spatial domain feature. Alternatively, the intermediate image feature is transformed between the spatial domain and the frequency domain to obtain the first spatial domain feature without changing the scale. The intermediate image feature and the second spatial domain feature are multiplied to obtain a fused feature, and the scale of the obtained fused feature is the same as the scale of the intermediate image feature. For example, Fig.13 As shown, the second spatial domain feature 57 and the intermediate image feature 13 are multiplied to obtain a fusion feature 16 with the same scale as the intermediate image feature 13.

[0127] In summary, the method provided in this embodiment enables the network to retain feature information from both local and global aspects by performing maximum sampling and average sampling on the first spatial domain feature. By multiplying the intermediate image feature and the second spatial domain feature, on the one hand, the feature information of the intermediate image feature is retained in the output fusion feature, and on the other hand, the scale of the final output fusion feature is regressed to the same scale as the intermediate image feature.

[0128] In some embodiments, the feature extraction network includes an n-level extraction network, each level of the n-level extraction network includes a convolution extraction unit and a fusion extraction unit, and the value of n is a positive integer greater than 1. Fig.14 As shown, the above method also includes:

[0129] Step 220: The i-th level fusion feature output by the fusion extraction unit of the i-th level extraction network is used as the i+1-th level initial image feature of the watermark image, and is input into the input end of the convolution extraction unit of the i+1-th level extraction network.

[0130] In some embodiments, the feature extraction network includes a first-level extraction network, a second-level extraction network to an nth-level extraction network. The input feature of the convolution extraction unit of the first-level extraction network is the first-level initial image feature of the watermark image, and the first-level initial image feature is obtained by feature extraction of the image data of the watermark image.

[0131] The first-level initial image features of the watermark image are input into the first-level extraction network to output the first-level fusion features; the first-level fusion features are input into the second-level extraction network as the second-level initial image features to output the second-level fusion features; until the n-1th-level fusion features are input into the nth-level extraction network as the nth-level initial image features to output the nth-level fusion features.

[0132] The first-level initial image features of the watermark image are input into the convolution extraction unit in the first-level extraction network to output the first-level intermediate image features, and the first-level intermediate image features are input into the fusion extraction unit in the first-level extraction network to output the first-level fusion features; the first-level fusion features are input into the convolution extraction unit in the second-level extraction network as the second-level initial image features, and the second-level intermediate image features are output; the second-level intermediate image features are input into the fusion extraction unit in the second-level extraction network to output the second-level fusion features; until the n-1-th level fusion features are input into the convolution extraction unit in the n-th level extraction network as the n-th level initial image features, and the n-th level intermediate image features are output; the n-th level intermediate image features are input into the fusion extraction unit in the n-th level extraction network to output the n-th level fusion features.

[0133] For example, Fig.15 As shown, the feature extraction network 10 includes a first-level extraction network 23, a second-level extraction network 24 to an n-th-level extraction network 25, and each extraction network includes a convolution extraction unit 11 and a fusion extraction unit 12. The first-level fusion feature output by the fusion extraction unit 12 in the first-level extraction network 23 is used as the second-level initial image feature of the watermark image and input into the input end of the convolution extraction unit 11 of the second-level extraction network 24; until the n-1-th-level fusion feature output by the fusion extraction unit 12 in the n-1-th-level extraction network is used as the n-th-level initial image feature of the watermark image and input into the input end of the convolution extraction unit 11 of the n-th-level extraction network 25.

[0134] Wherein, n is a positive integer greater than 1, and i is a positive integer less than n.

[0135] In some embodiments, the feature extraction network further includes a front-end extraction unit located before the convolution extraction unit and the fusion extraction unit. Fig.16 As shown, the above method also includes:

[0136] Step 240: Perform feature extraction on the image data of the watermark image through the front-end extraction unit to obtain the first level initial image features of the watermark image.

[0137] In some embodiments, the front-end extraction unit is used to extract features from the image data of the watermark image. The watermark image is input into the front-end extraction unit to obtain the first level initial image features of the watermark image corresponding to the image data of the watermark image.

[0138] For example, Fig.17 As shown, the feature extraction network 10 includes a front-end extraction unit 26, the watermark image is input into the front-end extraction unit 26 to obtain the first-level initial image features of the watermark image; the first-level initial image features of the watermark image are input into the convolution extraction unit 11 in the first-level extraction network 23.

[0139] In some embodiments, the front-end extraction unit includes p cascaded convolution activation units, and the value range of p is a positive integer greater than 0. In the embodiment of the present application, the front-end extraction unit includes two cascaded convolution activation units as an example. Fig.18 As shown, the front-end extraction unit 26 includes two cascaded convolution activation units 27. In the embodiment of the present application, the front-end extraction unit is taken as a CBS module for example.

[0140] In some embodiments, the convolution activation unit includes a cascaded convolution unit, a normalization unit, and an activation unit. The input of the convolution unit is connected to the input of the convolution activation unit; the output of the convolution unit is connected to the input of the normalization unit; the output of the normalization unit is connected to the input of the activation unit; and the output of the activation unit is the output of the convolution activation unit. For example, Fig.19 As shown, the convolution activation unit 27 includes a cascaded convolution unit 28, a normalization unit 29 and an activation unit 30, the input end of the convolution unit 28 is the input end of the convolution activation unit 27; the output end of the convolution unit 28 is connected to the input end of the normalization unit 29; the output end of the normalization unit 29 is connected to the input end of the activation unit 30; the output of the activation unit 30 is the output of the convolution activation unit 27. In the embodiment of the present application, the convolution unit is taken as an example of a Conv module, the normalization unit is taken as a data normalization (Batch Normalization, BN) module, and the activation unit is taken as an example of a SILU module.

[0141] In some embodiments, the fused features output by the n-level extraction network in the feature extraction network are fused features of the first scale, and the value of n is a positive integer greater than 1. The feature extraction network is a back-end extraction unit located before the convolution extraction unit and the fusion extraction unit. Fig. 20 As shown, the above method also includes:

[0142] Step 260: performing downsampling feature extraction on the fused features of the first scale through the back-end extraction unit to obtain fused features of the nth scale;

[0143] In some embodiments, the back-end extraction unit is used to perform downsampling feature extraction on the fused features of the first scale. The fused features of the first scale are input to the back-end extraction unit, and the fused features of the nth scale are output. For example, Fig.21 As shown, the feature extraction network 10 includes a back-end extraction unit 27, which inputs the first-scale fusion features output by the n-th level extraction network into the back-end extraction unit 27, and outputs the n-th scale fusion features.

[0144] In some embodiments, the fused features output by each level of extraction network are fused features of the first scale. Fig.21 As shown, the first-level fusion features output by the first-level extraction network 23 are fusion features of the first scale; the second-level fusion features output by the second-level extraction network 24 are fusion features of the first scale; and the n-th-level fusion features output by the n-th-level extraction network 25 are fusion features of the first scale. That is, the scales of the fusion features output by each level of extraction network are the same. In some embodiments, the first scale is the largest scale among the multiple levels of scales, and the multiple levels of scales decrease step by step. Assuming that a total of n levels of scales are included, the first scale is greater than the second scale, the second scale is greater than the third scale, and so on until the n-1th scale is greater than the nth scale.

[0145] In some embodiments, the back-end extraction unit includes a cascaded convolution extraction unit and a pooling unit, the input end of the convolution extraction unit is the input end of the back-end extraction unit, the output end of the convolution extraction unit is connected to the output end of the pooling unit, and the output end of the pooling unit is the output end of the back-end extraction unit. Fig. 22 As shown, the back-end extraction unit 27 includes a cascaded convolution extraction unit 11 and a pooling unit 28, the input end of the convolution extraction unit 11 is the input end of the back-end extraction unit 27, the output end of the convolution extraction unit 11 is connected to the output end of the pooling unit 28, and the output end of the pooling unit 28 is the output end of the back-end extraction unit 27. In the embodiment of the present application, the convolution extraction unit is an example of a C3 module, and the pooling unit is an SPPF (Spatial Pyramid Pooling-Fast) module.

[0146] In some embodiments, the convolution extraction unit includes a convolution activation unit and a fusion unit. The first feature is input into the convolution extraction unit, and after passing through at least two cascaded convolution activation units, a first activation feature is output; the first activation feature and the first feature are spliced ​​to obtain a spliced ​​feature. The first feature passes through a convolution activation unit, and a second activation feature is output. The spliced ​​feature and the second activation feature are input into the fusion unit for fusion to obtain a second feature. The second feature passes through a convolution activation unit to obtain a third feature. For example, Fig.23As shown, the convolution extraction unit 11 includes a convolution activation unit 27 and a fusion unit 24. The first feature is input into the convolution extraction unit 27, and after passing through at least two cascaded convolution activation units 27, a first activation feature is output; the first activation feature and the first feature are spliced ​​to obtain a spliced ​​feature. The first feature passes through a convolution activation unit 27, and a second activation feature is output. The spliced ​​feature and the second activation feature are input into the fusion unit 24 for fusion to obtain a second feature. The second feature passes through a convolution activation unit 27 to obtain a third feature. In the embodiment of the present application, the convolution activation unit is taken as a CBS module as an example for illustration, and the fusion unit is taken as a Concat module as an example for illustration.

[0147] In some embodiments, the pooling unit includes a sampling unit, a fusion unit, and a convolution activation unit that are cascaded to each other. The input of the pooling unit and the output of each level of the sampling unit are input to the fusion unit; the output of the fusion unit is input to the convolution activation unit; and the output of the convolution activation unit is the output of the pooling unit. For example, Fig.24 As shown, the pooling unit 28 includes a sampling unit 29, a fusion unit 24 and a convolution activation unit 27 which are cascaded to each other. The input of the pooling unit 28 and the output of each level of the sampling unit 29 are input to the fusion unit 24; the output of the fusion unit 24 is input to the convolution activation unit 27; and the output of the convolution activation unit 27 is the output of the pooling unit 28. In the embodiment of the present application, the sampling unit is taken as a Max Pool module as an example for illustration, and the sampling unit is used to perform maximum value sampling on the input of the sampling unit.

[0148] In summary, the method provided in this embodiment realizes n-level extraction of spatial domain features and frequency domain features through an n-level extraction network, so that the feature extraction network adapts to the watermark features in the watermark image during feature extraction, and the matching degree between the feature extraction network and the watermark features including spatial domain features and frequency domain features is further improved through n-level fusion extraction, thereby improving the accuracy of image watermark extraction by the feature extraction network.

[0149] In some embodiments, the watermark prediction network is a multi-scale watermark prediction network, which includes a sampling network and an n-level prediction network. Fig.25 As shown, the above step 160 includes the following sub-steps:

[0150] Step 161: using the sampling network to perform n-1 level upsampling using the fusion features of the n-th scale as the prediction features of the n-th scale, to obtain the prediction features of the n-1-th scale to the prediction features of the first scale;

[0151] In some embodiments, the sampling network includes n-1 levels of upsampling networks and n-1 levels of downsampling units corresponding to each level of upsampling networks in reverse order. The sampling network includes the n-1th level upsampling network, the n-2th level upsampling network, and the 1st level upsampling network, and the n-1th level downsampling unit corresponding to the n-1th level upsampling network, the n-2th level downsampling unit corresponding to the n-2th level upsampling network, and the 1st level downsampling unit corresponding to the 1st level upsampling network, where n is a positive integer greater than 1.

[0152] In some embodiments, the nx-th level upsampling network in the n-1-level upsampling network is used to upsample the first-scale fusion features output by the x-th level extraction network to obtain the x+1-th level intermediate features of the x+1-th scale, where x is a positive integer from 1 to n-1, and the value of n is a positive integer greater than 1.

[0153] In some embodiments, the nx-th level downsampling unit in the n-1-th level downsampling unit is used to fuse the x+1-th level intermediate features of the x+1-th scale output by the nx-th level upsampling network with the predicted features of the x+1-th scale and then downsample them to the predicted features of the x-th scale.

[0154] For example, Fig.26 As shown, the multi-scale watermark prediction network includes a sampling network 60, which includes an n-1th level upsampling network 61, an n-2th level upsampling network 62, up to a second level upsampling network 63 and a first level upsampling network 64, and also includes an n-1th level downsampling unit 91 corresponding to the n-1th level upsampling network 61, an n-2th level downsampling unit 92 corresponding to the n-2th level upsampling network 62, up to a second level downsampling unit 93 corresponding to the second level upsampling network 63, and a first level downsampling unit 94 corresponding to the first level upsampling network 64.

[0155] In some embodiments, the fused features of the first scale output by the x-th extraction network are upsampled to obtain the x+1th intermediate features of the x+1th scale through the nxth upsampling network in the n-1th upsampling network; the x+1th intermediate features of the x+1th scale are fused with the predicted features of the x+1th scale through the nxth downsampling unit in the n-1th downsampling unit corresponding to the nxth upsampling network, and then downsampled to the predicted features of the xth scale, where x is a positive integer from 1 to n-1, and n is a positive integer greater than 1. In some embodiments, the fused features of the nth scale are used as the predicted features of the nth scale, and the fused features of the nth scale are the output of the feature extraction network.

[0156] For example, Fig.26As shown, the first-scale fusion feature output by the n-1-level extraction network 66 is upsampled by the first-level upsampling network 64 to obtain the n-scale n-level intermediate feature 71; the n-scale n-level intermediate feature 71 and the n-scale prediction feature 72 are input into the first-level downsampling unit 94 to obtain the n-1-scale prediction feature 73. The n-scale prediction feature 72 is the n-scale fusion feature obtained by extracting the first-scale fusion feature output by the n-level extraction network 25 through downsampling by the back-end extraction unit 27.

[0157] The fused features of the first scale output by the n-2-th level extraction network 65 are upsampled through the second-level upsampling network 63 to obtain the n-1-th level intermediate features 70 of the n-1-th scale; the n-1-th level intermediate features 70 of the n-1-th scale and the predicted features 73 of the n-1-th scale are input into the second-level downsampling unit 93 to obtain the predicted features 74 of the n-2-th scale.

[0158] …

[0159] The fused features of the first scale output by the second-level extraction network 24 are upsampled through the n-2-level upsampling network 62 to obtain the third-level intermediate features 69 of the third scale; the third-level intermediate features 69 of the third scale and the predicted features 75 of the third scale are input into the n-2-level downsampling unit 92 to obtain the predicted features 76 of the second scale.

[0160] The fused features of the first scale output by the first-level extraction network 23 are upsampled through the n-1-th level upsampling network 61 to obtain the second-scale intermediate features 68 of the second scale; the second-scale intermediate features 68 of the second scale and the predicted features 76 of the second scale are input into the n-1-th level downsampling unit 91 to obtain the predicted features 77 of the first scale.

[0161] Step 162: fusing and downsampling convolution the fusion features of the first scale and the prediction features of the first scale through the first-level prediction network in the n-level prediction network to obtain the deep and shallow fusion features of the second scale;

[0162] In some embodiments, the fused features at the first scale and the predicted features at the first scale are input into the first-level prediction network to obtain the deep and shallow fused features at the second scale. The fused features at the first scale are the output of the nth-level extraction network.

[0163] In some embodiments, the fusion feature of the first scale and the prediction feature of the first scale are fused by a fusion unit. In some embodiments, the fusion feature of the first scale and the prediction feature of the first scale are downsampled by a convolution extraction unit. Fig. 27As shown, the first-scale fusion feature 78 and the first-scale prediction feature 77 are input into the first-level prediction network 87, and after the fusion unit 24 and the convolution extraction unit 11, the second-scale deep and shallow layer fusion feature 79 is obtained.

[0164] Step 163: predicting the deep and shallow layer fusion features of the second scale to obtain a watermark prediction result of the second scale;

[0165] In some embodiments, the deep and shallow fusion features of the second scale are predicted by the convolution unit to obtain the watermark prediction result of the second scale. Fig. 27 As shown, the deep and shallow layer fusion features 79 of the second scale are input into the convolution unit 28 to obtain the watermark prediction result 83 of the second scale.

[0166] Step 164: fusing and downsampling convolution the deep and shallow layer features of the k-th scale and the prediction features of the k-th scale through the k-th level prediction network in the n-level prediction network to obtain the deep and shallow layer fusion features of the k+1-th scale;

[0167] In some embodiments, n is a positive integer greater than 1, and k is a positive integer greater than 1 and less than or equal to n. The deep and shallow features of the kth scale are obtained from the k-1th level prediction network, and the prediction features of the kth scale are obtained based on the nkth level upsampling network.

[0168] In some embodiments, the second-level prediction network in the n-level prediction network fuses and downsamples the second-scale deep and shallow layer features and the second-scale prediction features to obtain the third-scale deep and shallow layer fusion features. Fig. 27 As shown, the second-scale deep and shallow layer features 79 and the second-scale prediction features 76 are input into the second-level prediction network 88, and after the fusion unit 24 and the convolution extraction unit 11, the third-scale deep and shallow layer fusion features 80 are obtained.

[0169] In some embodiments, the n-1th level prediction network in the n-level prediction network fuses and downsamples the n-1th scale deep and shallow features and the n-1th scale prediction features to obtain the nth scale deep and shallow fusion features. For example, Fig. 27 As shown, the deep and shallow layer features of the n-1th scale and the prediction features 73 of the n-1th scale are input into the n-1th level prediction network 89, and after the fusion unit 24 and the convolution extraction unit 11, the deep and shallow layer fusion features 81 of the n-1th scale are obtained.

[0170] In some embodiments, the nth level prediction network in the n-level prediction network fuses and downsamples the nth scale deep and shallow features and the nth scale prediction features to obtain the n+1th scale deep and shallow fusion features. For example, Fig. 27As shown, the deep and shallow layer features 81 of the nth scale and the prediction features 72 of the nth scale are input into the nth level prediction network 90, and after the fusion unit 24 and the convolution extraction unit 11, the deep and shallow layer fusion features 82 of the n+1th scale are obtained.

[0171] Step 165: predicting the deep and shallow fusion features of the k+1th scale to obtain the watermark prediction result of the k+1th scale;

[0172] In some embodiments, the deep and shallow fusion features of the k+1th scale are predicted by the convolution unit to obtain the watermark prediction result of the k+1th scale. Fig. 27 As shown, the deep and shallow layer fusion features 80 of the third scale are input into the convolution unit 28 to obtain the watermark prediction result 84 of the third scale; the deep and shallow layer fusion features 81 of the nth scale are input into the convolution unit 28 to obtain the watermark prediction result 85 of the nth scale; the deep and shallow layer fusion features 82 of the n+1th scale are input into the convolution unit 28 to obtain the watermark prediction result 86 of the n+1th scale.

[0173] Step 166: Obtain an image watermark based on the watermark prediction result.

[0174] In some embodiments, the image watermark is predicted based on the watermark prediction results at the second scale to the watermark prediction results at the (n+1)th scale.

[0175] In summary, the method provided in this embodiment, through the n-1 level upsampling network and the n level prediction network, can make full use of the output of the n-level extraction network when predicting the image watermark, thereby using more feature information when predicting the image watermark and improving the accuracy of the prediction.

[0176] In some embodiments, the present application embodiment takes n=3 as an example for illustration. Fig.28 As shown, the embodiment of the present application provides a method for extracting an image watermark, including a feature extraction network 10 and a watermark prediction network 20. The feature extraction network 10 is used to extract features, and the watermark prediction network 20 is used to predict image watermarks.

[0177] In some embodiments, the feature extraction network 10 uses a convolution extraction unit 11 with a cross-stage and local network structure, such as a C3 module, to divide the feature map output by the convolution layer into two parts, one part continues to perform the convolution operation, and the other part uses a residual network structure, and then fuses the two parts of the feature map to achieve rapid dimensionality reduction of the output feature map. While reducing the network size, the gradient vanishing problem is avoided, and the detection speed of the prediction stage can be improved.

[0178] In some embodiments, a plurality of small-size pooling cores are cascaded in the back-end extraction unit 27 to replace a single large-size pooling core in the SPP module, thereby further improving the running speed while retaining the original functions and enriching the expressive power of the feature map.

[0179] An image watermark extraction method provided in an embodiment of the present application includes a fusion extraction unit 12. Taking the FMSA module as an example, it is mainly divided into two parts: frequency domain mask and spatial attention fusion. The frequency domain mask part uses DFT transformation to convert the spatial domain features into frequency domain features composed of amplitude and phase, and then adds the learnable amplitude weight mask Mask A and phase weight mask Mask P respectively. The network learns the weights of the amplitude and phase feature maps, and then converts them back to the spatial domain through IDFT. In the spatial attention part, the spatial domain features are used to simultaneously undergo maximum sampling and average sampling operations, and then the results after the two dimensionality reductions are fused together, and then the number of channels is normalized through a layer of convolution, and then multiplied with the initial spatial domain features to achieve the fusion of spatial domain features and frequency domain features.

[0180] In some embodiments, in the watermark prediction network 20, watermark prediction results of different scales are predicted by fusing features of different depths, and the 640×640 resolution image is downsampled by 8 times, 16 times and 32 times respectively through multi-layer convolution, and corresponding watermark prediction results of 80×80, 40×40, and 20×20 are output.

[0181] In some embodiments, a 4×4 frequency domain watermark is embedded in the original image and used as the input image for training. The effects of converting the RGB format to the YCbCr format, adding the DFT amplitude mask, and adding the DFT phase mask on the prediction accuracy and recall rate are recorded respectively. The recorded results are shown in Table 1 below:

[0182] Table 1

[0183]

[0184] In some embodiments, adding a DFT magnitude mask helps the most in improving the accuracy. Converting the input image from RGB to YCbCr and adding a DFT magnitude mask both hurt the recall rate, and only adding a DFT phase mask helps the recall rate. The combination of the three methods can significantly improve the accuracy with a slight decrease in the recall rate.

[0185] Fig.29 A block diagram of an image watermark extraction device provided by an exemplary embodiment of the present application is shown. The device includes:

[0186] The acquisition module 2910 is used to acquire a watermark image. The watermark image is obtained by embedding watermark features corresponding to the watermark information in the original image. The watermark features include spatial domain features and frequency domain features.

[0187] The extraction module 2920 is used to extract fused features of spatial domain features and frequency domain features based on a feature extraction network. The feature extraction network is a neural network used to extract and fuse spatial domain features and frequency domain features through at least one network layer.

[0188] The prediction module 2930 is used to extract the watermark from the fused features based on the watermark prediction network to obtain the image watermark. The watermark prediction network is a neural network used to perform predictive reasoning of the image watermark using the fused features as input through at least one network layer.

[0189] In some embodiments, the feature extraction network includes: a convolution extraction unit and a fusion extraction unit.

[0190] The extraction module 2920 is further used to extract the initial image features of the watermark image through a convolution extraction unit to obtain the intermediate image features of the watermark image.

[0191] The extraction module 2920 is further used to perform fusion extraction on the spatial domain features and the frequency domain features in the intermediate image features through the fusion extraction unit to obtain fusion features.

[0192] The device also includes:

[0193] The extraction module 2920 is further used to convert the intermediate image features of the watermark image from the spatial domain to the frequency domain by fusing the extraction unit to obtain a first frequency domain feature, where the first frequency domain feature includes an amplitude feature and a phase feature.

[0194] The extraction module 2920 is further used to extract the amplitude feature and the phase feature in the first frequency domain feature to obtain the second frequency domain feature.

[0195] The extraction module 2920 is further used to convert the second frequency domain features from the frequency domain to the space domain to obtain fusion features.

[0196] The extraction module 2920 is further used to calculate the first product of the amplitude feature and the amplitude weight mask.

[0197] The extraction module 2920 is further used to calculate the second product of the phase feature and the phase weight mask.

[0198] The extraction module 2920 is further used to fuse the first product and the second product to obtain a second frequency domain feature.

[0199] The extraction module 2920 is further used to convert the second frequency domain feature from the frequency domain to the space domain to obtain the first space domain feature.

[0200] The extraction module 2920 is further used to perform maximum sampling on the first spatial domain feature to obtain a first sampling result; and perform average sampling on the first spatial domain feature to obtain a second sampling result.

[0201] The extraction module 2920 is further used to obtain a second spatial domain feature by fusing the first sampling result and the second sampling result.

[0202] The extraction module 2920 is further used to multiply the intermediate image feature and the second spatial domain feature to obtain a fused feature.

[0203] In some embodiments, the convolution extraction unit includes a convolution extraction branch and a residual extraction branch.

[0204] The extraction module 2920 is further used to perform convolution feature extraction on the initial image features of the watermark image through the convolution extraction branch to obtain a first feature map; and to extract the initial image features of the watermark image through the residual extraction branch to perform residual feature extraction to obtain a second feature map.

[0205] The extraction module 2920 is further used to fuse the first feature map and the second feature map to obtain the intermediate image features of the watermark image.

[0206] In some embodiments, the feature extraction network includes an n-level extraction network, and each level of the n-level extraction network includes a convolution extraction unit and a fusion extraction unit.

[0207] In some embodiments, the i-th level fusion feature output by the fusion extraction unit of the i-th level extraction network is used as the i+1-th level initial image feature of the watermark image and input into the input end of the convolution extraction unit of the i+1-th level extraction network.

[0208] Wherein, n is a positive integer greater than 1, and i is a positive integer less than n.

[0209] In some embodiments, the input features of the convolution extraction unit of the first-level extraction network are the first-level initial image features of the watermark image, and the first-level initial image features are obtained by performing feature extraction on the image data of the watermark image.

[0210] In some embodiments, the feature extraction network further includes: a front-end extraction unit located before the convolution extraction unit and the fusion extraction unit.

[0211] The extraction module 2920 is further used to perform feature extraction on the image data of the watermark image through the front-end extraction unit to obtain the first level initial image features of the watermark image.

[0212] In some embodiments, the fused features outputted from the n-stage extraction network are fused features of the first scale.

[0213] In some embodiments, the feature extraction network further includes: a back-end extraction unit located before the convolution extraction unit and the fusion extraction unit.

[0214] The extraction module 2920 is further configured to perform downsampling feature extraction on the fused features of the first scale through a back-end extraction unit to obtain fused features of the nth scale.

[0215] Among them, the first scale is the largest scale in the multi-level scales, and the multi-level scales decrease step by step.

[0216] In some embodiments, the watermark prediction network is a multi-scale watermark prediction network, and the multi-scale watermark prediction network includes: a sampling network and an n-level prediction network.

[0217] The prediction module 2930 is further used to perform n-1 level up-sampling using the fusion features of the n-th scale as the prediction features of the n-th scale through the sampling network to obtain the prediction features of the n-1-th scale to the prediction features of the first scale.

[0218] The prediction module 2930 is also used to fuse and downsample the fusion features of the first scale and the prediction features of the first scale through the first-level prediction network in the n-level prediction network to obtain the deep and shallow fusion features of the second scale; predict the deep and shallow fusion features of the second scale to obtain the watermark prediction result of the second scale.

[0219] The prediction module 2930 is also used to fuse and downsample the deep and shallow layer features of the k-th scale and the prediction features of the k-th scale through the k-th level prediction network in the n-level prediction network to obtain the deep and shallow layer fusion features of the k+1-th scale; predict the deep and shallow layer fusion features of the k+1-th scale to obtain the watermark prediction result of the k+1-th scale.

[0220] The prediction module 2930 is further used to obtain the image watermark based on the watermark prediction result.

[0221] Wherein, n is a positive integer greater than 1, and k is a positive integer greater than 1 and less than n.

[0222] In some embodiments, the sampling network includes an n-1 stage upsampling network arranged in reverse order.

[0223] The prediction module 2930 is further configured to use the fused features of the n-th scale as the prediction features of the n-th scale.

[0224] The prediction module 2930 is further used to upsample the fused features of the first scale output by the x-th extraction network through the nx-th upsampling network in the n-1-th upsampling network to obtain the x+1-th intermediate features of the x+1-th scale; and to downsample the x+1-th intermediate features of the x+1-th scale with the predicted features of the x+1-th scale, where x is a positive integer from 1 to n-1.

[0225] Fig.30 A schematic diagram of the structure of a computer device provided by an exemplary embodiment of the present application is shown.

[0226] In some embodiments, the computer device is a terminal device. In some embodiments, the computer device is a server.

[0227] Schematically, the computer device 3000 includes a central processing unit (CPU) 3001, a system memory 3004 including a random access memory (RAM) 3002 and a read-only memory (ROM) 3003, and a system bus 3005 connecting the system memory 3004 and the central processing unit 3001. The computer device 3000 also includes a basic input / output system 3006 for facilitating information transmission between various components in the computer, and a large-capacity storage device 3007 for storing an operating system 3013, application programs 3014, and other program modules 3015.

[0228] The basic input / output system 3006 includes a display 3008 for displaying information and an input device 3009 such as a mouse and a keyboard for user inputting information. The display 3008 and the input device 3009 are connected to the central processing unit 3001 through an input / output controller 3010 connected to the system bus 3005. The basic input / output system 3006 may also include an input / output controller 3010 for receiving and processing inputs from a plurality of other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 3010 also provides output to a display screen, a printer, or other types of output devices.

[0229] The mass storage device 3007 is connected to the central processing unit 3001 via a mass storage controller (not shown) connected to the system bus 3005. The mass storage device 3007 and its associated computer readable medium provide non-volatile storage for the computer device 3000. That is, the mass storage device 3007 may include a computer readable medium (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0230] The computer readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory or other solid-state storage technologies, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage medium is not limited to the above. The above-mentioned system memory 3004 and mass storage device 3007 can be collectively referred to as memory.

[0231] According to various embodiments of the present application, the computer device 3000 can also be connected to a remote computer on the network through a network such as the Internet. That is, the computer device 3000 can be connected to the network 3012 through the network interface unit 3011 connected to the system bus 3005, or the network interface unit 3011 can be used to connect to other types of networks or remote computer systems (not shown).

[0232] An exemplary embodiment of the present application further provides a computer-readable storage medium, in which at least one program is stored. The at least one program is loaded and executed by a processor to implement the image watermark extraction method provided by each of the above method embodiments.

[0233] An exemplary embodiment of the present application also provides a computer program product, which includes at least one program, and the at least one program is stored in a readable storage medium; a processor of a computer device reads at least one program from the readable storage medium, and the processor executes at least one program, so that the computer device executes to implement the image watermark extraction method provided by the above-mentioned various method embodiments.

[0234] It should be understood that the "plurality" mentioned in this article refers to two or more. Those skilled in the art will easily think of other embodiments of the present application after considering the specification and practicing the invention disclosed here. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary technical means in the art that are not disclosed in the present application. The description and examples are only regarded as exemplary, and the true scope and spirit of the present application are indicated by the following claims.

[0235] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0236] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent switching, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for extracting an image watermark, characterized in that: The method comprises: Acquire a watermark image, wherein the watermark image is obtained by embedding watermark features corresponding to the watermark information in the original image, wherein the watermark features include spatial domain features and frequency domain features; Extracting fusion features of the spatial domain features and the frequency domain features based on a feature extraction network, wherein the feature extraction network is a neural network for extracting and fusing the spatial domain features and the frequency domain features through at least one network layer; The fused features are subjected to watermark extraction based on a watermark prediction network to obtain the image watermark. The watermark prediction network is a neural network for performing predictive inference of the image watermark using the fused features as input through at least one network layer.

2. The method according to claim 1, characterized in that The feature extraction network includes: a convolution extraction unit and a fusion extraction unit; The extracting the fusion feature of the spatial domain feature and the frequency domain feature based on the feature extraction network includes: Extracting the initial image features of the watermark image by the convolution extraction unit to obtain the intermediate image features of the watermark image; The fusion extraction unit performs fusion extraction on the spatial domain features and the frequency domain features in the intermediate image features to obtain the fusion features.

3. The method according to claim 2, characterized in that The fusing and extracting the spatial domain features and the frequency domain features in the intermediate image features by the fusion extraction unit to obtain the fused features includes: Converting the intermediate image features of the watermark image from the spatial domain to the frequency domain by the fusion extraction unit to obtain first frequency domain features, where the first frequency domain features include amplitude features and phase features; performing feature extraction on the amplitude feature and the phase feature in the first frequency domain feature to obtain a second frequency domain feature; The second frequency domain feature is converted from the frequency domain to the spatial domain to obtain the fusion feature.

4. The method according to claim 3, characterized in that The extracting the amplitude feature and the phase feature from the first frequency domain feature to obtain a second frequency domain feature includes: Calculating a first product of the amplitude feature and the amplitude weight mask; calculating a second product of the phase feature and the phase weight mask; The first product and the second product are fused to obtain the second frequency domain feature.

5. The method according to claim 3, characterized in that The converting the second frequency domain feature from the frequency domain to the spatial domain to obtain the fused feature includes: Converting the second frequency domain feature from the frequency domain to the spatial domain to obtain a first spatial domain feature; Performing maximum sampling on the first spatial domain feature to obtain a first sampling result; performing average sampling on the first spatial domain feature to obtain a second sampling result; Fusing the first sampling result and the second sampling result to obtain a second spatial domain feature; The intermediate image feature and the second spatial domain feature are multiplied to obtain the fusion feature.

6. The method according to claim 2, characterized in that The convolution extraction unit includes a convolution extraction branch and a residual extraction branch; The step of extracting the initial image features of the watermark image by the convolution extraction unit to obtain the intermediate image features of the watermark image includes: Performing convolution feature extraction on the initial image features of the watermark image through the convolution extraction branch to obtain a first feature map; and extracting the initial image features of the watermark image through the residual extraction branch to perform residual feature extraction to obtain a second feature map; The first feature map and the second feature map are fused to obtain an intermediate image feature of the watermark image.

7. The method according to any one of claims 2 to 6, characterized in that: The feature extraction network includes an n-level extraction network, and each level of the n-level extraction network includes the convolution extraction unit and the fusion extraction unit; the method further includes: The i-th level fusion feature output by the fusion extraction unit of the i-th level extraction network is used as the i+1-th level initial image feature of the watermark image, and inputted into the input end of the convolution extraction unit of the i+1-th level extraction network; Wherein, n is a positive integer greater than 1, i is a positive integer less than n, the input feature of the convolution extraction unit of the first-level extraction network is the first-level initial image feature of the watermark image, and the first-level initial image feature is obtained by feature extraction of the image data of the watermark image.

8. The method according to claim 7, characterized in that The feature extraction network further includes: a front-end extraction unit located before the convolution extraction unit and the fusion extraction unit; The method further comprises: The front-end extraction unit performs feature extraction on the image data of the watermark image to obtain the first-level initial image features of the watermark image.

9. The method according to claim 7, characterized in that The fusion features outputted from the n-level extraction network are fusion features of the first scale; The feature extraction network further includes: a back-end extraction unit located before the convolution extraction unit and the fusion extraction unit; The method further comprises: Performing downsampling feature extraction on the fused features of the first scale by the back-end extraction unit to obtain fused features of the nth scale; The first scale is the largest scale among the multiple scales, and the multiple scales decrease step by step.

10. The method according to claim 9, characterized in that The watermark prediction network is a multi-scale watermark prediction network, and the multi-scale watermark prediction network includes: a sampling network and an n-level prediction network; The step of extracting the watermark from the fused features based on the watermark prediction network to obtain the image watermark includes: The fusion features of the nth scale are used as the prediction features of the nth scale through the sampling network to perform n-1 level upsampling to obtain the prediction features of the n-1th scale to the prediction features of the first scale; The first-level prediction network in the n-level prediction network fuses and downsamples the fused features of the first scale and the predicted features of the first scale to obtain a deep-shallow fused feature of the second scale; the deep-shallow fused feature of the second scale is predicted to obtain a watermark prediction result of the second scale; The k-th level prediction network in the n-level prediction network fuses and downsamples the k-th scale deep and shallow layer features and the k-th scale prediction features to obtain the k+1-th scale deep and shallow layer fusion features; the k+1-th scale deep and shallow layer fusion features are predicted to obtain the k+1-th scale watermark prediction result; Based on the watermark prediction result, obtaining the image watermark; Wherein, n is a positive integer greater than 1, and k is a positive integer greater than 1 and less than or equal to n.

11. An image watermark extraction device, characterized in that: The device comprises: An acquisition module is used to acquire a watermark image, wherein the watermark image is obtained by embedding watermark features corresponding to the watermark information into the original image, wherein the watermark features include spatial domain features and frequency domain features; An extraction module, configured to extract fusion features of the spatial domain features and the frequency domain features based on a feature extraction network; The prediction module is used to extract the watermark from the fusion feature based on the watermark prediction network to obtain the image watermark.

12. A computer device, characterized in that: The computer device includes: a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the image watermark extraction method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that The readable storage medium stores at least one program, and the at least one program is loaded and executed by a processor to implement the image watermark extraction method according to any one of claims 1 to 10.

14. A computer program product, characterized in that The computer program product includes at least one program segment, which is stored in a computer-readable storage medium; a processor of a computer device reads the at least one program segment from the computer-readable storage medium, and the processor executes the at least one program segment, so that the computer device executes the image watermark extraction method according to any one of claims 1 to 10.

Images