Method, apparatus, device and storage medium for embedding and extracting watermarks - Patents.com

Abstract

To provide a method, apparatus, device and computer-readable storage medium for watermark embedding and watermark extraction.SOLUTION: The method for watermark embedding includes: determining, based on a set of key points identified in an image to be processed, at least one image block to be processed in the image to be processed; generating, based on watermark information and the at least one image block to be processed, at least one target image block corresponding to the at least one image block to be processed; and generating a target image for the image to be processed by replacing the at least one image block to be processed with the at least one target image block in the image to be processed.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] Exemplary embodiments of the present invention relate generally to the field of image processing, and more particularly to methods, apparatus, devices, and computer-readable storage media for watermark embedding and watermark extraction. [Background technology]

[0002] With the popularization and rapid development of information technology, the propagation speed of digital carriers (e.g., images, videos, etc.) is getting faster and faster, and the propagation range is getting wider and wider. In order to better protect digital carriers, digital watermarking technologies such as dark watermarking or hidden watermarking have emerged. People can achieve the purpose of leakage traceability and copyright protection by embedding watermark information into digital carriers without affecting the use value of the original carriers. In view of this, a solution to realize reliable and robust watermarking is expected. Summary of the Invention

[0003] A first aspect of the present invention provides a watermark embedding method, the method including: determining at least one to-be-processed image block in the to-be-processed image based on a set of keypoints recognized in the to-be-processed image, generating at least one target image block respectively corresponding to the at least one to-be-processed image block based on watermark information and the at least one to-be-processed image block, each target image block having at least a part of the watermark information embedded therein, and generating a target image for the to-be-processed image by respectively replacing the at least one to-be-processed image block in the to-be-processed image with the at least one target image block.

[0004] A second aspect of the present invention provides a watermark extraction method, which includes: determining at least one target image block in the image to be extracted based on a set of keypoints recognized in the image to be extracted, and determining watermark information embedded in the image to be extracted based on the at least one target image block, respectively.

[0005] A third aspect of the present invention provides an apparatus for embedding a watermark, comprising: an image block determination module for determining at least one to-be-processed image block in the to-be-processed image based on a set of key points recognized in the to-be-processed image, an image block generation module for generating at least one target image block respectively corresponding to the at least one to-be-processed image block based on watermark information and the at least one to-be-processed image block, each target image block having at least a part of the watermark information embedded therein, and an image block replacement module for generating a target image for the to-be-processed image by replacing in the to-be-processed image at least one to-be-processed image block respectively with the at least one target image block.

[0006] A fourth aspect of the present invention provides an apparatus for extracting a watermark, comprising: a target image block determining module for determining at least one target image block in an image to be extracted based on a set of keypoints recognized in the image to be extracted, and a watermark information extracting module for determining watermark information embedded in the image to be extracted based on the at least one target image block, respectively.

[0007] A fifth aspect of the present invention provides an electronic device comprising at least one processing unit and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit that, when executed by the at least one processing unit, cause the device to perform a method of the first or second aspect.

[0008] A sixth aspect of the present invention provides a computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the method of the first or second aspect.

[0009] It should be understood that the contents described in the summary of the present invention are not intended to limit the main or important features of the embodiments of the present invention, and are not intended to limit the scope of the present invention. Other features of the present invention will be easily understood from the following description. [Brief description of the drawings]

[0010] The above and other features, advantages, and aspects of the embodiments of the present invention will become more apparent upon reference to the following detailed description in conjunction with the drawings, in which like or similar reference numerals indicate like or similar elements, and in which: [Figure 1A] FIG. 1 is a schematic diagram of an exemplary environment in which watermark embedding according to some embodiments of the present invention may be implemented; [Figure 1B] FIG. 1 is a schematic diagram of an exemplary environment in which watermark extraction according to some embodiments of the present invention may be implemented; [Diagram 2] 1 is a flow chart of a watermark embedding process according to some embodiments of the present invention; [Diagram 3] FIG. 2 is a schematic block diagram of a watermark embedding model according to some embodiments of the present invention; [Figure 4]1 is a flow chart of a process for watermark extraction in accordance with some embodiments of the present invention; [Diagram 5] FIG. 1 is a schematic diagram of a model training architecture according to some embodiments of the present invention; [Figure 6] FIG. 2 is a schematic diagram of an exemplary process for watermark embedding according to some embodiments of the present invention; [Figure 7] FIG. 2 is a schematic diagram of an exemplary process for watermark extraction according to some embodiments of the present invention; [Figure 8] FIG. 2 is a block diagram of a schematic structure of an apparatus for embedding watermarks according to a specific embodiment of the present invention; [Figure 9] FIG. 2 is a block diagram of a schematic structure of an apparatus for extracting a watermark according to a particular embodiment of the present invention; [Figure 10] 1 is a block diagram of an electronic device in which one or more embodiments of the present invention may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments of the present invention will be described in more detail with reference to the drawings. Although the drawings show specific embodiments of the present invention, it should be understood that the present invention can be realized in various forms and should not be construed as being limited to the embodiments described herein, but rather, these embodiments are provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are for illustrative purposes and are not used to limit the protection scope of the present invention.

[0012] In the description of the embodiments of the present invention, the term "comprises" and similar terms are open-ended inclusions including, but not limited to. The term "based on" is understood to mean "based at least in part on". The term "an embodiment" or "the embodiment" is understood to mean "at least one embodiment". The term "some embodiments" is understood to mean "at least some embodiments". In the following specification, explicit and implicit definitions may be included below. The term "model" as used herein may represent an association relationship between each data. For example, the above-mentioned association relationship may be obtained based on various technical solutions currently known and / or developed in the future.

[0013] It is understood that any data related to the technical solution (including but not limited to the data itself, the acquisition of the data, or the use of the data) should comply with the corresponding laws and regulations and related specified requirements.

[0014] It is understood that before using the technical solutions disclosed in each embodiment of the present invention, the types, scope of use, and scenarios of personal information related to the present invention should be notified to users in an appropriate manner in accordance with relevant laws and regulations, and consent from users should be obtained.

[0015] For example, in response to receiving an unsolicited request from a user, presenting information is sent to the user to explicitly present to the user that the requested operation requires the acquisition and use of the user's personal information, so that the user can independently choose whether or not to provide the personal information to software or hardware, such as an electronic device, application, server, or storage medium, that performs the operation of the technical solution of the present invention, based on the presenting information.

[0016] As an optional, but non-limiting implementation, the method of transmitting the presentation to the user in response to receiving the user's unsolicited request may be, for example, by utilizing a pop-up window in which the presentation may be displayed in the form of text and which may further include a selection control for the user to "agree" or "not agree" to providing the personal information to the electronic device.

[0017] It will be appreciated that the above-described notification and user authorization process is merely a general outline and is not intended to limit the implementation of the present invention, and other methods that comply with relevant laws and regulations may be applied to the implementation of the present invention.

[0018] The term "model" as used herein refers to a model that can learn the association relationship between corresponding inputs and outputs from training data and generate corresponding outputs for a given input after training is completed. The generation of the model can be based on machine learning techniques. Deep learning is a machine learning algorithm that uses multiple layers of processing units to process inputs and provide corresponding outputs. A neural network model is an example of a model based on deep learning. In this specification, a "model" may also be referred to as a "machine learning model", a "learning model", a "machine learning network", or a "learning network", and these terms are used interchangeably in this specification.

[0019] A "neural network" is a machine learning network based on deep learning. A neural network can process an input and provide a corresponding output, and typically includes an input layer, an output layer, and one or more hidden layers between the input layer and the output layer. Neural networks used in deep learning applications typically include multiple hidden layers, which increases the depth of the network. Each layer of a neural network is connected sequentially such that the output of the previous layer is provided as the input of the subsequent layer, the input layer receives the input of the neural network, and the output of the output layer serves as the final output of the neural network. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), and each node processes the input from the previous layer. Typically, machine learning may include three major stages, such as a training stage, a testing stage, and an application stage (also called an inference stage). In the training stage, a given model is trained using a large amount of training data, and parameter values ​​are repeatedly updated until the model can obtain consistent inferences from the training data that meet a desired goal. Through training, the model can be considered to be able to learn input-to-output associations (also called input-to-output mappings) from the training data. Parameter values ​​of the trained model are determined. In the testing stage, the performance of the model is determined by applying a test input to the trained model and testing whether the model can provide a correct output. In the application stage, the model can be used to process actual inputs based on the parameter values ​​obtained in training and determine the corresponding output.

[0020] 1A is a schematic diagram illustrating an example environment 100A in which watermark embedding according to some embodiments of the present invention can be implemented. As shown in FIG. 1A, the example environment 100A may include an electronic device 110.

[0021] The image 102 to be processed is an image in which to embed a watermark and may be an image of any format, any size, any color (e.g., color image, black and white image, grayscale image), etc. The image 102 to be processed may be an image that the electronic device 110 has obtained from another device, an image that is stored locally, or an image that the electronic device 110 has captured using its own image acquisition component (e.g., camera).

[0022] The watermark information 104 may be generally referred to as digital watermarking. The watermark information 104 may be any form of information, for example, a bit string. The electronic device 110 may generate a target image 112 for the image 102 to be processed based on the image 102 to be processed and the watermark information 104. That is, the electronic device 110 generates the target image 112 by embedding the watermark information 104 into the image 102 to be processed. In particular, in the target image 112, the watermark information may be invisible, i.e., imperceptible to the human eye.

[0023] In the environment 100A, the electronic device 110 may be any kind of device having computing capabilities, including a terminal device or a service side device. The terminal device may be any kind of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an e-book device, a game device, or any combination thereof, including accessories and peripherals of these devices, or any combination thereof.

[0024] The serving device may be an independent physical server, a server cluster or a distributed system consisting of multiple physical servers, or even a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks, big data and manual intelligence platforms, etc. The serving device may include computing systems / servers such as mainframes, edge computing nodes, electronic devices in a cloud environment, etc.

[0025] 1B is a schematic diagram illustrating an example environment 100B in which the watermark extraction of some embodiments of the present invention may be implemented. As shown in FIG. 1B, the example environment 100B may include an electronic device 120.

[0026] The image 106 to be extracted is an image in which a watermark may be embedded. The image 106 to be extracted may be any image, for example, of any format, any size, any color, etc. The image 106 to be extracted may be an image that the electronic device 120 obtains from another device via communication, or an image that is locally stored in the electronic device 120. The electronic device 120 may extract watermark information 122 from the image 106 to be extracted. The extracted watermark information 122 may be further verified to determine whether the extracted watermark is a correct watermark.

[0027] Similar to the electronic device 110 in the environment 100A, the electronic device 120 in the environment 100B may be any kind of device having computing capabilities, including a terminal device or a service device. In some embodiments, the electronic device 110 and the electronic device 120 may be the same device. For example, the owner of the image 102 to be processed may use the same device to perform watermark embedding on the image 102 to be processed and watermark extraction on the obtained image 106 to be extracted. In some embodiments, the electronic device 110 and the electronic device 120 may be different devices. For example, the owner of the image may use the electronic device 110 to embed a watermark in the image, and the user of the image may use the electronic device 120 to extract the watermark information in the image to verify the counterfeiting of the image.

[0028] It should be understood that the components and arrangements in environments 100A and 100B shown in Figures 1A and 1B are merely exemplary, and that a computing system suitable for implementing the exemplary embodiments described herein may include one or more different components, other components and / or different arrangements.

[0029] As mentioned above, watermark information that is imperceptible to the human eye can be added to an image, which is also called dark watermark. As a result, the desired purpose can be achieved without affecting the normal use of the image, for example, the purpose of leakage traceability and copyright protection can be achieved. Traditional solutions for dark watermark technology mainly include solutions based on image processing and solutions based on deep learning. Solutions based on image processing often need to be manually designed for different feature information of the image. Therefore, such solutions are complex to operate and cannot consider the robustness of the image against various changes. Solutions based on deep learning train an encoder-decoder model to realize watermark embedding and extraction, and introduce image changes into the training process to further highlight the data, thereby improving the robustness of the embedded watermark. However, traditional solutions based on deep learning can only embed watermarks for images that are consistent with the model input size, and cannot balance visual effect and robustness.

[0030] An exemplary embodiment of the present invention provides an improved solution for embedding and extracting watermarks, in which, at the watermark embedding side, a to-be-processed image block is determined based on keypoints recognized from the to-be-processed image; a target image block can be generated by embedding watermark information in the to-be-processed image block; and a target image with watermark information embedded therein can be obtained by replacing the to-be-processed image block with the target image block in the to-be-processed image. At the watermark extraction side, a target image block can be determined based on keypoints recognized from the to-be-processed image, and watermark information can be extracted from the target image block.

[0031] In an embodiment of the present invention, the keypoints are used to determine image blocks for embedding a watermark. The keypoints do not change obviously when the image is altered (e.g., compressed, noise added, rotated, etc.), which improves the robustness of the watermark embedding. Accordingly, when extracting a watermark, the keypoints are used to determine image blocks for extracting the watermark. This allows reliable location of image regions that may contain a watermark. It is noted that in an embodiment of the present invention, the watermark embedding and extraction are performed in units of image blocks, rather than the entire image. The method can process images of any resolution, thereby realizing a watermark embedding and extraction solution with broad applicability.

[0032] In the following, with continuing reference to the drawings, some exemplary embodiments of the present invention will be described.

[0033] 2 shows a flow chart illustrating a watermark embedding process 200 according to some embodiments of the present invention. Process 200 may be implemented in electronic device 110. For ease of discussion, process 200 will be described with reference to environment 100A of FIG. 1A.

[0034] In box 210, the electronic device 110 determines at least one image block to be processed in the image to be processed 102 based on a set of key points recognized in the image to be processed 102. In an embodiment of the present invention, a key point may be called a feature point, and refers to one or more pixels having visual characteristic properties. In some embodiments, the key point may be a corner point, and represents one or more pixels that are particularly emphasized on a certain attribute. For example, the key point may be an extreme point of a certain attribute. The image block to be processed may have a predetermined size, such as 128*128.

[0035] In some embodiments, these keypoints may be recognized by the electronic device 110. The electronic device 110 may utilize a keypoint detection algorithm to determine keypoints of the image 102 to be processed. Any suitable keypoint detection algorithm may be employed, such as, but not limited to, the Accelerated Segmentation Test by Feature (FAST) algorithm, the Scale Invariant Feature Transform (SIFT) algorithm, the Directional FAST, and the Rotational Binary Robust Basic Feature (ORB) algorithm. In some embodiments, the electronic device 110 may utilize an ORB keypoint detection algorithm to determine keypoints of the image 102 to be processed. The ORB keypoint detection algorithm has better robustness, and even if the image to be processed is subjected to various image processing operations such as compression, noise addition, cropping, panning, rotation, etc., the keypoints determined using the ORB keypoint detection algorithm will not undergo obvious changes. Therefore, in such embodiments, the use of the ORB algorithm helps to further improve the robustness of the watermark embedding.

[0036] In some embodiments, the electronic device 110 can further utilize the trained keypoint detection model to determine keypoints in the image 102 to be processed. The specific type of model structure can be selected according to actual application needs. In some embodiments, the keypoints in the image 102 to be processed can be recognized by other devices, and the electronic device 110 can obtain information about the recognized keypoints from the other devices.

[0037] The image blocks to be processed can be determined based on the key points in various suitable forms. In some embodiments, the electronic device 110 can directly determine at least one image block to be processed in the image to be processed 102 at a corresponding position in the image to be processed 102 based on the set of key points. For example, the electronic device 110 can determine a set of image blocks centered on each key point in the image to be processed 102 as the set of image blocks to be processed.

[0038] In some embodiments, the feature values ​​of the keypoints may be considered to determine the image block to be processed. The feature values ​​may be used to filter the recognized keypoints, and then the image block to be processed may be determined based on the filtered keypoints. The electronic device 110 may further determine a feature value for each keypoint while determining a set of keypoints. The feature value may represent the robustness of the corresponding keypoint, i.e., the probability that the point will be detected as a keypoint when the image is changed. Exemplarily, the electronic device 110 may utilize an ORB keypoint detection algorithm to determine a set of keypoints in the image 102 to be processed and the corresponding feature value for each keypoint.

[0039] The electronic device 110 can determine a set of candidate image blocks in the image 102 to be processed based on the corresponding feature values ​​of the recognized set of keypoints and their corresponding positions in the image 102 to be processed. For example, the set of keypoints can be sorted based on the corresponding feature values ​​of the set of keypoints. To ensure robustness of the keypoints obtained by sorting, the electronic device 110 can sort the set of keypoints in descending order based on the high and low corresponding feature values ​​of the set of keypoints. That is, the corresponding feature values ​​of the top keypoints are higher and the corresponding feature values ​​of the bottom keypoints are lower. For each keypoint, the electronic device 110 can determine an image block of a predetermined size (e.g., 128*128) that contains the keypoint in the image 102 to be processed. For example, the electronic device 110 can determine a set of image blocks centered around each keypoint of the set of keypoints in the image to be processed. The electronic device 110 may further sort the set of image blocks in order of image blocks having higher feature values ​​corresponding to the key points based on the sorting result of the key points, and accordingly, the image blocks corresponding to the key points having higher feature values ​​may be determined as candidate image blocks.

[0040] The candidate image blocks determined in the above manner may include multiple key points, and different candidate image blocks may overlap. In some embodiments, when selecting based on the feature values ​​of the key points, the overlapping between image blocks may be taken into consideration to obtain a set of candidate image blocks in which different candidate image blocks do not overlap with each other. For example, the electronic device 110 may determine whether image blocks overlap using an intersection-merging ratio (IOU) algorithm.

[0041] Further, the electronic device 110 can determine at least one image block to be processed from the set of candidate image blocks. In some embodiments, these candidate image blocks can be directly determined as the image block to be processed. In some embodiments, the image block to be processed can be determined from the set of candidate image blocks based on the sorting result of the feature values ​​corresponding to the key points, for example, a predetermined number of candidate image blocks can be selected as the image blocks to be processed. It should be noted that the number of image blocks to be processed should not be too large to ensure the visual effect of the final target image obtained by embedding the watermark. Exemplarily, the electronic device 110 can determine each of the candidate image blocks in the set of candidate image blocks, whose feature values ​​corresponding to the key points are sorted into the top 10, as the image block to be processed.

[0042] To improve the visual effect, in some embodiments, the electronic device 110 may further determine the texture complexity of each candidate image block of the set of candidate image blocks. The higher the texture complexity, the more difficult it is for the human eye to perceive the change of the corresponding image block. For example, the electronic device 110 may use any suitable texture calculation algorithm to determine the texture complexity of each candidate image block. The texture calculation algorithm may include, for example, a local binary mode (LBP) algorithm, a minimum perceptible error (JND) algorithm, etc. Illustratively, the electronic device 110 may use a JND algorithm to determine the texture complexity of each candidate image block. Alternatively or additionally, in some embodiments, the electronic device 110 may further determine the texture complexity of each candidate image block by using a trained texture calculation model. A specific type of model structure may be selected according to actual application needs.

[0043] The electronic device 110 may then select at least one image block to be processed from the set of candidate image blocks based on the corresponding texture complexities of the candidate image blocks. In some embodiments, the electronic device 110 may sort the set of candidate image blocks in descending order based on the corresponding texture complexities, and further select a predetermined number of the candidate image blocks from the set of candidate image blocks as the image blocks to be processed based on the sorting result. For example, the electronic device 110 may select the candidate image blocks with the top 10 corresponding texture complexities from the set of candidate image blocks, and determine the 10 candidate image blocks as the image blocks to be processed. In some embodiments, the electronic device 110 may compare the corresponding texture complexities of the set of candidate image blocks with a threshold, and select the image block to be processed from the set of candidate image blocks based on the comparison result. The threshold may be inputted into the electronic device 110 in advance by a user, or may be determined by the electronic device 110 itself. The electronic device 110 may determine the candidate image blocks with texture complexities higher than the threshold as the image blocks to be processed.

[0044] The higher the texture complexity, the more difficult it is for the human eye to perceive changes in the image. In view of this, selecting image blocks with high texture complexity to embed into the watermark can help obtain an embedded watermark that is difficult for the human eye to perceive. In this way, it can help improve the visual effect of the resulting target image.

[0045] Alternatively or additionally, in some embodiments, the image block to be processed can be selected from the candidate image blocks based on other attributes of the image block in addition to the texture complexity. Exemplarily, the color complexity of each candidate image block can be determined, for example, the more colors, the higher the complexity and / or the more chaotic the color distribution, the higher the complexity. Then, a predetermined number of candidate image blocks with high color complexity can be selected as the image block to be processed. Similar to the texture complexity, the higher the color complexity, the more difficult it is for the human eye to perceive changes in the image. The method can help improve the visual effect of the obtained target image.

[0046] It should be understood that the specific numbers and sizes of image blocks described above are merely exemplary and are not intended to limit the scope of the present invention. Other suitable numbers and sizes of image blocks are possible.

[0047] Continuing with the process 200, in box 220, the electronic device 110 generates at least one target image block, each corresponding to the at least one to-be-processed image block, based on the watermark information 104 and the at least one to-be-processed image block. Each target image block has at least a portion of the watermark information 104 embedded therein. For example, each target image block may have the entire watermark information 104 embedded therein. Also for example, the watermark information 104 may be divided into different portions, and each target image block may have one or more of these portions embedded therein. Embodiments of the present invention are not limited in this respect.

[0048] The electronic device 110 can employ any suitable watermark embedding technique to generate a corresponding target image block based on the watermark information and the image block to be processed, including, but not limited to, an overlay technique, a least significant bit technique (LSB), a quantization index modulation technique (QIM), a wavelet transform technique, etc.

[0049] In some implementations, the electronic device 110 can utilize one or more trained machine learning models to generate the target image blocks. The machine learning models may include, for example, a convolutional neural network (CNN), a feedforward neural network (FNN), a recurrent neural network (RNN), etc. The trained models may be deployed on the electronic device 110 or on other devices accessible by the electronic device 110.

[0050] In some embodiments, to further ensure sufficient integration of the watermark information with the image blocks and improve the robustness of the watermark embedding, the electronic device 110 may realize the watermark embedding through multi-level stitching of the watermark information with the image blocks. For example, the watermark information 104 (e.g., represented by a binary string) may be vectorized or tokenized to obtain a feature of the watermark information 104, which may be viewed as a vectorized representation of the watermark information 104 and may be referred to as a watermark feature. In some embodiments, a watermark binary string may be obtained by adding error correction bits using a Bose-Ray-Chad-Hooley-Hockholm (BCH) code to the binary watermark information. By way of example and not limitation, a 64-bit watermark binary string may be obtained, which may include 24 error correction bits. Then, a feature of the watermark binary string to which the error correction bits have been added may be determined.

[0051] Accordingly, for each image block to be processed, obtain features of the image block to be processed by performing feature extraction on the image block to be processed. Any suitable feature extraction algorithm is possible. Furthermore, the electronic device 110 can generate a fusion feature of the watermark information and the image block to be processed based on the features of the watermark information 104 and the features of the image block to be processed. It is understood that, before feature fusing the features of the watermark information 104 and the features of the image block to be processed, the electronic device 110 can first adjust the watermark features and the features of the image block to be processed to have the same dimension.

[0052] The electronic device 110 can further generate a target image block based on the fusion feature, the watermark feature, and the image block to be processed. In some embodiments, the electronic device 110 can obtain the target image block by fusing the fusion feature, the watermark feature, and the feature of the image block to be processed multiple times. In some embodiments, the electronic device 110 can utilize a trained watermark embedding model to realize embedding of the watermark information into the image block. Such an exemplary embodiment is described with reference to FIG. 3 below.

[0053] 3 shows a schematic block diagram of a watermark embedding model 300 according to some embodiments of the present invention. As shown in FIG. 3, the watermark embedding model 300 includes a linear layer 310, a feature extraction layer 320-1, a feature extraction layer 320-2 (collectively referred to as feature extraction layers 320), and feature fusion layers 330-1, 330-2, 330-3, 330-4, 330-5 (collectively referred to as feature fusion layers 330). In some embodiments, the feature extraction layer 320 and the feature fusion layer 330 may include at least one or more of a multi-channel attention (SeNet) layer, a convolution layer (Conv), an upsampling layer, a downsampling layer, a normalization (BN) layer, and an activation (Relu) layer. Overall, the watermark embedding model 300 may include a portion for extracting watermark information features (which in this example includes a linear layer 310 and a feature extraction layer 320-1), a portion for extracting image block features (which in this example includes a feature extraction layer 320-2), and a portion for performing multi-level stitching between the watermark information and the image blocks (which in this example includes a feature fusion layer 330-1, a feature fusion layer 330-2, and a feature fusion layer 330-3).

[0054] The watermark information 301 shown in Fig. 3 can be regarded as an example of the watermark information 104, and the image block to be processed 303 can be regarded as an example of the image block to be processed determined in box 201. The watermark information 301 and the image block to be processed 303 are input to a watermark embedding model 300. The watermark embedding model 300 obtains watermark features 302 of the watermark information 301 through a linear layer 310 and a feature extraction layer 320-1. Exemplarily, the linear layer 310 may be a simple fully connected neural network. The watermark embedding model 300 obtains features 304 of the image block to be processed through a feature extraction layer 320-2. The watermark features 302 and the features 304 of the image block to be processed may be provided together to a feature fusion layer 330-1 to obtain a fusion feature.

[0055] In order to enhance the expressiveness of the watermark information 301 in the final target image block 305, in some embodiments, the watermark embedding model 300 may generate intermediate features enhanced with the watermark information 301 based on the fusion features and the watermark features 302. As shown in Fig. 3, the fusion features and the watermark features 302 output by the feature fusion layer 330-1 may be provided together to a feature fusion layer 330-2 to obtain a first intermediate feature enhanced with the watermark information 301. The first intermediate feature and the watermark features 302 output by the feature fusion layer 330-2 may be provided together to a feature fusion layer 330-3 to obtain a second intermediate feature enhanced with the watermark information 301.

[0056] Further, the intermediate feature enhanced by the watermark information and the image block 303 to be processed may be combined to generate the target image block 305. In some embodiments, in order to ensure a higher similarity between the target image block 305 and the image block 303 to be processed, the watermark embedding model 300 may further generate the target image block 305 based on the intermediate feature and the image block 303 to be processed, and the target image block 305 thus generated is enhanced by the image block 303 to be processed. As shown in FIG. 3, the second intermediate feature output by the feature fusion layer 330-3 and the image block 303 to be processed may be provided together to the feature fusion layer 330-4 to obtain a first target feature. The first target feature output by the feature fusion layer 330-4 and the image block 303 to be processed may be provided together to the feature fusion layer 330-5 to obtain the target image block 305.

[0057] According to the method, the watermark information and the image block to be processed can be well fused, which helps to improve the robustness of watermark embedding. It should be understood that the types, numbers and connection relationships of layers of the watermark embedding model shown in Figure 3 are merely exemplary and are not intended to limit the scope of the present invention. Any suitable structure may be adopted to realize the watermark embedding model.

[0058] The above describes an example of first highlighting the fusion features in the watermark information and then highlighting them in the image block to be processed. In some other embodiments, other forms can also be adopted to generate a target image block based on the fusion features, the features of the watermark information, and the features of the image block to be processed. For example, the fusion features, the features of the watermark information, and the features of the image block to be processed can be stitched together, and then the stitched features can be used to generate a target image block.

[0059] Returning to FIG 2, for each to-be-processed image block in the at least one to-be-processed image block determined in box 210, the electronic device 110 may generate a corresponding target image block using the watermark embedding model 300, both of which have been described above, to obtain the at least one further target image block.

[0060] In box 230, the electronic device 110 replaces at least one to-be-processed image block in the to-be-processed image 102 with at least one target image block, respectively, to generate a target image 112 for the to-be-processed image 102. For each to-be-processed image block, the electronic device 110 can obtain the target image 112 by replacing the to-be-processed image block with the generated corresponding target image block. Since the target image blocks have watermark information embedded therein, the target image 112 including the target image blocks after replacement is an image with watermark information embedded therein.

[0061] Above, several exemplary embodiments of the watermark embedding of the present invention are described. In general, the key points in the image can be used to determine the image blocks for watermark embedding. This method can realize watermark embedding with high robustness and wide applicability. It should be noted that, compared with embedding the watermark in the entire image, this solution embeds the watermark in a part of the image that is not perceptible to the human eye, so that it has obvious improvement effects not only on visual effect but also on the increase of image data amount. In the following, several exemplary embodiments of the watermark extraction of the present invention will be described in conjunction with the drawings.

[0062] 4 shows a flow chart of a process 400 for watermark extraction according to some embodiments of the present invention. The process 400 may be implemented in the electronic device 120. For ease of discussion, the process 200 will be described with reference to the environment 100B of FIG. 1B.

[0063] In box 410, the electronic device 120 determines at least one target image block in the image 106 to be extracted based on the set of keypoints recognized in the image 106 to be extracted. Similar to the recognition of keypoints in the image 102 to be processed, the electronic device 120 can utilize any suitable keypoint detection algorithm or trained keypoint detection model to determine the set of keypoints in the image 106 to be extracted. No further description will be given here on the recognition of keypoints in the image 106 to be extracted.

[0064] The target image blocks can be determined based on the key points in various suitable forms. In some embodiments, the electronic device 120 can determine at least one target image block in the image 106 to be extracted based on the corresponding positions of the set of key points in the image 106 to be extracted directly. For example, the electronic device 120 can determine a set of image blocks centered on each key point in the image 106 to be extracted as a set of target image blocks.

[0065] In some embodiments, the feature values ​​of the keypoints may be considered to determine the target image block. The feature values ​​may be used to filter the recognized keypoints, and then the target image block may be determined based on the filtered keypoints. The electronic device 120 may further determine a feature value for each keypoint while determining a set of keypoints. The feature value may represent the robustness of the corresponding keypoint, i.e., the probability that the point will be detected as a keypoint when the image is changed. Exemplarily, the electronic device 120 may use an ORB keypoint detection algorithm to determine a set of keypoints in the image 106 to be extracted and the corresponding feature value for each keypoint.

[0066] The electronic device 120 can determine a set of candidate image blocks in the image 106 to be extracted based on the corresponding feature values ​​of the recognized set of keypoints and their corresponding positions in the image 106 to be extracted. For example, the set of keypoints can be sorted based on the corresponding feature values ​​of the set of keypoints. To ensure robustness of the keypoints obtained by sorting, the electronic device 120 can sort the set of keypoints in descending order based on the high and low corresponding feature values ​​of the set of keypoints. That is, the corresponding feature values ​​of the top keypoints are higher, and the corresponding feature values ​​of the bottom keypoints are lower. For each keypoint, the electronic device 120 can determine an image block of a first size (e.g., 64*64) that contains the keypoint in the image 106 to be extracted. For example, the electronic device 120 can determine a set of image blocks centered around each keypoint of the set of keypoints in the image 106 to be extracted. The electronic device 120 may further sort the set of image blocks in order of image blocks having higher feature values ​​corresponding to the key points based on the sorting result of the key points, and accordingly, the image blocks corresponding to the key points having higher feature values ​​may be determined as candidate image blocks.

[0067] The candidate image blocks determined in the above manner may include multiple key points, and different candidate image blocks may overlap. In some embodiments, when selecting based on the feature values ​​of the key points, the overlapping between image blocks may be considered, so that a set of candidate image blocks in which different candidate image blocks do not overlap with each other can be obtained. For example, the electronic device 110 may use an IOU algorithm to determine whether image blocks overlap.

[0068] Further, the electronic device 120 can determine at least one target image block from the set of candidate image blocks. In some embodiments, these candidate image blocks can be directly determined as image blocks to be processed. In some embodiments, the target image block can be determined from the set of candidate image blocks based on the sorting result of the feature values ​​corresponding to the keypoints, for example, a predetermined number of candidate image blocks can be selected as the target image blocks. It should be noted that the number of target image blocks selected in the watermark extraction process is larger than the number of image blocks to be processed selected in the watermark embedding process. This method can improve the probability of hitting the keypoints used in embedding. Exemplarily, the electronic device 120 can sort the top 50 feature values ​​corresponding to the keypoints in a set of candidate image blocks to determine 50 candidate image blocks as the target image blocks.

[0069] In some embodiments, the size of the target image block may be larger than the size of the candidate image block. When the candidate image block has a first size and the target image block has a second size, the second size is larger than the first size. In such embodiments, for each candidate image block, the electronic device 120 may determine, based on the position of the candidate image block in the image 106 to be extracted, an image block in the image 106 to be extracted that includes a key point in the candidate image block and has a second size as the target image block. For example, an image block centered on the key point and having the second size may be determined as the target image block. Exemplarily, when the candidate image block has a first size of 64*64, the target image block may have a second size of 128*128. In such an example, an image region centered on each key point and having a size of 128*128 may be determined as the target image block. In such embodiments, the candidate image block and the corresponding target image block may have the same center, for example, the center may be the corresponding key point. Note that different candidate image blocks are not overlapped with each other, but different target image blocks may be overlapped.

[0070] Continuing with the process 400, in box 420, the electronic device 120 determines the watermark information 122 embedded in the image 106 to be extracted based on at least one target image block each. That is, the watermark information can be extracted from each of these target image blocks. For example, if there are multiple target image blocks, multiple binary watermark sequences can be obtained. Then, a correction algorithm (e.g., BCH correction algorithm) can be used to verify, and the watermark sequence that is successfully verified can be the correct watermark sequence.

[0071] In box 420, the electronic device 120 may employ any suitable watermark extraction technique to extract the watermark information from the target image block, which may include, but is not limited to, for example, least significant bit (LSB) techniques, discrete cosine transform (DCT), etc.

[0072] In some embodiments, the electronic device 120 may utilize one or more trained machine learning models to extract the watermark information 122. The neural networks may include, for example, a convolutional neural network (CNN), a feedforward neural network (FNN), a recurrent neural network (RNN), etc. These trained models may be deployed on the electronic device 120 or on other devices accessible by the electronic device 120. In particular, in some embodiments, the machine learning models for the extraction of the watermark may be trained together with the machine learning models for the embedding of the watermark.

[0073] Above are described some exemplary embodiments of the watermark extraction of the present invention. Overall, the key points in the image can be used to determine the image blocks for extracting the watermark. This method can realize the watermark extraction conveniently and quickly, while ensuring the accuracy of the watermark extraction.

[0074] As mentioned above, in some embodiments, the electronic device 110 can use the trained watermark embedding model to embed a watermark into an image block to be processed, and the electronic device 120 can use the trained watermark extraction model to extract the watermark from the target image block. In some embodiments, the watermark embedding model and the watermark extraction model may be trained together.

[0075] 5 shows a schematic diagram of a model training architecture 500 according to some embodiments of the present invention. As shown in FIG. 5, the architecture 500 includes a watermark embedding model 510 (e.g., the watermark embedding model 300), a transformation module 520, and a watermark extraction model 530.

[0076] The reference watermark information 502 and the first image 504 are input to a watermark embedding model 510. The watermark embedding model 510 embeds the reference watermark information 502 into the first image 504 to obtain a second image 514.

[0077] The transformation module 520 can perform multiple transformation operations on the second image 514 to generate a third image 524. That is, the transformation module 520 is used to transform the second image 514 into the third image 524. The multiple transformation operations performed may be any combination of two or more of the candidate transformation operations. Examples of the candidate transformation operations may include, but are not limited to, compression, noise addition (e.g., high-speed noise addition), cropping, color conversion, rotation, scaling, etc. A random combination of multiple transformation operations can be adopted in one forward propagation. According to this method, the model can be made to have the ability to resist image changes, and the robustness of the model is effectively improved.

[0078] In some embodiments, the degree of change of the second image 514 due to each conversion operation in the multiple conversion operations may remain constant. In some other embodiments, the degree of change of the second image 514 due to at least one conversion operation of the multiple conversion operations increases as the number of training rounds increases. For example, the degree of change may be a linear growth mode. The degree of change of the second image 514 due to the conversion operation is associated with an intensity coefficient of the conversion operation. The degree of change increases as the number of training rounds increases, meaning that the intensity of the conversion operation may grow linearly as the number of training rounds increases. For example, if the conversion operation is trimming, the proportion of trimming may be increased as the number of training rounds increases. Also, for example, if the conversion operation is noise addition, the amount or intensity of the noise added may be increased as the number of training rounds increases. As an example, one or more conversion operations may be selected from the multiple conversion operations in each round of training and their degree of change may be increased. As another example, the degree of change of each conversion operation may be increased in each round of training.

[0079] As a result, in the early stage of model training, a weaker transformation operation is used for training, and as the number of training rounds increases, the strength of the transformation operation is increased, which makes the model easier to converge and improves the training effect of the model. This can be viewed as a gradual data enhancement policy, which can effectively improve the robustness of the model to different degrees of image changes.

[0080] Continuing with the architecture 500, the third image 524 is provided to a watermark extraction model 530. The watermark extraction model 530 can extract watermark information 532 from the third image 524.

[0081] In some embodiments, the watermark embedding model 510 and the watermark extraction model 530 may be trained based on an objective function related to the difference between the watermark information 532 and the reference watermark information 502. The objective function determines that the extracted watermark information matches the actual watermark information as closely as possible. The objective function here may be a loss function. The loss function used for the difference in the watermark information may include, for example, a mean squared error (MSE) loss function, but other loss functions are also possible, such as an L1 loss function, an L2 loss function, etc. In the example of FIG. 5, an extraction loss 508 may be determined based on the difference between the reference watermark information 502 and the extracted watermark information 532. Backpropagation of the extraction loss 508 may train the watermark embedding model 510 and the watermark extraction model 530.

[0082] In addition to the extraction loss 508, the training of the watermark embedding model 510 and / or the watermark extraction model 530 may use other objective functions or loss functions. As shown in FIG. 5, an embedding loss 541 can be determined based on the difference between the first image 504 and the second image 514. The embedding loss 541 is for ensuring that the image in which the watermark is embedded and the original image are as similar as possible. Exemplarily, the embedding loss 541 can adopt an MSE loss function. In some embodiments, the visual effect of the watermark image can be further improved by additionally adopting a multi-scale structural similarity (MS-SSIM) loss function.

[0083] At least one of a mean squared error (MSE) loss function, a structural similarity measure (SSIM) loss function, a binary cross entropy (BCE) loss function, etc. Specifically, the watermark embedding model 510 may be trained based on an embedding loss 541 between the first image 504 and the second image 514, and the watermark extraction model 530 may be trained based on an extraction loss 508 between the reference watermark information 502 and the watermark information 532. The training goal of the watermark embedding model 510 includes at least making the embedding loss 541 smaller than a first threshold, and the training goal of the watermark extraction model 530 includes at least making the extraction loss 508 smaller than a second threshold.

[0084] In some embodiments, the watermark embedding model 510 can be trained in an adversarial manner. In such embodiments, the architecture 500 may further include a classifier 540. Training the classifier 540 may include two goals. One goal is to have the classifier 540 determine the first image 504 (i.e., the original image) as true and the second image 514 (i.e., the image with the watermark embedded) as false. This goal is represented by the adversarial loss 542. Another goal is to have the classifier 540 determine the second image 514 as true. The other goal is represented by the embedding loss 543. Illustratively, a binary cross-entropy (BCE) loss function can be used for the classifier 540.

[0085] Any suitable training set and validation set can be adopted for training the model. The images in the training set and validation set are scaled to a predefined image block size (e.g., 128*128), and the watermark information also has the same length (e.g., 64 bits). Reasonable weights can then be assigned to the various loss functions mentioned above. Any suitable optimizer (e.g., Adam optimizer) plus learning rate decay policy (e.g., exponential decay policy) can be adopted.

[0086] Above, some exemplary embodiments of watermark embedding and watermark extraction are described with reference to Figures 2 to 5, respectively. Below, exemplary processes of watermark embedding and watermark extraction are described in conjunction with Figures 6 and 7, respectively.

[0087] 6 shows a schematic diagram of a watermark embedding process 600 according to some embodiments of the present invention. The process 600 may be implemented in the electronic device 110. For ease of discussion, the process 600 will be described with reference to the environment 100A of FIG. 1A.

[0088] In box 610, the electronic device 110 obtains the image 102 to be processed and the watermark information 104. The image 102 to be processed may be an image of any resolution.

[0089] In box 620, the electronic device 110 determines keypoints in the image 102 to be processed and sorts the keypoints in descending order based on their corresponding high and low feature values. For example, an ORB keypoint algorithm can be employed to recognize the keypoints and determine the feature value of each keypoint. Further, sorting can be done from high to low based on the feature value.

[0090] In box 630, the electronic device 110 ensures that candidate image blocks of size a*a centered on each keypoint are not overlapped with each other by starting from the keypoints corresponding to high feature values, where a*a may be, for example, 128*128. An IOU algorithm may be employed to determine whether the image blocks are overlapped.

[0091] In box 640, the electronic device 110 determines the corresponding texture complexity of each candidate image block of size a*a, and selects the N candidate image blocks with the highest complexity as the image blocks to be processed by sorting them in descending order based on the complexity, where N may be 10, for example. Illustratively, the texture complexity may be calculated using the JND algorithm.

[0092] In box 650, the electronic device 110 inputs the image block to be processed into the trained watermark embedding model to obtain a target image block, also called a watermark image block, in which watermark information is embedded. For example, the binary watermark information may be added with error correction bits using the BCH algorithm, and the resulting 64-bit watermark binary string may include 24 error correction bits. The selected image block is first pre-processed and then input into the trained watermark embedding model together with the watermark information.

[0093] In box 660, the electronic device 110 generates a watermark image by replacing the corresponding to-be-processed image blocks with the generated target image blocks, i.e., replacing the original image blocks with the watermark image blocks to obtain the watermark image.

[0094] 7 shows a schematic diagram of an exemplary process 700 for watermark extraction according to some embodiments of the present invention. Process 700 may be implemented in electronic device 120. For ease of discussion, process 700 will be described with reference to environment 100B of FIG. 1B.

[0095] In box 710, the electronic device 120 obtains the image 106 to be extracted. The image 106 to be extracted is an image that may contain a watermark.

[0096] In box 720, the electronic device 120 determines which keypoints in the image 106 to extract and sorts the keypoints in descending order based on their corresponding high and low feature values.

[0097] In box 730, the electronic device 120 ensures that candidate image blocks of size b*b centered on each keypoint are not overlapped with each other by starting from keypoints corresponding to high feature values, where b*b may be, for example, 64*64. An IOU algorithm may be employed to determine whether the image blocks are overlapped.

[0098] In box 740, the electronic device 120 determines the top M keypoints whose corresponding candidate image blocks are not overlapped with each other, and determines target image blocks of size c*c centered on each of these M keypoints, where M is, for example, 50 and c*c is, for example, 128*128.

[0099] In box 750, the electronic device 120 inputs the target image block into the trained watermark extraction model to obtain watermark information. For example, the selected image block is input into the trained watermark extraction model after preprocessing to obtain the probability orientation of the watermark length being the same. By setting an appropriate threshold, multiple binary watermark sequences can be obtained. Then, the BCH correction algorithm can be used to verify, and the watermark sequence that is successfully verified can be regarded as the correct watermark sequence.

[0100] 8 shows a schematic structural block diagram of an apparatus 800 for embedding a watermark in a particular embodiment according to the present invention. The apparatus 800 can be implemented as or included in the electronic device 110. Each module / component in the apparatus 800 can be implemented by hardware, software, firmware, or any combination thereof.

[0101] As shown in the drawing, the apparatus 800 comprises an image block determination module 810 for determining at least one to-be-processed image block in the to-be-processed image based on a set of key points recognized in the to-be-processed image. The apparatus 800 further comprises an image block generation module 820 for generating at least one target image block corresponding to the at least one to-be-processed image block respectively based on the watermark information and the at least one to-be-processed image block, where each target image block is embedded with at least a portion of the watermark information. The apparatus 800 further comprises an image block replacement module 830 for generating a target image for the to-be-processed image by respectively replacing the at least one to-be-processed image block in the to-be-processed image with the at least one target image block.

[0102] In some embodiments, the image block determination module 810 comprises: a candidate image block determination module for determining a set of candidate image blocks in the image to be processed based on corresponding feature values ​​of the set of key points and corresponding positions of the set of key points in the image to be processed, where each candidate image block includes at least one key point from the set of key points; and a to-be-processed image block determination module for determining at least one to-be-processed image block from the set of candidate image blocks.

[0103] In some embodiments, different candidate image blocks of a set of candidate image blocks are not overlapped with each other.

[0104] In some embodiments, the image block to be processed determination module comprises: a texture complexity determination module for determining corresponding texture complexities of a set of candidate image blocks; and an image block selection module for selecting at least one image block to be processed from the set of candidate image blocks based on the corresponding texture complexities.

[0105] In some embodiments, the image block generation module 820 further obtains a target image block corresponding to the image block to be processed by generating, for the image block to be processed of at least one image block to be processed, a fusion feature of the watermark information and the image block to be processed based on the features of the watermark information and the features of the image block to be processed, and generating a target image block based on the fusion feature, the features of the watermark information, and the image block to be processed.

[0106] In some embodiments, the image block generation module 820 comprises an intermediate feature generation module for generating intermediate features enhanced with the watermark information based on the fusion features and features of the watermark information, and a target image block generation module for generating a target image block based on the intermediate features and the image block to be processed.

[0107] In some embodiments, a target image block corresponding to the image block to be processed is obtained using a watermark embedding model, and training the watermark embedding model is performed by embedding reference watermark information in a first image using the watermark embedding model to obtain a second image, performing a number of transformation operations on the second image to generate a third image, determining a difference between the watermark information extracted from the third image and the reference watermark information, and training the watermark embedding model based on an objective function related to the difference.

[0108] In some embodiments, the degree of change of the second image due to at least one transformation operation of the plurality of transformation operations increases as the number of training rounds increases.

[0109] 9 shows a block diagram of a schematic structure of an apparatus 900 for extracting a watermark in a particular embodiment according to the present invention. The apparatus 900 can be implemented as or included in the terminal device 120. Each module / component in the apparatus 900 can be implemented by hardware, software, firmware, or any combination thereof.

[0110] As shown in the drawing, the apparatus 900 includes a target image block determination module 910 for determining at least one target image block in the image to be extracted based on a set of key points recognized in the image to be extracted. The apparatus 900 further includes a watermark information extraction module 920 for determining watermark information embedded in the image to be extracted based on the at least one target image block, respectively.

[0111] In some embodiments, the target image block determination module 910 comprises: a candidate image block determination module for determining a set of candidate image blocks in the image to be extracted based on corresponding feature values ​​of the set of key points and corresponding positions of the set of key points in the image to be extracted, where each candidate image block includes at least one key point from the set of key points; and an image block selection module for determining at least one target image block based on the set of candidate image blocks.

[0112] In some embodiments, different candidate image blocks of a set of candidate image blocks are not overlapped with each other.

[0113] In some embodiments, the set of candidate image blocks each have a first size, and the image block selection module further determines, for a candidate image block of the set of candidate image blocks, an image block that includes the keypoint and has a second size as one of the at least one target image block based on a position of the keypoint included in the candidate image block in the image to be extracted, where the second size is greater than the first size.

[0114] In some embodiments, the watermark information is determined using a watermark extraction model, and training the watermark extraction model is performed by obtaining a second image by embedding reference watermark information in a first image, generating a third image by performing a number of transformation operations on the second image, determining a difference between the watermark information extracted from the third image using the watermark extraction model and the reference watermark information, and training the watermark extraction model based on an objective function related to the difference.

[0115] In some embodiments, the degree of change of the second image due to at least one transformation operation of the plurality of transformation operations increases as the number of training rounds increases.

[0116] 10 illustrates a block diagram of an electronic device 1000 capable of implementing one or more embodiments of the present invention. It should be understood that the electronic device 1000 illustrated in FIG. 10 is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. The electronic device 1000 illustrated in FIG. 10 may be used to implement the electronic device 110 and / or the electronic device 120 of FIG. 1.

[0117] As shown in Fig. 10, the electronic device 1000 is in the form of a general-purpose electronic device. The components of the electronic device 1000 may include, but are not limited to, one or more processors or processing units 1010, a memory 1020, a storage device 1030, one or more communication units 1040, one or more input devices 1050, and one or more output devices 1060. The processing unit 1010 may be a real or virtual processor and may perform various processes based on a program stored in the memory 1020. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel, thereby improving the parallel processing capabilities of the electronic device 1000.

[0118] The electronic device 1000 typically includes a number of computer storage media. Such media may be any obtainable media accessible by the electronic device 1000, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 1020 may be volatile memory (e.g., registers, high-speed cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 1030 may be removable or non-removable media and may include machine-readable media, such as a flash memory drive, a magnetic disk, or any other media, that may be used to store information and / or data (e.g., training data for training) and may be accessible within the electronic device 1000.

[0119] The electronic device 1000 may further include other removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 10, a magnetic disk drive for reading from or writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a path (not shown) by one or more data media interfaces. The memory 1020 may include a computer program product 1025 having one or more program modules, which are configured to perform various methods or operations of various embodiments of the present invention.

[0120] The communication unit 1040 provides communication with other computing devices over a communication medium. Additionally, the functionality of the components of the electronic device 1000 may be implemented as a single computing cluster or multiple computing machines that can communicate over a communication connection. Thus, the electronic device 1000 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or other network nodes.

[0121] The input device 1050 may be one or more input devices such as a mouse, a keyboard, a trackball, etc. The output device 1060 may be one or more output devices such as a display, a speaker, a printer, etc. The electronic device 1000 may further communicate, as necessary, with one or more external devices (not shown), such as a storage device, a display device, etc., via a communication unit 1040, with one or more devices that allow a user to interact with the electronic device 1000, or with any device (e.g., a netbook card, a modem, etc.) that allows the electronic device 1000 to communicate with one or more other computing devices. Such communication may be performed via an input / output (I / O) interface (not shown).

[0122] An exemplary implementation according to the present invention provides a computer-readable storage medium having computer-executable instructions stored thereon, the computer-executable instructions being executed by a processor to implement the above-mentioned method.An exemplary implementation according to the present invention further provides a computer program product, the computer program product including computer-executable instructions being tangibly stored on a non-transitory computer-readable medium, the computer-executable instructions being executed by a processor to implement the above-mentioned method.

[0123] Aspects of the present invention have been described herein with reference to flowcharts and / or block diagrams of methods, apparatus, devices and computer program products implemented by the present invention. It will be understood that each box in the flowcharts and / or block diagrams, and combinations of boxes in the flowcharts and / or block diagrams, may be implemented by computer readable program instructions.

[0124] These computer readable program instructions may be provided to a processing unit of a general purpose computer, special purpose computer, or other programmable data processing apparatus to generate a machine that, when executed by a processing unit of the computer or other programmable data processing apparatus, generates an apparatus for implementing the functions / operations specified in one or more boxes in the flowcharts and / or block diagrams. These computer readable program instructions may be stored on a computer readable storage medium such that the instructions cause the computer, programmable data processing apparatus, and / or other device to operate in a particular manner such that the computer readable medium on which the instructions are stored constitutes an article of manufacture containing instructions that implement each aspect of the functions / operations specified in one or more boxes in the flowcharts and / or block diagrams.

[0125] Loading the computer-readable program instructions into a computer, other programmable data processing apparatus, or other device causes the computer, other programmable data processing apparatus, or other device to perform a series of operational steps to produce a computer-implemented process, such that the instructions executing on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in one or more boxes in the flowcharts and / or block diagrams.

[0126] The flowcharts and block diagrams in the figures illustrate possible architectures, functions, and operations of systems, methods, and computer program products of various implementations according to the present invention. In this regard, each box in the flowcharts or block diagrams may represent a module, program segment, or part of instructions, which includes one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions depicted in the boxes may occur in a different order than depicted in the figures. For example, two consecutive boxes may in fact be executed substantially in parallel, or may be executed in a reverse order depending on the functions involved. It should also be noted that each box in the block diagrams and / or flowcharts, and combinations of boxes in the block diagrams and / or flowcharts, may be implemented by a special-purpose hardware-based system that performs the specified functions or operations, or may be implemented by a combination of special-purpose hardware and computer instructions.

[0127] Although each implementation of the present invention has been described above, the above description is illustrative, not exhaustive, and is not limited to each disclosed implementation. Many modifications and changes will be apparent to those skilled in the art without departing from the scope and spirit of each described implementation. The selection of terms used in this specification is intended to best interpret the principles, practical applications, or improvements to the technology in the marketplace of each implementation, or to allow those skilled in the art to understand each implementation disclosed in this specification.

Claims

1. determining at least one image block to be processed in the image to be processed based on a set of keypoints recognized in the image to be processed; generating at least one target image block corresponding to the at least one image block to be processed based on watermark information and the at least one image block to be processed, each target image block having at least a portion of the watermark information embedded therein; generating a target image for the image to be processed by replacing in the image to be processed at least one of the at least one target image block, A target image block corresponding to the image block to be processed is obtained using a watermark embedding model; Training the watermark embedding model includes: Embedding reference watermark information into a first image using the watermark embedding model to obtain a second image; performing a plurality of transformation operations on the second image to generate a third image; determining a difference between the watermark information extracted from the third image and the reference watermark information; training the watermark embedding model based on an objective function related to the differences. Watermark embedding method.

2. Determining the at least one image block to be processed comprises: determining a set of candidate image blocks in the image to be processed based on corresponding feature values ​​of the set of keypoints and corresponding locations in the image to be processed, each candidate image block including at least one keypoint of the set of keypoints; determining the at least one image block to be processed from among the set of candidate image blocks. The watermark embedding method according to claim 1.

3. different candidate image blocks of the set of candidate image blocks are not overlapped with each other; The watermark embedding method according to claim 2.

4. Determining the at least one image block to be processed comprises: determining corresponding texture complexities of the set of candidate image blocks; selecting the at least one image block to process from the set of candidate image blocks based on the corresponding texture complexity. The watermark embedding method according to claim 2.

5. Generating the at least one target image block comprises: For an image block to be processed of the at least one image block to be processed, generating a fusion feature of the watermark information and the image block to be processed based on the feature of the watermark information and the feature of the image block to be processed; generating the target image block based on the fusion feature, the watermark information feature, and the image block to be processed; and obtaining a target image block corresponding to the image block to be processed. The watermark embedding method according to claim 1.

6. generating the target image block generating intermediate features enhanced with the watermark information based on the fusion features and features of the watermark information; generating the target image block based on the intermediate features and the image block to be processed. The watermark embedding method according to claim 5.

7. a degree of change of the second image due to at least one of the plurality of transformation operations increases as the number of training rounds increases; The watermark embedding method according to claim 1.

8. determining at least one target image block in the image to be extracted based on a set of keypoints recognized in the image to be extracted; determining watermark information embedded in the image to be extracted based on the at least one target image block, respectively; the watermark information is determined utilizing a watermark extraction model; Training the watermark extraction model includes: embedding the reference watermark information into the first image to obtain a second image; performing a plurality of transformation operations on the second image to generate a third image; determining a difference between watermark information extracted from the third image using the watermark extraction model and the reference watermark information; training the watermark extraction model based on an objective function related to the differences. Watermark extraction method.

9. Determining the at least one target image block comprises: determining a set of candidate image blocks in the image to be extracted based on corresponding feature values ​​of the set of keypoints and corresponding positions of the set of keypoints in the image to be extracted, each candidate image block including at least one keypoint of the set of keypoints; determining the at least one target image block based on the set of candidate image blocks. The method of claim 8 .

10. an image block determination module configured to determine at least one image block to be processed in the image to be processed based on a set of key points recognized in the image to be processed; an image block generation module configured to generate, based on watermark information and the at least one image block to be processed, at least one target image block corresponding to the at least one image block to be processed, each target image block having at least a portion of the watermark information embedded therein; an image block replacement module configured to generate a target image for the image to be processed by replacing in the image to be processed at least one image block to be processed with the at least one target image block, A target image block corresponding to the image block to be processed is obtained using a watermark embedding model; Training the watermark embedding model includes: Embedding reference watermark information into a first image using the watermark embedding model to obtain a second image; performing a plurality of transformation operations on the second image to generate a third image; determining a difference between the watermark information extracted from the third image and the reference watermark information; training the watermark embedding model based on an objective function related to the differences. Watermark embedder.

11. a target image block determination module configured to determine at least one target image block in the image to be extracted based on a set of key points recognized in the image to be extracted; a watermark information extraction module configured to determine watermark information embedded in the image to be extracted based on each of the at least one target image block; the watermark information is determined utilizing a watermark extraction model; Training the watermark extraction model includes: embedding the reference watermark information into the first image to obtain a second image; performing a plurality of transformation operations on the second image to generate a third image; determining a difference between watermark information extracted from the third image using the watermark extraction model and the reference watermark information; training the watermark extraction model based on an objective function related to the differences. Watermark extractor.

12. 1. An electronic device comprising: at least one processing unit; at least one memory coupled to said at least one processing unit and configured to store instructions for execution by said at least one processing unit, said instructions, when executed by said at least one processing unit, causing execution of the watermark embedding method according to any one of claims 1 to 7 or the watermark extraction method according to any one of claims 8 to 10. Electronic devices.

13. A computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to realize the watermark embedding method according to any one of claims 1 to 7 or the watermark extraction method according to any one of claims 8 to 10. A computer-readable storage medium.

Images