Method and device for extracting watermark

Abstract

The invention relates to the field of computers, in particular to a method and device for extracting watermarks. A digital watermark is embedded in a three-dimensional model, the risk that the three-dimensional model is illegally used can be reduced, however, the three-dimensional model is likely to be converted into two-dimensional data, an attack simulated by a neural network in the training process is generally an attack aiming at the three-dimensional model, and when the three-dimensional model is converted into the two-dimensional data, the form of the watermark is difficult to predict; effective watermarks are difficult to extract from two-dimensional data based on the neural network trained by the attack network. In the application, after the electronic device obtains the target file, the corresponding decoder in the neural network is used to extract the watermark based on the type of the target file, for example, the three-dimensional model decoder is used to process the three-dimensional model file, and the two-dimensional data decoder is used to process the two-dimensional data file, so that damage to the watermark caused by an attack means can be eliminated in a targeted manner, and the watermarking efficiency is improved. And the success rate of extracting the watermark from the attacked three-dimensional model is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and device for extracting a watermark. Background Art

[0002] A 3D digital model, often referred to as a 3D model, is a digital representation of the form represented in drawings of engineering or product designs, created using 3D modeling software. Compared to 2D data (such as images), 3D models visualize and visualize abstract spatial information, providing a richer display space.

[0003] A digital watermark, also known as a watermark, is information that can be embedded in a digital signal, such as audio, images, video, or a data structure. One important application of digital watermarks is copyright protection. For example, embedding a digital watermark in a 3D model or 2D data can reduce the risk of unauthorized use of the data.

[0004] Once publicly available, 3D models may be exposed to various attacks, such as geometric transformations, affine transformations, retopology, and mesh simplification. One approach to extracting watermarks is through neural networks. During neural network training, an attack network is added to simulate various potential attacks on 3D models, hoping to extract a valid watermark from the attacked 3D model. However, 3D models can be converted into 2D data, and the attacks simulated by the attack network are typically targeted at 3D models. When 3D models are converted into 2D data, the watermark's morphology is difficult to predict. Therefore, neural networks trained using this attack network struggle to extract a valid watermark from 2D data. Summary of the Invention

[0005] Embodiments of the present application provide a method, apparatus, computer-readable storage medium, and computer program product for extracting watermarks, as well as a method, apparatus, computer-readable storage medium, and computer program product for training neural networks, which can improve the success rate of extracting watermarks from attacked three-dimensional models.

[0006] In a first aspect, an embodiment of the present application provides a method for extracting a watermark. The execution subject of this method can be an electronic device or a chip applied to an electronic device. Hereinafter, the description will be given taking the execution subject as an electronic device as an example. The method includes: obtaining a target file, where the target file is a target 3D model or target 2D data, the target 2D data is an image or video generated by rendering a base 3D model and a 2D texture, the target 3D model carries a target watermark, the target 2D data carries a target watermark, and the target watermark carried by the target 2D data is the watermark carried by the base 3D model and / or the 2D texture; determining the type of the target file, where the type includes a 3D model or 2D data; when the type is a 3D model, extracting the target watermark carried by the target file through a 3D model decoder, and the 3D model decoder is a sub-network of a neural network; or when the type is 2D data, extracting the target watermark carried by the target file through at least one 2D data decoder, and the at least one 2D data decoder is a sub-network of a neural network.

[0007] After obtaining the target file, first identify the type of the target file, and then use the corresponding decoder in the neural network to extract the watermark based on the type of the target file, which can extract the watermark from the attacked 3D model or its rendered 2D data specifically, identify the tampering caused by the attack means to the data, and improve the success rate of extracting the watermark from various types of attacked 3D models.

[0008] In an optional implementation manner of the first aspect, before extracting the target watermark carried by the target file through at least one 2D data decoder, the method further includes: determining at least one lighting model corresponding to the target file; determining at least one 2D data decoder according to the at least one lighting model, and the at least one 2D data decoder is used to extract the target watermark carried by the target file according to its corresponding lighting model.

[0009] The 2D data is, for example, the data obtained by rendering a base 3D model and a 2D texture. Different lighting models are used in this rendering process. Determining the lighting model used by the target file in the rendering process can specifically analyze the influence of different lighting models on the target watermark and improve the success rate and accuracy of extracting the target watermark from the target file.

[0010] In an optional implementation of the first aspect, determining at least one illumination model corresponding to the target file includes: when receiving illumination model indication information, determining at least one illumination model according to the illumination model indication information, where the illumination model indication information is used to indicate the illumination model used when rendering and generating the target two-dimensional data; or, when not receiving the illumination model indication information, and when receiving watermark extraction indication information, determining at least one illumination model as a plurality of preset illumination models, where the illumination model indication information is used to indicate the illumination model used when rendering and generating the target two-dimensional data, and the watermark extraction indication information is used to indicate extracting the target watermark carried by the target file.

[0011] When the user determines the illumination model corresponding to the target file, extracting the watermark based on the illumination model indicated by the user can improve the accuracy of watermark extraction. When the user is not sure about the illumination model corresponding to the target file, extracting the watermark based on a plurality of preset illumination models can improve the success rate of watermark extraction.

[0012] In an optional implementation of the first aspect, when at least one illumination model is a plurality of preset illumination models, at least one two-dimensional data decoder is a plurality of two-dimensional data decoders, and the plurality of two-dimensional data decoders correspond to the plurality of preset illumination models one by one. Processing the target file through at least one two-dimensional data decoder includes: respectively using the plurality of two-dimensional data decoders to process the target file to determine a plurality of intermediate watermarks; respectively comparing the plurality of intermediate watermarks with a reference watermark to determine a plurality of similarities corresponding to the plurality of intermediate watermarks, where the plurality of similarities are used to indicate the similarity degree between the plurality of intermediate watermarks and the reference watermark; when at least one of the plurality of similarities is greater than or equal to a similarity threshold, determining the target watermark from the intermediate watermarks corresponding to at least one of the similarities, where the target watermark is the intermediate watermark corresponding to the highest value among the at least one similarity.

[0013] Using the two-dimensional data decoders corresponding to a plurality of illumination models to extract the watermark can improve the success rate of watermark extraction when the user is not sure about the illumination model of the target file.

[0014] In an optional implementation of the first aspect, before obtaining the target file, the method includes: obtaining an original watermark and an original three-dimensional model; encoding the original watermark and the original three-dimensional model through a three-dimensional model encoder to generate three-dimensional training data, where the three-dimensional training data is a three-dimensional model carrying the original watermark, and the three-dimensional model encoder is a sub-network of a neural network; attacking the three-dimensional training data to generate an attacked three-dimensional training data; using the attacked three-dimensional training data to train a three-dimensional model decoder, where the three-dimensional model decoder is used to extract a first watermark from the attacked three-dimensional training data.

[0015] The above attacks on the three-dimensional training data can be implemented by a three-dimensional model attack library. The three-dimensional model attack library can simulate the possible attacks on the target file. By training the neural network with the participation of the three-dimensional model attack library, the robustness of the three-dimensional model decoder can be improved.

[0016] In an optional implementation manner of the first aspect, before obtaining the target file, the method includes: obtaining an original watermark, an original three-dimensional model, and an original two-dimensional map; encoding the original watermark and the original two-dimensional map through a two-dimensional data encoder to generate a two-dimensional training map, where the two-dimensional training map is a two-dimensional map carrying the original watermark, and the two-dimensional data encoder is a sub-network of the neural network; attacking the three-dimensional model composed of the original three-dimensional model and the two-dimensional training map to generate an attacked three-dimensional model; rendering the attacked three-dimensional model using at least one differentiable renderer to obtain at least one two-dimensional training data, where the at least one two-dimensional training data is a picture or a video; training at least one two-dimensional data decoder using the at least one two-dimensional training data, and the at least one two-dimensional data decoder is used to extract a second watermark from the at least one two-dimensional training data.

[0017] The above attacks on the three-dimensional model composed of the original three-dimensional model and the two-dimensional training map can be implemented by a model attack library. The model attack library can simulate the possible attacks on the target file. The model attack library can be implemented by a two-dimensional model attack library or a three-dimensional model attack library. The two-dimensional model attack library can attack the two-dimensional map in the three-dimensional model to change the color texture, etc. of the two-dimensional map. The three-dimensional model attack library can attack the original three-dimensional model. The above two attack methods can also be implemented by the same model attack library. By training the neural network with the participation of the two-dimensional model attack library, the robustness of at least one two-dimensional data decoder can be improved, and the watermark extraction ability of the decoder for two-dimensional data under different attack methods can be enhanced.

[0018] In an optional implementation manner of the first aspect, training at least one two-dimensional model decoder using at least one two-dimensional training data includes: when the training degree of the three-dimensional model decoder meets the requirements, training at least one two-dimensional data decoder using the at least one two-dimensional training data.

[0019] The three-dimensional model decoder is a relatively independent sub-network. First training the three-dimensional model decoder requires fewer neural network parameters to be adjusted, which can make the local part of the neural network converge as soon as possible. In this way, when training the neural network with three-dimensional training data and two-dimensional training data, the overall convergence speed of the neural network can be accelerated.

[0020] In an optional implementation manner of the first aspect, at least one two-dimensional data decoder includes multiple two-dimensional data decoders, and the differentiable renderer includes multiple lighting models. The multiple two-dimensional data decoders correspond to the multiple lighting models one by one.

[0021] Multiple two-dimensional data decoders correspond one-to-one with multiple lighting models in the differentiable renderer, and can extract watermarks using the multiple two-dimensional data decoders when the lighting model of the target file is uncertain for the user, improving the success rate of watermark extraction.

[0022] In a second aspect, an embodiment of the present application provides a device for extracting watermarks. The device may include an input module and a processing module, configured to execute: any of the methods in the first aspect and its optional embodiments described above. Among them, the input module may execute the acquisition step under the control of the processing module.

[0023] The input module is configured to: acquire a target file, where the target file is a target three-dimensional model or target two-dimensional data, the target two-dimensional data is an image or video rendered based on a base three-dimensional model and a two-dimensional texture map, the target three-dimensional model carries a target watermark, the target two-dimensional data carries a target watermark, and the target watermark carried by the target two-dimensional data is the watermark carried by the base three-dimensional model and / or the two-dimensional texture map; the processing module is configured to: determine the type of the target file, where the type includes a three-dimensional model or two-dimensional data; when the type is a three-dimensional model, the processing module is further configured to extract the target watermark carried by the target file through a three-dimensional model decoder, and the three-dimensional model decoder is a sub-network of a neural network; or, when the type is two-dimensional data, the processing module is further configured to extract the target watermark carried by the target file through at least one two-dimensional data decoder, and the at least one two-dimensional data decoder is a sub-network of a neural network.

[0024] In an optional embodiment of the second aspect, before extracting the target watermark carried by the target file through at least one two-dimensional data decoder, the processing module is further configured to: determine at least one lighting model corresponding to the target file; determine at least one two-dimensional data decoder according to the at least one lighting model, and the at least one two-dimensional data decoder is configured to extract the target watermark carried by the target two-dimensional data according to its corresponding lighting model.

[0025] In an optional embodiment of the second aspect, the processing module is specifically configured to: when receiving lighting model indication information, determine at least one lighting model according to the lighting model indication information, and the lighting model indication information is used to indicate the lighting model used when rendering the target two-dimensional data; or, when not receiving the lighting model indication information, and when receiving watermark extraction indication information, determine at least one lighting model as a preset multiple lighting models, the lighting model indication information is used to indicate the lighting model used when rendering the target two-dimensional data, and the watermark extraction indication information is used to indicate extracting the target watermark carried by the target file.

[0026] In an optional implementation of the second aspect, when at least one lighting model is one of a plurality of preset lighting models, at least one two-dimensional data decoder is a plurality of two-dimensional data decoders, and the plurality of two-dimensional data decoders correspond to the plurality of preset lighting models one by one. The processing module is specifically configured to: process the target file using the plurality of two-dimensional data decoders respectively to determine a plurality of intermediate watermarks; compare the plurality of intermediate watermarks with the reference watermark respectively to determine a plurality of similarities corresponding to the plurality of intermediate watermarks, and the plurality of similarities are used to indicate the degree of similarity between the plurality of intermediate watermarks and the reference watermark; when at least one of the plurality of similarities is greater than or equal to the similarity threshold, determine the target watermark from the intermediate watermarks corresponding to the at least one similarity, where the target watermark is the intermediate watermark corresponding to the highest similarity value among the at least one similarity.

[0027] In an optional implementation of the second aspect, before obtaining the target file, the input module is further configured to: obtain the original watermark and the original 3D model; the processing module is further configured to: encode the original watermark and the original 3D model through a 3D model encoder to generate 3D training data, where the 3D training data is a 3D model carrying the original watermark, and the 3D model encoder is a sub-network of a neural network; attack the 3D training data to generate an attacked 3D training data; use the attacked 3D training data to train a 3D model decoder, and the 3D model decoder is used to extract the first watermark from the attacked 3D training data.

[0028] In an optional implementation of the second aspect, before obtaining the target file, the input module is further configured to: obtain the original watermark, the original 3D model and the original 2D texture; the processing module is further configured to: encode the original watermark and the original 2D texture through a 2D data encoder to generate 2D training textures, where the 2D training textures are 2D textures carrying the original watermark, and the 2D data encoder is a sub-network of a neural network; attack the 3D model composed of the original 3D model and the 2D training textures to generate an attacked 3D model; use at least one differentiable renderer to render the attacked 3D model to obtain at least one 2D training data, and the at least one 2D training data is a picture or a video; use the at least one 2D training data to train at least one 2D data decoder, and the at least one 2D data decoder is used to extract the second watermark from the at least one 2D training data.

[0029] In an optional implementation of the second aspect, the processing module is specifically configured to: when the training degree of the 3D model decoder meets the requirements, train at least one 2D data decoder through at least one 2D training data.

[0030] In an alternative embodiment of the second aspect, at least one two-dimensional data decoder includes a plurality of two-dimensional data decoders, and the differentiable renderer includes a plurality of lighting models. The plurality of two-dimensional data decoders correspond to the plurality of lighting models one by one.

[0031] In a third aspect, an embodiment of the present application provides a device for extracting a watermark. The device may be an electronic device or a chip applied to an electronic device. The device may include a processor configured to execute any of the methods in the first aspect and its alternative embodiments described above.

[0032] Optionally, when the device is an electronic device, the processor is, for example, a system on chip (SoC) or a central processing unit (CPU); when the device is a chip, the processor is, for example, a core, and the core may include at least one execution unit, and the execution unit is, for example, an arithmetic logic unit (ALU).

[0033] Optionally, the device may further include a transceiver. When the device is an electronic device, the transceiver may be a transceiver circuit, an antenna, etc.; when the device is a chip, the transceiver may be an input / output interface, a pin, a circuit, etc.

[0034] Optionally, the device may further include a memory for storing a computer program or instructions. The processor executes the computer program or instructions stored in the memory to enable the device to execute any of the methods in the first aspect and its alternative embodiments described above. When the device is an electronic device, the memory may be a read-only memory, a random access memory, etc.; when the device is a chip, the memory may be a register, a cache, etc.

[0035] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, which, when executed on a device for extracting a watermark, causes the device to execute any of the methods in the first aspect and its alternative embodiments.

[0036] In a fifth aspect, an embodiment of the present application provides a computer program product containing instructions, the computer program product including computer program code or computer program instructions, which, when run on a device for extracting a watermark, cause the device to execute any of the methods in the first aspect and its alternative embodiments. Description of the Drawings

[0037] Figure 1It is a schematic diagram of an architecture of a neural network provided by an embodiment of the present application;

[0038] Figure 2 It is a schematic diagram of a method for training a neural network provided by an embodiment of the present application;

[0039] Figure 3 It is a schematic diagram of a method for extracting a watermark based on a neural network provided by an embodiment of the present application;

[0040] Figure 4 It is a schematic diagram of a scenario where a 3D model is attacked provided by an embodiment of the present application;

[0041] Figure 5 It is a schematic diagram of a method for extracting a watermark provided by an embodiment of the present application;

[0042] Figure 6 It is a schematic diagram of a method for obtaining a target file provided by an embodiment of the present application;

[0043] Figure 7 It is a schematic diagram of a method for determining the type of a target file provided by an embodiment of the present application;

[0044] Figure 8 It is a schematic diagram of a method for determining the illumination model of a target file provided by an embodiment of the present application;

[0045] Figure 9 It is a schematic diagram of another method for determining the illumination model of a target file provided by an embodiment of the present application;

[0046] Figure 10 It is a schematic diagram of a method for training a neural network provided by an embodiment of the present application;

[0047] Figure 11 It is a schematic diagram of an inference process of a neural network provided by an embodiment of the present application;

[0048] Figure 12 It is a schematic diagram of a structure of a device for extracting a watermark provided by an embodiment of the present application;

[0049] Figure 13 It is a schematic diagram of a structure of a computer device for extracting a watermark provided by an embodiment of the present application;

[0050] Figure 14 It is a schematic diagram of a structure of a computer device cluster for extracting a watermark provided by an embodiment of the present application;

[0051] Figure 15 It is a schematic diagram of another structure of a computer device cluster for extracting a watermark provided by an embodiment of the present application. Detailed implementation manners

[0052] To facilitate understanding of the technical solution of this application, some terms and concepts related to this application are briefly introduced first. It should be noted that the descriptions of these terms and concepts are examples rather than limitations.

[0053] 1. Neural network (NN).

[0054] A neural network is also known as an artificial neural network (ANN) or a neural network-like. In the fields of machine learning and cognitive science, it is a mathematical model or computational model that mimics the structure and function of a biological neural network and is used to estimate and approximate functions. Neural networks can include multilayer perceptrons (MLPs), convolutional neural networks (CNNs), and deep neural networks (DNNs), etc.

[0055] 1.1 Architecture of the neural network.

[0056] A neural network can include multiple neural network layers. As Figure 1 shown, the neural network 100 can include n neural network layers, where n is a positive integer. Among them, the first layer is the input layer, the nth layer is the output layer, and the remaining layers are hidden layers. The arrows between the layers represent the direction of data transfer. In the embodiments of this application, a neural network layer is a logical probability. A neural network layer can refer to a set of computing nodes corresponding to one execution of neural network calculations, and the computing nodes can be called neurons. In practical applications, neural network layers can include convolutional layers, pooling layers, and fully connected layers, etc.

[0057] The working principle of each neural network layer can be described by the mathematical expression where, represents a sample (i.e., input data). A sample generally has multiple attributes, so a sample is usually a vector; w represents the weight value of each neuron in the neural network; b is the bias; a is the activation function; is the output vector. From a physical perspective, the work of each layer in the neural network can be understood as completing the transformation from the input space (i.e., the set of input vectors) to the output space (i.e., the set of output vectors) through five operations on the input space. Among them, determines the transformation from the input space to the output space. That is, the w of each layer controls how to transform the space. The above five operations include: 1. Dimensionality increase / dimensionality reduction; 2. Magnification / shrinkage; 3. Rotation; 4. Translation; 5. "Bending". Among them, operations 1, 2, and 3 are performed by Implementation: Operation 4 is implemented by +b, and operation 5 is implemented by a().

[0058] 1.2 Training of the neural network.

[0059] Since it is desired that the output of the neural network be as close as possible to the real situation, the difference between the predicted value and the target value of the neural network can be determined by comparison, and then the parameters of the neural network (such as weight values) can be adjusted according to the difference. There is usually an initialization process before the first parameter update, that is, parameters are preconfigured for each layer of the neural network. During the training process, if the predicted value is high, the parameters are adjusted to lower the predicted value of the neural network; if the predicted value is low, the parameters are adjusted to increase the predicted value of the neural network; until the output of the neural network meets the requirements. Therefore, it is necessary to predefine how to compare the difference between the predicted value and the target value. The loss function and the objective function are commonly used means to measure the difference between the predicted value and the target value. Taking the loss function as an example, the larger the output value (loss) of the loss function, the greater the difference; the smaller the output value of the loss function, the smaller the difference. Then, the training of the neural network becomes a process of minimizing this output value as much as possible.

[0060] As Figure 2 shown, the starting point corresponds to the output value of the loss function when the parameters of the neural network have not been updated. As the parameters of the neural network are updated, the output value of the loss function gradually decreases. The optimal point corresponds to the minimum value of the output value of the loss function. When the output value of the loss function reaches the minimum value, the training of the neural network is completed.

[0061] The loss function is usually a multi-variable function. The gradient can reflect the rate of change of the output value of the loss function when the variable changes. The larger the absolute value of the gradient, the greater the rate of change of the output value of the loss function. The gradients of the loss function when updating different parameters can be calculated, and the parameters can be updated along the direction where the gradient descends fastest to quickly reduce the output value of the loss function.

[0062] The output value of the loss function of the neural network can be obtained through forward propagation (FP) calculation. For example, the output result of the previous layer of the neural network can be input into the next layer until the output result of the output layer is obtained, and the output value of the loss function can be obtained based on this output result and the target value. Subsequently, based on the output value of the loss function, back propagation (BP) calculation is performed to obtain the gradients of each layer, and the parameters of the neural network are adjusted along the direction where the gradient descends fastest until the output value of the loss function reaches the minimum value.

[0063] 2. Three-dimensional model.

[0064] A 3D model is a polygonal representation of an object, usually displayed using a computer or other video device. The shape of a 3D model can be diverse. For example, it can be a sphere, a cone, a curved surface object, a planar object, or an object with an irregular surface, etc. The object represented by a 3D model can be a real-world entity or a fictional object. 3D models can be generated using specialized software such as 3D modeling tools, but can also be generated by other methods.

[0065] The basic structure of a 3D model is a mesh. The mesh can be composed of triangles, quadrilaterals, or other polygons, which can be called patches, and the intersection points of the edges of each patch are the vertices of each patch. Patches can be divided according to information such as the material of the 3D model. Material is a collection of attributes such as the appearance and optical properties of an object, which includes the color of the object, reflection attributes (such as diffuse reflection, specular reflection), transparency, refractive index, etc.

[0066] Material defines how an object interacts with light and determines the appearance effect of the object during rendering. For example, according to the different materials, the interaction between the patch and light can be divided into three cases: refraction, specular reflection, and diffuse reflection. Among them, when light contacts a transparent material, refraction occurs; when light contacts an opaque material with a smooth surface, reflection occurs; when light contacts an opaque material with a rough surface, diffuse reflection occurs.

[0067] Exemplarily, material can be expressed through texture, which is used to simulate the properties of the object surface and can contain color information, geometric information, process information, etc.

[0068] The texture containing color information is called color texture, which is used to represent patterns, pictures, and words presented on the object surface, such as marble walls, paintings and calligraphy on the wall, patterns on utensils, etc.

[0069] The texture containing geometric information is called geometric texture, which is used to represent the microscopic geometric shape of the object surface, such as an uneven surface.

[0070] The texture containing process information is called process texture, which is used to represent various regular or irregular dynamically changing scenes, such as water waves, clouds, fire, smoke, etc.

[0071] The process of applying texture to the model surface is texture mapping. The texture mapped to the 3D model surface can also be called a texture map. Texture mapping matches the texture with the vertices or pixels of the 3D model, so that the texture covers the model surface. During the rendering process, the color, geometry, and process of the model surface are determined according to the coordinate information of the texture. Mapping the texture to the model surface can endow the 3D model with a more realistic and detailed appearance.

[0072] 3. Two-dimensional data.

[0073] Two-dimensional data includes images or videos, which can be obtained after rendering a three-dimensional model.

[0074] In rendering, different material parameters can be specified for different objects. If the same effect needs to be shown at different positions of an object, the values of all material parameters can be constants. If different effects need to be shown at different positions of an object, then textures can be used to specify the material parameters for the object. There is a mapping relationship (i.e., UV unwrapping) from the surface of the three-dimensional geometric body to the two-dimensional plane between the object and the texture. U and V refer to the horizontal and vertical axes of the two-dimensional plane. Specifying different values for different positions in the unwrapped two-dimensional plane is equivalent to covering different textures on the surface of the object. Optionally, the texture can include color, surface bumpiness, metalness, roughness, opacity, etc.

[0075] Color can be expressed through a color map. The color map can be the pixel feature components on a two-dimensional image, such as red green blue (RGB), and is used to represent the color or refractive absorption coefficient of the surface of the three-dimensional model.

[0076] Surface bumpiness can be expressed through a normal map and a bump map. The normal map affects the normal orientation of the surface of the three-dimensional model and contains the angle information of the surface. The bump map is mainly used to express the height, pits, etc. of the surface and create an uneven surface effect.

[0077] Metalness can be expressed through a metalness map. The metalness map represents the specular reflection property of the surface of the object. The metalness map can define whether the three-dimensional model is metallic or non-metallic.

[0078] Roughness can be expressed through a roughness map. The roughness map represents the diffuse reflection property of the surface of the object.

[0079] Opacity can be expressed through an opacity map. The opacity map represents the transparency degree of the surface of the object.

[0080] In addition to rendering, the three-dimensional model can also be processed by shading, ray tracing, path tracing, etc., to make the finally generated two-dimensional data more realistic.

[0081] Shading is the process of calculating the pixel value of each pixel in the image based on the direction of the object relative to the light and its distance from the light source. The pixel value determines which graphics and light and dark effects are displayed in the final image.

[0082] The main idea of ray tracing is to emit a primary ray from the viewpoint to each pixel on the imaging plane, find the intersection point with the nearest object that intersects the primary ray, and calculate the color of this intersection point based on the properties of the object, the properties of the light source, and the lighting model. At the intersection point, the primary ray is reflected and / or refracted to form secondary rays. From this intersection point, the secondary rays are traced along all reflection and / or refraction directions to determine the intersection point of the secondary ray with the next surface, and the color of this intersection point is calculated. This recursive process continues until the ray escapes the scene or reaches a light source.

[0083] The idea of path tracing is similar to that of ray tracing. A ray of light is emitted from the viewpoint. When the ray encounters a surface point and needs to be reflected, a hemisphere is made with the surface point as the center. Several beams of light are drawn in several directions on the hemisphere, and then the Monte Carlo algorithm is used to calculate the illumination contribution of each of these beams of light to the surface point.

[0084] 4. Lighting model.

[0085] The lighting model is a mathematical model in computer graphics that is used to simulate and calculate the color and brightness of an object's surface under lighting.

[0086] When light strikes a surface, it may be absorbed, reflected, or transmitted. Some of the incident light energy is absorbed and converted into heat, while the remainder is reflected or transmitted. It is this reflected or transmitted portion of light that makes the object visible. The amount of light energy absorbed, reflected, or transmitted depends on the wavelength of the light. If approximately equal amounts of all wavelengths of incident light are absorbed, the object appears gray under white light. If only a small amount is absorbed, the object appears white. If certain wavelengths of light are selectively absorbed, the reflected and projected light leaving the surface will have different energy distributions, giving the object its appearance of color. The object's color depends on the wavelengths of light it selectively absorbs. The amount of light reflected or transmitted from an object's surface depends on the composition of the light source, its direction, the geometry of the light source, and the orientation and properties of the surface.

[0087] Lighting models are based on the laws of optics and describe the reflection, refraction, and scattering of light on surfaces. They can be categorized into two types: those based on physical theory and those based on experience. Physically based models use physical measurements and statistical methods to simulate real-world lighting effects, but they are computationally complex and difficult to implement. Empirical models, on the other hand, use specific probabilistic formulas tailored to a set of surface types. They are generally simpler and tend to produce idealized results.

[0088] Common experience-based lighting models include the Lambert model, the Phong model, and the Blinn-Phong model.

[0089] 4.1. The Lambert model.

[0090] The Lambert model simulates the intensity of diffuse light based on the angle between the incident light and the normal of the object surface. According to the Lambert cosine law, when light irradiates on the object surface, the intensity of its diffuse light is proportional to the cosine value of the above angle. The larger the angle, the smaller the intensity of the diffuse light.

[0091] 4.2. The Phong model.

[0092] In the real world, the light reaching the observer includes not only diffuse light but also ambient light and specular light.

[0093] The Phong model simulates the intensity of specular light based on the Fresnel equation, the angle of the incident light, the wavelength of the incident light, and the material properties of the reflecting surface, etc. The intensity of the specular light is determined by the angle of the incident light. For an ideal mirror, the angle of reflection is equal to the angle of incidence, so only an observer strictly in the direction of the reflected light can see the reflected light. For a non-ideal mirror, the light reaching the observer depends on the spatial distribution of the specular light. The spatial distribution of the reflected light on a smooth surface is relatively narrow or has better convergence; while for a rough surface, the scattering range of the reflected light is larger.

[0094] 4.3. The Blinn-Phong model.

[0095] The Blinn-Phong model is a modified model of the Phong model. The Blinn-Phong model introduces the concept of a half-angle vector, denoted by H. The half-angle vector H can be obtained by adding the light source direction L and the line-of-sight direction V and then normalizing. The Phong model uses the angle between the reflection direction R and the line-of-sight direction V to calculate the intensity of the specular light, while the Blinn-Phong model uses the angle between the half-angle vector H and the line-of-sight direction V to calculate the intensity of the specular light. Compared with the Phong model, the Blinn-Phong model simplifies the calculation process under the premise of similar effects.

[0096] 5. Digital watermarking.

[0097] With the popularization of smart terminals and the Internet, a large number of users have started to use terminals to create digital works. Digital works (such as the above-mentioned 3D models and 2D data) are often used for commercial purposes, such as advertising and selling, after being spread through the Internet. If the ownership of digital works cannot be proven, the interests of the rights holders will be damaged.

[0098] Digital watermarking is a technology that embeds specific digital signals into digital works to protect the copyright or integrity of digital works. It takes advantage of the limitations of the human senses to closely combine and hide digital signals, such as images, texts, symbols, numbers, and all other information that can be used as marks and identifiers, with the original data (such as 3D models and 2D data), and can survive some attacks.

[0099] There are many application scenarios for digital watermarking. According to the different characteristics of digital watermarks, watermarking algorithms suitable for different scenarios can be designed. For example, robust invisible watermarks can be used for copyright protection, copy control, fingerprint recognition, etc.; fragile invisible watermarks can be used for content authentication, file modification detection, etc. Generally, for copyright protection, digital watermarks should have the following basic characteristics:

[0100] 1) Provability: The watermark should be able to provide complete and reliable evidence for the ownership of the copyrighted digital work.

[0101] 2) Invisibility: Invisibility has two aspects. One refers to visual invisibility (the same requirement applies to hearing), that is, the change in the image caused by embedding the watermark should be imperceptible to the observer's visual system. The most ideal situation is that the watermarked image is exactly the same as the original image visually, which is the requirement that most watermarking algorithms should meet; on the other hand, the watermark cannot be recovered by statistical methods either. For example, even for a large number of information products processed with the same method and watermark, the watermark cannot be extracted or the existence of the watermark cannot be determined using statistical methods.

[0102] 3) Robustness: Robustness generally refers to the ability of the watermark to resist malicious erasure of the watermark. A watermark with strong robustness should be able to withstand a large number of different attacks, such as vertex rearrangement, patch rearrangement, rotation, scaling, translation, adding noise, smoothing, quantization, mesh simplification, retopology, local cropping, local deformation, affine transformation, etc.

[0103] Traditional 3D model watermarking algorithms can be divided into spatial domain watermarking algorithms and transform domain watermarking algorithms according to the way of watermark embedding. Spatial domain watermarking algorithms achieve the purpose of embedding watermarks by modifying the vertex coordinates, edge lengths, triangle areas, face vertex orders, etc. of the model. Transform domain watermarking algorithms add watermarks to transform domain coefficients, such as wavelet coefficients or spherical coordinate coefficients. According to whether the original 3D model is required as input information when extracting watermarks, 3D model digital watermarking algorithms are further divided into blind watermarking algorithms and non-blind watermarking algorithms.

[0104] Traditional 3D model watermarking algorithms have great advantages in watermark capacity. However, due to the influence of the limitations of manual design, an algorithm can often only resist a small number of attacks. In recent years, with the development of deep learning, more and more technologies for embedding and extracting watermarks in 3D models using deep learning have emerged. Digital watermarking technology based on deep learning uses deep neural networks to embed digital watermarks and has strong robustness. However, most of the work follows the paradigm that first embeds the watermark into the 3D model, and when the right of ownership needs to be confirmed, the watermark is extracted from the 3D model, as Figure 3 shown.

[0105] The original 3D model is a 3D model without watermarks. The original watermark is, for example, the author's name, company logo, etc. The deep neural network obtains the original 3D model and the original watermark, embeds the original watermark into the original 3D model, and generates a 3D model containing the watermark. After the 3D model containing the watermark is attacked, an attacked 3D model is generated, and the attacked 3D model may still contain the watermark. Subsequently, the attacked 3D model is processed by the deep neural network to extract the attacked watermark. After the 3D model is attacked, the watermark in it may have information loss. An attack network is added during the training process of the deep neural network to simulate various attacks that the 3D model may be subjected to, expecting to recover the missing information and extract the watermark that can prove the ownership of the 3D model.

[0106] The above method is effective in most scenarios. However, in some scenarios, the 3D model may be transformed into 2D data, and the attacks simulated by the attack network are usually attacks on 3D models and are difficult to handle this scenario.

[0107] Figure 4 is a schematic diagram of the process of transforming a 3D model into 2D data.

[0108] As Figure 4As shown, after texture mapping the three-dimensional model and then embedding the watermark, a three-dimensional model containing the watermark is generated. The three-dimensional model containing the watermark is attacked to obtain an attacked three-dimensional model, and the attacked three-dimensional model is transformed into two-dimensional data after being processed such as rendering, coloring, and ray tracing. After the three-dimensional model is transformed into two-dimensional data, the form of the watermark is difficult to predict, and it is difficult for the deep neural network trained based on the above attack network to extract an effective watermark from the two-dimensional data.

[0109] The method for extracting the watermark provided by the embodiment of the present application is introduced below.

[0110] Figure 5 It is a schematic diagram of the method 500 for expanding the kernel provided by the embodiment of the present application. The method 500 can be executed by an electronic device or a chip applied to an electronic device.

[0111] The electronic device can be a terminal, a server, or a chip applied to a terminal or a server.

[0112] As an optional example, the terminal can be a mobile phone, a tablet computer (Pad), a computer with wireless transceiver function, a virtual reality (VR) terminal, an augmented reality (AR) terminal, a wearable device, a wireless terminal in industrial control, a whole vehicle, a wireless communication module in the whole vehicle, a telematics box (T-box), a road site unit (RSU), a wireless terminal in unmanned driving, a smart speaker in the internet of things (IoT), a wireless terminal device in remote medical, a wireless terminal device in smart grid, a wireless terminal device in transportation safety, a wireless terminal device in smart city, or a wireless terminal device in smart home, etc. The embodiments of the present application do not limit this.

[0113] As an optional example, the server can be a tower server, a blade server, or a rack server.

[0114] The embodiments of the present application do not limit the specific form of the electronic device and the specific technology adopted.

[0115] As Figure 5 shown, the method 500 includes:

[0116] S510, Obtain a target file, where the target file is a target 3D model or target 2D data. The target 2D data is an image or video rendered based on a base 3D model and a 2D texture. The target 3D model carries a target watermark, and the target 2D data carries a target watermark. The target watermark carried by the target 2D data is the watermark carried by the base 3D model and / or the 2D texture.

[0117] The target file may be a target 3D model, which can be a 3D model without data such as textures (i.e., a bare model), or a 3D model containing data such as textures. For example, the target file can be an OSGB file, a PLY file, an OBJ file, or an S3MB file.

[0118] The target file may also be target 2D data, such as various image files or video files. Among them, the target 2D data is an image or video rendered based on a base 3D model and a 2D texture, and the base 3D model can be a bare model. Optionally, the base 3D model can also be processed such as coloring, ray tracing, and path tracing to generate the target 2D data. The embodiments of the present application do not limit the specific manner of generating the target 2D data based on the base 3D model and the 2D texture. Either the base 3D model and the 2D texture in the target 2D data can carry the target watermark, or only one of them can carry the target watermark.

[0119] The form of the target watermark can be one or more of audio, picture, video, and data structure. The target watermark can be embedded in the target 3D model or the target 2D data in the form of binary message bits. Among them, the data structure is, for example, the type of a single patch or the arrangement manner between multiple patches, etc. The form of the target watermark can also be other data forms that can be embedded in the 3D model. The content of the target watermark can be a company logo or a personal name, or other information that can prove the ownership of the target file. The embodiments of the present application do not limit the form and content of the target watermark.

[0120] The electronic device can obtain the target file locally. As Figure 6 shown, the user can drag the target file from a local folder to the watermark detection area. In this way, the electronic device can determine the location of the target file and read the target file from this location.

[0121] The electronic device can also receive the target file from the cloud through the network. The embodiments of the present application do not limit the specific manner for the electronic device to obtain the target file.

[0122] After obtaining the target file, the electronic device can perform the following steps.

[0123] S520, Determine the type of the target file, where the type includes 3D model or 2D data.

[0124] An electronic device can determine the type of a target file according to the format of the target file. For example, if the format of the target file is an OBJ file, the electronic device can determine that the type of the target file is a 3D model; if the format of the target file is a JPG file, the electronic device can determine that the type of the target file is 2D data. The electronic device can also open the target file and determine the type of the target file according to the content in the target file. For example, if after the electronic device opens the target file and determines that the target file contains 3D model content (such as a mesh file), the electronic device can determine that the type of the target file is a 3D model; if after the electronic device opens the target file and determines that the target file does not contain 3D model content (such as a mesh file), the electronic device can determine that the type of the target file is 2D data.

[0125] The electronic device can also determine the type of the target file according to the user's instruction. As Figure 7 shown, after the electronic device obtains the target file, it can pop up a dialog box of "Please select the file type", and the user selects the file type of the target file. After the user clicks the "3D model" option or the "2D data" option, the user can click the "Next" option to instruct the electronic device to perform subsequent processing. Since there are various types of 3D models and 2D data, the electronic device may not be able to recognize all types, or the types recognized by the electronic device are incorrect. Determining the type of the target file with the help of the user's experience can improve the accuracy of type judgment.

[0126] If the user is not sure about the type of the target file, the user can directly click the "Next" option, and the electronic device determines the type of the target file according to the format or content of the 3D model. The embodiments of the present application do not limit the specific manner of determining the type of the target file.

[0127] After determining the type of the target file, the electronic device can perform the following steps.

[0128] Optionally, method 500 further includes:

[0129] Determining at least one lighting model corresponding to the target file;

[0130] Determining at least one 2D data decoder according to the at least one lighting model, and the at least one 2D data decoder is used to extract the target watermark carried by the target file according to the corresponding lighting model.

[0131] The 2D data is the data obtained after rendering data such as a base 3D model and 2D textures. A lighting model may be used in the rendering process. Determining the lighting model used by the target file in the rendering process can use the 2D data decoder corresponding to the lighting model to specifically eliminate the influence of the lighting model on the target watermark and improve the success rate of extracting the target watermark from the target file.

[0132] At least one lighting model corresponding to the target file can be a Lambert model, a Phong model, or a Blinn-Phong model, or it can be other lighting models. Embodiments are not limited to the type of at least one lighting model in this application.

[0133] Optionally, determining at least one lighting model corresponding to the target file includes:

[0134] When receiving lighting model indication information, determining at least one lighting model according to the lighting model indication information, where the lighting model indication information is used to indicate the lighting model used when rendering and generating the target two-dimensional data; or, when not receiving the lighting model indication information, and when receiving watermark extraction indication information, determining at least one lighting model as a preset plurality of lighting models, where the lighting model indication information is used to indicate the lighting model used when rendering and generating the target two-dimensional data.

[0135] The lighting model indication information and the watermark extraction indication information can be indication information generated by a user clicking an icon on the interface.

[0136] As Figure 8 shown, after the electronic device determines the type of the target file, a dialog box of "Please select a lighting model" can be popped up, and the dialog box contains one or more lighting models used by the neural network during training. If the user determines the lighting model of the target file, one of the models can be selected, such as clicking the "Blinn-Phong model" option, and this click action triggers the electronic device to generate lighting model indication information, and the electronic device determines at least one lighting model as the Blinn-Phong model according to the lighting model indication information. Subsequently, the user can click the "Extract watermark" option to trigger the electronic device to generate watermark extraction indication information, instructing the electronic device to extract the watermark in the target model.

[0137] If the user is not sure about the lighting model of the target file, multiple lighting models that the target file may use can be selected, or the "Extract watermark" option can be directly clicked. As Figure 9 shown, at this time, when the electronic device receives the watermark extraction indication information without receiving the lighting model indication information, the electronic device determines at least one lighting model as a preset plurality of lighting models, and extracts the watermark in the target model based on the preset plurality of lighting models.

[0138] After determining to extract the watermark in the target model, the electronic device can execute S530 or S540.

[0139] S530, when the type is a three-dimensional model, extracting the target watermark carried by the target file through a three-dimensional model decoder, where the three-dimensional model decoder is a sub-network of the neural network.

[0140] S540, when the type is two-dimensional data, at least one two-dimensional data decoder is used to extract the target watermark carried in the target file, and at least one two-dimensional data decoder is a sub-network of a neural network.

[0141] Optionally, when at least one lighting model is a plurality of preset lighting models, at least one two-dimensional data decoder is a plurality of two-dimensional data decoders, and the plurality of two-dimensional data decoders correspond to the plurality of preset lighting models one by one. During the process of the electronic device extracting the target watermark, the following operations can be performed:

[0142] Process the target file using the plurality of two-dimensional data decoders respectively to determine a plurality of intermediate watermarks;

[0143] Compare the plurality of intermediate watermarks with the reference watermark respectively to determine a plurality of similarity degrees, and the plurality of similarity degrees are used to indicate the similarity between the plurality of intermediate watermarks and the reference watermark;

[0144] When at least one of the plurality of similarity degrees is greater than or equal to the similarity threshold, determine the target watermark from the intermediate watermarks corresponding to at least one of the similarity degrees, where the target watermark is the intermediate watermark corresponding to the highest value among at least one of the similarity degrees.

[0145] For example, the plurality of preset lighting models are Lambert model, Phong model, and Blinn-Phong model. The two-dimensional data decoders corresponding to these three lighting models are decoder A (corresponding to Lambert model), decoder B (corresponding to Phong model), and decoder C (corresponding to Blinn-Phong model) respectively. The neural network can use these three decoders to process the target file respectively to extract three intermediate watermarks; assume that these three intermediate watermarks are watermark A, watermark B, and watermark C respectively. The neural network can compare these three intermediate watermarks with the watermark registered in the database (i.e., the reference watermark) respectively to obtain three similarity degrees, namely similarity degree A, similarity degree B, and similarity degree C; if similarity degree B and similarity degree C among the three similarity degrees are greater than or equal to the similarity threshold, then the highest value among similarity degree B and similarity degree C can be determined, for example, similarity degree C, and the intermediate watermark (watermark C) corresponding to similarity degree C can be used as the target watermark and presented to the user.

[0146] The above reference watermark may be a watermark indicated by the user. The user can know the watermark embedded in the 3D model he owns. The target watermark determined based on the watermark indicated by the user has a high accuracy. The above reference watermark may also be a watermark determined by the electronic device itself. In some cases, the user is not sure which watermark is embedded in the 3D model he owns. Then the electronic device can determine the reference watermark according to the type of the intermediate watermark. For example, if watermark A, watermark B, and watermark C are all text, the electronic device can determine a text watermark as the reference watermark; if watermark A, watermark B, and watermark C are all pictures, the electronic device can determine a picture watermark as the reference watermark.

[0147] Using the two-dimensional data decoders corresponding to multiple lighting models to extract the watermark can improve the success rate of watermark extraction when the user is not sure about the lighting model of the target file.

[0148] The above text has introduced in detail the method for extracting the watermark provided by the embodiments of the present application. The forms of the watermarks in the 3D model and the two-dimensional data are different. For example, the form of the watermark in the 3D model may be the arrangement order of the patches, and the form of the watermark in the two-dimensional data may be pixels. Using the corresponding decoder to extract the watermark based on the type of the target file can specifically eliminate the damage caused by the attack means to the watermark and improve the success rate of extracting the watermark from the attacked 3D model.

[0149] The neural networks in S530 and S540 may be deep neural networks. Before executing S510, the deep neural network can be pre-trained using 3D models, two-dimensional data, and different attack means to enable it to have the ability to extract the target watermark from 3D models and two-dimensional data. The neural networks in S530 and S540 may also be other neural networks that can extract the target watermark from 3D models and two-dimensional data. The embodiments of the present application do not limit the specific type of the neural network in method 500.

[0150] In addition, the neural networks in S530 and S540 may be the same neural network or different neural networks. When the neural networks in S530 and S540 are the same neural network, the neural network has the ability to extract the watermark from 3D models and two-dimensional data; when the neural networks in S530 and S540 are different neural networks, different neural networks may have different abilities. For example, one neural network has the ability to extract the watermark from 3D models, and the other neural network has the ability to extract the watermark from two-dimensional data. Of course, both of these two neural networks may also have the ability to extract the watermark from 3D models and two-dimensional data.

[0151] The following introduces the method for training the neural network provided by the embodiments of the present application.

[0152] AsFigure 10 As shown, the neural network provided by the embodiments of the present application may include a 3D model encoder, an attack library, a 3D model decoder, a 2D data encoder, a differentiable renderer, and at least one 2D data decoder, where:

[0153] The 3D model encoder is used to encode the original watermark and the original 3D model (such as a nude model) to generate 3D training data, and the 3D training data is a 3D model carrying the original watermark. The 3D model encoder is a sub-network of the neural network;

[0154] The attack library 1 is used to simulate the attack means that the target file may be subjected to, and attack the 3D training data based on this attack means to generate the attacked 3D training data. Among them, the attack means simulated by the attack library 1 may be an attack on the 3D model;

[0155] The 3D model decoder is used to extract the first watermark from the attacked 3D training data;

[0156] The 2D data encoder is used to encode the original watermark and the original 2D texture map (such as an image) to generate 2D training texture maps, and the 2D training texture maps are 2D texture maps carrying the original watermark. The 2D data encoder is a sub-network of the neural network;

[0157] The attack library 2 is used to simulate the attack means that the target file may be subjected to, and attack the 3D model composed of the original 3D model and the 2D training texture maps based on this attack means to generate the attacked 3D model. Among them, the attack means simulated by the attack library 2 may be an attack on the 3D model, or an attack on the 2D data, or an attack that does not distinguish between the 3D model and the 2D data;

[0158] The differentiable renderer is used to render the attacked 3D model into at least one 2D training data, and the at least one 2D training data is a picture or a video;

[0159] At least one 2D data decoder is used to extract the second watermark from at least one 2D training data.

[0160] The original watermark may be one or more of text, pictures, audio, video, and data structures. During the training process, the 3D model encoder and the 3D model decoder can be trained first using the 3D training data, or, optionally, the 3D model decoder can be trained alone to facilitate the neural network to converge as soon as possible. Among them, the attack library 1 and the attack library 2 can simulate different attacks, such as vertex rearrangement, patch rearrangement, rotation, scaling, translation, adding noise, smoothing, quantization, mesh simplification, retopology, local cropping, local deformation, affine transformation, etc. Training with the participation of the attack library can improve the robustness of the 3D model decoder.

[0161] After the training of the 3D model encoder and the 3D model decoder meets the requirements, use the 2D training data to train the 2D data encoder and at least one 2D data decoder, or, optionally, train the 2D data decoder alone. During the training of the 2D data encoder and at least one 2D data decoder, the 3D model encoder and the 3D model decoder can also be fine-tuned. Among them, at least one 2D data decoder corresponds one-to-one with the lighting models in the differentiable renderer. For example, if the lighting models in the differentiable renderer include the Lambert model, the Phong model, and the Blinn-Phong model, then at least one 2D data decoder includes the 2D data decoder corresponding to the Lambert model, the 2D data decoder corresponding to the Phong model, and the 2D data decoder corresponding to the Blinn-Phong model, and these three 2D data decoders are trained separately during the training process.

[0162] In other embodiments provided by this application, the 3D model codec and the 2D model codec can also be trained separately, that is, directly use the 2D training data to train the 2D data encoder and at least one 2D data decoder, or, optionally, train the 2D data decoder alone. This application does not limit the training order of the encoders and decoders involved in the embodiments.

[0163] The first watermark and the second watermark are the predicted values of the neural network, and the original watermark is the true value. After inputting the first watermark, the second watermark, and the original watermark into the first loss function, determine the output value of the first loss function. Based on the output value of the first loss function, perform backpropagation calculation to obtain the gradients of each neural network layer in the 3D model encoder and the 3D model decoder, and adjust the neural network parameters along the direction of the steepest descent until the output value of the first loss function reaches the minimum. Similarly, after inputting the second watermark and the original watermark into the second loss function, determine the output value of the second loss function. Based on the output value of the second loss function, perform backpropagation calculation to obtain the gradients of each neural network layer in the 2D data encoder and at least one 2D data decoder, and adjust the neural network parameters along the direction of the steepest descent until the output value of the second loss function reaches the minimum.

[0164] The neural network can also be trained based on other methods, and the embodiments of this application do not limit the training method of the neural network.

[0165] In the above training method, both the 3D model encoder and the 2D data encoder are sub-networks to be trained. Optionally, when both the 3D model encoder and the 2D data encoder are sub-networks that have been trained, the parameters of the 3D model encoder and the 2D data encoder can be fixed during the training process to improve the training efficiency of the neural network.

[0166] After the neural network training is completed, inference operations can be performed.

[0167] As Figure 11 shown, in the inference stage, a 3D model containing a watermark or 2D data containing a watermark can be input. Optionally, the 3D model decoder and the 2D data decoder can share an input layer and an output layer. More neural network layers can also be included between the input layer and the 3D model decoder, between the output layer and the 3D model decoder, between the input layer and the 2D data decoder, and between the output layer and the 2D data decoder.

[0168] After the input layer obtains the 3D model containing the watermark, it transfers the 3D model containing the watermark to the 3D model decoder. The 3D model decoder extracts the watermark from the 3D model containing the watermark and outputs the watermark through the output layer.

[0169] After the input layer obtains the 2D data containing the watermark, it transfers the 2D data containing the watermark to the 2D data decoder. The 2D data decoder extracts the watermark from the 2D data containing the watermark and outputs the watermark through the output layer.

[0170] The method examples provided in the embodiments of the present application are introduced in detail above. It can be understood that for the corresponding device to implement the above functions, it includes the corresponding hardware structure and / or software module for executing each function. Those skilled in the art should easily realize that, combining the units and algorithm steps of each example described in the embodiments disclosed in this article, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the way of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0171] Figure 12 FIG. 18 is a schematic structural diagram of a watermark extraction device 1200 provided by an embodiment of the present application. The device 1200 includes a processing module 1210 and an input module 1220. The input module 1220 performs a receiving step or an input step under the control of the processing module 1210.

[0172] The input module 1220 is used to: obtain a target file, where the target file is a target 3D model or target 2D data, the target 2D data is an image or video generated by rendering a base 3D model and a 2D texture map, the target 3D model carries a target watermark, the target 2D data carries a target watermark, and the target watermark carried by the target 2D data is the watermark carried by the base 3D model and / or the 2D texture map; the processing module 1210 is used to: determine the type of the target file, where the type includes a 3D model or 2D data; when the type is a 3D model, the processing module 1210 is further used to extract the target watermark carried by the target file through a 3D model decoder, and the 3D model decoder is a sub-network of a neural network; or, when the type is 2D data, the processing module 1210 is further used to extract the target watermark carried by the target file through at least one 2D data decoder, and the at least one 2D data decoder is a sub-network of a neural network.

[0173] Among them, as an example of a software functional unit, the processing module 1210 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above computing instance may be one or more. For example, the processing module 1210 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers for running the code may be distributed in the same region, or may be distributed in different regions. Further, the multiple hosts / virtual machines / containers for running the code may be distributed in the same availability zone (AZ), or may be distributed in different AZs, and each AZ includes one data center or multiple geographically proximate data centers. Among them, usually one region may include multiple AZs.

[0174] Similarly, the multiple hosts / virtual machines / containers for running the code may be distributed in the same virtual private cloud (VPC), or may be distributed in multiple VPCs. Among them, usually one VPC is set within one region. For cross-region communication between two VPCs within the same region and between VPCs in different regions, a communication gateway needs to be set in each VPC, and the interconnection between VPCs is achieved through the communication gateway.

[0175] As an example of a hardware functional unit, the processing module 1210 may include at least one computing device, such as a server or the like. Alternatively, the processing module 1210 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). Among them, the above PLD may be implemented by a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0176] The multiple computing devices included in the processing module 1210 may be distributed in the same area or in different areas. The multiple computing devices included in the processing module 1210 may be distributed in the same availability zone (AZ) or in different AZs. Similarly, the multiple computing devices included in the processing module 1210 may be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Among them, the multiple computing devices may be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0177] It should be noted that in other embodiments, the processing module 1210 may be used to execute the processing steps in method 500, and the input module 1220 may be used to execute the input step, the receiving step, or the obtaining step in method 500. The steps to be implemented by the processing module 1210 and the input module 1220 can be specified as needed. By respectively implementing different steps in method 500 through the processing module 1210 and the input module 1220, all functions of the watermark extraction device 1200 are realized.

[0178] This application also provides a computing device 100. As Figure 13 shown, the computing device 100 includes: a bus 102, a processor 104, a memory 106, and a communication interface 108. The processor 104, the memory 106, and the communication interface 108 communicate with each other through the bus 102. The computing device 100 may be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 100.

[0179] The bus 102 can be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, Figure 13 only one line is used in Figure 13 , but it does not mean that there is only one bus or one type of bus. The bus 102 can include a path for transmitting information between various components of the computing device 100 (e.g., the memory 106, the processor 104, the communication interface 108).

[0180] The processor 104 can include any one or more of processors such as a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a Micro Processor (MP), or a Digital Signal Processor (DSP).

[0181] The memory 106 can include volatile memory, such as Random Access Memory (RAM). The processor 104 can also include non-volatile memory, such as Read-Only Memory (ROM), flash memory, a Hard Disk Drive (HDD), or a Solid State Drive (SSD).

[0182] The memory 106 stores executable program code, and the processor 104 executes the executable program code to respectively implement the functions of the aforementioned processing module 1210 and input module 1220, thereby implementing the method 500. That is, instructions for executing the method 500 are stored on the memory 106.

[0183] The communication interface 108 uses a transceiver module such as, but not limited to, a network interface card, a transceiver, to implement communication between the computing device 100 and other devices or communication networks.

[0184] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0185] As Figure 14 shown, the computing device cluster includes at least one computing device 100. In the memory 106 of one or more computing devices 100 in the computing device cluster, the same instructions for executing the method 500 can be stored.

[0186] In some possible implementation manners, the memory 106 of one or more computing devices 100 in the computing device cluster can also store partial instructions for executing the method 500 respectively. In other words, a combination of one or more computing devices 100 can jointly execute the instructions for executing the method 500.

[0187] It should be noted that the memories 106 in different computing devices 100 in the computing device cluster can store different instructions, which are respectively used to execute partial functions of the watermark extraction device 1200. That is, the instructions stored in the memories 106 of different computing devices 100 can implement the functions of one or more modules in the processing module 1210 and the input module 1220.

[0188] In some possible implementation manners, one or more computing devices in the computing device cluster can be connected through a network. Among them, the network can be a wide area network or a local area network, etc. Figure 15 Illustrates a possible implementation manner. As Figure 15 shown, two computing devices 100A and 100B are connected through a network. Specifically, they are connected to the network through the communication interfaces in each computing device. In this type of possible implementation manner, the memory 106 in the computing device 100A stores instructions for executing the function of the processing module 1210. At the same time, the memory 106 in the computing device 100B stores instructions for executing the function of the input module 1220.

[0189] Figure 15 The connection manner between the computing device clusters shown can be considered that since the method 500 provided in the present application requires a large amount of data storage, the function implemented by the input module 1220 is considered to be executed by the computing device 100B.

[0190] It should be understood that Figure 15 the function of the computing device 100A shown can also be completed by multiple computing devices 100. Similarly, the function of the computing device 100B can also be completed by multiple computing devices 100.

[0191] The embodiments of the present application also provide a computer program product containing instructions. The computer program product may be software or a program product containing instructions that can run on a computing device or be stored in any available medium. When the computer program product runs on at least one computing device, at least one computing device is caused to execute method 500.

[0192] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium may be any available medium that a computing device can store or a data storage device such as a data center containing one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a high-density digital video disc (DVD)), or a semiconductor medium (e.g., a solid-state drive), etc. The computer-readable storage medium includes instructions that instruct the computing device to execute method 500.

[0193] It should be understood that the "embodiments" mentioned throughout the specification mean that specific features, structures or characteristics related to the embodiments are included in at least one embodiment of the present application. Therefore, the various embodiments throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics may be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the terminal device and / or the network device may execute some or all of the steps in the embodiments. These steps or operations are only examples, and the embodiments of the present application may also execute other operations or various modifications of the operations. In addition, the various steps may be executed in different orders presented in the embodiments, and it is possible that not all the operations in the embodiments of the present application need to be executed. Moreover, the size of the serial numbers of the above processes does not mean the order of execution is prior or posterior, and the order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0194] It should also be understood that in the present application, "when", "if" and "in case" all mean that the execution subject will make corresponding processing under a certain objective situation, which is not a time limit, and it is not required that the execution subject must have a judgment action when implemented, nor does it mean that there are other limitations.

[0195] In addition, the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone.

[0196] It should be understood that in various embodiments of the present application, "B corresponding to A" means that B is associated with A, and B can be determined according to A. However, it should also be understood that determining B according to A does not mean determining B only according to A, and B can also be determined according to A and / or other information.

[0197] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for extracting a watermark, characterized in that: The method comprises: Obtaining a target file, where the target file is a target three-dimensional model or target two-dimensional data, where the target two-dimensional data is an image or video generated by rendering based on a basic three-dimensional model and a two-dimensional map, where the target three-dimensional model carries a target watermark, and the target two-dimensional data carries a target watermark, where the target watermark carried by the target two-dimensional data is the watermark carried by the basic three-dimensional model and / or the two-dimensional map; Determining the type of the target file, wherein the type includes a three-dimensional model or two-dimensional data; When the type is a three-dimensional model, the target watermark carried by the target file is extracted by a three-dimensional model decoder, wherein the three-dimensional model decoder is a sub-network of a neural network; or When the type is two-dimensional data, the target watermark carried by the target file is extracted by at least one two-dimensional data decoder, and the at least one two-dimensional data decoder is a sub-network of a neural network.

2. The method according to claim 1, characterized in that Before extracting the target watermark carried by the target file through at least one two-dimensional data decoder, the method further includes: Determining at least one lighting model corresponding to the target file; The at least one two-dimensional data decoder is determined according to the at least one illumination model, and the at least one two-dimensional data decoder is used to extract the target watermark carried by the target file according to its corresponding illumination model.

3. The method according to claim 2, characterized in that Determining at least one illumination model corresponding to the target file includes: When receiving illumination model indication information, determining the at least one illumination model according to the illumination model indication information, wherein the illumination model indication information is used to indicate the illumination model used when rendering and generating the target two-dimensional data; or When no illumination model indication information is received, and when watermark extraction indication information is received, it is determined that the at least one illumination model is a preset plurality of illumination models, the illumination model indication information is used to indicate the illumination model used when rendering and generating the target two-dimensional data, and the watermark extraction indication information is used to indicate the extraction of the target watermark carried by the target file.

4. The method according to claim 3, characterized in that When the at least one illumination model is the plurality of preset illumination models, the at least one two-dimensional data decoder is a plurality of two-dimensional data decoders, and the plurality of two-dimensional data decoders correspond one-to-one to the plurality of preset illumination models. Processing the target file by using the at least one two-dimensional data decoder includes: Processing the target file using the multiple two-dimensional data decoders respectively to determine multiple intermediate watermarks; Comparing the plurality of intermediate watermarks with a reference watermark, respectively, to determine a plurality of similarities corresponding to the plurality of intermediate watermarks, the plurality of similarities being used to indicate a degree of similarity between the plurality of intermediate watermarks and the reference watermark; When at least one similarity among the multiple similarities is greater than or equal to a similarity threshold, the target watermark is determined from the intermediate watermarks corresponding to the at least one similarity, wherein the target watermark is the intermediate watermark corresponding to a value with the highest similarity among the at least one similarity.

5. The method according to any one of claims 1 to 4, characterized in that Before obtaining the target file, the method includes: Obtain the original watermark and original 3D model; Encoding the original watermark and the original three-dimensional model through a three-dimensional model encoder to generate three-dimensional training data, wherein the three-dimensional training data is a three-dimensional model carrying the original watermark, and the three-dimensional model encoder is a subnetwork of the neural network; Attacking the three-dimensional training data to generate attacked three-dimensional training data; The 3D model decoder is trained using the attacked 3D training data, and the 3D model decoder is used to extract a first watermark from the attacked 3D training data.

6. The method according to any one of claims 1 to 4, characterized in that Before obtaining the target file, the method includes: Obtain the original watermark, original 3D model and original 2D map; Encoding the original watermark and the original two-dimensional map through a two-dimensional data encoder to generate a two-dimensional training map, wherein the two-dimensional training map is a two-dimensional map carrying the original watermark, and the two-dimensional data encoder is a subnetwork of the neural network; Attacking a three-dimensional model composed of the original three-dimensional model and the two-dimensional training map to generate an attacked three-dimensional model; Rendering the attacked three-dimensional model using at least one differentiable renderer to obtain at least one two-dimensional training data, where the at least one two-dimensional training data is a picture or a video; The at least one two-dimensional data decoder is trained using the at least one two-dimensional training data, and the at least one two-dimensional data decoder is used to extract a second watermark from the at least one two-dimensional training data.

7. The method according to claim 6, characterized in that The using the at least one two-dimensional training data to train the at least one two-dimensional model decoder comprises: When the training degree of the three-dimensional model decoder meets the requirement, the at least one two-dimensional data decoder is trained using the at least one two-dimensional training data.

8. The method according to claim 6 or 7, characterized in that The at least one two-dimensional data decoder includes a plurality of two-dimensional data decoders, the differentiable renderer includes a plurality of lighting models, and the plurality of two-dimensional data decoders correspond one-to-one to the plurality of lighting models.

9. A device for extracting watermarks, characterized in that: The device comprises: An input module is configured to obtain a target file, wherein the target file is a target three-dimensional model or target two-dimensional data, wherein the target two-dimensional data is an image or video generated by rendering based on a basic three-dimensional model and a two-dimensional map, wherein the target three-dimensional model carries a target watermark, and the target two-dimensional data carries a target watermark, wherein the target watermark carried by the target two-dimensional data is the watermark carried by the basic three-dimensional model and / or the two-dimensional map; A processing module, configured to determine a type of the target file, wherein the type includes a three-dimensional model or two-dimensional data; When the type is a three-dimensional model, the processing module is further configured to extract the target watermark carried by the target file through a three-dimensional model decoder, wherein the three-dimensional model decoder is a sub-network of a neural network; or When the type is two-dimensional data, the processing module is further used to extract the target watermark carried by the target file through at least one two-dimensional data decoder, and the at least one two-dimensional data decoder is a sub-network of a neural network.

10. The device according to claim 9, characterized in that Before extracting the target watermark carried by the target file through at least one two-dimensional data decoder, the processing module is further configured to: Determining at least one lighting model corresponding to the target file; The at least one two-dimensional data decoder is determined according to the at least one illumination model, and the at least one two-dimensional data decoder is used to extract the target watermark carried by the target two-dimensional data according to its corresponding illumination model.

11. The device according to claim 10, characterized in that The processing module is specifically used for: When receiving the illumination model indication information, determining the at least one illumination model according to the illumination model indication information, wherein the illumination model indication information is used to indicate the illumination model used when rendering and generating the target two-dimensional data; or, When no illumination model indication information is received, and when watermark extraction indication information is received, it is determined that the at least one illumination model is a preset plurality of illumination models, the illumination model indication information is used to indicate the illumination model used when rendering and generating the target two-dimensional data, and the watermark extraction indication information is used to indicate the extraction of the target watermark carried by the target file.

12. The device according to claim 11, characterized in that When the at least one illumination model is the plurality of preset illumination models, the at least one two-dimensional data decoder is a plurality of two-dimensional data decoders, and the plurality of two-dimensional data decoders correspond one-to-one to the plurality of preset illumination models, the processing module is specifically configured to: Processing the target file using the multiple two-dimensional data decoders respectively to determine multiple intermediate watermarks; Comparing the plurality of intermediate watermarks with a reference watermark, respectively, to determine a plurality of similarities corresponding to the plurality of intermediate watermarks, the plurality of similarities being used to indicate a degree of similarity between the plurality of intermediate watermarks and the reference watermark; When at least one similarity among the multiple similarities is greater than or equal to a similarity threshold, the target watermark is determined from the intermediate watermarks corresponding to the at least one similarity, wherein the target watermark is the intermediate watermark corresponding to a value with the highest similarity among the at least one similarity.

13. The device according to any one of claims 9 to 12, characterized in that Before obtaining the target file, The input module is further used to: obtain the original watermark and the original three-dimensional model; The processing module is further configured to: encode the original watermark and the original 3D model through a 3D model encoder to generate 3D training data, wherein the 3D training data is a 3D model carrying the original watermark, and the 3D model encoder is a subnetwork of the neural network; Attacking the three-dimensional training data to generate attacked three-dimensional training data; The 3D model decoder is trained using the attacked 3D training data, and the 3D model decoder is used to extract a first watermark from the attacked 3D training data.

14. The device according to any one of claims 9 to 12, characterized in that Before obtaining the target file, The input module is also used to: obtain the original watermark, the original three-dimensional model and the original two-dimensional map; The processing module is further configured to: encode the original watermark and the original two-dimensional map using a two-dimensional data encoder to generate a two-dimensional training map, wherein the two-dimensional training map is a two-dimensional map carrying the original watermark, and the two-dimensional data encoder is a subnetwork of the neural network; attack the three-dimensional model composed of the original three-dimensional model and the two-dimensional training map to generate an attacked three-dimensional model; Rendering the attacked three-dimensional model using at least one differentiable renderer to obtain at least one two-dimensional training data, where the at least one two-dimensional training data is a picture or a video; The at least one two-dimensional data decoder is trained using the at least one two-dimensional training data, and the at least one two-dimensional data decoder is used to extract a second watermark from the at least one two-dimensional training data.

15. The device according to claim 14, characterized in that The processing module is specifically used for: When the training degree of the three-dimensional model decoder meets the requirement, the at least one two-dimensional data decoder is trained using the at least one two-dimensional training data.

16. The device according to claim 14 or 15, characterized in that The at least one two-dimensional data decoder includes a plurality of two-dimensional data decoders, the differentiable renderer includes a plurality of lighting models, and the plurality of two-dimensional data decoders correspond one-to-one to the plurality of lighting models.

17. A computing device cluster, characterized in that: comprising at least one computing device, each computing device including a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in a memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that The method comprises computer program instructions, which, when executed by a computing device cluster, cause the computing device cluster to perform the method according to any one of claims 1 to 8.

19. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster is caused to perform the method according to any one of claims 1 to 8.

Images