A watermark recognition method and device, electronic device and storage medium

Abstract

The present application relates to a watermark recognition method and device, an electronic device and a storage medium. The method includes: obtaining an image to be recognized that needs to be watermark recognized; performing watermark detection on the image to be recognized to obtain a target watermark in the image to be recognized; determining a first watermark classification result of the target watermark by matching the first target watermark category corresponding to the target watermark in all first candidate watermark categories in the watermark feature database; and inputting the target watermark into a fully connected classification network to determine a second target watermark category of the target watermark and obtain a second watermark classification result of the target watermark; based on the first watermark classification result and the second watermark classification result, determining the target watermark classification result of the image to be recognized. The method provided in the embodiment of the present application can effectively solve the technical problems of low recognition accuracy and high probability of misrecognition when watermark classification is performed only through a watermark feature database.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a watermark recognition method and device, an electronic device and a storage medium. Background Art

[0002] In the process of watermark detection in video, a two-stage solution is usually adopted, in which the first stage is to detect the watermark in the video frame, and the second stage is to identify the detected watermark.

[0003] In the related art, for the detection results of the first stage, the extracted features are matched with the watermark features stored in the database through feature extraction, and the specific category of the detected watermark can be effectively identified through the matching results. By maintaining such a database, new watermark features to be matched can be added at any time. However, watermark recognition only through feature matching has the problems of low recognition accuracy and high probability of misrecognition.

[0004] There is a technical problem in the related art that watermark recognition is performed only by feature matching, which has low recognition accuracy and high probability of misrecognition. Currently, no effective solution has been provided. Summary of the invention

[0005] In order to solve the technical problem that watermark recognition is performed only by feature matching, which has low recognition accuracy and high probability of misrecognition, the present application provides a watermark recognition method and device, an electronic device and a storage medium.

[0006] In a first aspect, an embodiment of the present application provides a watermark recognition method, including:

[0007] Obtain an image to be identified that needs watermark identification;

[0008] Performing watermark detection on the image to be identified to obtain a target watermark in the image to be identified;

[0009] A first watermark classification result of the target watermark is determined by matching a first target watermark category corresponding to the target watermark among all first candidate watermark categories in a watermark feature database; and the target watermark is input into a fully connected classification network to determine a second target watermark category of the target watermark, and a second watermark classification result of the target watermark is obtained, wherein the first watermark classification result is used to indicate a first confidence level that the target watermark is of the first target watermark category, and the second watermark classification result is used to indicate a second confidence level that the target watermark is of the second target watermark category;

[0010] Based on the first watermark classification result and the second watermark classification result, a target watermark classification result of the image to be identified is determined.

[0011] Optionally, as in the aforementioned method, before inputting the target watermark into the fully connected classification network to obtain the second watermark classification result of the target watermark, the method further includes:

[0012] Obtaining a plurality of overlapping watermarks and a plurality of blurred watermarks for training;

[0013] Determine a training overlapping watermark and a verification overlapping watermark from the plurality of overlapping watermarks, and determine a training fuzzy watermark and a verification fuzzy watermark from the plurality of fuzzy watermarks;

[0014] After training the training network by using the training overlapping watermark and the training fuzzy watermark, a trained network is obtained;

[0015] When the accuracy rate obtained by verifying the trained network through the verification overlapping watermark and the verification fuzzy watermark is greater than or equal to the preset accuracy rate, the trained network is determined to be the fully connected classification network.

[0016] Optionally, as in the aforementioned method, inputting the target watermark into a fully connected classification network to obtain a second watermark classification result of the target watermark includes:

[0017] Inputting the target watermark into the fully connected classification network to obtain a second candidate confidence corresponding to each second candidate watermark category, wherein, for each second candidate watermark category, the second candidate confidence corresponding to the second candidate watermark category is the confidence that the target watermark is determined by the fully connected classification network to be the second candidate watermark category;

[0018] The second watermark classification result is obtained by determining the second target watermark category with the largest second candidate confidence among all the second candidate watermark categories, and determining the second candidate confidence corresponding to the second target watermark category as the second confidence.

[0019] Optionally, as in the aforementioned method, the first target watermark category corresponding to the target watermark is matched among all first candidate watermark categories in the watermark feature database to determine the first watermark classification result of the target watermark, including:

[0020] By performing feature extraction on the target watermark, target watermark feature information of the target watermark is obtained, wherein the target watermark feature information includes N-dimensional target watermark features, where N is an integer greater than or equal to 1;

[0021] For each of the first candidate watermark categories, using the calculated candidate cosine similarity value between the target watermark feature information and the candidate watermark feature information of the first candidate watermark category as the first candidate confidence corresponding to the first candidate watermark category, wherein the first candidate watermark category corresponds to the candidate watermark feature information one-to-one, and each of the candidate watermark feature information includes N-dimensional candidate watermark features;

[0022] A maximum first confidence is determined among all the first candidate confidences, and the first watermark classification result is obtained based on the first confidence and the first target watermark category corresponding to the first confidence.

[0023] Optionally, as in the aforementioned method, determining the target watermark classification result of the image to be identified based on the first watermark classification result and the second watermark classification result includes:

[0024] In a case where the first confidence level is greater than the second confidence level, determining the first watermark classification result as the target watermark classification result;

[0025] When the first confidence level is less than or equal to the second confidence level, the second watermark classification result is determined as the target watermark classification result.

[0026] Optionally, as in the aforementioned method, before inputting the target watermark into the fully connected classification network, determining the second target watermark category of the target watermark, and obtaining the second watermark classification result of the target watermark, the method further includes:

[0027] Determining a magnitude relationship between the first confidence level and a preset threshold;

[0028] When the size relationship indicates that the first confidence is greater than or equal to a preset threshold, stop executing the step of inputting the target watermark into a fully connected classification network, determine a second target watermark category of the target watermark, obtain a second watermark classification result of the target watermark, and determine a target watermark classification result of the image to be identified based on the first watermark classification result and the second watermark classification result;

[0029] The first watermark classification result is used as the target watermark classification result.

[0030] Optionally, as in the aforementioned method, inputting the target watermark into a fully connected classification network, determining a second target watermark category of the target watermark, and obtaining a second watermark classification result of the target watermark includes:

[0031] Determining a magnitude relationship between the first confidence level and a preset threshold;

[0032] When the size relationship indicates that the first confidence is less than a preset threshold, the target watermark is input into a fully connected classification network, a second target watermark category of the target watermark is determined, and a second watermark classification result of the target watermark is obtained.

[0033] In a second aspect, an embodiment of the present application provides a watermark recognition device, including:

[0034] An acquisition module is used to acquire an image to be identified that needs to be watermarked;

[0035] A detection module, used for performing watermark detection on the image to be identified to obtain a target watermark in the image to be identified;

[0036] A determination module, configured to determine a first watermark classification result of the target watermark by matching a first target watermark category corresponding to the target watermark among all first candidate watermark categories in a watermark feature database; and input the target watermark into a fully connected classification network to determine a second target watermark category of the target watermark, and obtain a second watermark classification result of the target watermark, wherein the first watermark classification result is used to indicate a first confidence level that the target watermark is of the first target watermark category, and the second watermark classification result is used to indicate a second confidence level that the target watermark is of the second target watermark category;

[0037] A result module is used to determine a target watermark classification result of the image to be identified based on the first watermark classification result and the second watermark classification result.

[0038] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0039] The memory is used to store computer programs;

[0040] The processor is used to implement any of the methods described above when executing the computer program.

[0041] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the storage medium includes a stored program, wherein the program executes a method as described in any of the preceding items when executed.

[0042] The above technical solution provided by the embodiment of the present application has the following advantages compared with the prior art:

[0043] The method provided in the embodiment of the present application determines the first watermark classification result of the target watermark according to the watermark feature database, and inputs the target watermark into the fully connected classification network to determine the second watermark classification result of the target watermark; thereby, by adding the fully connected classification network to classify the image to be identified, the technical problems of low recognition accuracy and high misrecognition probability existing in watermark classification only through the watermark feature database can be effectively solved; and by adopting two different watermark classification methods to obtain the classification results for classifying the image to be identified, the present application can effectively improve the recall rate of watermark recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0046] Figure 1 A schematic diagram of a watermark recognition method provided in an embodiment of the present application;

[0047] Figure 2 A schematic diagram of a flow chart of a watermark recognition method provided in another embodiment of the present application;

[0048] Figure 3 A schematic diagram of a flow chart of a watermark recognition method provided in another embodiment of the present application;

[0049] Figure 4 A schematic diagram of feature comparison in a watermark recognition method provided in an application example of the present application;

[0050] Figure 5 A block diagram of a watermark recognition device provided in an embodiment of the present application;

[0051] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0053] When there are multiple overlapping watermarks or the watermark is particularly blurred, it is usually impossible to return a result with accurate classification label and high confidence when the detected watermark features are matched with the database watermark features. If the features of such overlapping or blurred watermarks are forcibly added to the database, a large number of misidentifications will occur when matching the features extracted from the watermarks detected in the first stage.

[0054] In order to solve the above technical problems, according to one aspect of the embodiments of the present application, a watermark recognition method is provided. Optionally, in this embodiment, the above watermark recognition method can be applied to a hardware environment consisting of a terminal and a server. The server is connected to the terminal through a network, and can be used to provide services (such as data push services, application services, etc.) for the terminal or a client installed on the terminal. A database can be set on the server or independently of the server to provide data storage services for the server.

[0055] The above network may include but is not limited to at least one of the following: wired network, wireless network. The above wired network may include but is not limited to at least one of the following: wide area network, metropolitan area network, local area network, and the above wireless network may include but is not limited to at least one of the following: WIFI (Wireless Fidelity), Bluetooth. The terminal may not be limited to a PC, a mobile phone, a tablet computer, etc.

[0056] The watermark recognition method of the embodiment of the present application can be executed by a server, a terminal, or a server and a terminal together. The terminal can also execute the watermark recognition method of the embodiment of the present application by a client installed thereon.

[0057] Taking the watermark recognition method in this embodiment executed by the server as an example, Figure 1 A watermark recognition method provided in an embodiment of the present application includes the following steps:

[0058] Step S101, obtaining an image to be recognized that needs to be watermark recognized;

[0059] The watermark recognition method in this embodiment can be applied to scenarios where watermarks need to be recognized in media resources (e.g., videos or images), such as: scenarios for recognizing watermarks in videos, scenarios for recognizing watermarks in images, etc., and can also be scenarios for recognizing watermarks in media resources. In the embodiments of this application, the watermark recognition method described above is illustrated using a video as an example. For other types of media resources, the watermark recognition method described above is also applicable if there is no contradiction.

[0060] Taking the scene of identifying watermarks in videos as an example, watermark identification is performed on one or more frames of images in the video to determine the watermark classification result of the watermark in the video.

[0061] Before watermark detection is performed, the object to be detected for which watermark detection is required is obtained in advance. The object to be detected may be uploaded by the terminal user through a preset interface to the server for implementing watermark detection in this application.

[0062] For example, when a terminal user needs to upload a video (ie, an object to be detected) on a target platform, the video needs to be transmitted to a server via a client installed on a terminal used by the terminal user.

[0063] Step S102: Perform watermark detection on the image to be identified to obtain a target watermark in the image to be identified.

[0064] After obtaining the image to be identified, watermark detection can be performed on the image to be identified to determine the image of the watermark portion, that is, to obtain the target watermark in the image to be identified.

[0065] The target watermark may be a partial image corresponding to the watermark in the image to be identified. For example, when a watermark exists in the lower left corner of a frame image a1 of video A, the image including the lower left corner and the watermark is determined as the target watermark.

[0066] Optionally, a detection model for watermark detection may be obtained through pre-training, and then the image to be identified is input into the detection model to detect the target watermark in the image to be identified and select it through a rectangular detection frame.

[0067] Step S103, by matching the first target watermark category corresponding to the target watermark in all the first candidate watermark categories in the watermark feature database, determining the first watermark classification result of the target watermark; and inputting the target watermark into the fully connected classification network, determining the second target watermark category of the target watermark, and obtaining the second watermark classification result of the target watermark, wherein the first watermark classification result is used to indicate the first confidence that the target watermark is of the first target watermark category, and the second watermark classification result is used to indicate the second confidence that the target watermark is of the second target watermark category.

[0068] After the target watermark is detected, the first watermark category of the target watermark can be determined by matching; further, the first target watermark category with the highest feature similarity with the target watermark can be determined among all the first candidate watermark categories by feature matching, and the first watermark classification result of the target watermark can be determined.

[0069] The first watermark classification result includes the probability that the target watermark is of the first target watermark category, for example, the probability that the target watermark is "XX video" is 95%.

[0070] At the same time, the target watermark can also be identified through a fully connected classification network, so as to determine the second target watermark category of the target watermark through the identification result output by the fully connected classification network, and obtain the second watermark classification result of the target watermark.

[0071] The fully connected classification network includes at least a hidden layer (the hidden layer may include one or more layers), the hidden layer includes multiple nodes, and each node of the hidden layer is connected to all nodes of the previous layer.

[0072] Similar to the first watermark classification result, the second watermark classification result includes the probability that the target watermark is a second target watermark category, for example, the probability that the target watermark is "YY video" is 90%. And the first watermark category corresponding to the first watermark classification result can be the same as the second target watermark category of the second watermark classification result.

[0073] Step S104: determining a target watermark classification result of the image to be identified based on the first watermark classification result and the second watermark classification result.

[0074] After obtaining the first watermark classification result and the second watermark classification result, different watermark classification results obtained by performing different recognition methods on the image to be recognized can be obtained. Therefore, based on the above two watermark classification results, processing can be performed to obtain the target watermark classification result of the image to be recognized.

[0075] For example, the target watermark classification result can be obtained by weighting the first watermark classification result and the second watermark classification result; the first confidence of the first watermark classification result and the second confidence of the second watermark classification result can also be determined, and then the largest target confidence between the first confidence and the second confidence is determined, and the watermark classification result corresponding to the target confidence is used as the target watermark classification result.

[0076] As an optional implementation, as in the above method, the step S104 determines the target watermark classification result of the image to be identified based on the first watermark classification result and the second watermark classification result, including the following steps:

[0077] Step S501, when the first confidence level is greater than the second confidence level, determining the first watermark classification result as the target watermark classification result;

[0078] Step S502: when the first confidence level is less than or equal to the second confidence level, determine the second watermark classification result as the target watermark classification result.

[0079] After determining the first confidence and the second confidence, the relationship between the first confidence and the second confidence can be compared. When the first confidence is greater than the second confidence, the first watermark classification result is determined as the target watermark classification result. Conversely, since the target watermark can be classified more accurately through the fully connected classification network, therefore, when the first confidence is less than or equal to the second confidence, the second watermark classification result is determined as the target watermark classification result.

[0080] Through the method in this embodiment, the first watermark classification result of the target watermark is determined according to the watermark feature database, and the target watermark is input into the fully connected classification network to determine the second watermark classification result of the target watermark; thereby, by adding a fully connected classification network to classify the image to be identified, the technical problems of low recognition accuracy and high misrecognition probability that only watermark classification through the watermark feature database is solved; and by adopting two different watermark classification methods to obtain the classification results for classifying the image to be identified, the present application can effectively improve the recall rate of watermark recognition.

[0081] like Figure 2 As shown, as an optional implementation, as in the aforementioned method, before the target watermark is input into the fully connected classification network in step S103 to obtain the second watermark classification result of the target watermark, the method further includes the following steps:

[0082] Step S201, obtaining a plurality of overlapping watermarks and a plurality of fuzzy watermarks for training.

[0083] When watermarks are overlapping or blurred, the related art is also unable to accurately identify this type of watermark. Therefore, multiple overlapping watermarks and multiple blurred watermarks can be obtained for training.

[0084] An overlapping watermark may be a watermark image in which multiple watermarks overlap; a fuzzy watermark may be a watermark image in which the watermark clarity is low or the watermark image is unclear due to the influence of the background.

[0085] Optionally, the overlapping watermark may be obtained by overlapping the clear watermark or by blurring the clear watermark to obtain the fuzzy watermark; multiple overlapping watermarks and multiple fuzzy watermarks may also be obtained by a preset query method (eg, searching on the Internet).

[0086] Step S202: determining a training overlapping watermark and a verification overlapping watermark from a plurality of overlapping watermarks, and determining a training fuzzy watermark and a verification fuzzy watermark from a plurality of fuzzy watermarks.

[0087] After obtaining multiple overlapping watermarks and multiple fuzzy watermarks, the multiple overlapping watermarks and multiple fuzzy watermarks can be grouped to obtain overlapping watermarks for training and overlapping watermarks for verification, and fuzzy watermarks for training and fuzzy watermarks for verification.

[0088] Optionally, a first quantity ratio between overlapping watermarks for training and overlapping watermarks for verification, and a second quantity ratio between fuzzy watermarks for training and fuzzy watermarks for verification can be set according to actual model training requirements.

[0089] Step S203, after training the training network with the training overlapping watermark and the training fuzzy watermark, a trained network is obtained.

[0090] After the overlapping watermark for training and the fuzzy watermark for training are determined, the preset network to be trained can be trained by using the overlapping watermark for training and the fuzzy watermark for training to obtain a trained network.

[0091] Optionally, the network to be trained is also a fully connected network, and the fully connected network includes at least one hidden layer.

[0092] Step S204: when the accuracy rate obtained by verifying the trained network through the verification overlapping watermark and the verification fuzzy watermark is greater than or equal to the preset accuracy rate, the trained network is determined to be a fully connected classification network.

[0093] After the trained network is obtained through training, the trained network can be verified by the verification overlapping watermark and the verification fuzzy watermark obtained in step S202. If the accuracy rate obtained by verifying the trained network through the verification overlapping watermark and the verification fuzzy watermark is greater than or equal to the preset accuracy rate, the trained network is determined to be a fully connected classification network.

[0094] If the accuracy of the trained network verified by the verification overlapping watermark and the verification fuzzy watermark is less than the preset accuracy, the trained network is trained again according to the data not used for training in the training overlapping watermark and the training fuzzy watermark, and / or the new training overlapping watermark and the new training fuzzy watermark, and this cycle is repeated until the accuracy of the final network is greater than or equal to the preset accuracy, thereby obtaining a fully connected classification network.

[0095] The preset accuracy rate may be a preset value used to indicate the minimum accuracy rate that the fully connected classification network needs to achieve.

[0096] For example, when the preset accuracy is 98%, if the accuracy of the trained network is 99% after verification by verifying the overlapping watermark for verification and the fuzzy watermark for verification, the trained network is determined to be a fully connected classification network.

[0097] Through the method in this embodiment, the training network is trained by multiple overlapping watermarks and multiple fuzzy watermarks, so that the final fully connected classification network can be used to classify overlapping watermarks and fuzzy watermarks, so as to improve the accuracy of classification when the watermarks are overlapping or blurred.

[0098] like Figure 3 As shown, as an optional implementation, as in the aforementioned method, the step S103 inputs the target watermark into the fully connected classification network to obtain the second watermark classification result of the target watermark, including the following steps:

[0099] Step S301, inputting the target watermark into the fully connected classification network to obtain the second candidate confidence corresponding to each second candidate watermark category, wherein, for each second candidate watermark category, the second candidate confidence corresponding to the second candidate watermark category is the confidence that the target watermark is determined by the fully connected classification network as the second candidate watermark category;

[0100] Step S302: obtaining a second watermark classification result by determining a second target watermark category with the largest second candidate confidence among all second candidate watermark categories, and determining the second candidate confidence corresponding to the second target watermark category as the second confidence.

[0101] After the fully connected classification network is determined, the target watermark can be identified through the fully connected classification network to obtain a second classification result corresponding to the target watermark.

[0102] After the target watermark is input into the fully connected classification network, the fully connected classification network can analyze the target watermark and obtain the second candidate confidence corresponding to each second candidate watermark category; for example, when the second candidate watermark categories include: "XX video", "YY video" and "ZZ video", the second classification results of the target watermark corresponding to "XX video", "YY video" and "ZZ video" can be obtained, and one characterization method of the second classification result can be: the second candidate confidence of "XX video" is 0.95, the second candidate confidence of "YY video" is 0.02, and the second candidate confidence of "ZZ video" is 0.03.

[0103] After the second candidate confidences corresponding to each second candidate watermark category are determined, the second watermark classification result of the second candidate watermark can be determined based on each second candidate confidence.

[0104] Since each second candidate watermark category has obtained a corresponding second candidate confidence, after determining the maximum second confidence among all second candidate confidences, a second watermark classification result can be obtained based on the second confidences and the second candidate watermark categories corresponding to the second confidences.

[0105] Based on the above example, since the second candidate confidence corresponding to "XX video" is 0.95, which is greater than the second candidate confidence of "YY video" and the second candidate confidence of "ZZ video", the second watermark classification result can be: the second candidate confidence of "XX video" is 0.95.

[0106] By using the method in this embodiment, when multiple results are obtained through the fully connected classification network analysis, the second watermark classification result of the target watermark can be determined from all the results.

[0107] As an optional implementation, as in the aforementioned method, step 103 determines the first watermark classification result of the target watermark by matching the first target watermark category corresponding to the target watermark in all first candidate watermark categories in the watermark feature database, and includes the following steps:

[0108] Step S401: extracting features from a target watermark to obtain target watermark feature information of the target watermark, wherein the target watermark feature information includes N-dimensional target watermark features, where N is an integer greater than or equal to 1.

[0109] After the target watermark is obtained, a first target watermark category of the target watermark may be determined by feature matching.

[0110] In order to perform feature matching, it is necessary to extract features of the target watermark. Optionally, the target watermark may be extracted using a network model for watermark feature extraction. Further, the extracted target watermark feature information may include 2048-dimensional target watermark features.

[0111] Step S402: for each first candidate watermark category, the candidate cosine similarity value between the calculated target watermark feature information and the candidate watermark feature information of the first candidate watermark category is used as the first candidate confidence corresponding to the first candidate watermark category, wherein the first candidate watermark category corresponds to the candidate watermark feature information one-to-one, and each candidate watermark feature information includes N-dimensional candidate watermark features.

[0112] After the target watermark characteristic information of the target watermark is obtained, a first target watermark category corresponding to the target watermark can be determined from among all first candidate watermark categories based on the target watermark characteristic information.

[0113] In order to determine the similarity between the target watermark and any first candidate watermark category, the candidate cosine similarity value between the target watermark feature information and the candidate watermark feature information of the first candidate watermark category can be calculated in advance. Optionally, the candidate watermark feature information can also be obtained by extracting the feature of the watermark of the first candidate watermark category through the aforementioned network model for watermark feature extraction. Therefore, the dimension of the candidate watermark feature included in the candidate watermark feature information is completely consistent with the dimension of the target watermark feature included in the target watermark feature information.

[0114] Furthermore, the first candidate watermark category may correspond to multiple candidate watermark feature information, and each candidate watermark feature information may be extracted based on different watermark images of the first candidate watermark category; therefore, the similarity between the target watermark and any first candidate watermark category may include multiple ones. For example, when the first candidate watermark category is "XX video", and the watermark pictures corresponding to "XX video" include: X1 (corresponding candidate watermark feature information is x1), X2 (corresponding candidate watermark feature information is x2), X3 (corresponding candidate watermark feature information is x3), X4 (corresponding candidate watermark feature information is x4), X5 (corresponding candidate watermark feature information is x5); then the candidate cosine similarity values ​​between the target watermark and the first candidate watermark category may include: the sub-candidate cosine similarity value between the target watermark feature information and x1, the sub-candidate cosine similarity value between the target watermark feature information and x2, the sub-candidate cosine similarity value between the target watermark feature information and x3, the sub-candidate cosine similarity value between the target watermark feature information and x4, and the sub-candidate cosine similarity value between the target watermark feature information and x5. Further, the candidate cosine similarity value may be the average of the sub-candidate cosine similarity values, or the maximum value of the sub-candidate cosine similarity values.

[0115] After determining the target watermark feature information and the candidate watermark feature information of the first candidate watermark category, the candidate cosine similarity value between the target watermark feature information and the candidate watermark feature information can be calculated as the first confidence corresponding to the first candidate watermark category. According to this method, the first confidence between the target watermark and each first candidate watermark category can be calculated. Therefore, each first candidate watermark category can obtain the corresponding first confidence, and the confidence can be used to indicate the probability that the target watermark is the first candidate watermark category.

[0116] Step S403: determining the largest first confidence among all first candidate confidences, and obtaining a first watermark classification result based on the first confidence and the first target watermark category corresponding to the first confidence.

[0117] After obtaining all the first confidences, the largest first confidence among all the first confidences can be determined, and then the first candidate watermark category corresponding to the largest first confidence is determined as the first target watermark category of the target watermark, and the first watermark classification result is obtained based on the largest first confidence and the first target watermark category corresponding to the first confidence.

[0118] For example, when the first candidate watermark categories include: "XX video", "YY video" and "ZZ video", the target watermark feature information can be obtained, which corresponds to the first candidate confidence cosine1 (e.g., 0.96) of the first candidate watermark category of "XX video", the first candidate confidence cosine2 (e.g., 0.02) of the first candidate watermark category of "YY video" and the first candidate confidence cosine3 (e.g., 0.02) of the first candidate watermark category of "ZZ video" respectively; then the first candidate confidence cosine1 is determined to be the first confidence, the first candidate watermark category is watermarked as the first target watermark category, and the first watermark classification result is obtained based on the first confidence 0.96 and the first target watermark category.

[0119] Through the method in this embodiment, a method is provided for determining the first target watermark category and the first confidence of the target watermark by feature matching; and the method in this embodiment can obtain the first confidence by calculating the cosine similarity, which has the advantage of high processing efficiency.

[0120] As an optional implementation, as in the aforementioned method, before the target watermark is input into the fully connected classification network in step S103, the second target watermark category of the target watermark is determined, and the second watermark classification result of the target watermark is obtained, the following steps are also included:

[0121] Step S601, determine the size relationship between the first confidence and the preset threshold. Step S602, when the size relationship indicates that the first confidence is greater than or equal to the preset threshold, stop executing the step of inputting the target watermark into the fully connected classification network, determine the second target watermark category of the target watermark, and obtain the second watermark classification result of the target watermark, and determine the target watermark classification result of the image to be identified based on the first watermark classification result and the second watermark classification result. Step S603, use the first watermark classification result as the target watermark classification result.

[0122] After the first confidence level is determined, the first confidence level may be compared with a preset threshold value to obtain a magnitude relationship between the first confidence level and the preset threshold value.

[0123] The preset threshold may be a minimum value of the first confidence level for indicating that the first watermark classification result may be directly used as the target watermark classification result.

[0124] When the first confidence level is greater than or equal to the preset threshold, the subsequent step S103 of inputting the target watermark into the fully connected classification network, determining the second target watermark category of the target watermark, and obtaining the second watermark classification result of the target watermark can be stopped, and S104 determines the target watermark classification result of the image to be identified based on the first watermark classification result and the second watermark classification result.

[0125] And the first watermark classification result is directly used as the target watermark classification result.

[0126] For example, when the first confidence is 0.98 and the preset threshold is 0.95, since the first confidence is greater than the preset threshold, the first watermark classification result corresponding to the first confidence can be used as the target watermark classification result.

[0127] Through the method in this embodiment, when the first confidence reaches the preset threshold, the second target watermark classification result can be determined directly based on the first watermark classification result corresponding to the first confidence, without going through a fully connected classification network, so as to improve the efficiency of finally obtaining the target watermark classification result.

[0128] As an optional implementation, as in the aforementioned method, the step S103 inputs the target watermark into the fully connected classification network, determines the second target watermark category of the target watermark, and obtains the second watermark classification result of the target watermark, including the following steps:

[0129] Step S701, determining a magnitude relationship between a first confidence level and a preset threshold;

[0130] Step S702: when the size relationship indicates that the first confidence is less than a preset threshold, the target watermark is input into a fully connected classification network, a second target watermark category of the target watermark is determined, and a second watermark classification result of the target watermark is obtained.

[0131] After the first confidence level is determined, the first confidence level may be compared with a preset threshold value to obtain a magnitude relationship between the first confidence level and the preset threshold value.

[0132] The preset threshold may be a minimum value of the first confidence level for indicating that the first watermark classification result may be directly used as the target watermark classification result.

[0133] When the size relationship indicates that the first confidence is less than the preset threshold, it means that the similarity between the target watermark and any of the first candidate watermark categories does not meet the requirement indicated by the preset threshold. Therefore, it is necessary to input the target watermark into the fully connected classification network, determine the second target watermark category of the target watermark, and obtain the second watermark classification result of the target watermark.

[0134] After obtaining the second watermark classification result, a target watermark classification result of the image to be identified is determined based on the first watermark classification result and the second watermark classification result.

[0135] Through the method in this embodiment, the second target watermark classification result can be determined through the fully connected classification network only when the first confidence level does not reach the preset threshold, thereby effectively ensuring the accuracy of the final target watermark classification result.

[0136] As described below, an application example of applying any of the above embodiments is provided:

[0137] Create and maintain a watermark feature database for storing watermark features. For the second candidate watermark category to be identified, collect a sufficient number of clear watermark images (usually 50-300 watermark images are collected according to each second candidate watermark category). Through a network model for extracting watermark features, extract 2048-dimensional sub-candidate watermark feature information from the collected watermark images and store it in the watermark feature database. For any second candidate watermark category, obtain the candidate watermark feature information based on all the sub-candidate watermark feature information corresponding to the second candidate watermark category.

[0138] In the application process of watermark recognition for the image to be identified, the target watermark detected in the image to be identified is subjected to watermark feature extraction using the same network model for extracting watermark features, and target watermark feature information of 2048-dimensional target watermark features is obtained. The target watermark feature information is subjected to cosine feature matching with each candidate watermark feature information in the watermark feature database (i.e., the candidate cosine similarity value is calculated) and the best matching result is found.

[0139] like Figure 4 As shown in the figure, the matching results of the first target image "TikTok" and the third target image "Xigua Video" are correct, and the matching result confidence is high. The second target image "Xigua Video" has a blurred watermark, an incorrect matching result, and a low matching confidence, which is a recall failure case.

[0140] To solve the error cases in the matching process, Figure 4For overlapping watermarks similar to the second target image “Xigua Video” in the figure, a fully connected classification network is added to improve the classification recall of overlapping and blurred watermarks.

[0141] In the process of training the fully connected classification network, both clear watermark data and overlapping or blurred watermark data (i.e., overlapping watermarks and blurred watermarks) should be used for learning difficult samples.

[0142] After extracting features from a detected watermark image, a first watermark classification result can be obtained by matching in the watermark feature database, and a second watermark classification result can also be obtained through the fully connected classification network. Finally, the result of watermark feature matching is merged with the result obtained by the fully connected classification network to obtain the final result. This can ensure the accuracy of the algorithm and improve the recall rate of the algorithm. At the same time, it also meets the need to dynamically add watermark features that need to be identified in the database in the watermark detection project.

[0143] like Figure 5 As shown, according to an embodiment of another aspect of the present application, a watermark recognition device is also provided, including:

[0144] The acquisition module 1 is used to acquire the image to be identified that needs to be watermarked;

[0145] Detection module 2, used to perform watermark detection on the image to be identified, and obtain the target watermark in the image to be identified;

[0146] Determination module 3, used for determining a first watermark classification result of the target watermark by matching a first target watermark category corresponding to the target watermark among all first candidate watermark categories in the watermark feature database, and inputting the target watermark into a fully connected classification network to determine a second target watermark category of the target watermark, and obtaining a second watermark classification result of the target watermark, wherein the first watermark classification result is used to indicate a first confidence level that the target watermark is of the first target watermark category, and the second watermark classification result is used to indicate a second confidence level that the target watermark is of the second target watermark category;

[0147] The result module 4 is used to determine the target watermark classification result of the image to be identified based on the first watermark classification result and the second watermark classification result.

[0148] Specifically, the specific process of each module in the device of the embodiment of the present invention realizing its function can be referred to the relevant description in the method embodiment, which will not be repeated here.

[0149] According to another embodiment of the present application, there is also provided an electronic device, including: Figure 6As shown, the electronic device may include: a processor 1501 , a communication interface 1502 , a memory 1503 and a communication bus 1504 , wherein the processor 1501 , the communication interface 1502 , and the memory 1503 communicate with each other via the communication bus 1504 .

[0150] Memory 1503, used for storing computer programs;

[0151] The processor 1501 is used to implement the steps of the above method embodiment when executing the program stored in the memory 1503.

[0152] The bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0153] The communication interface is used for communication between the above electronic device and other devices.

[0154] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0155] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0156] An embodiment of the present application further provides a computer-readable storage medium, the storage medium including a stored program, wherein the method steps of the above method embodiment are executed when the program is run.

[0157] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0158] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A watermark recognition method, characterized in that: include: Obtain an image to be identified that needs watermark identification; Performing watermark detection on the image to be identified to obtain a target watermark in the image to be identified; Determine a first watermark classification result of the target watermark by matching a first target watermark category corresponding to the target watermark among all first candidate watermark categories in a watermark feature database; and inputting the target watermark into a fully connected classification network, determining a second target watermark category of the target watermark, and obtaining a second watermark classification result of the target watermark, wherein the first watermark classification result is used to indicate a first confidence level that the target watermark is the first target watermark category, and the second watermark classification result is used to indicate a second confidence level that the target watermark is the second target watermark category; Determining a target watermark classification result of the image to be identified based on the first watermark classification result and the second watermark classification result; Before inputting the target watermark into the fully connected classification network to obtain the second watermark classification result of the target watermark, the method further includes: obtaining multiple overlapping watermarks and multiple fuzzy watermarks for training; determining overlapping watermarks for training and overlapping watermarks for verification from the multiple overlapping watermarks, and determining fuzzy watermarks for training and fuzzy watermarks for verification from the multiple fuzzy watermarks; training the network to be trained with the overlapping watermarks for training and the fuzzy watermarks for training to obtain a trained network; and determining the trained network as the fully connected classification network when the accuracy rate obtained by verifying the trained network with the overlapping watermarks for verification and the fuzzy watermarks for verification is greater than or equal to a preset accuracy rate.

2. The method according to claim 1, characterized in that The step of inputting the target watermark into a fully connected classification network to obtain a second watermark classification result of the target watermark includes: Inputting the target watermark into the fully connected classification network to obtain a second candidate confidence corresponding to each second candidate watermark category, wherein, for each second candidate watermark category, the second candidate confidence corresponding to the second candidate watermark category is the confidence that the target watermark is determined by the fully connected classification network to be the second candidate watermark category; The second watermark classification result is obtained by determining the second target watermark category with the largest second candidate confidence among all the second candidate watermark categories, and determining the second candidate confidence corresponding to the second target watermark category as the second confidence.

3. The method according to claim 1, characterized in that The first target watermark category corresponding to the target watermark is matched in all first candidate watermark categories in the watermark feature database to determine the first watermark classification result of the target watermark, including: By performing feature extraction on the target watermark, target watermark feature information of the target watermark is obtained, wherein the target watermark feature information includes N-dimensional target watermark features, where N is an integer greater than or equal to 1; For each of the first candidate watermark categories, using the calculated candidate cosine similarity value between the target watermark feature information and the candidate watermark feature information of the first candidate watermark category as the first candidate confidence corresponding to the first candidate watermark category, wherein the first candidate watermark category corresponds to the candidate watermark feature information one-to-one, and each of the candidate watermark feature information includes N-dimensional candidate watermark features; A maximum first confidence is determined among all the first candidate confidences, and the first watermark classification result is obtained based on the first confidence and the first target watermark category corresponding to the first confidence.

4. The method according to claim 1, characterized in that The determining, based on the first watermark classification result and the second watermark classification result, of a target watermark classification result of the image to be identified includes: In a case where the first confidence level is greater than the second confidence level, determining the first watermark classification result as the target watermark classification result; When the first confidence level is less than or equal to the second confidence level, the second watermark classification result is determined as the target watermark classification result.

5. The method according to any one of claims 1 to 4, characterized in that: Before inputting the target watermark into the fully connected classification network, determining the second target watermark category of the target watermark, and obtaining the second watermark classification result of the target watermark, the method further includes: Determining a magnitude relationship between the first confidence level and a preset threshold; When the size relationship indicates that the first confidence is greater than or equal to a preset threshold, stop executing the step of inputting the target watermark into a fully connected classification network, determine a second target watermark category of the target watermark, obtain a second watermark classification result of the target watermark, and determine a target watermark classification result of the image to be identified based on the first watermark classification result and the second watermark classification result; The first watermark classification result is used as the target watermark classification result.

6. The method according to any one of claims 1 to 4, characterized in that: The step of inputting the target watermark into a fully connected classification network, determining a second target watermark category of the target watermark, and obtaining a second watermark classification result of the target watermark includes: Determining a magnitude relationship between the first confidence level and a preset threshold; When the size relationship indicates that the first confidence is less than a preset threshold, the target watermark is input into a fully connected classification network, a second target watermark category of the target watermark is determined, and a second watermark classification result of the target watermark is obtained.

7. A watermark recognition device, characterized in that: include: An acquisition module is used to acquire an image to be identified that needs to be watermarked; A detection module, used for performing watermark detection on the image to be identified to obtain a target watermark in the image to be identified; A determination module, configured to determine a first watermark classification result of the target watermark by matching a first target watermark category corresponding to the target watermark in all first candidate watermark categories in a watermark feature database; and inputting the target watermark into a fully connected classification network, determining a second target watermark category of the target watermark, and obtaining a second watermark classification result of the target watermark, wherein the first watermark classification result is used to indicate a first confidence level that the target watermark is the first target watermark category, and the second watermark classification result is used to indicate a second confidence level that the target watermark is the second target watermark category; A result module, used to determine a target watermark classification result of the image to be identified based on the first watermark classification result and the second watermark classification result; It also includes a module for obtaining a plurality of overlapping watermarks and a plurality of fuzzy watermarks for training before inputting the target watermark into the fully connected classification network to obtain the second watermark classification result of the target watermark; determining overlapping watermarks for training and overlapping watermarks for verification from the plurality of overlapping watermarks, and determining fuzzy watermarks for training and fuzzy watermarks for verification from the plurality of fuzzy watermarks; training the to-be-trained network with the overlapping watermarks for training and the fuzzy watermarks for training to obtain a trained network; and determining the trained network as the fully connected classification network when the accuracy rate obtained by verifying the trained network with the overlapping watermarks for verification and the fuzzy watermarks for verification is greater than or equal to a preset accuracy rate.

8. An electronic device, characterized in that: include: A processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The storage medium includes a stored program, wherein the program executes the method according to any one of claims 1 to 6 when executed.

Images