Watermarking method based on high-concurrency real-time encoding and decoding of video

Abstract

The application discloses a watermarking method based on high-concurrency real-time encoding and decoding of video, which directly gives each video frame in a second set corresponding historical watermark information through a second strategy, ensures the efficiency of watermarking in the high-concurrency real-time encoding and decoding scene of video, and further improves the speed of encoding watermark information into video encoding data; the first strategy is used to judge the reason for the failure of historical watermarking of each video frame in a first set, and the anti-fake level of watermarking is adaptively adjusted, so that the anti-fake effect of watermarking is ensured while the watermarking efficiency is considered. By introducing random encryption watermark, the anti-fake level of watermark is increased and the security of video decoding is improved without modifying the watermarking algorithm. By verifying the decoding correctness, the provided watermarking method meets the real-time and error rate requirements of video encoding and decoding in a high-concurrency scene, and avoids sacrificing the decoding accuracy when pursuing the anti-fake effect.

Description

Technical Field

[0001] This application relates to the field of video data processing technology, specifically to a watermarking method based on high-concurrency real-time video encoding and decoding. Background Technology

[0002] High-concurrency video processing refers to scenarios where a large number of users simultaneously access or request video content, placing immense pressure on servers, network bandwidth, and system architecture. The primary purpose of adding watermarks to videos is anti-counterfeiting. After the generated watermark is added to the video encoding data, its authenticity can be verified during video decoding and playback. Anti-counterfeiting watermarks are typically invisible. While there are many existing methods for generating anti-counterfeiting watermarks, current watermarking technologies for high-concurrency videos have the following technical shortcomings when dealing with large-scale real-time processing demands:

[0003] 1. To ensure anti-counterfeiting effectiveness, videos in high-concurrency video applications, such as those transmitted on social media platforms, are generally subjected to high-intensity compression. When the quantization parameters of existing watermarks increase, the watermark information of medium and high frequency coefficients is destroyed due to quantization noise, and the failure rate of watermark information extraction increases significantly. That is, excessive video compression and re-editing may make the watermark unrecoverable. Therefore, how to avoid excessively increasing the quantization parameters of the watermark when compressing videos with high intensity, while still meeting the anti-counterfeiting level requirements of the video and reducing the failure rate of watermark information extraction, is a technical problem that urgently needs to be solved in this field.

[0004] 2. Overly complex watermarking addition and extraction algorithms cannot meet the real-time requirements of high-concurrency video processing. However, watermarks generated using simple algorithms are more vulnerable to targeted attacks, such as adversarial sample perturbations (adding noise invisible to the human eye) or collusion attacks (merging multiple watermarked copies), which can lead to a significant drop in the watermark signal-to-noise ratio. Therefore, about 90% of watermarks generated using existing high-concurrency video watermarking technologies can be removed using open-source tools. Thus, there is a need in this field for a high-concurrency video watermarking generation technology that can meet the real-time requirements of high-concurrency video processing and also has strong defense capabilities against attacks. Summary of the Invention

[0005] This application aims to prevent excessive increases in the quantization parameters of watermarks during high-intensity compression of high-concurrency videos, meet the real-time requirements of high-concurrency video transmission, reduce the failure rate of watermark information extraction, and ensure that the watermark information added to the video encoding data has strong defense capabilities against attacks, thereby ensuring the security of high-concurrency video transmission. It provides a watermarking method based on real-time encoding and decoding of high-concurrency videos.

[0006] To achieve this objective, the following technical solution is adopted in this application:

[0007] A watermarking method based on high-concurrency real-time video encoding and decoding is provided, including the following steps:

[0008] S1, for the current user who intends to transmit video, filter out at least one watermark that has a similar feature relationship with the current user and assign it to the set of failed frames;

[0009] S2, perform frame splitting on the video to be transmitted to the current user, and determine whether each split video frame is added to the filtered set of failed watermark assignment frames;

[0010] If so, the currently determined video frame is added to the first set;

[0011] If not, the currently determined video frame is added to the second set;

[0012] S3, for each video frame in the first set and the second set, an invisible watermark is generated using the first strategy and the second strategy, respectively;

[0013] S4, extract the neighboring video frames of the current video frame to be transmitted, and perform random watermark swap encryption to obtain a random encrypted watermark of the current video frame, which is then added to the encoded data of the current video frame.

[0014] S5, verify the correctness of decoding of the current video frame transmitted to the current user based on the random encrypted watermark;

[0015] If the verification is successful, the decoded current video frame will be played to the current user.

[0016] If the verification fails, a video playback error alarm is issued to the current user, and the current video frame is added to the set of failed watermark assignment frames associated with the current user.

[0017] Preferably, the feature similarity relationship between other users and the current user includes any one or more of user feature similarity, failure frame information similarity, and decoding feature similarity;

[0018] The user feature similarity includes one or more similarities between the current user and other users when playing the same video, such as device parameters, network jitter, and geographical location.

[0019] The similarity of the failed frame information is the similarity of the watermark information of the same frames in the history of the same video played by each of the other users who have similar user characteristics to the current user and failed to assign watermarks.

[0020] The decoding feature similarity is the similarity of the decoding delay distribution and / or the dependence strength on the random encrypted watermark when other users with the same user feature similarity as the current user decode the same video frame in their history of playing the same video.

[0021] Preferably, step S1, the method for selecting at least one set of failed frames for watermarking that has a feature similarity relationship with the current user, includes the following steps:

[0022] S11, filter out other users who have similar user characteristics to the current user. When the similarity of user characteristics between the current user and the other users is greater than a preset first similarity threshold, determine that the current user and the other users have similar user characteristics.

[0023] S12, calculate the watermark information similarity of the same frames in the past when the watermark assignment failed to be applied during the playback of the same video by other users who have similar user characteristics to the current user, and filter out the other users whose watermark information similarity is less than the second similarity threshold.

[0024] S13, calculate the decoding feature similarity for each pair of the other users who have played the same video in history and decoded the same video frame, and then extract the set of watermark-assigned failed frames associated with each other user whose decoding feature similarity is greater than the third similarity threshold as the filtering result of the watermark-assigned failed frame set.

[0025] Preferably, the user feature similarity is a weighted sum of the similarity of device parameters, network jitter, and geographical location between the current user and the other users when playing the same video. Geographical location similarity is represented by the straight-line distance between two users at the same geographical level. The greater the straight-line distance, the greater the geographical location similarity between the two users.

[0026] The decoding feature similarity is a weighted sum of the decoding delay distribution and the dependence strength on the random encrypted watermark when two other users decode the same video frame in their history of playing the same video.

[0027] Preferably, devices that have historically played the same video in different geographical locations are divided into several geographical tiers based on their distance from the server that remotely stores the video, from closest to furthest.

[0028] Preferably, step S12, the method for calculating the similarity of the watermark information, includes the following steps:

[0029] S121, calculate the consistency of watermark information for the same failure frames in the first watermark set and the second watermark set generated by the first other user and the second other user respectively when they played the same video in the past.

[0030] S122, calculate the number of failed frames where the consistency of the watermark information is greater than the consistency threshold, and calculate the union of the number of failed frames in the first set of failed frames and the second set of failed frames.

[0031] S123, calculate the ratio of the number of failed frames to the union of the numbers as the watermark information similarity.

[0032] Preferably, the watermark features on which the watermark information similarity calculation depends include the random encrypted watermark;

[0033] The neighboring video frames of the current video frame are: the previous and next historical video frames of the historical video frame that are the same as the current video frame, which are selected in step S1 and have the greatest feature similarity with the current user.

[0034] The maximum feature similarity relationship is the maximum value among one or more similarity fusion values ​​calculated for the same other user, including the user feature similarity, the failure frame information similarity, and the decoding feature similarity.

[0035] Preferably, the method for generating a watermark for each video frame in the first set using the first strategy includes the following steps:

[0036] A1, for each video frame in the first set, determine whether the texture complexity is greater than a preset complexity threshold;

[0037] If so, proceed to step A3;

[0038] If not, proceed to step A2;

[0039] A2, determine whether there is a sudden increase in the historical decoding latency of the video frame.

[0040] If so, it is determined that the playback device malfunctioned when the video frame was played in the past, and then the watermark information of the historical failed frame is directly assigned to the video frame that is the same frame as the historical failed frame.

[0041] If not, it is determined that abnormal network jitter occurred when the video frame was played in the past, and then proceed to step A3;

[0042] A3. Determine whether the historical failed frames in the set of failed frames selected in step S1, which are the same as the video frames in the first set, have been added with additional watermark features.

[0043] If so, then after removing the additional watermark features from the watermark information assigned to the historical failure frame, the remaining watermark features are assigned to the video frame.

[0044] If not, then after removing the random encrypted watermark from the watermark information assigned to the historical failed frame, the remaining watermark features are assigned to the video frame.

[0045] Preferably, the method for generating a watermark for each video frame in the second set using the second strategy includes the following steps:

[0046] B1, the set of successful historical frames for each watermark that has a feature similarity relationship with the current user is arranged into a frame set list according to the similarity fusion value from largest to smallest;

[0047] B2, extract the first watermark assigned to the successful historical frame set from the frame set list;

[0048] B3, extract each historical video frame that also exists in the second set of the filtered remaining from the extracted watermark successful historical frame set and add it to the second strategy watermark frame set, and then filter out each video frame that is the same as the extracted historical video frame from the second set of the filtered remaining.

[0049] B4, determine whether the elements in the second set are empty;

[0050] If so, the watermark information assigned to each of the historical video frames in the second strategy watermark frame set will be directly assigned to the video frame that was initially added to the second set and is the same as the historical video frame.

[0051] If not, return to step B2.

[0052] Preferably, the additional watermark features include the similarity of the failure frame information and / or the similarity of the decoding features associated with the historical failure frames;

[0053] In step S4, the method for generating the random encrypted watermark is as follows: after performing chaotic permutation on the watermark information of the current video frame and the neighboring video frames, the random encrypted watermark to be added to the encoded data of the current video frame is generated.

[0054] In step S5, the method for verifying the correctness of decoding of the current video frame transmitted to the current user includes the following steps:

[0055] S51, the random encrypted watermark added to the current video frame and the watermark information generated on the current video frame using the first strategy or the second strategy are used as the initial state for entropy decoding.

[0056] S52, For the current video frame, reconstruct the correlation between the frame quality and the initial state of the entropy decoding;

[0057] S53, determine whether the correlation is greater than a preset correlation threshold;

[0058] If so, the verification of the correctness of decoding the current video frame is considered successful;

[0059] If not, the verification of the correctness of decoding the current video frame is deemed to have failed.

[0060] This application has the following beneficial effects:

[0061] 1. Devices that have historically played the same video in various geographical locations are divided into several geographical tiers based on their distance from the server storing the video, from closest to furthest. Other users in the same geographical tier as the current user are used as the objects for calculating user feature similarity with the current user. Since the distance between devices in the same geographical tier and the server is similar, the network jitter caused by the transmission distance is small when transmitting the same video from the server to devices in the same geographical tier. This helps to ensure the consistency of the decoding delay distribution when decoding the same video by devices in the same geographical tier. As a result, the other users matched are more similar to the current user in terms of user features. Thus, the prediction of the set of failed watermarking frames that the current user may get when playing the video is more accurate through the filtering in steps S11-S13. This is conducive to adjusting the watermark generation method for the failed frames predicted for the current user in advance in step S2, improving the success rate and efficiency of watermarking, while ensuring the anti-counterfeiting effect.

[0062] 2. To increase the difference in watermark information for the same video played on devices within the same geographic region and improve anti-counterfeiting effectiveness, geographical similarity is represented by the straight-line distance between two users (devices) at the same geographic region. The greater the straight-line distance, the greater the geographical similarity between the two users, and vice versa. When attacking watermarks, watermark information added historically by neighboring devices to the same video may become an important basis for cracking the watermark. Therefore, this application avoids the above-mentioned risk by setting a greater straight-line distance between two devices at the same geographic region, which in turn indicates a greater geographical similarity between the two devices. Furthermore, based on this technology, when matching other users with similar user characteristics to the current user, the randomness of matching other users can be increased or decreased by setting a distance threshold between the current user and other users at the same geographic region. The set of failed frames for assigning watermarks to the current user, filtered through steps S11-S13, is more random while ensuring accuracy, thereby improving the anti-counterfeiting effect.

[0063] 3. By employing the second strategy, corresponding historical watermark information is directly assigned to each video frame in the second set, ensuring the efficiency of watermark assignment in high-concurrency real-time video encoding and decoding scenarios and improving the speed of encoding watermark information into video encoding data. The first strategy determines the reasons for failed historical watermark assignments to each video frame in the first set and adaptively adjusts the anti-counterfeiting level of the reassigned watermark, ensuring both efficiency and anti-counterfeiting effectiveness. When using the first and second strategies for watermark assignment, the introduction of a random encrypted watermark increases the anti-counterfeiting level and enhances video decoding security without modifying the watermark assignment algorithm or with minimal changes.

[0064] 4. The innovative technology of generating random encrypted watermarks lies in the fact that by relying on the watermarks selected in step S1 to be assigned to the set of failed frames and the set of historical frames, the neighboring video frames of the current video frame can be extracted. The watermark information of the neighboring video frames may include historical additional watermark features and / or historical random encrypted watermarks. The watermark information of the current video frame may also include historical additional watermark features and / or historical random encrypted watermarks. The newly generated random encrypted watermarks by chaotic permutation of the current video frame and the neighboring video frames are therefore more random, increasing the difficulty of watermark attacks.

[0065] 5. Steps S51-S53 verify the correctness of decoding the current video frame, further verifying whether the watermark features assigned to the current video frame can meet the efficiency and error rate requirements of real-time video encoding and decoding in high-concurrency scenarios, thus avoiding excessive sacrifice of decoding accuracy when pursuing the anti-counterfeiting effect of watermark in steps S1-S4. Attached Figure Description

[0066] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly described below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 This is a diagram illustrating the implementation steps of the watermarking method based on high-concurrency real-time video encoding and decoding provided in this embodiment;

[0068] Figure 2 This is an example diagram illustrating the geographical similarity between users at each geographic level. Detailed Implementation

[0069] The technical solution of this application will be further described below with reference to the accompanying drawings and specific embodiments.

[0070] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual images. They should not be construed as limiting the scope of this patent. To better illustrate the embodiments of this application, some components in the drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions of the product. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0071] In the accompanying drawings of the embodiments of this application, the same or similar reference numerals correspond to the same or similar components. In the description of this application, it should be understood that if terms such as "upper," "lower," "left," "right," "inner," and "outer" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting this patent. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0072] In the description of this application, unless otherwise expressly specified and limited, the term "connection" or similar designation indicating a connection between components should be interpreted broadly. For example, it can refer to a fixed connection, a detachable connection, or an integral part; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can refer to the internal communication between two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0073] The watermarking method based on high-concurrency real-time video encoding and decoding provided in this embodiment, such as... Figure 1 As shown, the steps include:

[0074] S1, for the current user who is to transmit video, select at least one watermark that has a feature similarity relationship with the current user and assign it to the set of failure frames; for example, if the video to be transmitted to the current user AU is video, then select at least one watermark that has a feature similarity relationship with the current user AU and assign it to the set of failure frames.

[0075] The feature similarity relationship with the current user AU includes any one or more of the following: user feature similarity, failure frame information similarity, and decoding feature similarity. User feature similarity is calculated as one or more similarities between the current user AU and other users OU in terms of device parameters, network jitter, and geographical location when playing the same video. When multiple similarities exist, the device parameter similarity, network jitter similarity, and geographical location similarity are weighted and summed after equalization to obtain the user feature similarity. Device parameters include one or more of the following: the type and performance parameters of the device playing the video, such as the type and performance parameters of the computer, or the type and performance parameters of the media player. Device parameter similarity is also preferably a weighted fusion result after equalization of the parameter values ​​for each type of device. Network jitter refers to the network jitter during video transmission, and geographical location refers to the geographical location of the device playing the video.

[0076] To increase the difference in watermark information added to the same video played on different devices within the same geographic level and improve the anti-counterfeiting level, preferably, the similarity of the geographical locations of the devices played by the current user AU and other users OU for the same video is defined as the straight-line distance between the current user AU and other users OU within the same geographic level, with the greater the straight-line distance, the greater the geographical similarity between the two users. For example, Figure 2 In this context, if the straight-line distance between the current user AU and the first other user OU1 is longer than the straight-line distance between the current user AU and the second other user OU2, then the geographical similarity between the current user AU and the first other user OU1 is greater than the geographical similarity between the current user AU and the second other user OU2.

[0077] In this embodiment, the method for dividing geographical levels is as follows:

[0078] Devices located in different geographical locations that have historically played the same video are divided into several geographical tiers based on their distance from the server remotely storing the video, from closest to furthest. For example... Figure 2 As shown, with the video storage server 100 as the center of the outward expansion of the geographic hierarchy, a geographic hierarchy with a width of 100 meters is constructed for every 100 meters of outward expansion (e.g., ...). Figure 2 The first, second, third, and fourth levels are shown in the diagram.

[0079] Dividing devices into geographical tiers can improve the success rate and efficiency of watermarking. Devices that have historically played the same video are divided into several geographical tiers based on their distance from the server storing the video, from closest to furthest. Other users within the same geographical tier as the current user are used to calculate user feature similarity. Because devices within the same geographical tier are similarly located from the server, the network jitter caused by transmission distance is smaller when transmitting the same video from the server to devices within the same geographical tier. This helps ensure the consistency of decoding latency distribution when decoding the same video on devices within the same geographical tier. Consequently, the matched other users have a higher degree of user feature similarity to the current user. Therefore, the prediction of the set of watermarking failure frames that might be obtained when the current user plays the video, achieved through the filtering in steps S11-S13, is more accurate. This allows for the pre-adjustment of the watermark generation method for the predicted failure frames in step S2, improving the success rate and efficiency of watermarking while ensuring anti-counterfeiting effectiveness.

[0080] Furthermore, when attacking watermarks, watermark information from neighboring devices under the same video historical conditions may become an important basis for cracking the watermark. Therefore, this application avoids the above-mentioned risks by setting the straight-line distance between two devices under the same geographical level to be greater and the geographical similarity between the two devices to be greater. Based on this technology, when matching other users with similar user characteristics to the current user, the randomness of matching other users can be increased or decreased by setting the distance threshold between the current user and other users under the same geographical level. The set of failed frames for watermarking the current user predicted by the filter in steps S11-S13 is more random while ensuring accuracy, thereby improving the anti-counterfeiting effect.

[0081] The similarity of failed frame information is defined as the similarity of watermark information among identical frames in the same video that failed to be watermarked when played by other users with similar user characteristics to the current user. For example, suppose that other users with similar user characteristics selected from the current user include OU1 and OU2. OU1 is assumed to have generated a first set of failed watermarking frames when playing the video in the past, and OU2 is assumed to have generated a second set of failed watermarking frames when playing the video in the past. The first set of failed watermarking frames is assumed to include failed frames f1-f5, and the second set is assumed to include failed frames f1, f3, f4, f6, f7, and f8. First, the consistency of the watermark information of identical failed frames in the first and second sets of failed watermarking frames is calculated, specifically the consistency of the watermark information of two failed frames f1, two failed frames f3, and two failed frames f4. Then, the consistency of each calculated watermark information is compared with a preset consistency threshold, and the number of failed frames with a watermark information consistency greater than the consistency threshold is calculated. Assuming the watermark information consistency of two failed frames f1 and two failed frames f3 is greater than the consistency threshold, while the watermark information consistency of two failed frames f4 is less than the consistency threshold, the calculated number of failed frames with watermark information consistency greater than the consistency threshold is 2. Then, the union of the number of failed frames assigned to the first and second watermark sets is calculated. For example, in the above example, the union of the first and second watermark sets is f1-f8, so the union of the number of failed frames is 8. Finally, the ratio of the number of failed frames to the union of the number of failed frames is calculated as the watermark information similarity between the first and second watermark sets.

[0082] It's important to note that the consistency of watermark information is related to the method used to assign watermarks to the same failed frames. For example, if the device assigns watermark to a failed frame f1 played by another user OU1 using the following method: watermark feature a is assigned to the first region of failed frame f1, watermark feature b is assigned to the second region, and watermark feature c is assigned to the third region; if the device assigns watermark to a failed frame f1 played by another user OU2 using the following method: watermark feature a is assigned to the first region of failed frame f1, watermark feature c is assigned to the second region, and watermark feature b is assigned to the third region, then the consistency between the first watermark information assigned to failed frame f1 when the device plays the video for another user OU1 and the second watermark information assigned to failed frame f1 when the device plays the same video for another user OU2 is one-third. That is, the ratio of the number of identical regions assigned the same watermark feature to the total number of regions where watermark information is assigned in the failed frame.

[0083] The above method for calculating the consistency of watermark information is just one example; there are many other methods, which will not be elaborated on here.

[0084] To enhance the anti-counterfeiting effect of the video, preferably, the watermark features relied upon for watermark information similarity calculation include randomly encrypted watermarks. The generation of the randomly encrypted watermark depends on the neighboring video frames of the current video frame. Here, "neighborhood" refers to the previous and next historical video frames of the same historical video frame as the current video frame. Specifically, when adding a randomly encrypted watermark to the current video frame, firstly, a set of historical frames with the greatest feature similarity to the current user is selected for watermark assignment. This set of historical frames consists of video frames where other users with the greatest feature similarity to the current user have watermarked each frame during video encoding. Then, historical current video frames that are the same as the current video frame to be transmitted to the current user are extracted from this set of historical frames. Furthermore, the previous and next frames of these historical current video frames are extracted as the neighboring video frames of the current video frame to be transmitted to the current user. The set of historical frames with the greatest feature similarity to the current user includes two sets: one where watermark assignment to the same video historical segment was successful, and the other where it failed.

[0085] In this embodiment, the numerical value of the feature similarity relationship in step S1 is represented by a similarity fusion value of one or more of the following: user feature similarity, failure frame information similarity, and decoding feature similarity between other users and the current user. When the feature similarity relationship includes user feature similarity, failure frame information similarity, and decoding feature similarity, the similarity fusion value is a weighted sum of user feature similarity, failure frame information similarity, and decoding feature similarity.

[0086] The method for generating random encrypted watermarks will be explained in later sections and will not be discussed here.

[0087] In this embodiment, the decoding feature similarity is: the similarity of the decoding delay distribution and / or the dependence strength on the random encrypted watermark when other users with similar user features to the current user decode the same video frames in their history.

[0088] The preferred decoding delay distribution in the decoding feature similarity calculation is the average decoding delay of different users decoding the same video frame. The method for calculating the dependence strength of the random encrypted watermark is as follows: In steps S51-S53 below, if the absolute value of the difference between the correlation calculated using the random encrypted watermark added to the current video frame and the watermark information generated for the current video frame using the first strategy or the second strategy as the initial state of entropy decoding (defined as the first correlation) and the correlation calculated using only the watermark information generated for the current video frame using the first strategy or the second strategy as the initial state of entropy decoding (defined as the second correlation) is less than a preset absolute value threshold, then the dependence strength of the random encrypted watermark when decoding the current video frame is assigned as "0", otherwise it is "1".

[0089] In this embodiment, the decoding feature similarity is preferably a weighted sum of the similarity of the decoding delay distribution and the similarity of the dependence strength on the random encrypted watermark.

[0090] In summary, step S1, the method of selecting at least one watermark that has a feature similarity relationship with the current user and assigning it to the set of failed frames, includes the following steps:

[0091] S11, filter out other users who have similar user characteristics to the current user. When the similarity of user characteristics between the current user and other users is greater than the preset first similarity threshold, it is determined that the current user has similar user characteristics to other users.

[0092] S12, calculate the watermark information similarity of the same frames in the past when watermarking failed to be assigned when other users with similar user characteristics played the same video with the current user, and filter out other users whose watermark information similarity is less than the second similarity threshold.

[0093] S13, calculate the decoding feature similarity of the other users remaining after filtering in step S12 when decoding the same video frames in the past, and then extract the set of failed watermark assignments associated with each other user whose decoding feature similarity is greater than the third similarity threshold as the filtering result of the failed watermark assignment set.

[0094] It should be noted that the decoding feature similarity calculation applies to two sets of historical watermarked frames, such as the first set and the second set. The difference between the set of historical watermarked frames and the set of failed watermarking frames is that one set includes all video frames in the same video after watermarking, while the other set includes video frames in the same video where the initial watermarking attempt failed. The calculation principle for decoding feature similarity is the same as that for watermark information similarity, using a ratio to represent decoding feature similarity. The calculation principle for the consistency of decoded information is the same as that for watermark information consistency. However, since the calculation process for the consistency of decoded information is not within the scope of protection claimed in this application and does not affect the technical effect achieved by the technical solution provided in this application, nor does it solve the corresponding technical problem, it will not be elaborated further.

[0095] The method for calculating the similarity of watermark information provided in this embodiment is briefly described again here, including the following steps:

[0096] S121, calculate the watermark information consistency for the same failed frames in the first watermark set and the second watermark set generated by the first other user and the second other user respectively when they played the same video in the past; how the watermark information consistency is calculated has been exemplified in the above content and will not be repeated here.

[0097] S122, calculate the number of failed frames where the watermark information consistency is greater than the consistency threshold, and calculate the union of the number of failed frames in the first watermark set and the second watermark set; the meaning of the union of the number has been explained above and will not be repeated here.

[0098] S123, calculate the ratio of the number of failed frames to the union of the number of failed frames as the watermark information similarity.

[0099] After step S1, the watermark selected by the current user is assigned to a set of failed frames, as follows: Figure 1 As shown, the watermarking method based on high-concurrency real-time video encoding and decoding provided in this embodiment proceeds to the following steps:

[0100] S2, perform frame splitting on the video to be transmitted to the current user, that is, split the video into each frame. The frame splitting method is the same for the same video to be transmitted to different users to ensure that the type and number of split frames are the same. Then, determine whether each split video frame is added to the set of failed watermark assignment frames selected in step S1.

[0101] If so, add the currently determined video frame to the first set;

[0102] If not, add the currently determined video frame to the second set;

[0103] For example, after splitting the video (video) intended to be transmitted to the current user into frames, suppose we get 100 video frames (f1-f100) arranged consecutively in playback order. Suppose this video was previously transmitted to 100 other users (OU1-OU100), and before being transmitted to these other users, the video was split into these 100 frames (f1-f100).

[0104] Suppose that the set of failed watermarking frames selected in step S1 includes two sets: a first set of failed watermarking frames and a second set of failed watermarking frames. These sets were generated during the most recent playback of the same video by the first "other user" and the second "other user," respectively. Here, "last playback" refers to the playback of the video that resulted in a failed watermarking frame and is most recent in time.

[0105] Assume that the first watermarking failure frame set selected in step S1 includes 5 video frames (f10, f18, f21, f33, f41), and the second watermarking failure frame set includes 4 video frames (f21, f29, f41, f78). Then, add the 7 video frames (f10, f18, f21, f29, f33, f41, f78) to the first set; add the remaining 93 video frames (excluding these 7) from the 100 frames to the second set.

[0106] After obtaining the first set and the second set to be watermarked respectively in step S2 for the video to be transmitted to the current user, the watermarking method based on high-concurrency real-time video encoding and decoding provided in this embodiment is as follows: Figure 1 As shown, proceed to the following steps:

[0107] S3, for each video frame in the first set and the second set, generate an invisible watermark using the first strategy and the second strategy respectively;

[0108] The method for generating a watermark for each video frame in the first set using the first strategy includes the following steps:

[0109] A1, for each video frame in the first set, determine whether the texture complexity is greater than the preset complexity threshold;

[0110] If so, proceed to step A3;

[0111] If not, proceed to step A2;

[0112] There are many existing methods for identifying the texture of video frames and calculating texture complexity, which will not be discussed in detail here.

[0113] A2, determine if there is a sudden increase in the historical decoding latency of video frames;

[0114] If so, it is determined that the playback device was faulty when the video frame was played in the past, and then the watermark information of the historical failed frame is directly assigned to the video frame that is the same frame as the historical failed frame.

[0115] If not, it is determined that network jitter occurred when the video frame was played in the past, and then proceed to step A3;

[0116] The method for determining whether there is a sudden increase in historical decoding latency for video frames in the first set is briefly described below:

[0117] Assuming that video frame f1 in the first set also exists in the first and second watermark-application failure frame sets selected in step S1, and assuming that the first other user associated with the first watermark-application failure frame set has a more similar feature relationship to the current user than the second other user associated with the second watermark-application failure frame set, then extract the historical failure frame f1' that is the same frame as video frame f1 from the first watermark-application failure frame set. Then, extract the preceding and following historical failure frames of historical failure frame f1', and calculate the preceding historical failure frame... The absolute value of the difference between the historical decoding delay corresponding to the next historical failure frame and the previous historical failure frame is defined as the first absolute value of the difference. The absolute value of the difference between the historical failure frame f1' and the previous historical failure frame is defined as the second absolute value of the difference. Then, it is determined whether the first absolute value of the difference is less than the preset first absolute value threshold and the second absolute value of the difference is greater than the preset second absolute value threshold. If so, it is determined that the historical failure frame f1' has experienced a sudden increase in decoding delay, that is, it is determined that the video frame in the first set that is the same as the historical failure frame f1' has experienced a sudden increase in historical decoding delay.

[0118] Under the "if" condition in step A2, the method of assigning watermark information to the video frames in the first set is the same as the method of assigning it to the historical failed frame f1', thereby saving time in assigning watermarks to the video frames.

[0119] A3. Determine whether the historical failed frames in the set of failed frames selected in step S1 that are the same as the video frames in the first set have been added with additional watermark features.

[0120] If so, remove the additional watermark features from the watermark information assigned to the historical failed frame, and then assign the remaining watermark features to the video frame.

[0121] If not, then after removing the random encrypted watermark (if any) from the watermark information assigned to the historical failed frame, the remaining watermark features are assigned to the failed frame.

[0122] In this embodiment, the additional watermark features include the similarity of failure frame information and / or decoding feature similarity associated with historical failure frames. The calculation methods for failure frame information similarity and decoding feature similarity have been specifically described above and will not be repeated here. Adding additional watermark features to the invisible watermark generated by the conventional watermarking method can enhance the watermark anti-counterfeiting effect. Users do not need to make technical improvements to the conventional watermarking algorithm, and the generation of additional watermark features integrates the common features of watermark information from other users with similar user characteristics to the current user. However, by defining geographical location similarity in reverse, it increases the difference in additional watermark features generated when other users with similar device parameters and network jitter play videos, thus making the generation of additional watermark features more efficient yet with better anti-counterfeiting effects. Furthermore, by analyzing the reasons for the failure to watermark historical failure frames in step A3, and by removing additional watermark features and / or randomly encrypting watermarks, the anti-counterfeiting level of the watermark on video frames that are the same as the historical failure frames is reduced, improving the success rate of decoding watermark information in video frames.

[0123] The method for generating a watermark for each video frame in the second set using the second strategy includes the following steps:

[0124] B1, assign a set of successful historical frames to each watermark that has a feature similarity relationship with the current user, and arrange them into a frame set list according to the similarity fusion value from largest to smallest;

[0125] Assume that the users with similar characteristics to the current user selected in step S1 include two other users, OU1 and OU2. Let's assume that the sets of successfully watermarked historical frames associated with OU1 and OU2 are respectively the first and second sets of successfully watermarked historical frames, and the set of successfully watermarked historical frames is the set of historical video frames that were successfully watermarked after video splitting. The similarity fusion value is a weighted sum of one or more of the following: the similarity of user characteristics with similar characteristics to the current user, the similarity of failed frame information, and the similarity of decoding characteristics.

[0126] B2, extract the first watermark from the frame set list and assign it to the successful historical frame set;

[0127] Assuming the frame set list is arranged in descending order of similarity fusion value, there are three sets of historical frames with successful first watermarking, successful second watermarking, and successful third watermarking. In step B2, the set of historical frames with successful first watermarking is extracted.

[0128] B3, extract each historical video frame that also exists in the second set of the filtered remaining from the extracted watermarked historical frame set and add it to the second strategy watermarked frame set. Then, filter out each video frame that is the same as the extracted historical video frame from the second set of the filtered remaining.

[0129] For example, the second set includes video frames f1-f5. Assuming the first watermark-acquired historical frame set extracted in step B2 includes historical video frames f1'-f3' and f4'-f100', and f1'-f3' are the same video frames as f1-f3, then the historical video frames f1'-f3' from the first watermark-acquired historical frame set are added to the second strategy watermarking frame set. Then, f1-f3 are filtered out from the second set, leaving only video frames f4-f5 in the second set.

[0130] B4, determine whether the elements in the second set are empty;

[0131] If so, the watermark information assigned to each historical video frame in the second strategy watermark frame set will be directly assigned to the video frame that was initially added to the second set and is the same as the historical video frame.

[0132] If not, return to step B2.

[0133] Continuing with the example above, since there are still two video frames f4-f5 left in the second set, the "if not" condition in step B4 is met, and we return to step B2 to extract the second watermark that is currently ranked first and assign it to the successful historical frame set.

[0134] Assuming that the elements in the second set are empty at this time, the historical watermark information of each historical video frame in f1'-f5' of the second strategy watermark frame set will be directly assigned to video frames f1-f5 in the second set.

[0135] After generating watermark information for the video frames in the first and second sets in step S3, as follows: Figure 1 As shown, the watermarking method based on high-concurrency real-time video encoding and decoding provided in this embodiment proceeds to the following steps:

[0136] S4. Extract the neighboring video frames of the current video frame to be transmitted, and perform random watermark swap encryption to obtain a random encrypted watermark of the current video frame, which is then added to the encoded data of the current video frame.

[0137] In this embodiment, the method for generating a random encrypted watermark is as follows: after performing a chaotic permutation between the watermark information of the current video frame and the neighboring video frames, a random encrypted watermark to be added to the encoded data of the current video frame is generated.

[0138] In this embodiment, the innovative technology for generating random encrypted watermarks lies in the fact that the watermarks selected in step S1 can be applied to the set of failed frames and the set of historical frames to extract the neighboring video frames of the current video frame. The watermark information of the neighboring video frames may include historical additional watermark features and / or historical random encrypted watermarks. Similarly, the watermark information of the current video frame may also include historical additional watermark features and / or historical random encrypted watermarks. The random encrypted watermarks newly generated by chaotic permutation of the current video frame and the neighboring video frames are therefore more random, increasing the difficulty of watermark attacks.

[0139] It is important to emphasize that the neighboring video frames of the current video frame are not the previous and next video frames, but rather the previous and next historical video frames played by other users who have the greatest feature similarity to the current user and are playing historical video frames that are the same as the current video frame.

[0140] The existing method is to generate a random password for the current frame based on the encrypted data of the previous and next frames through chaotic permutation. The process of generating a random encrypted watermark can refer to the random password generation process through chaotic permutation, and will not be explained in detail here.

[0141] After generating a random encrypted watermark for the current video frame in step S4, the watermarking method based on high-concurrency real-time video encoding and decoding provided in this embodiment is as follows: Figure 1 As shown, proceed to the following steps:

[0142] S5, verify the correctness of decoding of the current video frame transmitted to the current user based on the random encrypted watermark;

[0143] If the verification is successful, the decoded current video frame will be played to the current user.

[0144] If verification fails, a video playback error alert will be issued to the current user, and the current video frame will be added to the set of failed watermark assignment frames associated with the current user.

[0145] In this embodiment, the method for verifying the correctness of decoding the current video frame transmitted to the current user includes the following steps:

[0146] S51, the random encrypted watermark added to the current video frame and the watermark information generated on the current video frame using the first strategy or the second strategy are used as the initial state for entropy decoding.

[0147] S52, the correlation between the reconstructed frame quality of the current video frame and the initial state of entropy decoding, the reconstruction method is as follows:

[0148] Assuming the frame quality of the current video frame is Qt, the random encrypted watermark generated for the current video frame is m1t, and the watermark information generated for the current video frame using either the first or second strategy is m2t, then the correlation st between frame quality and the initial state of entropy decoding can be expressed as: st = F(Qt, (m1t, m2t)), where F is the correlation function between frame quality and the initial state of entropy decoding. There are many correlation functions that express the relationship between frame quality and the entropy decoding state, which will not be detailed here.

[0149] S53, determine whether the correlation is greater than the preset correlation threshold;

[0150] If so, the verification of the correctness of decoding the current video frame is considered successful;

[0151] If not, the verification of the correctness of decoding the current video frame is deemed to have failed.

[0152] Steps S51-S53 verify the correctness of decoding the current video frame, further verifying whether the watermark features assigned to the current video frame can meet the efficiency and error rate requirements of real-time video encoding and decoding in high-concurrency scenarios, thus avoiding excessive sacrifice of decoding accuracy when pursuing watermark anti-counterfeiting effects in steps S1-S4.

[0153] It should be stated that the above-described specific embodiments are merely preferred embodiments and technical principles applied in this application. Those skilled in the art should understand that various modifications, equivalent substitutions, and variations can be made to this application. However, such variations, as long as they do not depart from the spirit of this application, should be within the scope of protection of this application. Furthermore, some terminology used in this application's specification and claims is not limiting but merely for ease of description.

Claims

1. A watermarking method based on high-concurrency real-time video encoding and decoding, characterized in that, Including the following steps: S1, for the current user who intends to transmit video, filter out at least one watermark that has a similar feature relationship with the current user and assign it to the set of failed frames; S2, perform frame splitting on the video to be transmitted to the current user, and determine whether each split video frame is added to the filtered set of failed watermark assignment frames; If so, the currently determined video frame is added to the first set; If not, the currently determined video frame is added to the second set; S3, for each video frame in the first set and the second set, an invisible watermark is generated using the first strategy and the second strategy, respectively; S4, extract the neighboring video frames of the current video frame to be transmitted, and perform random watermark swap encryption to obtain a random encrypted watermark of the current video frame, which is then added to the encoded data of the current video frame. S5, verify the correctness of decoding of the current video frame transmitted to the current user based on the random encrypted watermark; If the verification is successful, the decoded current video frame will be played to the current user. If the verification fails, a video playback error alarm will be issued to the current user, and the current video frame will be added to the set of failed watermark assignment frames associated with the current user. The feature similarity relationship between other users and the current user includes any one or more of the following: user feature similarity, failure frame information similarity, and decoding feature similarity. The user feature similarity includes one or more similarities between the current user and other users when playing the same video, such as device parameters, network jitter, and geographical location. The similarity of the failed frame information is the similarity of the watermark information of the same frames in the history of the same video played by each of the other users who have similar user characteristics to the current user and failed to assign watermarks. The decoding feature similarity is the similarity of the decoding delay distribution and / or the dependence strength on the random encrypted watermark when other users with the same user feature similarity as the current user decode the same video frame in their history of playing the same video.

2. The watermarking method based on high-concurrency real-time video encoding and decoding according to claim 1, characterized in that, In step S1, the method for selecting at least one set of failed frames for watermarking that has a feature similarity relationship with the current user includes the following steps: S11, filter out other users who have similar user characteristics to the current user. When the similarity of user characteristics between the current user and the other users is greater than a preset first similarity threshold, determine that the current user and the other users have similar user characteristics. S12, calculate the watermark information similarity of the same frames in the past when the watermark assignment failed to be applied during the playback of the same video by other users who have similar user characteristics to the current user, and filter out the other users whose watermark information similarity is less than the second similarity threshold. S13, calculate the decoding feature similarity for each pair of the other users who have played the same video in history and decoded the same video frame, and then extract the set of watermark-assigned failed frames associated with each other user whose decoding feature similarity is greater than the third similarity threshold as the filtering result of the watermark-assigned failed frame set.

3. The watermarking method based on high-concurrency real-time video encoding and decoding according to claim 1 or 2, characterized in that, The user feature similarity is a weighted sum of the similarity of device parameters, network jitter, and geographical location between the current user and the other users when playing the same video. Geographical location similarity is represented by the straight-line distance between two users at the same geographic level. The greater the straight-line distance, the greater the geographical location similarity between the two users. The decoding feature similarity is a weighted sum of the decoding delay distribution and the dependence strength on the random encrypted watermark when two other users decode the same video frame in their history of playing the same video.

4. The watermarking method based on high-concurrency real-time video encoding and decoding according to claim 3, characterized in that, Devices that have historically played the same video in different geographical locations are divided into several geographical tiers based on their distance from the server that remotely stores the video, from closest to furthest.

5. The watermarking method based on high-concurrency real-time video encoding and decoding according to claim 2, characterized in that, In step S12, the method for calculating the similarity of the watermark information includes the following steps: S121, calculate the consistency of watermark information for the same failure frames in the first watermark set and the second watermark set generated by the first other user and the second other user respectively when they played the same video in the past. S122, calculate the number of failed frames where the consistency of the watermark information is greater than the consistency threshold, and calculate the union of the number of failed frames in the first set of failed frames and the second set of failed frames. S123, calculate the ratio of the number of failed frames to the union of the numbers as the watermark information similarity.

6. The watermarking method based on high-concurrency real-time video encoding and decoding according to claim 2, characterized in that, The watermark features on which the watermark information similarity calculation depends include the random encrypted watermark; The neighboring video frames of the current video frame are: the previous and next historical video frames of the historical video frame that are the same as the current video frame, which are selected in step S1 and have the greatest feature similarity with the current user. The maximum feature similarity relationship is the maximum value among one or more similarity fusion values ​​calculated for the same other user, including the user feature similarity, the failure frame information similarity, and the decoding feature similarity.

7. The watermarking method based on high-concurrency real-time video encoding and decoding according to claim 6, characterized in that, The method for generating a watermark for each video frame in the first set using the first strategy includes the following steps: A1, for each video frame in the first set, determine whether the texture complexity is greater than a preset complexity threshold; If so, proceed to step A3; If not, proceed to step A2; A2, determine whether the historical decoding delay of the video frame has suddenly increased; If so, it is determined that the playback device malfunctioned when the video frame was played in the past, and then the watermark information of the historical failed frame is directly assigned to the video frame that is the same frame as the historical failed frame. If not, it is determined that abnormal network jitter occurred when the video frame was played in the past, and then proceed to step A3; A3. Determine whether the historical failed frames in the set of failed frames selected in step S1, which are the same as the video frames in the first set, have been added with additional watermark features. If so, then after removing the additional watermark features from the watermark information assigned to the historical failure frame, the remaining watermark features are assigned to the video frame. If not, then after removing the random encrypted watermark from the watermark information assigned to the historical failed frame, the remaining watermark features are assigned to the video frame.

8. The watermarking method based on high-concurrency real-time video encoding and decoding according to claim 6, characterized in that, The method for generating a watermark for each video frame in the second set using the second strategy includes the following steps: B1, assign a set of successful historical frames to each watermark that has a feature similarity relationship with the current user, and arrange them into a frame set list according to the similarity fusion value from largest to smallest; B2, extract the first watermark assigned to the successful historical frame set from the frame set list; B3, extract each historical video frame that also exists in the second set of the filtered remaining from the extracted watermark successful historical frame set and add it to the second strategy watermark frame set, and then filter out each video frame that is the same as the extracted historical video frame from the second set of the filtered remaining. B4, determine whether the elements in the second set are empty; If so, the watermark information assigned to each of the historical video frames in the second strategy watermark frame set will be directly assigned to the video frame that was initially added to the second set and is the same as the historical video frame. If not, return to step B2.

9. The watermarking method based on high-concurrency real-time video encoding and decoding according to claim 7, characterized in that, The additional watermark features include the similarity of the failure frame information associated with the historical failure frames and / or the similarity of the decoding features; In step S4, the method for obtaining the random encrypted watermark is as follows: after performing chaotic permutation on the watermark information of the current video frame and the neighboring video frames, the random encrypted watermark to be added to the encoded data of the current video frame is generated. In step S5, the method for verifying the correctness of decoding of the current video frame transmitted to the current user includes the following steps: S51, the random encrypted watermark added to the current video frame and the invisible watermark generated on the current video frame using the first strategy or the second strategy are used as the initial state for entropy decoding. S52, For the current video frame, reconstruct the correlation between the frame quality and the initial state of the entropy decoding; S53, determine whether the correlation is greater than a preset correlation threshold; If so, the verification of the correctness of decoding the current video frame is considered successful; If not, the verification of the correctness of decoding the current video frame is deemed to have failed.

Images