Watermark endowing method based on video high-concurrency real-time coding and decoding

Abstract

The invention discloses a watermark endowing method based on video high-concurrency real-time coding and decoding, and the method comprises the steps: directly endowing each video frame in a second set with corresponding historical watermark information through a second strategy, and guaranteeing the watermark endowing efficiency in a video high-concurrency real-time coding and decoding scene. The speed of encoding the watermark information into the video encoding data is increased; according to the method and the device, the reason for historical watermark endowing failure of each video frame in the first set is judged through the first strategy, and the anti-counterfeiting level of watermark endowing is adaptively adjusted, so that the watermark endowing efficiency is considered, and meanwhile, the anti-counterfeiting effect of watermark endowing is ensured. By introducing the random encryption watermark, the watermark anti-counterfeiting level is increased and the video decoding security is improved on the premise of not modifying the watermark endowing algorithm. By verifying the decoding correctness, the provided watermark endowing method meets the requirements for the real-time performance and error rate of video coding and decoding in a high-concurrency scene, and the decoding accuracy is prevented from being sacrificed when the anti-counterfeiting effect is pursued.

Description

Technical Field

[0001] The present application relates to the technical field of video data processing, and in particular to a watermarking method based on high-concurrency real-time encoding and decoding of videos. Background Art

[0002] High video concurrency refers to scenarios where a large number of users simultaneously access or request video content, placing significant pressure on servers, network bandwidth, and system architecture. The primary purpose of adding watermarks to videos is to prevent counterfeiting. After the generated watermarks are added to the video encoding data, the authenticity of the video can be verified when it is decoded and played. Anti-counterfeiting watermarks are typically invisible. While there are many existing methods for generating anti-counterfeiting watermarks, current watermarking technologies for high-concurrency videos have the following technical drawbacks when addressing large-scale real-time processing requirements: 1. To ensure anti-counterfeiting effectiveness, some high-concurrency video application scenarios, such as videos transmitted on social platforms, are generally highly compressed. When the quantization parameters of existing watermarks increase, the watermark information of the mid- and high-frequency coefficients is destroyed due to quantization noise, and the failure rate of watermark information extraction increases significantly. In other words, excessive video compression and re-editing may make the watermark impossible to recover. Therefore, when highly compressing videos, how to avoid excessive increases in the quantization parameters of the watermark while meeting the anti-counterfeiting level requirements for the video and reducing the failure rate of watermark information extraction is a technical problem that needs to be urgently solved in this field.

[0003] 2. Overly complex watermark addition and extraction algorithms cannot meet the real-time requirements of high-concurrency video. However, watermarks generated using simple algorithms are even more vulnerable to targeted attacks, such as adversarial sample perturbations (adding noise invisible to the human eye) or collusion attacks (merging multiple watermarked copies), which can significantly reduce the watermark signal-to-noise ratio. Therefore, approximately 90% of watermarks generated using existing high-concurrency video watermarking technologies can be removed using open-source tools. Therefore, the field needs a high-concurrency video watermark generation technology that can not only meet the real-time requirements of high-concurrency video but also have strong defense capabilities against attacks. Summary of the Invention This application aims to avoid excessive increase in the quantization parameter of the watermark when performing high-intensity compression on high-concurrency videos, meet the real-time requirements of high-concurrency videos, reduce the failure rate of watermark information extraction, and ensure that the watermark information added to the video encoding data has a high-intensity defense capability against attacks, thereby ensuring the security of high-concurrency video transmission. A watermark assignment method based on high-concurrency real-time encoding and decoding of videos is provided.

[0004] To achieve this goal, this application adopts the following technical solutions: A watermarking method based on high-concurrency real-time encoding and decoding of video is provided, comprising the steps of: S1, for a current user to be used for video transmission, screening out at least one watermarking failure frame set having features similar to those of the current user; S2, splitting the video to be transmitted to the current user into frames, and determining whether each split video frame is added to the filtered set of frames that failed to be watermarked; If yes, then adding the currently determined video frame to the first set; If not, adding the currently determined video frame to the second set; S3, for each of the video frames in the first set and the second set, generating an invisible watermark using a first strategy and a second strategy respectively; S4, extracting neighboring video frames of the current video frame to be transmitted, performing random watermark exchange encryption to obtain a random encrypted watermark of the current video frame, and adding the watermark to the encoded data of the current video frame; S5, verifying decoding correctness of the current video frame transmitted to the current user according to the random encrypted watermark; If the verification is successful, the decoded current video frame is played to the current user; If the verification fails, a video playback abnormality alarm is issued to the current user, and the current video frame is added to the watermark assignment failure frame set associated with the current user.

[0005] Preferably, the feature similarity relationship between other users and the current user includes any one or more of user feature similarity, failed frame information similarity, and decoding feature similarity; The user feature similarity includes one or more similarities of device parameters, network jitter, and geographic location between the current user and the other users playing the same video; The failed frame information similarity is the watermark information similarity of the same frame where watermarking failed when playing the same video in pairs of the other users who have user feature similarity with the current user; The decoding feature similarity is the similarity of decoding delay distribution and / or dependence strength on the random encrypted watermark when decoding the same video frame when each of the other users having the user feature similarity with the current user historically plays the same video.

[0006] Preferably, in step S1, the method for screening out at least one watermarking failure frame set having a feature similarity relationship with the current user comprises the steps of: S11, screening out other users having user feature similarity with the current user, and when the user feature similarity between the current user and the other user is greater than a preset first similarity threshold, determining that the current user and the other user have the user feature similarity relationship; S12, calculating the watermark information similarity of the same frames of the video that failed to be watermarked when the other users who have similar user characteristics to the current user played the same video in history, and filtering out the other users whose watermark information similarity is less than a second similarity threshold; S13, calculating the decoding feature similarity of each of the other users remaining after filtering in step S12 when decoding the same video frame during historical playback of the same video, and then extracting the watermark failure frame set associated with each of the other users whose decoding feature similarity is greater than a third similarity threshold as the screening result of the watermark failure frame set.

[0007] Preferably, the user feature similarity is a weighted sum of the similarities of device parameters, network jitter, and geographic location between the current user and the other users playing the same video. Geographic location similarity is represented by the straight-line distance between two users at the same geographical level. The greater the straight-line distance, the greater the geographic location similarity between the two users. The decoding feature similarity is a weighted sum of the decoding delay distribution and the dependence strength on the random encrypted watermark when two other users decode the same video frame when playing the same video in the past.

[0008] Preferably, devices at various geographical locations that have historically played the same video are divided into several geographical levels from near to far based on their distance from a server that remotely stores the video.

[0009] Preferably, in step S12, the method for calculating the watermark information similarity comprises the steps of: S121, calculating watermark information consistency for identical failed frames in a first watermarking failure frame set and a second watermarking failure frame set generated by historical playback of the same video by the first other user and the second other user, respectively; S122, calculating the number of failed frames whose watermark information consistency is greater than a consistency threshold, and calculating the union of the number of failed frames in the first watermark assignment failure frame set and the second watermark assignment failure frame set; S123: Calculate the ratio of the number of failed frames to the union of the numbers as the watermark information similarity.

[0010] Preferably, the watermark feature on which the watermark information similarity calculation depends includes the random encrypted watermark; The neighborhood video frames of the current video frame are: the previous historical video frame and the next historical video frame of the historical video frame that is the same frame as the current video frame, in the watermarked historical frame set that has the greatest feature similarity with the current user selected in step S1; The maximum feature similarity relationship is: the maximum value of one or more similarity fusion values ​​of the user feature similarity, the failed frame information similarity, and the decoding feature similarity calculated for the same other user.

[0011] Preferably, the method for generating a watermark for each of the video frames in the first set using the first strategy comprises the steps of: A1, for each of the video frames in the first set, determining whether the texture complexity is greater than a preset complexity threshold; If yes, jump to step A3; If not, go to step A2; A2, determining whether the historical decoding delay of the video frame increases suddenly, If so, it is determined that the playback device failed when the video frame was played in the past, and then the watermark information of the historical failed frame is directly assigned to the video frame that is the same frame as the historical failed frame; If not, it is determined that abnormal network jitter occurred when the video frame was played historically, and then the process goes to step A3; A3, determining whether an additional watermark feature is added to a historical failure frame in the watermarking failure frame set screened in step S1 that is the same as the video frame in the first set; If yes, after removing the additional watermark feature from the watermark information assigned to the historical failed frame, assign the remaining watermark feature to the video frame; If not, after removing the random encrypted watermark in the watermark information assigned to the historical failed frame, the remaining watermark features are assigned to the video frame.

[0012] Preferably, the method for generating a watermark for each of the video frames in the second set using the second strategy comprises the steps of: B1, arranging the historical frame sets for which each watermark has been successfully assigned and has a feature similarity relationship with the current user into a frame set list according to the similarity fusion value from large to small; B2, extracting the first historical frame set with successful watermarking from the frame set list; B3, extracting from the extracted watermarked historical frame set each historical video frame that is also present in the second filtered remaining set and adding the frame to the second strategic watermarked frame set, and then filtering out from the second filtered remaining set each video frame that is the same frame as each extracted historical video frame; B4, determining whether the element in the second set is empty; If so, directly assigning the watermark information assigned to each of the historical video frames in the second strategy watermarked frame set to the video frame initially added to the second set that is the same frame as the historical video frame; If not, return to step B2.

[0013] Preferably, the additional watermark feature includes the similarity of the failed frame information and / or the similarity of the decoding features associated with the historical failed frames; In step S4, the method for generating the random encrypted watermark is: performing chaotic permutation on the watermark information of the current video frame and the neighboring video frame to generate the random encrypted watermark to be added to the encoded data of the current video frame; In step S5, the method for verifying the decoding correctness of the current video frame transmitted to the current user includes the steps of: S51, using the randomly encrypted watermark added to the current video frame and the watermark information generated for the current video frame using the first strategy or the second strategy as an initial state of entropy decoding; S52, reconstructing, for the current video frame, a correlation between frame quality and an initial state of entropy decoding; S53, determining whether the correlation is greater than a preset correlation threshold; If yes, it is determined that the decoding correctness verification of the current video frame is successful; If not, it is determined that the decoding correctness verification of the current video frame fails.

[0014] This application has the following beneficial effects: 1. Devices in various geographical locations that have historically played the same video are divided into several regional levels based on their distance from the server that remotely stores the video. Other users in the same regional level as the current user are used as objects for calculating user feature similarity with the current user. Since the distances between devices in the same regional level and the server are similar, when the same video is transmitted from the server to devices in the same regional level, the difference in network jitter caused by the transmission distance is small, which is conducive to ensuring the consistency of the decoding delay distribution when devices in the same regional level decode the same video. The other users matched thereby have higher user feature similarity with the current user. In this way, the prediction of the set of failed watermarking frames that may be obtained when the current user plays the video is made more accurate through the screening of steps S11-S13, which is conducive to adjusting the watermark generation method for the failed frames predicted for the current user in advance in step S2, thereby improving the success rate and efficiency of watermarking, and ensuring the anti-counterfeiting effect.

[0015] 2. In order to increase the difference in watermark information of the same video condition played by each device at the same geographical level and improve the anti-counterfeiting effect, the geographical location similarity is characterized by the straight-line distance between two users (devices) at the same geographical level. The greater the straight-line distance, the greater the geographical location similarity of the two users, and vice versa. When attacking the watermark, the watermark information added to the same video history by neighboring devices may become an important basis for cracking the watermark. Therefore, this application circumvents the above risks by setting the greater the straight-line distance between two devices at the same geographical level, the greater the geographical location similarity of the two devices. And based on this technology, when matching other users with user feature similarity to the current user, the randomness of matching other users can be increased or decreased by setting the distance threshold between the current user and other users at the same geographical level. The set of failed frames for the watermark predicted for the current user screened by steps S11-S13 is more random while ensuring accuracy, thereby improving the anti-counterfeiting effect.

[0016] 3. The second strategy directly assigns corresponding historical watermark information to each video frame in the second set, ensuring the efficiency of watermark assignment in high-concurrency real-time video encoding and decoding scenarios, and improving the speed of encoding watermark information into video encoding data; the first strategy determines the reason for the failure of historical watermark assignment to each video frame in the first set, and adaptively adjusts the anti-counterfeiting level of the re-assigned watermark, ensuring the anti-counterfeiting effect of watermark assignment while taking into account the efficiency of watermark assignment. When assigning watermarks using the first and second strategies, by introducing random encrypted watermarks, the watermark anti-counterfeiting level is increased and the security of video decoding is improved without modifying the watermark assignment algorithm or with only minor changes; 4. The technical creativity of generating random encrypted watermarks lies in that, relying on the watermark-unable frame set and the watermark-assigned historical frame set screened out in step S1, the neighboring video frames of the current video frame can be extracted. The watermark information of the neighboring video frames may include historical additional watermark features and / or historical random encrypted watermarks. The watermark information of the current video frame may also include historical additional watermark features and / or historical random encrypted watermarks. The random encrypted watermark newly generated by chaotic permutation for the current video frame and the neighboring video frame is therefore more random, which increases the difficulty of the watermark being attacked.

[0017] 5. Steps S51-S53 verify the decoding correctness of the current video frame, further verifying whether the watermark features assigned to the current video frame can meet the efficiency and error rate requirements of real-time encoding and decoding of videos in high-concurrency scenarios, avoiding excessive sacrifice of decoding accuracy in pursuit of watermark anti-counterfeiting effects in steps S1-S4. BRIEF DESCRIPTION OF THE DRAWINGS To more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0018] Figure 1 1 is a diagram illustrating the steps for implementing a watermarking method based on high-concurrency real-time video encoding and decoding provided by this embodiment; Figure 2 This is an example graph showing the geographic location similarity between users at each regional level. DETAILED DESCRIPTION

[0019] The technical solution of the present application will be further explained below with reference to the accompanying drawings and through specific implementation methods.

[0020] Among them, the drawings are only used for illustrative purposes and represent only schematic diagrams rather than actual pictures, and should not be understood as limiting this patent; in order to better illustrate the embodiments of this application, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0021] The same or similar numbers in the drawings of the embodiments of the present application correspond to the same or similar parts; in the description of the present application, it should be understood that if the terms "upper", "lower", "left", "right", "inside", "outside", etc. appear, the orientation or position relationship indicated is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting this patent. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0022] In the description of this application, unless otherwise expressly specified or limited, when the term "connection" or the like appears to indicate a connection relationship between components, such term should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be internal communication between two components or an interaction between two components. For those skilled in the art, the specific meanings of the above terms in this application can be understood according to the specific circumstances.

[0023] This embodiment provides a watermarking method based on high-concurrency real-time video encoding and decoding, such as Figure 1 As shown, the steps include: S1, for a current user to be transmitted video, screen out at least one watermark-failed frame set having features similar to those of the current user; for example, if the video to be transmitted by the current user AU is a video, screen out at least one watermark-failed frame set having features similar to those of the current user AU; The feature similarity relationship with the current user AU includes any one or more of user feature similarity, failed frame information similarity, and decoding feature similarity. User feature similarity is: one or more of the similarities in device parameters, network jitter, and geographical location between the current user AU and other users OU playing the same video. When there are multiple types, the device parameter similarity, network jitter similarity, and geographical location similarity are subjected to equal-dimensional processing and weighted summation to obtain user feature similarity. Device parameters include one or more of the type and performance parameters of the device that plays the video, such as a computer, and the type and performance parameters of the player. Device parameter similarity is also preferably a weighted fusion result of equal-dimensional processing of the parameter values ​​of each type of device parameters. Network jitter is the network jitter of transmitting the video, and geographical location is the geographical location of the device that plays the video.

[0024] In order to increase the difference of watermark information added to the same video played by each device in the same geographical level and improve the anti-counterfeiting level, preferably, the similarity of the geographical location of the device where the current user AU and the other user OU play the same video is: the straight-line distance between the current user AU and the other user OU in the same geographical level, and the greater the straight-line distance, the greater the similarity of the geographical location of the two users. For example, Figure 2 In the example, the straight-line distance between the current user AU and the first other user OU1 is longer than the straight-line distance between the current user AU and the second other user OU2, and the geographical location similarity between the current user AU and the first other user OU1 is greater than the geographical location similarity between the current user AU and the second other user OU2.

[0025] In this embodiment, the method of dividing the regional levels is as follows: Devices in different geographical locations that have played the same video in the past are divided into several geographical levels based on their distance from the server that stores the video. Figure 2 As shown in FIG, the server 100 storing the video is used as the center of the regional hierarchy expansion. For example, a regional hierarchy with a width of 100 meters is constructed with each expansion of 100 meters (eg Figure 2 tiers 1, 2, 3, and 4 shown in ).

[0026] Dividing the geographical levels can improve the success rate and efficiency of watermark assignment. Devices in various geographical locations that have played the same video in the past are divided into several geographical levels based on their distance from the server storing the video, and other users in the same geographical level as the current user are used as the objects for calculating the user feature similarity with the current user. Since the distance between devices in the same geographical level and the server is similar, when the same video is transmitted from the server to each device in the same geographical level, the network jitter caused by the transmission distance is small, which is conducive to ensuring the consistency of the decoding delay distribution when each device in the same geographical level decodes the same video. The other users matched thereby have a higher user feature similarity with the current user. In this way, the prediction of the set of failed frames that may be obtained when the current user plays the video is more accurate through the screening of steps S11-S13, which is conducive to adjusting the watermark generation method for the failed frames predicted for the current user in advance in step S2, improving the success rate and efficiency of watermark assignment, and ensuring the anti-counterfeiting effect.

[0027] In addition, when attacking the watermark, the watermark information of the same video historical conditions of the neighboring devices may become an important basis for cracking the watermark. Therefore, the present application reversely avoids the above risks by setting that the greater the straight-line distance between two devices at the same geographical level, the greater the geographical similarity of the two devices. On the basis of this technology, when matching other users with user characteristics similar to the current user, the randomness of matching other users can be increased or decreased by setting the distance threshold between the current user and other users at the same geographical level. The set of failed frames for the watermark predicted for the current user screened through steps S11-S13 is more random while ensuring accuracy, thereby improving the anti-counterfeiting effect.

[0028] The similarity of failed frame information is: the similarity of the watermark information of the same frame in which watermarking failed when playing the same video in the past by other users with similar user characteristics as the current user. For example, assuming that the other users with similar user characteristics screened out for the current user include OU1 and OU2, OU1 is assumed to have generated a first set of failed watermarking frames when playing the video in the past, and OU2 is assumed to have generated a second set of failed watermarking frames when playing the video in the past. The first set of failed watermarking frames is assumed to include failed frames f1-f5, and the second set of failed watermarking frames is assumed to include failed frames f1, f3, f4, f6, f7, and f8. Then, the consistency of the watermark information of the same failed frames in the first set of failed watermarking frames and the second set of failed watermarking frames is first calculated, that is, the consistency of the watermark information of the two failed frames f1, the consistency of the watermark information of the two failed frames f3, and the consistency of the watermark information of the two failed frames f4 are calculated. Then, the consistency of each calculated watermark information is compared with the preset consistency threshold, and the number of failed frames whose watermark information consistency is greater than the consistency threshold is calculated. Assuming that the watermark information consistency of the two failed frames f1 and the two failed frames f3 is greater than the consistency threshold, and the watermark information consistency of the two failed frames f4 is less than the consistency threshold, the calculated number of failed frames whose watermark information consistency is greater than the consistency threshold is 2. Then, the union of the number of failed frames in the first watermark assigned failed frame set and the second watermark assigned failed frame set is calculated. For example, in the above example, the union of the failed frames in the first watermark assigned failed frame set and the second watermark assigned failed frame set is f1-f8, so the union of the number is 8. Finally, the ratio of the number of failed frames to the union of the number is calculated as the watermark information similarity between the first watermark assigned failed frame set and the second watermark assigned failed frame set.

[0029] It should be noted here that the consistency of watermark information is related to the method of assigning watermarks to the same failed frame. For example, the method of assigning watermarks to failed frame f1 played by another user OU1 is: assigning watermark feature a to the first area of ​​failed frame f1, assigning watermark feature b to the second area, and assigning watermark feature c to the third area; the method of assigning watermarks to failed frame f1 played by another user OU2 is: assigning watermark feature a to the first area of ​​the failed frame f1, assigning watermark feature c to the second area, and assigning watermark feature b to the third area. Then, the consistency of the first watermark information assigned to failed frame f1 when the device plays the video for another user OU1 and the second watermark information assigned to the failed frame f1 when the device plays the same video for another user OU2 is one-third, that is, the ratio of the number of identical areas assigned the same watermark feature to the total number of areas with watermark information set in the failed frame.

[0030] The above method for calculating the consistency of watermark information is only an example, and there are many other methods, which will not be elaborated in detail.

[0031] In order to improve the anti-counterfeiting effect of the video, preferably, the watermark features that the watermark information similarity calculation relies on include randomly encrypted watermarks. The generation of the randomly encrypted watermark depends on the neighborhood video frames of the current video frame. Here, the neighborhood refers to the previous historical video frame and the next historical video frame of the historical current video frame that is the same frame as the current video frame. Specifically, when adding a randomly encrypted watermark to the current video frame, the watermark-assigned historical frame set with the greatest feature similarity to the current user is first screened out. Here, the watermark-assigned historical frame set is the set of video frames after other users with the greatest feature similarity to the current user have assigned watermarks to each frame in the video when encoding the video. Then, from the watermark-assigned historical frame set, the historical current video frame that is the same frame as the current video frame to be transmitted to the current user is extracted, and then the previous frame and the next frame of the historical current video frame are extracted as the neighborhood video frames of the current video frame to be transmitted to the current user. Here, the watermark-assigned historical frame set with the greatest feature similarity to the current user includes two sets of successful or failed watermark assignments for the same video history split.

[0032] In this embodiment, the numerical value of the feature similarity relationship in step S1 is represented by a similarity fusion value of one or more of the following: user feature similarity, failed frame information similarity, and decoding feature similarity between the other users and the current user. When the feature similarity relationship includes user feature similarity, failed frame information similarity, and decoding feature similarity, the similarity fusion value is the weighted sum of the user feature similarity, failed frame information similarity, and decoding feature similarity.

[0033] The method of generating random encrypted watermarks will be described in the subsequent content and will not be explained here.

[0034] In this embodiment, the decoding feature similarity is: the similarity of the decoding delay distribution and / or the dependence strength on the random encrypted watermark when decoding the same video frame when other users with user feature similarity with the current user historically play the same video.

[0035] The decoding delay distribution in the decoding feature similarity calculation is preferably: the average of the decoding delays when different users decode the same video frame; the method for calculating the dependence strength on the random encrypted watermark is: in the following steps S51-S53, if the correlation obtained by taking the random encrypted watermark added to the current video frame and the watermark information generated for the current video frame using the first strategy or the second strategy as the initial state of entropy decoding (defined as the first correlation), and the correlation obtained by taking only the watermark information generated for the current video frame using the first strategy or the second strategy as the initial state of entropy decoding (defined as the second correlation) as the absolute value of the difference is less than the preset difference absolute value threshold, then the dependence strength on the random encrypted watermark when decoding the current video frame is assigned to "0", otherwise it is assigned to "1".

[0036] In this embodiment, the decoding feature similarity is preferably a weighted sum of the similarity of the decoding delay distribution and the similarity of the dependence strength on the random encrypted watermark.

[0037] In summary, in step S1, the method for screening out at least one watermarking failure frame set having a feature similarity relationship with the current user includes the following steps: S11, screening out other users who have user feature similarity with the current user, and determining that the current user has a user feature similarity with the other user when the user feature similarity between the current user and the other user is greater than a preset first similarity threshold; S12, calculating the watermark information similarity of the same frames where watermarking failed when other users with similar user characteristics to the current user played the same video in the past, and filtering out other users whose watermark information similarity is less than a second similarity threshold; S13, for the other users remaining after filtering in step S12, the decoding feature similarity when decoding the same video frame when the same video is played historically is calculated for each of the other users, and then the watermark assignment failure frame set associated with each other user whose decoding feature similarity is greater than the third similarity threshold is extracted as the screening result of the watermark assignment failure frame set.

[0038] It should be noted here that the decoding feature similarity calculation is for two watermark assigned historical frame sets, such as the first watermark assigned historical frame set and the second watermark assigned historical frame set. The difference between the watermark assigned historical frame set and the watermark assigned failed frame set is that one is a collection of video frames after all frames in the same video are watermarked, and the other is a collection of video frames in the same video that failed to be watermarked for the first time. The calculation principle of decoding feature similarity is the same as the calculation principle of watermark information similarity mentioned above, and the decoding feature similarity is also characterized by a ratio. The calculation principle of decoding information consistency is the same as the calculation principle of watermark information consistency. At the same time, since the calculation process of decoding information consistency is not within the scope of the rights protection required by this application, and does not affect the technical solution provided by this application to achieve the corresponding technical effects and solve the corresponding technical problems, it will not be repeated here.

[0039] The method for calculating the watermark information similarity provided in this embodiment is briefly described as follows, including the following steps: S121, calculating watermark information consistency for identical failed frames in a first watermarking failure frame set and a second watermarking failure frame set generated by historical playback of the same video by the first other user and the second other user, respectively. The calculation of watermark information consistency has been exemplified in the above content and will not be further described. S122, calculating the number of failed frames whose watermark information consistency is greater than a consistency threshold, and calculating the union of the number of failed frames in the first watermark assignment failure frame set and the second watermark assignment failure frame set; the meaning of the union of the number has been explained in the above content and will not be repeated here; S123, calculating the ratio of the number of failed frames to the union of the number as the watermark information similarity.

[0040] After the watermarking failure frame set is screened for the current user through step S1, Figure 1 As shown, the watermarking method based on high-concurrency real-time video encoding and decoding provided by this embodiment includes the following steps: S2, splitting the video to be transmitted to the current user into frames, that is, splitting the video into each frame. The same frame splitting method is used for the same video to be transmitted to different users to ensure that the type and number of the split frames are the same. Then, it is determined whether each split video frame is added to the set of frames that failed to be watermarked as screened in step S1; If yes, then add the currently judged video frame to the first set; If not, the currently judged video frame is added to the second set; For example, after splitting the video to be transmitted to the current user, suppose that the resulting video frames, f1-f100, are arranged consecutively in playback order. Suppose that the video was previously transmitted to 100 other users, OU1-OU100. Before transmitting the video to these 100 other users, the video was split into the 100 frames f1-f100.

[0041] Assume that the watermarking failure frame sets filtered in step S1 include two sets: a first watermarking failure frame set and a second watermarking failure frame set, which are generated when the first other user and the second other user respectively played the same video most recently. The last playback here refers to the most recent playback behavior that generated a watermarking failure frame during video playback and is closest to the current time.

[0042] Assume that the first watermarking failure frame set screened in step S1 includes five video frames, f10, f18, f21, f33, and f41, and the second watermarking failure frame set includes four video frames, f21, f29, f41, and f78. Then, add the seven video frames, f10, f18, f21, f29, f33, f41, and f78, to the first set; and add the remaining 93 video frames, excluding these seven frames, to the second set.

[0043] After obtaining the first set and the second set of watermarks to be assigned to the video to be transmitted to the current user in step S2, the watermark assignment method based on high-concurrency real-time encoding and decoding of video provided in this embodiment is as follows: Figure 1 As shown, go to step: S3, for each video frame in the first set and the second set, generating an invisible watermark using the first strategy and the second strategy respectively; The method for generating a watermark for each video frame in the first set using the first strategy comprises the following steps: A1, for each video frame in the first set, determining whether the texture complexity is greater than a preset complexity threshold; If yes, jump to step A3; If not, go to step A2; There are many existing methods for identifying the texture of video frames and calculating texture complexity, which will not be explained in detail.

[0044] A2, determining whether the historical decoding delay of the video frame has increased suddenly; If so, it is determined that the playback device failed when the video frame was played in the past, and then the watermark information of the historical failed frame is directly assigned to the video frame that is the same frame as the historical failed frame; If not, it is determined that network jitter occurred when the video frame was played historically, and then the process goes to step A3; The method for determining whether the historical decoding delay of the video frames in the first set has increased suddenly is briefly described as follows: Assume that the video frame f1 in the first set also exists in the first watermark assignment failure frame set and the second watermark assignment failure frame set selected in step S1, and assume that the first other user associated with the first watermark assignment failure frame set has a more characteristic similarity with the current user than the second other user associated with the second watermark assignment failure frame set. Then, extract the historical failure frame f1' which is the same frame as the video frame f1 from the first watermark assignment failure frame set, then extract the previous historical failure frame and the next historical failure frame of the historical failure frame f1', and calculate the previous historical failure frame The absolute value of the difference in historical decoding delay corresponding to the subsequent historical failed frame (defined as the first absolute value of the difference) is calculated, and the absolute value of the difference between the historical failed frame f1' and the previous historical failed frame (defined as the second absolute value of the difference) is calculated. Then, it is determined whether the first absolute value of the difference is less than the preset first absolute value of the difference threshold and the second absolute value of the difference is greater than the preset second absolute value of the difference threshold. If so, it is determined that the historical failed frame f1' has experienced a sudden increase in decoding delay, that is, it is determined that the video frame in the first set that is the same frame as the historical failed frame f1' has experienced a sudden increase in historical decoding delay.

[0045] Under the "if" judgment condition in step A2, the method of assigning watermark information to the video frames in the first set is the same as the method of historically assigning watermark information to the historical failed frame f1', thereby saving time for assigning watermarks to the video frames.

[0046] A3, determining whether an additional watermark feature is added to a historical failed frame in the watermark assignment failure frame set screened in step S1 that is the same as the video frame in the first set; If yes, then after removing the additional watermark features in the watermark information assigned to the historical failed frame, the remaining watermark features are assigned to the video frame; If not, after removing the random encrypted watermark (if any) in the watermark information assigned to the historical failed frame, the remaining watermark features are assigned to the failed frame.

[0047] In this embodiment, the additional watermark features include the similarity of the failure frame information and / or the similarity of the decoding features of the associated historical failure frames. The calculation method of the failure frame information similarity and the decoding feature similarity has been specifically described in the above content and will not be repeated here. Adding additional watermark features to the invisible watermark generated by the conventional watermark assignment method can increase the watermark anti-counterfeiting effect. The user does not need to make technical improvements to the conventional watermark assignment algorithm, and the generation of the additional watermark features incorporates the common features of the watermark information of other users with similar user features to the current user, but by reversely defining the geographic location similarity, the difference of the additional watermark features generated when other users with similar device parameters and network jitter play the video is increased, thereby making the generation of the additional watermark features more efficient and having a better anti-counterfeiting effect. In addition, through step A3, the reason for the failure to assign watermarks to the historical failure frames is analyzed, and by removing the additional watermark features and / or randomly encrypting the watermark, the watermark anti-counterfeiting level of the video frame that is the same frame as the historical failure frame is reduced, thereby improving the success rate of decoding the watermark information in the video frame.

[0048] The method for generating a watermark for each video frame in the second set using the second strategy comprises the following steps: B1, the historical frames that have been successfully assigned watermarks with similar features to the current user are arranged into a frame set list according to the similarity fusion value from large to small; Assume that the users screened in step S1 with similar features to the current user include two other users, OU1 and OU2. The historical frames with successful watermarking associated with OU1 and OU2 are assumed to be the first and second historical frames with successful watermarking, respectively. The historical frames with successful watermarking are the set of historical video frames with successfully watermarked information after video segmentation. The similarity fusion value is a weighted sum of one or more of the feature similarities of users with similar features to the current user, the similarity of failed frame information, and the similarity of decoded features.

[0049] B2, extract the first historical frame set with successful watermark assignment in the frame set list; Assuming that the frame set list includes a first watermark successfully assigned historical frame set, a second watermark successfully assigned historical frame set, and a third watermark successfully assigned historical frame set arranged in descending order according to similarity fusion values, then in step B2, the first watermark successfully assigned historical frame set is extracted; B3, extracting from the extracted watermarked historical frame set the historical video frames that are also present in the second filtered remaining set and adding them to the second strategic watermarked frame set, and then filtering out from the second filtered remaining set the video frames that are the same as the extracted historical video frames; For example, the second set includes video frames f1-f5. Assume that the first watermark successfully assigned historical frame set extracted in step B2 includes historical video frames f1'-f3' and f4'-f100'. F1'-f3' are the same video frames as f1-f3. Then, historical video frames f1'-f3' in the first watermark successfully assigned historical frame set are added to the second strategically watermarked frame set. Then, f1-f3 are filtered out from the second set, leaving video frames f4-f5 in the second set.

[0050] B4, determine whether the elements in the second set are empty; If so, the watermark information assigned to each historical video frame in the second strategy watermarked frame set is directly assigned to the video frame initially added to the second set that is the same frame as the historical video frame; If not, return to step B2.

[0051] Continuing with the above example, since there are still two video frames f4-f5 left in the second set, the "if no" condition of step B4 is met, and the process returns to step B2 to extract the historical frame set with the second watermark successfully assigned that is ranked first at this time.

[0052] Assuming that the elements in the second set are empty at this time, the historical watermark information of each historical video frame in f1'-f5' in the second strategy watermark frame set is directly assigned to the video frames f1-f5 in the second set respectively. After generating watermark information for the video frames in the first set and the second set through step S3, as shown in FIG. Figure 1 As shown, the watermarking method based on high-concurrency real-time video encoding and decoding provided by this embodiment includes the following steps: S4, extracting neighboring video frames of the current video frame to be transmitted, performing random watermark exchange encryption to obtain a random encrypted watermark of the current video frame, and adding it to the encoded data of the current video frame; In this embodiment, the method for generating the random encrypted watermark is as follows: performing chaotic permutation on the watermark information of the current video frame and the neighboring video frames to generate the random encrypted watermark to be added to the encoded data of the current video frame; In this embodiment, the technical creativity of generating random encrypted watermarks lies in that, relying on the watermark-assignment failure frame set and the watermark-assignment history frame set screened out in step S1, the neighboring video frames of the current video frame can be extracted. The watermark information of the neighboring video frames may include historical additional watermark features and / or historical random encrypted watermarks. The watermark information of the current video frame may also include historical additional watermark features and / or historical random encrypted watermarks. The random encrypted watermark newly generated by chaotic permutation for the current video frame and the neighboring video frame is therefore more random, which increases the difficulty of the watermark being attacked.

[0053] It should be emphasized here that the neighborhood video frames of the current video frame are not the previous video frame and the next video frame of the current video frame, but the previous historical video frame and the next historical video frame of the historical video frame played by other users who have the greatest feature similarity with the current user.

[0054] The existing method is to generate a random password for the current frame through chaotic permutation based on the encrypted data of the previous frame and the next frame. The process of generating a random encrypted watermark can refer to the random password generation process through chaotic permutation, and no further detailed description is given.

[0055] After the random encrypted watermark is generated for the current video frame through step S4, the watermark assignment method based on high concurrent real-time video encoding and decoding provided by this embodiment is as follows: Figure 1 As shown, go to step: S5, verifying the decoding correctness of the current video frame transmitted to the current user based on the random encrypted watermark; If the verification is successful, the decoded current video frame will be played to the current user; If the verification fails, a video playback abnormality alarm is issued to the current user, and the current video frame is added to the watermark assignment failure frame set associated with the current user.

[0056] In this embodiment, the method for verifying the decoding correctness of the current video frame transmitted to the current user includes the steps of: S51, using the randomly encrypted watermark added to the current video frame and the watermark information generated for the current video frame using the first strategy or the second strategy as the initial state of entropy decoding; S52, reconstructing the correlation between the frame quality and the initial state of entropy decoding for the current video frame, the reconstruction method is, for example: Assume that the frame quality of the current video frame is Qt, the randomly encrypted watermark generated for the current video frame is m1t, and the watermark information generated for the current video frame using the first or second strategy is m2t. Then, the correlation between the frame quality and the initial state of entropy decoding, st, can be expressed as: st = F(Qt, (m1t, m2t)), where F is the correlation function between the frame quality and the initial state of entropy decoding. There are many correlation functions that express the relationship between frame quality and entropy decoding state, which are not explained in detail here.

[0057] S53, determining whether the correlation is greater than a preset correlation threshold; If yes, it is determined that the decoding correctness verification of the current video frame is successful; If not, it is determined that the decoding correctness verification of the current video frame has failed.

[0058] Steps S51-S53 verify the decoding correctness of the current video frame, and further verify whether the watermark features assigned to the current video frame can meet the efficiency and error rate requirements of real-time encoding and decoding of videos in high-concurrency scenarios, avoiding excessive sacrifice of decoding accuracy when pursuing the watermark anti-counterfeiting effect in steps S1-S4.

[0059] It should be noted that the above-described specific embodiments are merely preferred embodiments of the present application and the technical principles employed. Those skilled in the art will appreciate that various modifications, equivalent substitutions, and variations may be made to the present application. However, as long as these modifications do not depart from the spirit of the present application, they are intended to be within the scope of protection of the present application. Furthermore, certain terms used in the specification and claims of this application are not intended to be limiting but are provided for ease of description only.

Claims

1. A watermarking method based on high-concurrency real-time video encoding and decoding, characterized in that: Including steps: S1, for a current user to be used for video transmission, screening out at least one watermarking failure frame set having features similar to those of the current user; S2, splitting the video to be transmitted to the current user into frames, and determining whether each split video frame is added to the filtered set of frames that failed to be watermarked; If yes, then adding the currently determined video frame to the first set; If not, adding the currently determined video frame to the second set; S3, for each of the video frames in the first set and the second set, generating an invisible watermark using a first strategy and a second strategy respectively; S4, extracting neighboring video frames of the current video frame to be transmitted, performing random watermark exchange encryption to obtain a random encrypted watermark of the current video frame, and adding the watermark to the encoded data of the current video frame; S5, verifying decoding correctness of the current video frame transmitted to the current user according to the random encrypted watermark; If the verification is successful, the decoded current video frame is played to the current user; If the verification fails, a video playback abnormality alarm is issued to the current user, and the current video frame is added to the watermark assignment failure frame set associated with the current user.

2. The watermarking method based on high-concurrency real-time encoding and decoding of video according to claim 1 is characterized in that: The feature similarity relationship between other users and the current user includes any one or more of user feature similarity, failed frame information similarity, and decoding feature similarity; The user feature similarity includes one or more similarities of device parameters, network jitter, and geographic location between the current user and the other users playing the same video; The failed frame information similarity is the watermark information similarity of the same frame where watermarking failed when playing the same video in pairs of the other users who have user feature similarity with the current user; The decoding feature similarity is the similarity of decoding delay distribution and / or dependence strength on the random encrypted watermark when decoding the same video frame when each of the other users having the user feature similarity with the current user historically plays the same video.

3. The watermarking method based on high-concurrency real-time encoding and decoding of video according to claim 2 is characterized in that: In step S1, the method for screening out at least one watermarking failure frame set having a feature similarity relationship with the current user comprises the steps of: S11, screening out other users having user feature similarity with the current user, and when the user feature similarity between the current user and the other user is greater than a preset first similarity threshold, determining that the current user and the other user have the user feature similarity relationship; S12, calculating the watermark information similarity of the same frames of the video that failed to be watermarked when the other users who have similar user characteristics to the current user played the same video in history, and filtering out the other users whose watermark information similarity is less than a second similarity threshold; S13, calculating the decoding feature similarity of each of the other users remaining after filtering in step S12 when decoding the same video frame during historical playback of the same video, and then extracting the watermark failure frame set associated with each of the other users whose decoding feature similarity is greater than a third similarity threshold as the screening result of the watermark failure frame set.

4. The watermarking method based on high-concurrency real-time encoding and decoding of video according to claim 2 or 3, characterized in that: The user feature similarity is a weighted sum of the similarities in device parameters, network jitter, and geographic location between the current user and the other users playing the same video. Geographic location similarity is represented by the straight-line distance between two users in the same geographical level. The greater the straight-line distance, the greater the geographic location similarity between the two users. The decoding feature similarity is a weighted sum of the decoding delay distribution and the dependence strength on the random encrypted watermark when two other users decode the same video frame when playing the same video in the past.

5. The watermarking method based on high-concurrency real-time video encoding and decoding according to claim 4 is characterized in that: Devices in various geographical locations that have historically played the same video are divided into several regional levels based on their distance from a server that remotely stores the video.

6. The watermarking method based on high-concurrency real-time video encoding and decoding according to claim 3 is characterized in that: In step S12, the method for calculating the watermark information similarity includes the following steps: S121, calculating watermark information consistency for identical failed frames in a first watermarking failure frame set and a second watermarking failure frame set generated by historical playback of the same video by the first other user and the second other user, respectively; S122, calculating the number of failed frames whose watermark information consistency is greater than a consistency threshold, and calculating the union of the number of failed frames in the first watermark assignment failure frame set and the second watermark assignment failure frame set; S123: Calculate the ratio of the number of failed frames to the union of the numbers as the watermark information similarity.

7. The watermarking method based on high-concurrency real-time video encoding and decoding according to claim 2 is characterized in that: The watermark features that the watermark information similarity calculation relies on include the random encrypted watermark; The neighborhood video frames of the current video frame are: the previous historical video frame and the next historical video frame of the historical video frame that is the same frame as the current video frame, in the watermarked historical frame set that has the greatest feature similarity with the current user selected in step S1; The maximum feature similarity relationship is: the maximum value of one or more similarity fusion values ​​of the user feature similarity, the failed frame information similarity, and the decoding feature similarity calculated for the same other user.

8. The watermarking method based on high-concurrency real-time video encoding and decoding according to claim 7 is characterized in that: The method for generating a watermark for each of the video frames in the first set using the first strategy comprises the steps of: A1, for each of the video frames in the first set, determining whether the texture complexity is greater than a preset complexity threshold; If yes, jump to step A3; If not, go to step A2; A2, determining whether the historical decoding delay of the video frame increases suddenly; If so, it is determined that the playback device failed when the video frame was played in the past, and then the watermark information of the historical failed frame is directly assigned to the video frame that is the same frame as the historical failed frame; If not, it is determined that abnormal network jitter occurred when the video frame was played historically, and then the process goes to step A3; A3, determining whether an additional watermark feature is added to a historical failure frame in the watermarking failure frame set screened in step S1 that is the same as the video frame in the first set; If yes, after removing the additional watermark feature from the watermark information assigned to the historical failed frame, assign the remaining watermark feature to the video frame; If not, after removing the random encrypted watermark from the watermark information assigned to the historical failed frame, the remaining watermark features are assigned to the video frame.

9. The watermarking method based on high-concurrency real-time encoding and decoding of video according to claim 7, characterized in that: The method for generating a watermark for each of the video frames in the second set using the second strategy comprises the steps of: B1, arranging the historical frame sets for which each watermark has been successfully assigned and has a feature similarity relationship with the current user into a frame set list according to the similarity fusion value from large to small; B2, extracting the first historical frame set with successful watermarking from the frame set list; B3, extracting from the extracted watermarked historical frame set each historical video frame that is also present in the second filtered remaining set and adding the frame to the second strategic watermarked frame set, and then filtering out from the second filtered remaining set each video frame that is the same frame as each extracted historical video frame; B4, determining whether the element in the second set is empty; If so, directly assigning the watermark information assigned to each of the historical video frames in the second strategy watermarked frame set to the video frame initially added to the second set that is the same frame as the historical video frame; If not, return to step B2.

10. The watermarking method based on high-concurrency real-time video encoding and decoding according to claim 8, characterized in that: The additional watermark feature includes the similarity of the failed frame information and / or the similarity of the decoding features associated with the historical failed frames; In step S4, the method for generating the random encrypted watermark is: performing chaotic permutation on the watermark information of the current video frame and the neighboring video frame to generate the random encrypted watermark to be added to the encoded data of the current video frame; In step S5, the method for verifying the decoding correctness of the current video frame transmitted to the current user includes the steps of: S51, using the randomly encrypted watermark added to the current video frame and the watermark information generated for the current video frame using the first strategy or the second strategy as an initial state of entropy decoding; S52, reconstructing the correlation between the frame quality and the initial state of the entropy decoding for the current video frame; S53, determining whether the correlation is greater than a preset correlation threshold; If yes, it is determined that the decoding correctness verification of the current video frame is successful; If not, it is determined that the decoding correctness verification of the current video frame fails.

Images