Multimedia copyright protection method and apparatus

Abstract

The application relates to the multimedia field and provides a multimedia copyright protection method and device. The method comprises the following steps: decoding and separating a multimedia to be protected to obtain an audio stream to be protected and a video stream to be protected; extracting scene features of the video stream to be protected according to a histogram-local binary pattern scene classification algorithm, and obtaining a first watermark according to the scene features; embedding the first watermark into the audio stream to be protected to obtain a watermark-containing audio stream; generating a second watermark according to the watermark-containing audio stream; embedding the second watermark into the video stream to be protected to obtain a watermark-containing video stream; and obtaining a watermark-containing multimedia according to the watermark-containing audio stream and the watermark-containing video stream. The multimedia copyright protection method and device provided in the application embodiment generate the first watermark and the second watermark according to audio and video global features, and form mutual nesting of watermark generation and embedding, improve the sensitivity of the watermark to various tampering operations of the audio and video stream, and further improve the integrity protection capability of the watermark.

Description

Technical Field

[0001] This application relates to the field of multimedia technology, specifically to a multimedia copyright protection method and apparatus. Background Technology

[0002] Encryption and digital watermarking have long been the primary means of digital rights protection. Encryption technology transforms multimedia data into incomprehensible data during data transmission using encryption algorithms, thereby achieving the purpose of copyright protection. However, once the encrypted data is successfully decoded, it is identical to the original data and no longer possesses any protective capabilities. Therefore, encryption technology cannot fundamentally solve the problem of infringement.

[0003] Digital watermarking technology embeds authentication information into multimedia data through embedding algorithms. Common operations (transmission, copying, etc.) do not affect the embedded information. Generally, it has the ability to authenticate copyright within the lifespan of digital media. However, most existing digital technologies rely on the local features of audio and video streams to generate watermarks. Therefore, they are not very sensitive to various tampering operations on audio and video streams, resulting in weak watermark integrity protection capabilities. Summary of the Invention

[0004] This application provides a multimedia copyright protection method and apparatus to address the technical problem that existing digital technologies mostly rely on local features of audio and video streams to generate watermarks, thus having poor sensitivity to various tampering operations on audio and video streams, resulting in weak watermark integrity protection capabilities.

[0005] In a first aspect, embodiments of this application provide a multimedia copyright protection method, including:

[0006] Decode and separate the multimedia to be protected to obtain the audio stream and the video stream to be protected;

[0007] The scene features of the video stream to be protected are extracted according to the histogram-local binary pattern scene classification algorithm, and the first watermark is obtained according to the scene features;

[0008] The first watermark is embedded into the audio stream to be protected to obtain a watermarked audio stream;

[0009] A second watermark is generated based on the watermarked audio stream;

[0010] The second watermark is embedded into the video stream to be protected to obtain a watermarked video stream;

[0011] Watermarked multimedia is obtained from the watermarked audio stream and the watermarked video stream.

[0012] In one embodiment, extracting scene features of the video stream to be protected based on the histogram-local binary pattern scene classification algorithm includes:

[0013] The video stream to be protected is classified into scenes using a histogram scene classification algorithm to obtain the histogram inter-frame difference sum;

[0014] The video stream to be protected is classified into scenes using a local binary mode scene classification algorithm to obtain the inter-frame difference of texture maps.

[0015] The scene features of the video stream to be protected are extracted based on the harmonic mean of the sum of the inter-frame differences in the histogram and the sum of the inter-frame differences in the texture map.

[0016] In one embodiment, embedding the first watermark into the audio stream to be protected to obtain a watermarked audio stream includes:

[0017] Decode the audio stream to be protected to obtain audio stream data;

[0018] The left channel data in the audio stream is divided into non-overlapping frames to obtain the audio frame data to be processed.

[0019] Perform a three-level discrete wavelet transform on the audio frame data to be processed to obtain multiple first low-frequency coefficients;

[0020] The plurality of first low-frequency coefficients are sorted, and the plurality of first low-frequency coefficients are grouped according to the sorting. By modifying the last first low-frequency coefficient in the previous group, 1 bit of the first watermark data is embedded in each group to obtain a watermarked low-frequency coefficient group.

[0021] Merge all groups of watermarked low-frequency coefficients to obtain the watermarked low-frequency coefficient set;

[0022] The watermarked audio stream is obtained from the data after the inverse discrete wavelet transform of the watermarked low-frequency coefficient set.

[0023] In one embodiment, generating a second watermark based on the watermarked audio stream includes:

[0024] Extract the one-dimensional MFCC coefficients of the watermarked audio stream, and divide the one-dimensional MFCC coefficients into multiple MFCC coefficient segments according to a specific length;

[0025] A corresponding digital watermark is generated based on the variance of each MFCC coefficient segment, and all digital watermarks are merged to obtain the second watermark.

[0026] In one embodiment, embedding the second watermark into the video stream to be protected to obtain a watermarked video stream includes:

[0027] The video stream to be protected is segmented according to the scene characteristics to obtain L video scenes to be processed; where L is an integer greater than or equal to 2.

[0028] The second watermark is divided into L watermark segments, and an end marker is added at the end of each watermark segment;

[0029] For any video frame in the L video scenes to be processed, the regions are divided according to the different salience of each region of the video frame. The three regions with the highest salience are excluded to obtain the first non-salience region of the video frame.

[0030] An 8*8 discrete cosine transform is performed on the luminance component of the first non-significant region to obtain the first DC coefficient. By modifying the first DC coefficient of each video frame in each video scene to be processed, the L second watermark segments are embedded one-to-one into the L video scenes to be processed to obtain a set of watermarked video scenes.

[0031] Perform an inverse discrete cosine transform on the watermarked video scene set, and merge the watermarked video scene sets after the inverse transform to obtain the watermarked video stream.

[0032] In one embodiment, obtaining the watermarked multimedia includes:

[0033] Decode and separate the watermarked multimedia to obtain the audio stream and video stream to be verified;

[0034] The video stream to be verified is segmented into L video scenes based on the histogram-local binary pattern scene classification algorithm.

[0035] For the first three video frames in any video scene to be verified, the regions are divided according to the different salience of each region in any video frame. The three regions with the highest salience are excluded to obtain the second non-salience region of the video frame.

[0036] Perform an 8*8 discrete cosine transform on the luminance component of the second non-significant region to obtain the second DC coefficient. Extract the watermark fragments on the first three video frames in any video scene to be verified using the second DC coefficients, and stop extraction when the end flag is encountered.

[0037] The L video scenes to be verified are sorted in chronological order.

[0038] Based on the order of sorting, the second watermark segments to be verified in two adjacent video scenes to be verified are selected and discarded.

[0039] If the i-th second watermark fragment in the l-th video scene to be verified is w l(i) The i-th second watermark fragment w in the (l-1)-th video scene to be verified l-1 (i) If they are not the same, then keep w. l (i) and w l-1 (i); where l is an integer greater than or equal to 2 and less than or equal to L, i is an integer greater than or equal to 1, and w l (i) is obtained from the watermark fragments on the first three video frames of the l-th video scene to be verified;

[0040] If the i-th second watermark fragment in the l-th video scene to be verified is w l (i) The i-th second watermark fragment w in the (l-1)-th video scene to be verified l-1 (i) If they are the same, then keep w. l (i), discard w l-1 (i);

[0041] After replacing l with l+1, return to the step of selecting the second watermark segments of two adjacent video scenes to be verified in order of sorting, until l=L, and complete the selection of the second watermark segments of all adjacent video scenes to be verified.

[0042] All the retained second watermark fragments to be verified are merged to obtain the second watermark to be verified;

[0043] The system retrieves a pre-stored trusted second watermark of the watermarked multimedia from the blockchain. If the second watermark to be verified is inconsistent with the trusted second watermark, it determines that the audio stream to be verified has been tampered with, and identifies the type and location of the tampering.

[0044] In one embodiment, after obtaining the pre-stored trusted second watermark of the watermarked multimedia from the blockchain, the process includes:

[0045] If the second watermark to be verified is consistent with the trusted second watermark, then it is determined that the audio stream to be verified has not been tampered with;

[0046] The left channel data in the audio stream to be verified is divided into non-overlapping frames to obtain the audio frame data to be verified.

[0047] Perform a three-level discrete wavelet transform on the audio frame data to be verified to obtain multiple second low-frequency coefficients;

[0048] The plurality of second low-frequency coefficients are sorted, and the plurality of second low-frequency coefficients are grouped according to the sorting.

[0049] Based on the relationship between the average value of the second low-frequency coefficients in any group and the last sorted second low-frequency coefficient in the group preceding the group, extract the first watermark fragment to be verified in any group.

[0050] Merge all the first watermark fragments to be verified to obtain the first watermark to be verified;

[0051] The system retrieves a pre-stored trusted first watermark of the watermarked multimedia from the blockchain. If the first watermark to be verified is inconsistent with the trusted first watermark, it determines that the video stream to be verified has been tampered with, and identifies the type and location of the tampering.

[0052] Secondly, embodiments of this application provide a multimedia copyright protection device, including:

[0053] The decoding and separation module is used to decode and separate the multimedia to be protected, obtaining the audio stream and the video stream to be protected.

[0054] The first watermark generation module is used to: extract scene features of the video stream to be protected according to the histogram-local binary mode scene classification algorithm, and obtain the first watermark according to the scene features;

[0055] A watermarked audio stream generation module is used to: embed the first watermark into the audio stream to be protected to obtain a watermarked audio stream;

[0056] The second watermark generation module is used to: generate a second watermark based on the watermarked audio stream;

[0057] A watermarked video stream generation module is used to: embed the second watermark into the video stream to be protected to obtain a watermarked video stream;

[0058] A watermarked multimedia generation module is used to: obtain watermarked multimedia based on the watermarked audio stream and the watermarked video stream.

[0059] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory storing a computer program, wherein the processor executes the program to implement the steps of the multimedia copyright protection method described in the first aspect.

[0060] Fourthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the multimedia copyright protection method described in the first aspect.

[0061] The multimedia copyright protection method and apparatus provided in this application first decode and separate the multimedia to be protected, obtaining an audio stream and a video stream to be protected. Then, scene features of the video stream to be protected are extracted using a histogram-local binary pattern scene classification algorithm. A first watermark is obtained based on the scene features, and the first watermark is embedded into the audio stream to obtain a watermarked audio stream. A second watermark is then generated from the watermarked audio stream and embedded into the video stream to obtain a watermarked video stream. Finally, the watermarked multimedia is obtained from the watermarked audio stream and the watermarked video stream. This application utilizes the scene features of the video stream to obtain the first watermark, embeds the first watermark into the audio stream to be protected, generates the second watermark from the audio stream with the first watermark embedded, and then embeds the second watermark into the video stream to be protected. By generating the first and second watermarks using global audio and video features, a nested process of watermark generation and embedding is formed, improving the watermark's sensitivity to various tampering operations on the audio and video streams, thereby enhancing the watermark's integrity protection capability. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 This is one of the flowcharts illustrating the multimedia copyright protection method provided in this application embodiment;

[0064] Figure 2 This is a second schematic flowchart of the multimedia copyright protection method provided in the embodiments of this application;

[0065] Figure 3 This is the third flowchart illustrating the multimedia copyright protection method provided in this application embodiment;

[0066] Figure 4 This is the fourth flowchart illustrating the multimedia copyright protection method provided in this application embodiment;

[0067] Figure 5 This is the fifth flowchart illustrating the multimedia copyright protection method provided in this application embodiment;

[0068] Figure 6 This is the sixth flowchart illustrating the multimedia copyright protection method provided in this application embodiment;

[0069] Figure 7 This is the seventh flowchart illustrating the multimedia copyright protection method provided in this application embodiment;

[0070] Figure 8 This is a schematic diagram of the structure of the multimedia copyright protection device provided in the embodiments of this application;

[0071] Figure 9 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0073] Figure 1 This is one of the flowcharts illustrating the multimedia copyright protection method provided in this application. (Refer to...) Figure 1 This application provides a multimedia copyright protection method, which may include:

[0074] 101. Decode and separate the multimedia to be protected to obtain the audio stream and video stream to be protected;

[0075] 102. Extract scene features of the video stream to be protected based on the histogram-local binary pattern scene classification algorithm, and obtain the first watermark based on the scene features;

[0076] 103. Embed the first watermark into the audio stream to be protected to obtain the watermarked audio stream;

[0077] 104. Generate a second watermark based on the watermarked audio stream;

[0078] 105. Embed the second watermark into the video stream to be protected to obtain the watermarked video stream;

[0079] 106. Obtain watermarked multimedia based on the watermarked audio stream and the watermarked video stream.

[0080] In step 102, the scene features can be scrambled, and then the first watermark can be obtained based on the scrambled scene features, thereby improving the confidentiality of the first watermark.

[0081] The multimedia copyright protection method provided in this embodiment first decodes and separates the multimedia to be protected, obtaining the audio stream and video stream to be protected. Then, it extracts scene features of the video stream to be protected using a histogram-local binary pattern scene classification algorithm, obtains a first watermark based on the scene features, embeds the first watermark into the audio stream to be protected, obtaining a watermarked audio stream. Next, it generates a second watermark based on the watermarked audio stream, embeds the second watermark into the video stream to be protected, obtaining a watermarked video stream. Finally, it obtains the watermarked multimedia based on the watermarked audio stream and the watermarked video stream. This embodiment utilizes the scene features of the video stream to be protected to obtain the first watermark, embeds the first watermark into the audio stream to be protected, generates the second watermark based on the audio stream with the first watermark embedded, and then embeds the second watermark into the video stream to be protected. It generates the first and second watermarks using global audio and video features, forming a nested process of watermark generation and embedding. This improves the sensitivity of the watermark to various tampering operations on the audio and video streams, thereby enhancing the integrity protection capability of the watermark.

[0082] Figure 2 This is the second flowchart illustrating the multimedia copyright protection method provided in this application. (Refer to...) Figure 2 In one embodiment, extracting scene features of the video stream to be protected based on the histogram-local binary pattern scene classification algorithm may include:

[0083] 201. Use the histogram scene classification algorithm to classify the video stream to be protected into scenes and obtain the histogram inter-frame difference sum;

[0084] 202. Use the local binary mode scene classification algorithm to classify the video stream to be protected and obtain the inter-frame difference of texture map;

[0085] 203. Extract scene features of the video stream to be protected based on the harmonic mean of the sum of the inter-frame differences in the histogram and the sum of the inter-frame differences in the texture map.

[0086] In step 201, the histogram scene classification algorithm can intuitively describe the proportion of different color values ​​in an image, and is the main method for expressing image color features. When classifying video scenes, the histogram scene classification algorithm is used to statistically analyze the grayscale values ​​of two consecutive frames of the video. Then, by comparing the sum of the histogram inter-frame differences between the two frames with a threshold (empirical value), it is determined whether the video scene has changed. Its drawback is that it is difficult to distinguish different scenes when the colors are similar, such as a beach and a desert. The histogram inter-frame difference sum in this step is the sum of the histogram inter-frame differences obtained by summing the histogram inter-frame differences of all two consecutive frames in the video stream to be protected.

[0087] In step 202, the basic idea of ​​the local binary mode scene classification algorithm is to compare the gray value of each pixel with the gray values ​​of its neighbors. Neighboring pixels with gray values ​​smaller than the center point are set to 0, and vice versa. This binary number is then converted into a decimal gray value. Finally, the inter-frame difference of the texture map between two adjacent frames is calculated and compared with an empirical value to determine the video scene transition. Its drawback is that it cannot distinguish scenes with the same texture but different colors, such as black and white scenes. The inter-frame difference of the texture map in this step is the sum of the inter-frame differences of the texture map of all adjacent frames in the video stream to be protected.

[0088] In step 203, the harmonic mean of the sum of histogram inter-frame differences and the sum of texture map inter-frame differences can be calculated using the following formula:

[0089]

[0090] Among them, D H-LBP is the harmonic mean of the sum of histogram frame differences and the sum of texture map frame differences, where Ds is the sum of histogram frame differences and DLs is the sum of texture map frame differences.

[0091] D H-LBP Compare with the threshold value (empirical value 0.18), if D H-LBP If the value exceeds the threshold, it is determined that the scene has changed; otherwise, it is determined that the scene has not changed. Based on this, the scene features of the video stream to be protected are identified and extracted.

[0092] In practical applications, there is no strict timing relationship between steps 201 and 202; that is, they can be executed simultaneously, or either step can be executed first, depending on the actual needs, and no restrictions are made here.

[0093] This embodiment utilizes a histogram-local binary mode scene classification algorithm to extract scene features from the video stream to be protected. By comparing the histogram scene classification algorithm and the local binary mode scene classification algorithm, it can determine whether the scene has changed from both color and texture features. This effectively solves the problems of missed detection in scenes with similar colors and false detection in scenes with rapidly changing content in the histogram scene classification algorithm, and also makes up for the shortcomings of the local binary mode scene classification algorithm in scene color feature detection.

[0094] Figure 3 This is the third flowchart illustrating the multimedia copyright protection method provided in this application. (Refer to...) Figure 3 In one embodiment, embedding a first watermark into the audio stream to be protected to obtain a watermarked audio stream may include:

[0095] 301. Decode the audio stream to be protected to obtain the audio stream data;

[0096] 302. Perform non-overlapping frame segmentation on the left channel data in the audio stream to obtain the audio frame data to be processed;

[0097] 303. Perform a three-level discrete wavelet transform on the audio frame data to be processed to obtain multiple first low-frequency coefficients;

[0098] 304. Sort the multiple first low-frequency coefficients, group the multiple first low-frequency coefficients according to the sorting, and embed 1 bit of first watermark data into each group by modifying the last first low-frequency coefficient in the previous group, to obtain the watermarked low-frequency coefficient group.

[0099] 305. Merge all groups of low-frequency coefficients containing watermarks to obtain a set of low-frequency coefficients containing watermarks;

[0100] 306. Based on the data obtained by inverse discrete wavelet transform of the low-frequency coefficient set containing the watermark, the watermarked audio stream is obtained.

[0101] In step 302, the left channel data in the audio stream can be non-overlappingly framed every 1600 frames.

[0102] In step 303, a three-level discrete wavelet transform is performed on the audio frame data to be processed, which can also yield multiple high-frequency coefficients.

[0103] In step 304, the multiple first low-frequency coefficients can be sorted according to their chronological order. When the total length of the multiple first low-frequency coefficients is 200 bits, they are divided into 10 groups according to the sorting, with 20 coefficients in each group and each coefficient having a length of 1 bit. By modifying the last first low-frequency coefficient in the previous group, 1 bit of first watermark data is embedded in each group. For example, the first group has first low-frequency coefficients 1 to 20, the second group has first low-frequency coefficients 21 to 40, and so on. By modifying the 20th first low-frequency coefficient in the first group, 1 bit of first watermark data is embedded in the second group.

[0104] It should be noted that the right channel data in the audio stream can also be processed using the same steps to obtain a complete watermarked audio stream.

[0105] This embodiment improves watermark embedding efficiency by grouping low-frequency coefficients and embedding portions of the first watermark separately, thereby completing the embedding of the entire first watermark. Furthermore, it can locate the attacked position when the watermark is attacked.

[0106] Figure 4 This is the fourth flowchart illustrating the multimedia copyright protection method provided in this application. (Refer to...) Figure 4 In one embodiment, generating a second watermark based on the watermarked audio stream may include:

[0107] 401. Extract the one-dimensional MFCC coefficients of the watermarked audio stream, and divide the one-dimensional MFCC coefficients into multiple MFCC coefficient segments according to a specific length;

[0108] 402. Generate a corresponding digital watermark based on the variance of each MFCC coefficient segment, and merge all digital watermarks to obtain the second watermark.

[0109] MFCC coefficients, or Mel Frequency Cepstrum Coefficients (MFCCs), are derived from the Mel frequency, which is based on the characteristics of human hearing and has a non-linear relationship with Hertzian frequency. MFCCs utilize this relationship to calculate the Hertzian spectrum characteristics and are widely used in speech recognition. However, due to the non-linear relationship between Mel and Hertzian frequencies, the accuracy of MFCC calculation decreases as the frequency increases. Therefore, in applications, only low-frequency MFCCs are often used, while mid- to high-frequency MFCCs are discarded.

[0110] In step 401, the one-dimensional MFCC coefficients can be divided into segments of 64 bits in length.

[0111] In step 402, a code can be generated based on the variance of each MFCC coefficient segment. This code is the corresponding digital watermark. Specifically, the code can be generated based on the following formula:

[0112]

[0113] Where T is the code, i.e. digital watermark, D(n) is the variance of the nth MFCC coefficient segment, n is an integer greater than or equal to 1, and M is the mean of the variances of all MFCC coefficient segments.

[0114] This embodiment can encode based on the variance of multiple MFCC coefficient segments, use the encoding as the corresponding digital watermark, and then merge multiple digital watermarks to obtain a second watermark.

[0115] Figure 5 This is the fifth flowchart illustrating the multimedia copyright protection method provided in this application. (Refer to...) Figure 5 In one embodiment, embedding the second watermark into the video stream to be protected to obtain a watermarked video stream may include:

[0116] 501. Segment the video stream to be protected according to scene characteristics to obtain L video scenes to be processed;

[0117] Where L is an integer greater than or equal to 2;

[0118] 502. Divide the second watermark into L second watermark segments, and add an end marker at the end of each second watermark segment;

[0119] 503. For any video frame in L video scenes to be processed, divide the video frame into regions according to the different salience of each region. Exclude the three regions with the highest salience to obtain the first non-salience region of any video frame.

[0120] 504. Perform an 8*8 discrete cosine transform on the luminance component of the first non-significant region to obtain the first DC coefficient. By modifying the first DC coefficient of each video frame in each video scene to be processed, embed the L second watermark segments into the L video scenes to be processed one by one to obtain a set of watermarked video scenes.

[0121] 505. Perform inverse discrete cosine transform on the set of watermarked video scenes, and merge the watermarked video scene sets after the inverse transform to obtain the watermarked video stream.

[0122] In step 502, the length of the first L-1 second watermark segments can be a first specific integer, and the length of the last second watermark segment can be the sum of the first specific integer and the second specific integer; the first specific integer is the quotient of the total length of the second watermark divided by L, the second specific integer is the remainder of the total length of the second watermark divided by L, and the length of the end flag can be 8 bits.

[0123] In step 503, the Itti visual saliency model can be used to perform saliency analysis on each frame in the L video scenes to be processed. For any video frame in the L video scenes to be processed, the regions are divided according to the different saliency of each region in any video frame. The three regions with the highest saliency are excluded to obtain the first non-saliency region of any video frame.

[0124] The Itti visual saliency model is a visual attention model designed based on the visual nervous system of early primates. This model first constructs Gaussian pyramids for the color, brightness, and orientation of an image using Gaussian sampling. Then, it uses these Gaussian pyramids to calculate the brightness, color, and orientation feature maps of the image. Finally, by combining these feature maps at different scales, the brightness, color, and orientation saliency maps are obtained, and summing them yields the final visual saliency map.

[0125] In step 504, a binary parity quantization modulation algorithm can be used to embed the L second watermark segments into the L video scenes to be processed one by one by modifying the first DC coefficient of each video frame in each video scene to be processed.

[0126] This embodiment improves watermark embedding efficiency by segmenting the video stream to be protected and the second watermark, and embedding the segmented second watermark fragments one-to-one into the segmented video scene to be processed. Furthermore, it can locate the attacked position when the watermark is attacked.

[0127] Figure 6 This is the sixth flowchart illustrating the multimedia copyright protection method provided in this application. (Refer to...) Figure 6 In one embodiment, obtaining a watermarked video stream after obtaining the watermarked multimedia may include:

[0128] 601. Decode and separate the watermarked multimedia to obtain the audio stream and video stream to be verified;

[0129] 602. Based on the histogram-local binary pattern scene classification algorithm, perform scene segmentation on the video stream to be verified to obtain L video scenes to be verified;

[0130] 603. For the first three video frames in any video scene to be verified, divide the regions according to the different salience of each region in any video frame, exclude the three regions with the highest salience, and obtain the second non-salience region of any video frame.

[0131] 604. Perform an 8*8 discrete cosine transform on the luminance component of the second non-significant region to obtain the second DC coefficient. Extract the watermark fragments on the first three video frames in any video scene to be verified using the second DC coefficients of the first three video frames respectively, until the extraction stops when the end flag is encountered.

[0132] 605. Sort the L video scenes to be verified in ascending order of time;

[0133] 606. Based on the order of sorting, select the second watermark segments to be verified from two adjacent video scenes to be verified.

[0134] 607. If the i-th second watermark segment to be verified in the l-th video scene to be verified is w l (i) The i-th second watermark fragment w in the (l-1)-th video scene to be verified l-1 (i) If they are not the same, then keep w. l (i) and w l-1 (i);

[0135] Where l is an integer greater than or equal to 2 and less than or equal to L, i is an integer greater than or equal to 1, and w l (i) is obtained from the watermark fragments on the first three video frames of the l-th video scene to be verified;

[0136] 608. If the i-th second watermark segment in the l-th video scene to be verified is wl (i) The i-th second watermark fragment w in the (l-1)-th video scene to be verified l-1 (i) If they are the same, then keep w. l (i), discard w l-1 (i);

[0137] 609. After replacing l with l+1, return to step 606 until l = L, and complete the selection of the second watermark segments to be verified for all adjacent video scenes to be verified.

[0138] 610. Merge all the retained second watermark fragments to be verified to obtain the second watermark to be verified;

[0139] 611. Obtain the trusted second watermark of the pre-stored watermarked multimedia from the blockchain. If the second watermark to be verified is inconsistent with the trusted second watermark, it is determined that the audio stream to be verified has been tampered with, and the tampering type and tampering location are located.

[0140] In step 602, the Itti visual saliency model can be used to perform saliency analysis on each frame of the L video scenes to be verified. For the first three video frames of any video scene to be verified, the regions are divided according to the different saliency of each region of any video frame. The three regions with the highest saliency are excluded to obtain the second non-saliency region of any video frame.

[0141] In step 604, the watermark fragment on any of the first three video frames of each video scene to be verified can be represented in encoded form as follows:

[0142]

[0143] Among them, w g (i) represents the i-th watermark fragment on the g-th video frame in the first three video frames of any video scene to be verified, where d g Let be the second DC coefficient on the g-th video frame, Δ be a specific constant, and floor be the floor sign for rounding down. Not greater than The largest integer.

[0144] In step 607, the i-th second watermark segment to be verified in the l-th video scene to be verified can be represented in encoded form as follows:

[0145]

[0146] In step 611, after obtaining the watermarked multimedia, a user-friendly interface is provided for users to fill in relevant copyright information, such as creator, creation time, and work description. Simultaneously, the watermarked multimedia is sent to a file server for storage, and address information is generated. Furthermore, based on the watermarked video and audio streams in the watermarked multimedia, the previously embedded first and second watermarks are regenerated. These first and second watermarks are considered reliable. The reliable first and second watermarks, the copyright information of the watermarked multimedia, and the address information of the watermarked multimedia are uploaded to the blockchain for storage, ensuring the security and immutability of the multimedia and effectively solving problems such as digital watermark attacks and damage.

[0147] If it is determined that the audio stream to be verified has been tampered with, the type and location of the tampering can be located based on the inconsistencies in the second watermark to be verified.

[0148] For example, if the length of the trusted second watermark is longer than the length of the second watermark to be verified, it is determined that the audio stream to be verified has been cut. Alternatively, the variances of the second watermark to be verified and the trusted second watermark can be reversed to be the variances at the time of encoding. The location where the variances of the two are deviated can be used to determine the location where the audio stream to be verified has been tampered with. In addition, if all the variances of the second watermark to be verified are less than the variances of the trusted second watermark after low-pass filtering, it is determined that the attack area of ​​the audio stream to be verified is the entire audio stream and the attack type is low-pass filtering.

[0149] In practical applications, there is no strict timing relationship between steps 607 and 608; that is, they can be executed simultaneously, or either step can be executed first, depending on the actual needs, and no restrictions are made here.

[0150] This embodiment can verify the integrity of the audio stream by comparing the extracted second watermark to be verified with the regenerated trusted second watermark.

[0151] Figure 7 This is the seventh flowchart illustrating the multimedia copyright protection method provided in this application. (Refer to...) Figure 7 In one embodiment, after obtaining a pre-stored trusted second watermark of the watermarked multimedia from the blockchain, the process includes:

[0152] 701. If the second watermark to be verified is consistent with the trusted second watermark, then it is determined that the audio stream to be verified has not been tampered with;

[0153] 702. Perform non-overlapping frame segmentation on the left channel data in the audio stream to be verified to obtain the audio frame data to be verified;

[0154] 703. Perform a three-level discrete wavelet transform on the audio frame data to be verified to obtain multiple second low-frequency coefficients;

[0155] 704. Sort the multiple second low-frequency coefficients and group them according to the sorting.

[0156] 705. Based on the relationship between the average value of the second low-frequency coefficient in any group and the last ranked second low-frequency coefficient in the previous group, extract the first watermark fragment to be verified in that group.

[0157] 706. Merge all the first watermark fragments to be verified to obtain the first watermark to be verified;

[0158] 707. Obtain the trusted first watermark of the pre-stored watermarked multimedia from the blockchain. If the first watermark to be verified is inconsistent with the trusted first watermark, it is determined that the video stream to be verified has been tampered with, and the tampering type and tampering location are located.

[0159] In step 704, multiple second low-frequency coefficients can be sorted in chronological order from first to last, and then grouped according to the sorting. When the total length of the multiple second low-frequency coefficients is 200 bits, the multiple second low-frequency coefficients are divided into 10 groups according to the sorting, with 20 coefficients in each group and each coefficient having a length of 1 bit.

[0160] In step 705, the first watermark fragment to be verified is extracted from the k-th group, which can be represented in encoded form as follows:

[0161]

[0162] Where W(k) is the watermark code extracted in the k-th group, i.e. the first watermark segment to be verified, DWT0(20×(k-1)) is the 20×(k-1)th second low-frequency coefficient, and temp_ave(k) is the average value of the second low-frequency coefficient in the k-th group.

[0163] After step 706, the right channel data in the audio stream data can be processed according to the same steps to obtain the complete first watermark to be verified.

[0164] In step 707, if it is determined that the video stream to be verified has been tampered with, the type and location of the tampering can be located based on the inconsistencies in the first watermark to be verified.

[0165] For example:

[0166] 1. Frame deletion attack:

[0167] Table 1. Experimental Results of Frame Deletion Attack

[0168]

[0169]

[0170] The frame deletion experiment results in Table 1 show that when the video stream to be verified is subjected to a frame deletion attack, it is possible to pinpoint whether a specific scene has been modified with 100% accuracy. For example, when frame 184 is deleted from the video stream to be verified, the encoding of bits 178 to 193 of the first watermark to be verified and the trusted first watermark is shown in the table below:

[0171] Table 2 Comparison of Watermark Removal Attack Information in Frame 2

[0172] First watermark to be verified 1 1 1 1 1 1 1 1 Trustworthy First Watermark 1 1 1 1 1 1 1 0 Watermark location 186 187 188 189 190 191 192 193 First watermark to be verified 0 0 0 0 0 0 0 0 Trustworthy First Watermark 0 0 0 0 0 0 0 0

[0173] Looking at the watermark information as a whole, the first watermark to be verified has 919 bits, while the trusted first watermark has 918 bits. This indicates that the video stream has been subjected to a frame deletion attack. Table 2 shows that the 185th watermark bit has changed, indicating that a frame of data has been deleted from the video stream in scene 184. Deleting multiple frames is similar to deleting a single frame; both methods can pinpoint the attacked scene.

[0174] 2. Frame-adding attack:

[0175] Frame insertion is similar to frame deletion. When an attack occurs, the difference in length between the regenerated trusted first watermark and the extracted first watermark to be verified can accurately detect whether an attack has occurred. For example, if a completely black image is inserted between frames 180 and 181, bits 178 to 193 of the first watermark to be verified and the trusted first watermark are shown in Table 3. Table 3 clearly shows that the video scene changes before and after frame 181, thus indicating that frame 181 is the inserted frame. Inserting multiple frames is similar to inserting a single frame; by comparing the two sets of watermarks, the location of the inserted frame can be accurately determined.

[0176] Table 3 Comparison of Attack Watermark Information Added to Frames

[0177] First watermark to be verified 1 1 1 1 1 1 1 1 Trustworthy First Watermark 1 1 0 1 1 1 1 1 Watermark location 186 187 188 189 190 191 192 193 First watermark to be verified 0 0 0 0 0 0 0 0 Trustworthy First Watermark 0 0 0 0 0 0 0 0

[0178] The detection process for frame addition attacks and frame deletion attacks is similar, and both can pinpoint the attack location with 100% accuracy.

[0179] 3. Frame modification attack:

[0180] Frame modification refers to altering video frames. Since frame modification is generally partial and does not affect the viewing experience, the experiment simulated the video frame modification process by changing the R and B values ​​of some pixels in the video frame of the video stream to zero, while leaving the G value unchanged. Different degrees of modification were applied to certain consecutive frames during the experiment, and the results are shown in the table below:

[0181] Table 4. Experimental Results of Frame Deletion Attack

[0182]

[0183]

[0184] Experimental results show that the algorithm can detect tampering with 100% accuracy when the degree of tampering reaches 16.7%. When a large number of frames are modified, and the degree of tampering reaches 16.7%, the algorithm can detect that the video has been attacked and can locate some of the attacked scenes.

[0185] This embodiment can verify the integrity of the video stream by comparing the extracted first watermark to be verified with the regenerated trusted first watermark.

[0186] The multimedia copyright protection device provided in the embodiments of this application is described below. The multimedia copyright protection device described below can be referred to in correspondence with the multimedia copyright protection method described above.

[0187] Figure 8 A schematic diagram of the multimedia copyright protection device provided in an embodiment of this application. (Refer to...) Figure 8 This application provides a multimedia copyright protection device, which may include:

[0188] The decoding and separation module 801 is used to: decode and separate the multimedia to be protected to obtain the audio stream and the video stream to be protected;

[0189] The first watermark generation module 802 is used to: extract scene features of the video stream to be protected according to the histogram-local binary mode scene classification algorithm, and obtain the first watermark according to the scene features;

[0190] The watermarked audio stream generation module 803 is used to: embed the first watermark into the audio stream to be protected to obtain a watermarked audio stream;

[0191] The second watermark generation module 804 is used to: generate a second watermark based on the watermarked audio stream;

[0192] The watermarked video stream generation module 805 is used to: embed the second watermark into the video stream to be protected to obtain a watermarked video stream;

[0193] The watermarked multimedia generation module 806 is used to: obtain watermarked multimedia based on the watermarked audio stream and the watermarked video stream.

[0194] The multimedia copyright protection device provided in this embodiment first decodes and separates the multimedia to be protected, obtaining the audio stream and video stream to be protected. Then, it extracts the scene features of the video stream to be protected according to the histogram-local binary mode scene classification algorithm, obtains a first watermark based on the scene features, embeds the first watermark into the audio stream to be protected, obtaining a watermarked audio stream, generates a second watermark based on the watermarked audio stream, embeds the second watermark into the video stream to be protected, obtaining a watermarked video stream, and finally obtains the watermarked multimedia based on the watermarked audio stream and the watermarked video stream. This embodiment utilizes the scene features of the video stream to be protected to obtain the first watermark, embeds the first watermark into the audio stream to be protected, generates the second watermark based on the audio stream to be protected with the first watermark embedded, and then embeds the second watermark into the video stream to be protected. It generates the first and second watermarks based on the global features of the audio and video, forming a nested process of watermark generation and embedding, which improves the sensitivity of the watermark to various tampering operations on the audio and video streams, thereby improving the integrity protection capability of the watermark.

[0195] In one embodiment, the first watermark generation module 802 is specifically used for:

[0196] The video stream to be protected is classified into scenes using a histogram scene classification algorithm to obtain the histogram inter-frame difference sum;

[0197] The video stream to be protected is classified into scenes using a local binary mode scene classification algorithm to obtain the inter-frame difference of texture maps.

[0198] The scene features of the video stream to be protected are extracted based on the harmonic mean of the sum of the inter-frame differences in the histogram and the sum of the inter-frame differences in the texture map.

[0199] In one embodiment, the watermarked audio stream generation module 803 is specifically used for:

[0200] Decode the audio stream to be protected to obtain audio stream data;

[0201] The left channel data in the audio stream is divided into non-overlapping frames to obtain the audio frame data to be processed.

[0202] Perform a three-level discrete wavelet transform on the audio frame data to be processed to obtain multiple first low-frequency coefficients;

[0203] The plurality of first low-frequency coefficients are sorted, and the plurality of first low-frequency coefficients are grouped according to the sorting. By modifying the last first low-frequency coefficient in the previous group, 1 bit of the first watermark data is embedded in each group to obtain a watermarked low-frequency coefficient group.

[0204] Merge all groups of watermarked low-frequency coefficients to obtain the watermarked low-frequency coefficient set;

[0205] The watermarked audio stream is obtained from the data after the inverse discrete wavelet transform of the watermarked low-frequency coefficient set.

[0206] In one embodiment, the second watermark generation module 804 is specifically used for:

[0207] Extract the one-dimensional MFCC coefficients of the watermarked audio stream, and divide the one-dimensional MFCC coefficients into multiple MFCC coefficient segments according to a specific length;

[0208] A corresponding digital watermark is generated based on the variance of each MFCC coefficient segment, and all digital watermarks are merged to obtain the second watermark.

[0209] In one embodiment, the watermarked video stream generation module 805 is specifically used for:

[0210] The video stream to be protected is segmented according to the scene characteristics to obtain L video scenes to be processed; where L is an integer greater than or equal to 2.

[0211] The second watermark is divided into L watermark segments, and an end marker is added at the end of each watermark segment;

[0212] For any video frame in the L video scenes to be processed, the regions are divided according to the different salience of each region of the video frame. The three regions with the highest salience are excluded to obtain the first non-salience region of the video frame.

[0213] An 8*8 discrete cosine transform is performed on the luminance component of the first non-significant region to obtain the first DC coefficient. By modifying the first DC coefficient of each video frame in each video scene to be processed, the L second watermark segments are embedded one-to-one into the L video scenes to be processed to obtain a set of watermarked video scenes.

[0214] Perform an inverse discrete cosine transform on the watermarked video scene set, and merge the watermarked video scene sets after the inverse transform to obtain the watermarked video stream.

[0215] In one embodiment, an audio integrity verification module (not shown in the figure) is further included, for:

[0216] Decode and separate the watermarked multimedia to obtain the audio stream and video stream to be verified;

[0217] The video stream to be verified is segmented into L video scenes based on the histogram-local binary pattern scene classification algorithm.

[0218] For the first three video frames in any video scene to be verified, the regions are divided according to the different salience of each region in any video frame. The three regions with the highest salience are excluded to obtain the second non-salience region of the video frame.

[0219] Perform an 8*8 discrete cosine transform on the luminance component of the second non-significant region to obtain the second DC coefficient. Extract the watermark fragments on the first three video frames in any video scene to be verified using the second DC coefficients, and stop extraction when the end flag is encountered.

[0220] The L video scenes to be verified are sorted in chronological order.

[0221] Based on the order of sorting, the second watermark segments to be verified in two adjacent video scenes to be verified are selected and discarded.

[0222] If the i-th second watermark fragment in the l-th video scene to be verified is w l (i) The i-th second watermark fragment w in the (l-1)-th video scene to be verified l-1 (i) If they are not the same, then keep w. l (i) and w l-1 (i); where l is an integer greater than or equal to 2 and less than or equal to L, i is an integer greater than or equal to 1, and w l (i) is obtained from the watermark fragments on the first three video frames of the l-th video scene to be verified;

[0223] If the i-th second watermark fragment in the l-th video scene to be verified is w l (i) The i-th second watermark fragment w in the (l-1)-th video scene to be verified l-1 (i) If they are the same, then keep w. l (i), discard w l-1 (i);

[0224] After replacing l with l+1, return to the step of selecting the second watermark segments of two adjacent video scenes to be verified in order of sorting, until l=L, and complete the selection of the second watermark segments of all adjacent video scenes to be verified.

[0225] All the retained second watermark fragments to be verified are merged to obtain the second watermark to be verified;

[0226] The system retrieves a pre-stored trusted second watermark of the watermarked multimedia from the blockchain. If the second watermark to be verified is inconsistent with the trusted second watermark, it determines that the audio stream to be verified has been tampered with, and identifies the type and location of the tampering.

[0227] In one embodiment, a video integrity verification module (not shown in the figure) is further included, for:

[0228] If the second watermark to be verified is consistent with the trusted second watermark, then it is determined that the audio stream to be verified has not been tampered with;

[0229] The left channel data in the audio stream to be verified is divided into non-overlapping frames to obtain the audio frame data to be verified.

[0230] Perform a three-level discrete wavelet transform on the audio frame data to be verified to obtain multiple second low-frequency coefficients;

[0231] The plurality of second low-frequency coefficients are sorted, and the plurality of second low-frequency coefficients are grouped according to the sorting.

[0232] Based on the relationship between the average value of the second low-frequency coefficients in any group and the last sorted second low-frequency coefficient in the group preceding the group, extract the first watermark fragment to be verified in any group.

[0233] Merge all the first watermark fragments to be verified to obtain the first watermark to be verified;

[0234] The system retrieves a pre-stored trusted first watermark of the watermarked multimedia from the blockchain. If the first watermark to be verified is inconsistent with the trusted first watermark, it determines that the video stream to be verified has been tampered with, and identifies the type and location of the tampering.

[0235] Figure 9 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 9 As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communication interface 920, and the memory 930 communicate with each other via the communication bus 940. The processor 910 can call a computer program stored in the memory 930 to execute steps of a multimedia copyright protection method, such as including:

[0236] Decode and separate the multimedia to be protected to obtain the audio stream and the video stream to be protected;

[0237] The scene features of the video stream to be protected are extracted according to the histogram-local binary pattern scene classification algorithm, and the first watermark is obtained according to the scene features;

[0238] The first watermark is embedded into the audio stream to be protected to obtain a watermarked audio stream;

[0239] A second watermark is generated based on the watermarked audio stream;

[0240] The second watermark is embedded into the video stream to be protected to obtain a watermarked video stream;

[0241] Watermarked multimedia is obtained from the watermarked audio stream and the watermarked video stream.

[0242] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0243] On the other hand, embodiments of this application also provide a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the steps of the multimedia copyright protection method provided in the above embodiments, such as including:

[0244] Decode and separate the multimedia to be protected to obtain the audio stream and the video stream to be protected;

[0245] The scene features of the video stream to be protected are extracted according to the histogram-local binary pattern scene classification algorithm, and the first watermark is obtained according to the scene features;

[0246] The first watermark is embedded into the audio stream to be protected to obtain a watermarked audio stream;

[0247] A second watermark is generated based on the watermarked audio stream;

[0248] The second watermark is embedded into the video stream to be protected to obtain a watermarked video stream;

[0249] Watermarked multimedia is obtained from the watermarked audio stream and the watermarked video stream.

[0250] On the other hand, embodiments of this application also provide a processor-readable storage medium storing a computer program for causing a processor to perform the steps of the methods provided in the above embodiments, such as including:

[0251] Decode and separate the multimedia to be protected to obtain the audio stream and the video stream to be protected;

[0252] The scene features of the video stream to be protected are extracted according to the histogram-local binary pattern scene classification algorithm, and the first watermark is obtained according to the scene features;

[0253] The first watermark is embedded into the audio stream to be protected to obtain a watermarked audio stream;

[0254] A second watermark is generated based on the watermarked audio stream;

[0255] The second watermark is embedded into the video stream to be protected to obtain a watermarked video stream;

[0256] Watermarked multimedia is obtained from the watermarked audio stream and the watermarked video stream.

[0257] The processor-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical memory (e.g., CD, DVD, BD, HVD), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0258] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0259] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0260] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A multimedia copyright protection method, characterized in that, include: Decode and separate the multimedia to be protected to obtain the audio stream and the video stream to be protected; The scene features of the video stream to be protected are extracted according to the histogram-local binary mode scene classification algorithm. After the scene features are scrambled, the first watermark is obtained based on the scrambled scene features. The first watermark is embedded into the audio stream to be protected to obtain a watermarked audio stream; A second watermark is generated based on the watermarked audio stream; Embedding the second watermark into the video stream to be protected to obtain a watermarked video stream includes: The video stream to be protected is segmented according to the scene characteristics to obtain L video scenes to be processed; where L is an integer greater than or equal to 2. The second watermark is divided into L watermark segments, and an end marker is added at the end of each watermark segment; For any video frame in the L video scenes to be processed, the regions are divided according to the different salience of each region of the video frame. The three regions with the highest salience are excluded to obtain the first non-salience region of the video frame. The luminance component of the first non-significant region is subjected to 8 The discrete cosine transform of 8 is used to obtain the first DC coefficient. By modifying the first DC coefficient of each video frame in each video scene to be processed, the L second watermark segments are embedded one-to-one into the L video scenes to be processed, thus obtaining a set of watermarked video scenes. Perform an inverse discrete cosine transform on the watermarked video scene set, and merge the watermarked video scene sets after the inverse transform to obtain a watermarked video stream. Watermarked multimedia is obtained from the watermarked audio stream and the watermarked video stream.

2. The multimedia copyright protection method according to claim 1, characterized in that, The step of extracting scene features from the video stream to be protected based on the histogram-local binary pattern scene classification algorithm includes: The video stream to be protected is classified into scenes using a histogram scene classification algorithm to obtain the histogram inter-frame difference sum; The video stream to be protected is classified into scenes using a local binary mode scene classification algorithm to obtain the inter-frame difference of texture maps. The scene features of the video stream to be protected are extracted based on the harmonic mean of the sum of the inter-frame differences in the histogram and the sum of the inter-frame differences in the texture map.

3. The multimedia copyright protection method according to claim 1, characterized in that, The step of embedding the first watermark into the audio stream to be protected to obtain a watermarked audio stream includes: Decode the audio stream to be protected to obtain audio stream data; The left channel data in the audio stream is divided into non-overlapping frames to obtain the audio frame data to be processed. Perform a three-level discrete wavelet transform on the audio frame data to be processed to obtain multiple first low-frequency coefficients; The plurality of first low-frequency coefficients are sorted, and the plurality of first low-frequency coefficients are grouped according to the sorting. By modifying the last first low-frequency coefficient in the previous group, 1 bit of the first watermark data is embedded in each group to obtain a watermarked low-frequency coefficient group. Merge all groups of watermarked low-frequency coefficients to obtain the watermarked low-frequency coefficient set; The watermarked audio stream is obtained from the data after the inverse discrete wavelet transform of the watermarked low-frequency coefficient set.

4. The multimedia copyright protection method according to claim 1, characterized in that, The step of generating a second watermark based on the watermarked audio stream includes: Extract the one-dimensional MFCC coefficients of the watermarked audio stream, and divide the one-dimensional MFCC coefficients into multiple MFCC coefficient segments according to a specific length; A corresponding digital watermark is generated based on the variance of each MFCC coefficient segment, and all digital watermarks are merged to obtain the second watermark.

5. The multimedia copyright protection method according to claim 1, characterized in that, After obtaining the watermarked multimedia, the process includes: Decode and separate the watermarked multimedia to obtain the audio stream and video stream to be verified; The video stream to be verified is segmented into L video scenes based on the histogram-local binary pattern scene classification algorithm. For the first three video frames in any video scene to be verified, the regions are divided according to the different salience of each region in any video frame. The three regions with the highest salience are excluded to obtain the second non-salience region of the video frame. The luminance component of the second insignificant region is subjected to 8 The discrete cosine transform of 8 is used to obtain the second DC coefficient. The watermark fragments on the first three video frames are extracted by the second DC coefficients of the first three video frames in any video scene to be verified, until the extraction stops when the end flag is encountered. The L video scenes to be verified are sorted in chronological order. Based on the order of sorting, the second watermark segments to be verified in two adjacent video scenes to be verified are selected and discarded. If the first l The first video scene to be verified i A second watermark fragment awaiting verification. With the l -1 of the video scenes to be verified i A second watermark fragment awaiting verification. If they are different, retain them. and ;in, l It is an integer greater than or equal to 2 and less than or equal to L. i For integers greater than or equal to 1 It is based on the first l The watermark is obtained from the first three video frames of the video scene to be verified. If the first l The first video scene to be verified i A second watermark fragment awaiting verification. With the l -1 of the video scenes to be verified i A second watermark fragment awaiting verification. If they are the same, keep them. ,give up ; use l+ 1 replacement l Then, return to the step of selecting the second watermark segments of two adjacent video scenes to be verified in order of priority, until... At that time, the selection of the second watermark segments to be verified for all two adjacent video scenes to be verified is completed; All the retained second watermark fragments to be verified are merged to obtain the second watermark to be verified; The system retrieves a pre-stored trusted second watermark of the watermarked multimedia from the blockchain. If the second watermark to be verified is inconsistent with the trusted second watermark, it determines that the audio stream to be verified has been tampered with, and identifies the type and location of the tampering.

6. The multimedia copyright protection method according to claim 5, characterized in that, After obtaining the pre-stored trusted second watermark of the watermarked multimedia from the blockchain, the process includes: If the second watermark to be verified is consistent with the trusted second watermark, then it is determined that the audio stream to be verified has not been tampered with; The left channel data in the audio stream to be verified is divided into non-overlapping frames to obtain the audio frame data to be verified. Perform a three-level discrete wavelet transform on the audio frame data to be verified to obtain multiple second low-frequency coefficients; The plurality of second low-frequency coefficients are sorted, and the plurality of second low-frequency coefficients are grouped according to the sorting. Based on the relationship between the average value of the second low-frequency coefficients in any group and the last sorted second low-frequency coefficient in the group preceding the group, extract the first watermark fragment to be verified in any group. Merge all the first watermark fragments to be verified to obtain the first watermark to be verified; The system retrieves a pre-stored trusted first watermark of the watermarked multimedia from the blockchain. If the first watermark to be verified is inconsistent with the trusted first watermark, it determines that the video stream to be verified has been tampered with, and identifies the type and location of the tampering.

7. A multimedia copyright protection device, characterized in that, The method for performing the multimedia copyright protection method of claim 1 includes: The decoding and separation module is used to decode and separate the multimedia to be protected, obtaining the audio stream and the video stream to be protected. The first watermark generation module is used to: extract scene features of the video stream to be protected according to the histogram-local binary mode scene classification algorithm, and obtain the first watermark according to the scene features; A watermarked audio stream generation module is used to: embed the first watermark into the audio stream to be protected to obtain a watermarked audio stream; The second watermark generation module is used to: generate a second watermark based on the watermarked audio stream; A watermarked video stream generation module is used to: embed the second watermark into the video stream to be protected to obtain a watermarked video stream; A watermarked multimedia generation module is used to: obtain watermarked multimedia based on the watermarked audio stream and the watermarked video stream.

8. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the multimedia copyright protection method according to any one of claims 1 to 6.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the multimedia copyright protection method according to any one of claims 1 to 6.

Images