Dataset copyright auditing method and device for video recognition system

Abstract

The application discloses a kind of video recognition system's data set copyright audit method and device. First, an evaluation model is trained based on original video data set, and the output probability of original video is obtained based on it;Then, a modified data set is obtained by slightly modifying the original video data set, and the output probability of the modified video is obtained by inputting it into the evaluation model;After that, the difference between the two output probabilities is compared to select a specified number of samples, and then a published data set and an unpublished data set are obtained;Finally, by comparing the behavior differences of the suspect model on the published data set and the unpublished data set, the copyright audit of the video recognition data set is realized.

Description

Technical Field

[0001] This invention relates to the technical field of data security, and in particular to a method and apparatus for copyright auditing of datasets for video recognition systems. Background Technology

[0002] Video recognition systems are increasingly widely used in real-world applications. For example, by identifying activity in surveillance systems, they can effectively determine whether abnormal situations have occurred and provide users with timely warnings. To promote video recognition research, research institutions release a series of high-quality datasets. However, these video datasets are released under strict open-source licenses, designed to protect the intellectual property rights of data owners. Unauthorized commercial use can easily violate these license terms and may lead to serious legal and ethical issues. Therefore, protecting the copyright of video recognition datasets is crucial.

[0003] Existing research on copyright protection for datasets primarily focuses on the image domain, and these methods are not effectively applicable to video data. On one hand, video data has an additional temporal dimension compared to image data; on the other hand, neural network models for video recognition exhibit greater robustness. Therefore, designing an effective dataset copyright auditing scheme tailored to the characteristics of video recognition datasets is a pressing issue that needs to be addressed. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing dataset copyright auditing methods that do not take into account the characteristics of video data, and to provide a dataset copyright auditing method and apparatus for video recognition systems.

[0005] The objective of this invention is achieved through the following technical solution: In a first aspect, the present invention provides a data set copyright auditing method for video recognition systems, the method comprising: Step 1: Train an evaluation model based on the original video dataset, and then obtain the posterior probability of the original video data based on the evaluation model. Step 2: Modify each video in the original video dataset to obtain the modified video dataset, and obtain the posterior probability corresponding to the modified video dataset by evaluating the model query. Step 3: For each video in the video dataset, calculate the difference between the posterior probability of the original video and the ground truth label corresponding to its modified video. Then, determine the candidate set based on this. By sampling the candidate set, obtain the modified set and the reference set, and further obtain the published video dataset and the unpublished video dataset. This includes the following sub-steps: 3.1 For each video in the video dataset, calculate and record the difference between the posterior probability of the original video and the corresponding ground truth label of the modified video; 3.2 Sort all the differences in descending order, and then select the top-ranked video pairs as the candidate set; 3.3 Perform non-repeating random sampling of video pairs from the candidate set and select... Each video pair was used as a modification set. Each video pair serves as a reference set; 3.4 The published video dataset consists of modified videos from the modification set and other original videos, while the unpublished video dataset consists of original videos from the modification set and other modified videos. Step four involves querying the suspected model using both published and unpublished video datasets, and then performing copyright auditing on the video recognition dataset based on the behavioral differences of the suspected model. This includes the following sub-steps: 4.1 Input all video pairs from the reference set into the suspect model for querying. Calculate the probability output difference between the original video and the modified video for the true label in each video pair and record the result to obtain a probability sequence. Based on this, an upper limit threshold is calculated. ; 4.2 Input all video pairs in the modified set into the suspect model for querying. Calculate the difference in probability output between the original video and the modified video for the true label in each video pair, and then perform post-processing to obtain the probability sequence. ; 4.3 Combining upper limit threshold For probability sequences Perform hypothesis testing to determine whether the video recognition dataset has been stolen.

[0006] Furthermore, in step one, when training the evaluation model, the evaluation model is a neural network architecture. The input of the model is preprocessed video frames, and the output of the model is a set of vectors, where the value corresponding to the real video label is 1, and the value corresponding to the other labels is 0.

[0007] Furthermore, in step two, procedural noise is injected into each frame of the video to obtain a modified video dataset.

[0008] Further, in step 4.1, the upper limit threshold is calculated. The process is as follows: First, the mean of the probability sequence is calculated as follows: Next, the range of the obtained mean is restricted as follows: In the formula The function represents the The range is limited to the minimum value To the maximum value between.

[0009] Furthermore, in step 4.2, the post-processing procedure is as follows: If the probability output values ​​of both the original video and the modified video for the true label are less than... , If the probability difference between video pairs is the reciprocal of the number of label categories in the original video dataset, then the probability difference between the video pairs is modified to... ,in The adjustment coefficient is greater than 0.

[0010] Furthermore, in step 4.3, the hypothesis test performed is a Wilcoxon signed-rank test, and the null hypothesis... for Alternative hypothesis for .

[0011] Secondly, the present invention provides a dataset copyright auditing device for video recognition systems, comprising: Model training module: Trains an evaluation model based on the original video dataset; Video editing module: Injects noise into each frame of the original video to obtain the modified video dataset; Probability calculation module: Based on the trained evaluation model, calculates the output probabilities of the original video and the modified video on the ground truth labels, and calculates the difference between them; Video determination module: Sort the probability differences in descending order, then perform non-repeating random sampling from the top-ranked video set to select the modification set and the reference set. The published video dataset consists of the modified videos from the modification set and other original videos. Probability determination module: By querying the probability output of the suspected model on the modified set and the reference set, and then performing the Wilcoxon signed-rank test, it determines whether the video recognition dataset to be audited has been stolen.

[0012] Thirdly, the present invention provides a dataset copyright auditing device for a video recognition system, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the dataset copyright auditing method for a video recognition system as described in the first aspect.

[0013] Fourthly, the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the dataset copyright auditing method for video recognition systems as described in the first aspect.

[0014] The beneficial effects of this invention are as follows: 1. By injecting noise into the original video and selecting from it using an evaluation model, video pairs with significant behavioral differences can be effectively selected while making minor modifications; 2. By querying the probability output of the suspected model on the modified set and the reference set, and performing a signed-rank test, it is possible to effectively determine whether the dataset has been stolen; 3. Based on the characteristics of video data, a dataset copyright auditing method for video recognition systems is designed to improve auditing accuracy while minimizing the degree of modification to the original video. Attached Figure Description

[0015] Figure 1 A flowchart of a data set copyright auditing method for a video recognition system provided in this embodiment of the invention; Figure 2 This is an example of the probability output of an evaluation model for an original video set provided in an embodiment of the present invention; Figure 3 This is an example of modifying the probability output of an evaluation model for a video set, as provided in an embodiment of the present invention. Figure 4 This is a structural block diagram of a data set copyright auditing device for a video recognition system, provided as an embodiment of the present invention. Detailed Implementation

[0016] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0017] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0018] like Figure 1 As shown in the figure, this embodiment provides a data set copyright auditing method for video recognition systems, with the following steps: Step 1: Train an evaluation model based on the original video dataset, and obtain the posterior probability of the original video data based on this evaluation model. This includes the following sub-steps: 1.1 An evaluation model is obtained by training the original video dataset; The specific implementation details are as follows: When training the evaluation model, the evaluation model is a neural network architecture, such as the I3D model. The input of the model is preprocessed video frames, and the output of the model is a set of vectors (the value corresponding to the real video label is 1, and the value corresponding to other labels is 0).

[0019] 1.2 Input the raw video data into the trained evaluation model for querying, and record the posterior probability value corresponding to the true label of the raw video, for example... Figure 2 As shown, the original video 1 is labeled as "running". When this video is input into the evaluation model, the posterior probability of "running" is 0.87. Step two: Modify each video in the original video dataset (the modification of each pixel should not exceed a certain value). ,For example To obtain a modified video dataset, the model queries are used to obtain the posterior probability corresponding to the modified video dataset, including the following sub-steps: 2.1 For each video in the original video dataset, noise is injected into each frame to form modified video data; The specific implementation details are as follows: The noise injected into each frame of the video is procedural noise (such as Burmester noise).

[0020] 2.2 Input the modified video data into the evaluation model for querying, and record the posterior probability of the corresponding ground truth label for the modified video, for example... Figure 3 As shown, modifying the input of video 1 into the evaluation model query yields a posterior probability of 0.17 for running; Step 3: For each video in the video dataset, calculate the difference between the posterior probability of the original video and the ground truth label corresponding to its modified video. Then, determine the candidate set based on this. By sampling the candidate set, obtain the modified set and the reference set, and further obtain the published video dataset and the unpublished video dataset. This includes the following sub-steps: 3.1 For each video in the video dataset, calculate and record the difference between the posterior probability of the original video and the corresponding ground truth label of the modified video. For example, based on... Figure 2 and Figure 3 The difference between video pair 1 and video pair 1 can be calculated as 0.87 - 0.17 = 0.70. 3.2 Sort all differences in descending order, then select the one with the highest rank. % of video pairs (including the original video and the corresponding modified video) are used as the candidate set, for example, based on Figure 2 and Figure 3 There are a total of 1000 video pairs, set up 100 video pairs were selected as the candidate set; 3.3 Perform non-repeating random sampling of video pairs from the candidate set and select... Each video pair was used as a modification set. Each video pair serves as a reference set. The modification set and the reference set are mutually exclusive; for example, setting... and This means that a total of 40 video pairs were sampled in the candidate set, with 20 video pairs located in the modification set and the other 20 video pairs located in the reference set; 3.4 The published video dataset consists of modified videos from the modification set and other original videos (the result of removing the original videos from the modification set from the original video dataset). The unpublished video dataset consists of the original videos from the modification set and other modified videos (the result of removing the modified videos from the modification set from the modified video dataset). The published video dataset is public, while the unpublished video dataset is private. Step four involves querying the suspected model using both published and unpublished video datasets, and then performing copyright auditing on the video recognition dataset based on the behavioral differences of the suspected model. This includes the following sub-steps: 4.1 Input all video pairs from the reference set into the suspect model for querying. Calculate the probability output difference between the original video and the modified video for the true label in each video pair and record the result to obtain a probability sequence. Based on this, an upper limit threshold is calculated. ; Calculate the upper limit threshold The specific implementation details are as follows: First, the mean of the probability sequence is calculated as follows: In the formula, the sum function represents the summation function. Then, the range of the obtained mean is restricted as follows: In the formula The function represents the The range is limited to the minimum value To the maximum value Between, for example, setting and ,For example Output , Then output .

[0021] 4.2 Input all video pairs in the modified set into the suspect model for querying. Calculate the difference in probability output between the original video and the modified video for the true label in each video pair, and then perform post-processing to obtain the probability sequence. ; The specific implementation details of the post-processing are as follows: If the probability output values ​​of both the original video and the modified video for the true label are less than... ( If the number of label categories in the original video dataset is the reciprocal, then the probability difference of the video pair is modified to... ,in An adjustment coefficient greater than 0, for example, set .

[0022] 4.3 Combining upper limit threshold For probability sequences Perform hypothesis testing to determine whether the video recognition dataset has been stolen.

[0023] The specific implementation details are as follows: The hypothesis test performed was the Wilcoxon signed-rank test, and the null hypothesis was... for Alternative hypothesis for An example of the test is as follows: Step 1, Calculation Each value in the sequence relative to the upper threshold Sign differences: In the formula express The first of the sequence One value; The second step is to All samples were excluded; The third step is to calculate the absolute value of the remaining non-zero values ​​in the sequence. And assign corresponding ranks based on these absolute values ​​(or average rank if there are ties). The smallest absolute value has a rank of 1, the next smallest is 2, and so on. Fourth step, according to The sign is restored to the signed rank for each rank, and the positive rank sum is calculated as follows: Fifth step, use statistics Calculate the one-sided p-value. If the one-sided p-value is less than the significance level (e.g., 0.01), reject the null hypothesis. If the copyright of the dataset is stolen, it is determined that the dataset copyright has been stolen; otherwise, it is considered that the dataset copyright has not been stolen.

[0024] Another aspect of the present invention discloses a dataset copyright auditing device for video recognition systems, such as... Figure 4 As shown, the device includes the following modules: Model training module: Trains an evaluation model based on the original video dataset; Video modification module: Noise is injected into each frame of the original video to obtain a modified video recognition dataset; Probability calculation module: Based on the trained evaluation model, calculates the output probabilities of the original video and the modified video on the ground truth labels, and calculates the difference between them; Video determination module: Sort the probability differences in descending order, and then randomly sample and select the modification set and reference set from the top-ranked video set. The published video dataset consists of the modified videos of the modification set and other original videos (the original videos of the modification set are removed from all the original videos). Probability determination module: By querying the probability output of the suspected model on the modified set and the reference set, and then performing the Wilcoxon signed-rank test, it determines whether the video recognition dataset to be audited has been stolen.

[0025] The present invention also discloses a dataset copyright auditing device for video recognition systems, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements the dataset copyright auditing method for video recognition systems as described above.

[0026] The present invention also discloses a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the above-described data set copyright auditing method for video recognition systems.

[0027] The above description is merely a preferred embodiment of the present invention. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make many possible variations and modifications to the technical solutions of the present invention using the methods and techniques disclosed above, or modify them into equivalent embodiments with equivalent changes, without departing from the scope of the technical solutions of the present invention. Therefore, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall still fall within the protection scope of the technical solutions of the present invention.

Claims

1. A method for copyright auditing of datasets for video recognition systems, characterized in that, The method includes: Step 1: Train an evaluation model based on the original video dataset, and then obtain the posterior probability of the original video data based on the evaluation model. Step 2: Modify each video in the original video dataset to obtain the modified video dataset, and obtain the posterior probability corresponding to the modified video dataset by evaluating the model query. Step 3: For each video in the video dataset, calculate the difference between the posterior probability of the original video and the ground truth label corresponding to its modified video. Then, determine the candidate set based on this. By sampling the candidate set, obtain the modified set and the reference set, and further obtain the published video dataset and the unpublished video dataset. This includes the following sub-steps: 3.1 For each video in the video dataset, calculate and record the difference between the posterior probability of the original video and the corresponding ground truth label of the modified video; 3.2 Sort all the differences in descending order, and then select the top-ranked video pairs as the candidate set; 3.3 Perform non-repeating random sampling of video pairs from the candidate set and select... Each video pair was used as a modification set. Each video pair serves as a reference set; 3.4 The published video dataset consists of modified videos from the modification set and other original videos, while the unpublished video dataset consists of original videos from the modification set and other modified videos. Step four involves querying the suspected model using both published and unpublished video datasets, and then performing copyright auditing on the video recognition dataset based on the behavioral differences of the suspected model. This includes the following sub-steps: 4.1 Input all video pairs from the reference set into the suspect model for querying. Calculate the probability output difference between the original video and the modified video for the true label in each video pair and record the result to obtain a probability sequence. Based on this, an upper limit threshold is calculated. ; 4.2 Input all video pairs in the modified set into the suspect model for querying. Calculate the difference in probability output between the original video and the modified video for the true label in each video pair, and then perform post-processing to obtain the probability sequence. ; 4.3 Combining upper limit threshold For probability sequences Perform hypothesis testing to determine whether the video recognition dataset has been stolen.

2. The data set copyright auditing method for video recognition systems according to claim 1, characterized in that, In step one, when training the evaluation model, the evaluation model is a neural network architecture. The input of the model is preprocessed video frames, and the output of the model is a set of vectors. The value corresponding to the real video label is 1, and the value corresponding to the other labels is 0.

3. The data set copyright auditing method for video recognition systems according to claim 1, characterized in that, In step two, procedural noise is injected into each frame of the video to obtain a modified video dataset.

4. The data set copyright auditing method for video recognition systems according to claim 1, characterized in that, In step 4.1, the upper limit threshold is calculated. The process is as follows: First, the mean of the probability sequence is calculated as follows: Next, the range of the obtained mean is restricted as follows: In the formula The function represents the The range is limited to the minimum value To the maximum value between.

5. The data set copyright auditing method for video recognition systems according to claim 1, characterized in that, In step 4.2, the post-processing procedure is as follows: If the probability output values ​​of both the original video and the modified video for the true label are less than... , If the probability difference between video pairs is the reciprocal of the number of label categories in the original video dataset, then the probability difference between the video pairs is modified to... ,in The adjustment coefficient is greater than 0.

6. The data set copyright auditing method for video recognition systems according to claim 1, characterized in that, In step 4.3, the hypothesis test performed is the Wilcoxon signed-rank test, and the null hypothesis is... for Alternative hypothesis for .

7. A dataset copyright auditing device for a video recognition system implemented using the method described in any one of claims 1-6, characterized in that, Includes the following modules: Model training module: Trains an evaluation model based on the original video dataset; Video modification module: Noise is injected into each frame of the original video to obtain the modified video dataset; Probability calculation module: Based on the trained evaluation model, calculates the output probabilities of the original video and the modified video on the ground truth labels, and calculates the difference between them; Video determination module: Sort the probability differences in descending order, then perform non-repeating random sampling from the top-ranked video set to select the modification set and the reference set. The published video dataset consists of the modified videos from the modification set and other original videos. Probability determination module: By querying the probability output of the suspected model on the modified set and the reference set, and then performing the Wilcoxon signed-rank test, it determines whether the video recognition dataset to be audited has been stolen.

8. A dataset copyright auditing device for a video recognition system, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that... When the processor executes the executable code, it implements the dataset copyright auditing method for video recognition systems as described in any one of claims 1-6.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the data set copyright auditing method for video recognition systems as described in any one of claims 1-6.

Images