Video replacement detection and model training method and device

Abstract

One or more embodiments of the present specification provide a method and device for video replacement detection and its model training. The method includes: extracting multiple frames of images from a video to be detected, and extracting an image feature vector of each frame of the image; generating a video feature vector corresponding to the video to be detected, wherein the video feature vector is related to a fusion result of the image feature vectors corresponding to the multiple frames of the image; and determining whether the video to be detected has been replaced according to the video feature vector.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of terminal technology, and more particularly, to a method and device for video replacement detection and model training thereof. Background Art

[0002] As the threshold for using video replacement algorithms and tools such as Deepfake decreases, more and more replaced fake videos and fake news are circulating on the Internet, and the demand for video replacement detection technology is increasing.

[0003] In the prior art, video replacement detection is generally performed with the help of a classifier. Specifically, each frame image extracted from the video to be detected is input into the classifier for detection, and the detection results of each frame image are weighted fused. According to the fusion result, it is determined whether the video to be detected has been replaced. Summary of the invention

[0004] In view of this, one or more embodiments of the present specification provide a method and device for video replacement detection and model training.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of this specification, a video replacement detection method is proposed, including:

[0007] Extract multiple frames of images from the video to be detected, and extract the image feature vector of each frame of image;

[0008] Generate a video feature vector corresponding to the video to be detected, wherein the video feature vector is related to a fusion result of image feature vectors corresponding to the multiple frames of images;

[0009] Determine whether the video to be detected has been replaced according to the video feature vector.

[0010] According to a second aspect of one or more embodiments of this specification, a model training method for video replacement detection is proposed, including:

[0011] Obtain multiple video samples, each of which contains replacement annotation information;

[0012] Extracting multiple frames of images from each video sample, and generating a video feature vector corresponding to each video sample, wherein the video feature vector is related to a fusion result of image feature vectors corresponding to the multiple frames of images;

[0013] The video feature vector is input into a classifier for training, and the parameters of the classifier are adjusted according to the replacement annotation information and the output result of the classifier.

[0014] According to a third aspect of one or more embodiments of this specification, a model training method for video replacement detection is provided, wherein the model comprises a cascaded feature extraction sub-model, a feature fusion sub-model, and a detection sub-model;

[0015] The method comprises:

[0016] Inputting multiple frames of images extracted from each video sample into the feature extraction submodel for feature extraction, wherein the video sample includes replacement annotation information;

[0017] Inputting the image feature vector of each frame image output by the feature extraction sub-model into the feature fusion sub-model to perform feature vector fusion;

[0018] Inputting the video feature vectors corresponding to each video sample output by the feature fusion sub-model into the detection sub-model;

[0019] The model parameters of the feature extraction sub-model, the feature fusion sub-model and the detection sub-model are adjusted according to the replacement annotation information and the output result of the detection sub-model.

[0020] According to a fourth aspect of one or more embodiments of this specification, a video replacement detection device is provided, including:

[0021] An extraction module is used to extract multiple frames of images from the video to be detected and extract the image feature vector of each frame of image;

[0022] A generating module, used for generating a video feature vector corresponding to the video to be detected, wherein the video feature vector is related to a fusion result of the image feature vectors corresponding to the multiple frames of images;

[0023] A determination module is used to determine whether the video to be detected has been replaced according to the video feature vector.

[0024] According to a fifth aspect of one or more embodiments of this specification, a model training device for video replacement detection is provided, including:

[0025] An acquisition module is used to acquire multiple video samples, each of which contains replacement annotation information;

[0026] A generating module, used for extracting multiple frames of images from each video sample, and generating a video feature vector corresponding to each video sample, wherein the video feature vector is related to a fusion result of the image feature vectors corresponding to the multiple frames of images;

[0027] The training module is used to input the video feature vector into a classifier for training, and adjust the parameters of the classifier according to the replacement annotation information and the output result of the classifier.

[0028] According to a sixth aspect of one or more embodiments of this specification, there is provided a model training device for video replacement detection, wherein the model comprises a cascaded feature extraction sub-model, a feature fusion sub-model and a detection sub-model;

[0029] The device comprises:

[0030] An extraction module, used for inputting multiple frames of images extracted from each video sample into the feature extraction sub-model for feature extraction, wherein the video sample includes replacement annotation information;

[0031] An input module, used for inputting the image feature vector of each frame image output by the feature extraction sub-model into the feature fusion sub-model for feature vector fusion;

[0032] The input module is further used to input the video feature vectors corresponding to each video sample output by the feature fusion sub-model into the detection sub-model;

[0033] An adjustment module is used to adjust the model parameters of the feature extraction sub-model, the feature fusion sub-model and the detection sub-model according to the replacement annotation information and the output result of the detection sub-model.

[0034] According to a seventh aspect of one or more embodiments of this specification, an electronic device is provided, including:

[0035] Replace the detector;

[0036] a memory for storing replacement detector executable instructions;

[0037] The replacement detector implements any of the above methods by running the executable instructions.

[0038] According to an eighth aspect of one or more embodiments of the present specification, there is provided a computer-readable storage medium having computer instructions stored thereon, which, when executed by a replacement detector, implement the steps of any of the above-described methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the architecture of a video replacement detection system provided by an exemplary embodiment.

[0040] Figure 2 The present invention is a flowchart of a video replacement detection method provided by an exemplary embodiment.

[0041] Figure 3a A flowchart of generating a video feature vector corresponding to a video to be detected is provided by an exemplary embodiment.

[0042] Figure 3b An exemplary embodiment provides a Figure 3a Flowchart of the convolution process.

[0043] Figure 4a It is another flowchart of fusing image feature vectors corresponding to each frame image provided by an exemplary embodiment.

[0044] Figure 4b A flowchart of fusing temporal feature vectors corresponding to each image grouping is provided by an exemplary embodiment.

[0045] Figure 4c An exemplary embodiment provides a Figure 4b Flowchart of the convolution process.

[0046] Figure 5 It is a flowchart of a model training method for video replacement detection provided by an exemplary embodiment.

[0047] Figure 6 is a flowchart of another model training method for video replacement detection provided by an exemplary embodiment

[0048] Figure 7 It is a structural schematic diagram of a device provided by an exemplary embodiment.

[0049] Figure 8 It is a block diagram of a video replacement detection device provided by an exemplary embodiment.

[0050] Fig. 9 This is a schematic diagram of another device provided by an exemplary embodiment.

[0051] Fig.10 A model training device for video replacement detection provided by an exemplary embodiment

[0052] Fig.11 Another model training device for video replacement detection is provided by an exemplary embodiment. DETAILED DESCRIPTION

[0053] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0054] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0055] In one embodiment, the screenshot processing scheme of this specification can be applied to electronic devices, for example, the electronic devices can include any type of mobile phones, tablet devices, laptops, PDAs (Personal Digital Assistants), wearable devices (such as smart glasses, smart watches, etc.), etc., and this specification does not limit this. During operation, the electronic device can obtain screenshots and process the screenshots through human-computer interaction with the user.

[0056] Figure 1 FIG. 1 is a schematic diagram of the architecture of a video replacement detection system provided by an exemplary embodiment. Figure 1 As shown, the system may include a server 11, a network 12, and several electronic devices, such as a mobile phone 13, a mobile phone 14, and a mobile phone 15.

[0057] Mobile phones 13-15 are just one type of electronic device that users can use. In fact, users can obviously also use electronic devices such as tablet devices, laptops, PDAs (Personal Digital Assistants), wearable devices (such as smart glasses, smart watches, etc.), etc., and one or more embodiments of this specification do not limit this. The user can use an electronic device to obtain the video to be detected and send it to the server 11 for video replacement detection, wherein the video to be detected can be a video shot by the user using an electronic device, a video downloaded from the Internet, or a video call video saved during a video call with other electronic devices.

[0058] The server 11 may be a physical server including an independent host, or the server 11 may be a virtual server carried by a host cluster. During operation, the server 11 may detect whether the received video to be detected has been replaced.

[0059] The network 12 for interaction between the mobile phones 13-15 and the server 11 may include various types of wired or wireless networks. In one embodiment, the network 12 may include a public switched telephone network (PSTN) and the Internet. At the same time, electronic devices such as the mobile phones 13-15 may also communicate and interact with each other through the network 12, for example, the mobile phone 13 sends the acquired video to the mobile phone 14 and / or the mobile phone 15 for video replacement detection.

[0060] Figure 2 1 is a flowchart of a video replacement detection method provided by an exemplary embodiment, which may include the following steps:

[0061] Step 202: extract multiple frames of images from the video to be detected, and extract the image feature vector of each frame of image.

[0062] Among them, when extracting images, the time interval between two adjacent frames of images can be set according to actual needs. It can be understood that the shorter the time interval, the more images can be obtained, and each frame of image can better reflect the video characteristics.

[0063] If n frames of images are extracted from the video to be detected, correspondingly, n image feature vectors can be obtained.

[0064] At present, the replacement of videos generally involves replacing all or part of the N consecutive frames of images that make up the video, and generally the entire frame of the image is not replaced, but the area where the target object is located in the image is replaced. For example, a false facial image is used to replace the face (target object) area in each frame of the video, or the features of the face area in each frame of the image are modified (such as enlarging the eyes of the face, shrinking the cheeks, etc.). Therefore, before extracting the image feature vector, the area where the target object is located in the image can be identified first, and then the image features of the area where the target object is located can be extracted. This can reduce the amount of calculation for feature vector extraction on the one hand, and improve the accuracy of feature vector extraction on the other hand.

[0065] The target object is an object that may be replaced in the pre-detection video, which may be, but is not limited to, a human face, an animal, a vehicle, etc.

[0066] In one embodiment, the identification of the area where the target object is located can be achieved with the help of a multi-task convolutional neural network MTCNN. MTCNN uses three cascaded networks and adopts the idea of ​​candidate boxes plus classifiers to perform fast and efficient face detection. The three cascaded networks are P-Net for quickly generating candidate windows, R-Net for high-precision candidate window filtering and selection, and O-Net for generating the final bounding box and face key points.

[0067] The basic structure of P-Net is a fully connected network, which is a regional proposal network for face areas. After the features are input into the result three convolutional layers, a face classifier is used to determine whether the area is a face. At the same time, bounding box regression and a facial key point locator are used to make preliminary proposals for face areas. This part will eventually output many face areas where faces may exist, and these areas will be input into R-Net for further processing.

[0068] The basic structure of P-Net is a convolutional neural network, which adds a fully connected layer compared to the first layer of P-Net. The idea of ​​R-Net is to use a more complex network structure than P-Net to further select and adjust the possible face area windows generated by P-Net, so as to achieve high-precision filtering and face area optimization.

[0069] The basic structure of O-Net is a more complex convolutional neural network, which has one more convolutional layer than R-Net. The difference between O-Net and R-Net is that this layer of structure will recognize the facial area through more supervision, and will regress the facial feature points of a person, and finally output multiple, such as five facial feature points.

[0070] MTCNN can avoid the huge performance consumption brought by traditional ideas such as sliding window plus classifier, taking into account both performance and accuracy, and can quickly and efficiently identify the area where the target object is located in the image.

[0071] In another embodiment, the extraction of image feature vectors can be achieved with the help of a deep convolutional network. The regional image corresponding to the area where the target object is located identified by MTCNN can be input into a trained deep convolutional network for feature extraction to obtain an image feature vector.

[0072] Step 204: Generate a video feature vector corresponding to the video to be detected, where the video feature vector is related to a fusion result of image feature vectors corresponding to multiple frames of images.

[0073] In this embodiment, the image feature vectors corresponding to each frame image extracted from the video to be detected are fused to obtain the video feature vector of the video to be detected, so as to detect whether the video to be detected has been replaced according to the video feature vector. Compared with the video detection method of the related art, the detection result has higher accuracy. This is because in the related art, each frame image extracted from the video to be detected is input into the classifier for replacement detection, that is, a single frame image is detected, and multiple detection results for the single frame image are weighted and fused to perform video detection according to the fusion result. The detection result may have errors due to factors such as the fluctuation of the number of replaced images in the video.

[0074] Step 206: Determine whether the video to be detected has been replaced according to the video feature vector.

[0075] When performing detection, a classifier can be used to input the video feature vector into a trained classifier for recognition, and the classifier outputs a recognition result of whether the video to be detected has been replaced.

[0076] In this embodiment, whether the video has been replaced is determined based on the video feature vector of the video to be detected, thereby realizing video-level detection. Compared with detection based on single-frame images, it avoids errors caused by fluctuations in the number of replaced images, and can improve the accuracy and reliability of the detection results.

[0077] In another embodiment, Figure 2 The video replacement detection method shown can be implemented by a video replacement detection model, which includes a feature extraction sub-model, a feature fusion sub-model and a detection sub-model. The video replacement detection model can be obtained by, but is not limited to, training a convolutional neural network, a recurrent neural network or a deep neural network.

[0078] Regarding the extraction of image feature vectors, a feature extraction sub-model can be used. Specifically: multiple frames of images are extracted from the video to be detected and input into the feature extraction sub-model. After the features of the image are extracted, the feature extraction sub-model outputs the image feature vector of the image.

[0079] Regarding the generation of video feature vectors, a feature fusion sub-model can be used. Specifically: the image feature vector of each frame image output by the feature extraction sub-model is input into the feature fusion sub-model. After processing the image feature vector of each frame image, the feature fusion sub-model outputs the video feature vector corresponding to the video to be detected.

[0080] Regarding video detection, a detection sub-model may be used. Specifically, the video feature vector output by the feature fusion sub-model is input into the detection sub-model, and the detection sub-model outputs a detection result of whether the video to be detected has been replaced.

[0081] Figure 3a is a flowchart of generating a video feature vector corresponding to a video to be detected provided by an exemplary embodiment. In this embodiment, the image feature vectors corresponding to each frame image extracted from the video to be detected are fused, and the fusion result of the image feature vectors is determined as the video feature vector of the video to be detected. Figure 3a , which may include the following steps:

[0082] Step 302: sequentially use the image feature vector corresponding to each frame of the multiple frames as a convolution kernel.

[0083] For example, suppose that image extraction is performed on the video to be detected to obtain n frames of images. Correspondingly, n image feature vectors can be obtained, which are [A 11 ,A 12 ,A 13 ,……,A 1i ],[A 21 ,A 22 ,A 23 ,……,A 2i ],[A 31 ,A 32 ,A 33 ,……,A 3i ], ..., [A n1 ,A n2 ,A n3 ,……,A ni ], where i means that each image feature vector contains i-dimensional features, and the above n image feature vectors are used as convolution kernels in turn.

[0084] Step 304: Use each convolution kernel to perform convolution processing on the image feature vectors corresponding to the multiple frames of images to obtain the convolution results corresponding to each convolution kernel.

[0085] In one embodiment, when performing convolution processing, a convolution kernel may be used to perform convolution processing on a matrix composed of image feature vectors corresponding to multiple frames of images, so as to determine a convolution result based on the result of the convolution processing.

[0086] Still taking the above n image feature vectors as an example, the matrix composed of n image feature vectors is expressed as follows:

[0087]

[0088] In step 304, n vectors [A 11 ,A 12 ,A 13 ,……,A 1i ],[A 21 ,A 22 ,A 23 ,……,A 2i ],[A 31 ,A 32 ,A 33 ,……,A 3i ], ..., [A n1 ,A n2 ,A n3 ,……,A ni ]By performing convolution processing on the above n×i matrix, n convolution results can be obtained.

[0089] In another embodiment, when performing convolution processing, a convolution kernel can be used to perform convolution processing on the image feature vectors corresponding to each frame image respectively, and the obtained initial convolution results corresponding to each frame image are spliced ​​into a convolution result corresponding to the convolution kernel.

[0090] Still taking the above n image feature vectors as an example, first use [A 11 ,A 12 ,A 13 ,……,A 1i ] as the convolution kernel, respectively [A 11 ,A 12 ,A 13 ,……,A 1i ],[A 21 ,A 22 ,A 23 ,……,A 2i ],[A 31 ,A 32 ,A 33 ,……,A 3i ], ..., [A n1 ,A n2 ,A n3 ,……,A ni ] to perform convolution processing, we can get n initial convolution results, and these n initial convolution results can be spliced ​​to get the same convolution kernel [A 11 ,A 12 ,A 13 ,……,A 1i ] corresponding to the convolution result; and [A 11 ,A 12 ,A 13 ,……,A 1i ]Similarly, use [A 21 ,A 22 ,A 23 ,……,A 2i ],[A 31 ,A 32 ,A 33 ,……,A 3i ], ..., [A n1 ,A n2 ,A n3 ,……,A ni ] as the convolution kernel, the convolution result corresponding to n convolution kernels can be obtained.

[0091] Step 306: Perform weighted fusion on each convolution result, and determine the weighted fusion result as a video feature vector.

[0092] When weighted fusion is performed on multiple convolution results, the weight of each convolution result can be determined based on the attention mechanism, and the multiple convolution results can be weighted fused based on the weight, so that the weight corresponding to each convolution result can more accurately reflect the impact of the convolution result on the video feature vector.

[0093] For ease of understanding, the following Figure 3b Taking the flowchart of the convolution processing shown as an example, the implementation process of steps 302 to 306 is further explained. The 6×5 matrix in the figure represents a matrix composed of image feature vectors corresponding to 6 images. The 6 images are obtained by image extraction of the video to be detected, and each image feature vector contains 5-dimensional features. The image feature vectors corresponding to the 6 images are used as convolution kernels to perform convolution processing on the 6×5 matrix, and 6 vectors (convolution results) can be obtained. These 6 convolutions are weighted and fused. The result of the weighted fusion is the video feature vector of the video to be detected, which is used for replacement detection of the video to be detected.

[0094] Figure 4a is another flowchart of fusing the image feature vectors corresponding to each frame image provided by an exemplary embodiment. In this embodiment, based on the fusion result of the image feature vector, the fusion result of the time series feature vectors of multiple frames of images is combined, and the splicing result of the two fusion results is determined as the video feature vector of the video to be detected. Figure 4a , the method may include the following steps:

[0095] Step 402: Fuse the image feature vectors corresponding to each frame image to obtain a fusion result of the image feature vectors.

[0096] Step 402 is to fuse the image feature vectors corresponding to each frame image. The specific implementation process of step 402 is similar to steps 302 to 306 and will not be repeated here.

[0097] Step 404: Group the multiple image frames based on at least one dimension.

[0098] Each image group includes at least two adjacent frames of images, and the dimension corresponds to the number of images included in each image group. The number of dimensions and the number of images k included in each image group can be set according to actual needs.

[0099] There is a one-to-one correspondence between images and image feature vectors. Grouping images is equivalent to grouping image feature vectors. For example, if multiple frames of images are grouped based on two dimensions, the number of images corresponding to the first dimension is 2, and the number of images corresponding to the second dimension is 3. The number of n vectors [A 11 ,A 12 ,A 13 ,……,A 1i],[A 21 ,A 22 ,A 23 ,……,A 2i ],[A 31 ,A 32 ,A 33 ,……,A 3i ], ..., [A n1 ,A n2 ,A n3 ,……,A ni The grouping result of ] can be expressed as:

[0100] For the first dimension, we get n-1 image groups, represented as {[A 11 ,A 12 ,A 13 ,……,A 1i ]、[A 21 ,A 22 ,A 23 ,……,A 2i ]}、{[A 21 ,A 22 ,A 23 ,……,A 2i ]、[A 31 ,A 32 ,A 33 ,……,A 3i ]}、……、{[A (n-1)1 ,A (n-1)2 ,A (n-1)3 ,……,A (n-1)i ]、[A n1 ,A n2 ,A n3 ,……,A ni ]};

[0101] For the second dimension, we get n-2 image groups, represented as {[A 11 ,A 12 ,A 13 ,……,A 1i ]、[A 21 ,A 22 ,A 23 ,……,A 2i ]、[A 31 ,A 32 ,A 33 ,……,A 3i ]}、{[A 21 ,A 22 ,A 23 ,……,A 2i ]、[A 31 ,A 32,A 33 ,……,A 3i ]、[A 41 ,A 42 ,A 43 ,……,A 4i ]}、……、{[A (n-2)1 ,A (n-1)2 ,A (n -2)3,……,A (n-2)i ]、[A (n-1)1 ,A (n-1)2 ,A (n-1)3 ,……,A (n-1)i ]、[A n1 ,A n2 ,A n3 ,……,A ni ]}.

[0102] Step 406: Fuse the time series feature vectors corresponding to each image group to obtain a fusion result of the time series feature vectors.

[0103] The time series feature vector is composed of image feature vectors corresponding to the images contained in the image group.

[0104] Taking the target object as a face, for example, if the face in the video is replaced, there are inconsistencies in the fit of the faces in the two frames or traces of artificial replacement. The fusion results of the temporal feature vectors can learn the discriminative features in the video, and then, based on the fusion results of the image feature vectors, the fusion results of the temporal feature vectors of multiple frames of images can be combined to determine whether the video has been replaced. Taking into account the information in the temporal and spatial dimensions of the video, the accuracy of detection can be further improved.

[0105] It should be noted that step 402, step 404 and step 406 are not limited to Figure 4a In the order shown in FIG. 4 , step 402 and step 404 and step 406 may be executed synchronously, that is, step 404 and step 406 are executed first, and then step 402 is executed.

[0106] Figure 4b A flowchart of fusing temporal feature vectors corresponding to each image group provided by an exemplary embodiment may include the following steps:

[0107] Step 406-2: Use the temporal feature row vector corresponding to each image group as a convolution kernel.

[0108] Taking the grouping result of the first dimension as an example, n-1 image groups correspond to n-1 convolution kernels, which are

[0109] Step 406-4: Use each convolution kernel to perform convolution processing on the image feature vectors corresponding to the multiple frames of images to obtain the convolution results corresponding to each convolution kernel.

[0110] When performing the convolution process, similar to step 304, in one embodiment, a convolution kernel may be used to perform convolution processing on a matrix composed of image feature vectors corresponding to multiple frames of images.

[0111] In another embodiment, the image feature vectors corresponding to each frame image may be convolved using a convolution kernel, and the initial convolution results corresponding to each frame image may be concatenated to obtain a convolution result corresponding to the convolution kernel. In this embodiment, the row and column dimensions of the image feature vector may not match the dimension of the convolution kernel, and the image feature vector may be padded with zeros before convolution.

[0112] Step 406-6: Perform weighted fusion on the convolution results of each convolution kernel to obtain a fusion result of the time series feature vector.

[0113] In step 406-6, when weighted fusion is performed on the convolution results of each convolution kernel, the weight of each convolution result can be determined based on the attention mechanism, and multiple convolution results can be weightedly fused based on the weight, so that the weight corresponding to each convolution result can more accurately reflect the impact of the convolution result on the fusion result of the time series feature vector.

[0114] Step 408: Concatenate the fusion result of the image feature vector and the fusion result of the time series feature vector to obtain a video feature vector.

[0115] For ease of understanding, the following Figure 4c Taking the convolution processing flowchart provided as an example, the implementation process of steps 402 to 408 is further explained. Taking the 6×5 matrix as an example, if two dimensions are used to group images, the number of images K corresponding to the two dimensions are 2 and 3 respectively, the process of generating the video feature vector is as follows:

[0116] Step a: Use the image feature vectors corresponding to the six images as convolution kernels to perform convolution processing on the 6×5 matrix. In this case, the six images are divided into six image groups. The number of images contained in each image group is K=1. After the convolution processing, six convolution results (vectors) can be obtained. These six convolution results are weighted fused to obtain the fusion result of the image feature vector.

[0117] Step b, respectively using the image feature vectors corresponding to two adjacent images as convolution kernels to perform convolution processing on the 6×5 matrix, that is, dividing the six images into five image groups, each image group contains K=2 images, and five convolution results can be obtained after the convolution processing, and the five convolution results are weightedly fused to obtain the first fusion result of the time series feature vector;

[0118] Step c, respectively using the image feature vectors corresponding to the three adjacent images as convolution kernels to perform convolution processing on the 6×5 matrix, that is, dividing the six images into four image groups, each image group contains K=3 images, and four convolution results can be obtained after the convolution processing, and the four convolution results are weightedly fused to obtain a second fusion result of the time series feature vector;

[0119] It should be noted that steps a to c can be executed synchronously or sequentially, and the execution order is not limited to executing the fusion result of K=1 first, then executing the fusion result of K=2, and finally executing the fusion result of K=3. The execution order can be set arbitrarily. The convolution step size for convolution processing can be set according to actual needs, and the convolution step size can be 1 or other values.

[0120] Step d: concatenate the fusion result of the image feature vector, the first fusion result of the time series feature vector, and the second fusion result of the time series feature vector to obtain a video feature vector.

[0121] In another embodiment, generating a video feature vector corresponding to the video to be detected can also be achieved using a long short-term memory network or a 3D convolutional neural network. Specifically, the image feature vector corresponding to each frame image is input into the long short-term memory network or the 3D convolutional neural network for feature vector fusion processing, and the long short-term memory network or the 3D convolutional neural network outputs the video feature vector.

[0122] Figure 2 The video replacement detection method shown can be implemented based on a video replacement detection model, and the video replacement detection model can include a feature extraction sub-model, a feature fusion sub-model and a detection sub-model.

[0123] When extracting the image feature vector of an image, the image is input into the feature extraction sub-model, and the image feature vector of the image is obtained after feature extraction by the feature extraction sub-model;

[0124] When generating a video feature vector corresponding to the video to be detected, the image feature vector corresponding to each frame image is input into the feature fusion sub-model, and after being processed by the feature fusion sub-model, the feature fusion sub-model outputs the video feature vector corresponding to the video to be detected;

[0125] When determining whether the video to be detected has been replaced, the video feature vector is input into the detection sub-model, and after being processed by the detection sub-model, the detection sub-model outputs a detection result of whether the video to be detected has been replaced.

[0126] Different sub-model parts in the above-mentioned video replacement detection model can be trained and adjusted in parameters during the model training phase to obtain better output results.

[0127] Figure 5 The following is a flowchart of a model training method for video replacement detection provided by an exemplary embodiment, which may include the following steps:

[0128] Step 502: Input n frames of images extracted from each video sample into the feature extraction sub-model for feature extraction.

[0129] The video sample includes replacement annotation information. The feature extraction sub-model can be but is not limited to using a deep convolutional network.

[0130] The video samples in this embodiment may include a part of positive samples and a part of negative samples. The video samples as positive samples are videos that have not been replaced, and the video samples as negative samples are videos that have been replaced.

[0131] Step 504: input the image feature vector of each frame image output by the feature extraction sub-model into the feature fusion sub-model for feature vector fusion.

[0132] Among them, the feature fusion sub-model can be but is not limited to using a long short-term memory network (LSTM) and a 3D convolutional neural network.

[0133] In one embodiment, the feature fusion submodel can be composed of a two-branch multi-instance network. The multi-instance network of one branch treats n frames of images as n instances and fuses the image feature vectors of the n frames of images; the multi-instance network of the other branch treats multiple image groups as multiple instances, and for each dimension, fuses the time series feature vectors corresponding to each image group; the feature fusion submodel splices the fusion results of the image feature vector and the fusion results of the time series feature vector to obtain the video feature vector of the video sample. All instances in the video sample share the replacement annotation information of the video sample as a whole.

[0134] Step 506: Input the video feature vectors corresponding to each video sample segment output by the feature fusion sub-model into the detection sub-model.

[0135] In one embodiment, the feature fusion sub-model may, but is not limited to, use a classifier.

[0136] Step 508: Adjust the model parameters of the feature extraction sub-model, the feature fusion sub-model and the detection sub-model according to the replacement annotation information and the output result of the detection sub-model.

[0137] Therefore, replacement detection can be achieved by inputting the video to be detected into the trained model.

[0138] In one embodiment, three loss functions can be established corresponding to the feature extraction sub-model, feature fusion sub-model and detection sub-model respectively, and the loss values ​​of the three loss functions can be determined respectively according to the replacement annotation information and the output results of the detection sub-model, and then the model parameters of the three sub-models can be adjusted respectively.

[0139] In another embodiment, a loss function may be established, and the loss value of the loss function may be determined based on the replacement annotation information and the output result of the detection sub-model, and then the model parameters of the three sub-models may be adjusted based on the loss value.

[0140] In another embodiment, the feature extraction sub-model, the feature fusion sub-model and the detection sub-model may be trained separately, and the three trained sub-models may be spliced ​​into a model for video replacement detection.

[0141] The training method of this embodiment, based on the fusion result of the image feature vector, combines the fusion result of the time series feature vectors of multiple frames of images, and determines the splicing result of the two fusion results as the video feature vector of the video to be detected, so as to perform replacement recognition based on the video feature vector. It takes into account the information in the spatiotemporal dimensions of the video, and can learn discriminative features, which can significantly improve the precision and recall of the model.

[0142] Figure 6 1 is a flowchart of another model training method for video replacement detection provided by an exemplary embodiment, which may include the following steps:

[0143] Step 602: Acquire multiple video samples, each of which contains replacement annotation information.

[0144] Step 604: extract multiple frames of images from each video sample, and generate a video feature vector corresponding to each video sample.

[0145] The video feature vector is related to the fusion result of the image feature vectors corresponding to the multiple frames of images.

[0146] In one embodiment, the image feature vectors of multiple frames of images can be fused, and the fusion result of the image feature vectors can be determined as the video feature vector. For the specific implementation process, see Figure 3a As shown, no further description is given here.

[0147] In another embodiment, based on the fusion result of the image feature vector, the fusion result of the time series feature vector of multiple frames of images can be combined to determine the splicing result of the two fusion results as the video feature vector of the video to be detected. For the specific implementation process, see Figure 4a As shown, no further description is given here.

[0148] Step 606: Input the video feature vector into the classifier for training, and adjust the parameters of the classifier according to the replacement annotation information and the output result of the classifier.

[0149] Among them, the classifier can be trained using but is not limited to the cross entropy loss function.

[0150] When performing video replacement detection, it is necessary to extract images and image features from the video to be detected, fuse the image feature vectors corresponding to each frame of the image into a video feature vector, and input the video feature vector into the trained classifier to achieve replacement detection.

[0151] Figure 7 This is a schematic diagram of a device structure provided by an exemplary embodiment. Figure 7 At the hardware level, the device includes a processor 702, an internal bus 704, a network interface 706, a memory 708, and a non-volatile memory 710, and may also include hardware required for other services. The processor 702 reads the corresponding computer program from the non-volatile memory 710 into the memory 708 and then runs it, forming a video replacement detection device at the logical level. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0152] Please refer to Figure 8 In a software implementation, the video replacement detection device may include:

[0153] The extraction module 81 is used to extract multiple frames of images from the video to be detected and extract the image feature vector of each frame of image;

[0154] A generating module 82, configured to generate a video feature vector corresponding to the video to be detected, wherein the video feature vector is related to a fusion result of image feature vectors corresponding to the multiple frames of images;

[0155] The determination module 83 is used to determine whether the video to be detected has been replaced according to the video feature vector.

[0156] Optionally, the generating module has a function for:

[0157] Grouping the multiple frames of images based on at least one dimension, each image grouping includes at least two adjacent frames of images, and the dimension corresponds to the number of images included in each image group;

[0158] Fusion of time series feature vectors corresponding to each image group, wherein the time series feature vector is formed by combining image feature vectors corresponding to images included in the image group;

[0159] The fusion result of the time series feature vector is concatenated with the fusion result of the image feature vector to obtain the video feature vector.

[0160] Optionally, when fusing the temporal feature vectors corresponding to each image group, the generating module is used to:

[0161] The temporal feature row vector corresponding to each image group is used as the convolution kernel;

[0162] Using each convolution kernel to perform convolution processing on the image feature vectors corresponding to the multiple frames of images, respectively, to obtain a first convolution result corresponding to each convolution kernel;

[0163] The first convolution results are weightedly fused to obtain a fusion result of the time series feature vector.

[0164] Optionally, when performing weighted fusion on each first convolution result, the generating module is used to:

[0165] Determine the weights of each first convolution result based on the attention mechanism;

[0166] The first convolution results are weightedly fused based on the weights.

[0167] Optionally, when fusing the image feature vectors corresponding to the multiple frames of images, the generating module is used to:

[0168] Sequentially using the image feature vector corresponding to each frame of the multiple frames of images as a convolution kernel;

[0169] Using each convolution kernel to perform convolution processing on the image feature vectors corresponding to the multiple frames of images, respectively, to obtain second convolution results corresponding to each convolution kernel;

[0170] The second convolution results are weightedly fused, and the weighted fusion result is used to determine the video feature vector.

[0171] Optionally, when the convolution kernel is used to perform convolution processing on the image feature vectors corresponding to the multiple frames of images, the generating module is used to:

[0172] Using the convolution kernel to perform convolution processing on the matrix composed of image feature vectors corresponding to the multiple frames of images, so as to determine the second convolution result according to the result of the convolution processing; or,

[0173] The convolution kernel is used to perform convolution processing on the image feature vector corresponding to each frame image, and the obtained initial convolution results corresponding to each frame image are spliced ​​into a second convolution result corresponding to the convolution kernel.

[0174] Optionally, when performing weighted fusion on each second convolution result, the generating module is used to:

[0175] Determine the weight of each second convolution result based on the attention mechanism;

[0176] The second convolution results are weightedly fused based on the weights.

[0177] Optionally, when generating the video feature vector corresponding to the video to be detected, the generating module is used to:

[0178] The image feature vectors corresponding to the frames of images are input into a long short-term memory network or a 3D convolutional neural network for feature vector fusion processing, and the long short-term memory network or the 3D convolutional neural network outputs the video feature vector.

[0179] Optionally, the determining module is specifically used to:

[0180] The video feature vector is input into a classifier for identification, and the classifier outputs a result of whether the video to be detected has been replaced.

[0181] Optionally, the video replacement detection method is performed by a video replacement detection model;

[0182] The video replacement detection model includes a feature extraction sub-model, a feature fusion sub-model and a detection sub-model;

[0183] The feature extraction sub-model is used to extract the image feature vector of each frame of image;

[0184] The feature fusion sub-model is used to generate a video feature vector corresponding to the video to be detected;

[0185] The detection sub-model is used to determine whether the video to be detected has been replaced according to the video feature vector.

[0186] Optionally, when extracting the image feature vector of each frame of image, the extraction module is used to:

[0187] Identify the area where the target object is located in each frame image;

[0188] An image feature vector of the region is extracted.

[0189] Fig. 9 This is a schematic diagram of a device structure provided by an exemplary embodiment. Fig. 9 At the hardware level, the device includes a processor 902, an internal bus 904, a network interface 906, a memory 908, and a non-volatile memory 910, and may also include hardware required for other services. The processor 902 reads the corresponding computer program from the non-volatile memory 910 into the memory 908 and then runs it, forming a model training device for video replacement detection at the logical level. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0190] Please refer to Fig.10 In a software implementation, a model training device for video replacement detection may include:

[0191] The acquisition module 1001 is used to acquire multiple video samples, each of which contains replacement annotation information;

[0192] The generating module 1002 is used to extract multiple frames of images from each video sample and generate a video feature vector corresponding to each video sample, wherein the video feature vector is related to the fusion result of the image feature vector corresponding to the multiple frames of images.

[0193] The training module 1003 is used to input the video feature vector into a classifier for training, and adjust the parameters of the classifier according to the replacement annotation information and the output result of the classifier.

[0194] Please refer to Fig.11 In a software implementation, another model training device for video replacement detection may include:

[0195] An extraction module 1101 is used to input multiple frames of images extracted from each video sample into the feature extraction sub-model for feature extraction, wherein the video sample includes replacement annotation information;

[0196] An input module 1102 is used to input the image feature vector of each frame image output by the feature extraction sub-model into the feature fusion sub-model to perform feature vector fusion;

[0197] The input module 1103 is further used to input the video feature vectors corresponding to each video sample output by the feature fusion sub-model into the detection sub-model;

[0198] The adjustment module 1104 is used to adjust the model parameters of the feature extraction sub-model, the feature fusion sub-model and the detection sub-model according to the replacement annotation information and the output result of the detection sub-model.

[0199] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0200] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0201] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0202] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology, on which a computer program (information) is stored, and when the program is executed by the processor, the method steps provided in any of the above embodiments are implemented. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0203] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0204] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0205] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0206] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0207] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A video replacement detection method, characterized in that: include: Extract multiple frames of images from the video to be detected, and extract the image feature vector of each frame of image; Generate a video feature vector corresponding to the video to be detected, wherein the video feature vector is related to a fusion result of image feature vectors corresponding to the multiple frames of images; Determining whether the video to be detected has been replaced according to the video feature vector; The image feature vectors corresponding to the multiple frames of images are integrated, including: Sequentially using the image feature vector corresponding to each frame of the multiple frames of images as a convolution kernel; Using each convolution kernel to perform convolution processing on the image feature vectors corresponding to the multiple frames of images, respectively, to obtain second convolution results corresponding to each convolution kernel; The second convolution results are weightedly fused, and the weighted fusion result is used to determine the video feature vector.

2. The video replacement detection method according to claim 1, characterized in that: Generating a video feature vector corresponding to the video to be detected, including: Grouping the multiple frames of images based on at least one dimension, each image grouping includes at least two adjacent frames of images, and the dimension corresponds to the number of images included in each image group; Fusion of time series feature vectors corresponding to each image group, wherein the time series feature vector is formed by combining image feature vectors corresponding to images included in the image group; The fusion result of the time series feature vector is concatenated with the fusion result of the image feature vector to obtain the video feature vector.

3. The video replacement detection method according to claim 2, characterized in that: The temporal feature vectors corresponding to each image group are integrated, including: The temporal feature row vector corresponding to each image group is used as the convolution kernel; Using each convolution kernel to perform convolution processing on the image feature vectors corresponding to the multiple frames of images, respectively, to obtain a first convolution result corresponding to each convolution kernel; The first convolution results are weightedly fused to obtain a fusion result of the time series feature vector.

4. The video replacement detection method according to claim 3, characterized in that: The first convolution results are weighted and fused, including: Determine the weights of each first convolution result based on the attention mechanism; The first convolution results are weightedly fused based on the weights.

5. The video replacement detection method according to claim 1, characterized in that: The convolution kernel is used to perform convolution processing on the image feature vectors corresponding to the multiple frames of images, including: Using the convolution kernel to perform convolution processing on the matrix composed of image feature vectors corresponding to the multiple frames of images, so as to determine the second convolution result according to the result of the convolution processing; or, The convolution kernel is used to perform convolution processing on the image feature vector corresponding to each frame image, and the obtained initial convolution results corresponding to each frame image are spliced ​​into a second convolution result corresponding to the convolution kernel.

6. The video replacement detection method according to claim 5, characterized in that: The second convolution results are weighted and fused, including: Determine the weight of each second convolution result based on the attention mechanism; The second convolution results are weightedly fused based on the weights.

7. The video replacement detection method according to claim 1, characterized in that: Generating a video feature vector corresponding to the video to be detected, including: The image feature vectors corresponding to the frames of images are input into a long short-term memory network or a 3D convolutional neural network for feature vector fusion processing, and the long short-term memory network or the 3D convolutional neural network outputs the video feature vector.

8. The video replacement detection method according to claim 1, characterized in that: Determining whether the video to be detected has been replaced according to the video feature vector includes: The video feature vector is input into a classifier for identification, and the classifier outputs a result of whether the video to be detected has been replaced.

9. The video replacement detection method according to claim 1, characterized in that: The video replacement detection method is performed by a video replacement detection model; The video replacement detection model includes a feature extraction sub-model, a feature fusion sub-model and a detection sub-model; The extracting of the image feature vector of each frame of the image comprises: extracting the image through the feature extraction sub-model to obtain the image feature vector of the image; The generating of the video feature vector corresponding to the video to be detected comprises: processing the image feature vector corresponding to each frame image by the feature fusion sub-model, and outputting the video feature vector corresponding to the video to be detected by the feature fusion sub-model; Determining whether the video to be detected has been replaced according to the video feature vector includes: the video feature vector is processed by the detection sub-model, and the detection sub-model outputs a detection result of whether the video to be detected has been replaced.

10. The video replacement detection method according to claim 1, characterized in that: Extract the image feature vector of each frame, including: Identify the area where the target object is located in each frame image; An image feature vector of the region is extracted.

11. A model training method for video replacement detection, characterized in that: include: Obtain multiple video samples, each of which contains replacement annotation information; Extracting multiple frames of images from each video sample, and generating a video feature vector corresponding to each video sample, wherein the video feature vector is related to a fusion result of image feature vectors corresponding to the multiple frames of images; Inputting the video feature vector into a classifier for training, and adjusting parameters of the classifier according to the replacement annotation information and an output result of the classifier; The image feature vectors corresponding to the multiple frames of images are integrated, including: Sequentially using the image feature vector corresponding to each frame of the multiple frames of images as a convolution kernel; Using each convolution kernel to perform convolution processing on the image feature vectors corresponding to the multiple frames of images, respectively, to obtain second convolution results corresponding to each convolution kernel; The second convolution results are weightedly fused, and the weighted fusion result is used to determine the video feature vector.

12. A model training method for video replacement detection, characterized in that: The model includes a cascaded feature extraction sub-model, a feature fusion sub-model and a detection sub-model; The method comprises: Inputting multiple frames of images extracted from each video sample into the feature extraction submodel for feature extraction, wherein the video sample includes replacement annotation information; Inputting the image feature vector of each frame image output by the feature extraction sub-model into the feature fusion sub-model to perform feature vector fusion; Inputting the video feature vectors corresponding to each video sample output by the feature fusion sub-model into the detection sub-model; Adjusting model parameters of the feature extraction sub-model, the feature fusion sub-model and the detection sub-model according to the replacement annotation information and the output result of the detection sub-model; The image feature vectors corresponding to the multiple frames of images are integrated, including: Sequentially using the image feature vector corresponding to each frame of the multiple frames of images as a convolution kernel; Using each convolution kernel to perform convolution processing on the image feature vectors corresponding to the multiple frames of images, respectively, to obtain second convolution results corresponding to each convolution kernel; The second convolution results are weightedly fused, and the weighted fusion result is used to determine the video feature vector.

13. A video replacement detection device, characterized in that: include: An extraction module is used to extract multiple frames of images from the video to be detected and extract the image feature vector of each frame of image; A generating module, used for generating a video feature vector corresponding to the video to be detected, wherein the video feature vector is related to a fusion result of the image feature vectors corresponding to the multiple frames of images; A determination module, used for determining whether the video to be detected has been replaced according to the video feature vector; When fusing the image feature vectors corresponding to the multiple frames of images, the generating module is used to: Sequentially using the image feature vector corresponding to each frame of the multiple frames of images as a convolution kernel; Using each convolution kernel to perform convolution processing on the image feature vectors corresponding to the multiple frames of images, respectively, to obtain second convolution results corresponding to each convolution kernel; The second convolution results are weightedly fused, and the weighted fusion result is used to determine the video feature vector.

14. A model training device for video replacement detection, characterized in that: include: An acquisition module is used to acquire multiple video samples, each of which contains replacement annotation information; A generating module, used for extracting multiple frames of images from each video sample, and generating a video feature vector corresponding to each video sample, wherein the video feature vector is related to a fusion result of the image feature vectors corresponding to the multiple frames of images; A training module, used for inputting the video feature vector into a classifier for training, and adjusting parameters of the classifier according to the replacement annotation information and an output result of the classifier; When fusing the image feature vectors corresponding to the multiple frames of images, the generating module is used to: Sequentially using the image feature vector corresponding to each frame of the multiple frames of images as a convolution kernel; Using each convolution kernel to perform convolution processing on the image feature vectors corresponding to the multiple frames of images, respectively, to obtain second convolution results corresponding to each convolution kernel; The second convolution results are weightedly fused, and the weighted fusion result is used to determine the video feature vector.

15. A model training device for video replacement detection, characterized in that: The model includes a cascaded feature extraction sub-model, a feature fusion sub-model and a detection sub-model; The device comprises: An extraction module, used for inputting multiple frames of images extracted from each video sample into the feature extraction sub-model for feature extraction, wherein the video sample includes replacement annotation information; An input module, used for inputting the image feature vector of each frame image output by the feature extraction sub-model into the feature fusion sub-model for feature vector fusion; The input module is further used to input the video feature vectors corresponding to each video sample output by the feature fusion sub-model into the detection sub-model; An adjustment module, used to adjust the model parameters of the feature extraction sub-model, the feature fusion sub-model and the detection sub-model according to the replacement annotation information and the output result of the detection sub-model; When fusing the image feature vectors corresponding to the multiple frames of images, the input module is used to: Sequentially using the image feature vector corresponding to each frame of the multiple frames of images as a convolution kernel; Using each convolution kernel to perform convolution processing on the image feature vectors corresponding to the multiple frames of images, respectively, to obtain second convolution results corresponding to each convolution kernel; The second convolution results are weightedly fused, and the weighted fusion result is used to determine the video feature vector.

16. An electronic device, characterized in that: include: Replace the detector; a memory for storing replacement detector executable instructions; The replacement detector implements the method according to any one of claims 1 to 12 by running the executable instructions.

17. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by the replacement detector, the steps of the method according to any one of claims 1 to 12 are implemented.

Images