Video authenticity judgment method and system based on global-local spatial-temporal feature alignment

Through the global-local spatiotemporal feature alignment method, the problem of insufficient connection between local features and global features in the existing technology is solved, and the accuracy and generalization ability of the judgment of video authenticity is improved, so that better identify the deep fake video Exception characteristics.

CN119091344APending Publication Date: 2024-12-06CHINA CITIC BANK CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411023474.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The existing video authenticity judgment methods fail to fully consider the connection between local features and global features, resulting in inaccurate results in video authenticity judgment, and it is difficult to identify abnormal spatio-temporal features in finely faked videos, and lack generalization capabilities.

Method used

Using a global-local spatiotemporal feature alignment method, the domain feature distribution is adjusted to align the global spatiotemporal features and the aligned Features are mapped to binary classification space for authenticity judgment.

Benefits of technology

It improves the accuracy and generalization ability of video authenticity judgment, can accurately identify abnormal spatial and temporal features, and enhances the detection ability of deep fake videos.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091344A_ABST
    Figure CN119091344A_ABST
Patent Text Reader

Abstract

The invention relates to a video authenticity judgment method and system based on global-local spatial-temporal feature alignment, electronic equipment and a computer readable storage medium. The method comprises the following steps: sampling a target video, obtaining video groups containing continuous frames, extracting local spatial-temporal features of each video group, fusing the local spatial-temporal features to form global spatial-temporal features, performing domain division and distribution difference calculation on the global spatial-temporal features, adjusting distribution of the domain features to realize feature alignment, and finally, obtaining the target video. And mapping the aligned global spatial-temporal characteristics to a dichotomy space, and judging the authenticity of the video by using an authenticity judgment model. According to the video authenticity judgment scheme disclosed by the invention, the global features of the whole video group are fused under the condition of considering the local features, the abnormal spatio-temporal features can be accurately identified, and the accuracy and generalization ability of video authenticity judgment are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of video processing and recognition technology, and in particular to a video authenticity judgment method, system, electronic device and computer-readable storage medium based on global-local spatiotemporal feature alignment. Background Art

[0002] In recent years, with the rapid development of digital technology, video editing and tampering technology has become increasingly mature. Forged videos are made by replacing the face of the target video character with a specified original video face, or by having the target face reenact or imitate the movements and expressions of the original face, thereby creating a forged video of the target face. However, forged videos are currently difficult to identify with the naked eye.

[0003] The identification of investment risk preferences is a very important part of financial investment, especially for investors' emotions. For example, the Chinese invention patent disclosure CN115861068A discloses a spatiotemporal hybrid video super-resolution method based on deformable attention, including: extracting local features and global features from low-resolution video data streams, obtaining video frame feature vectors that aggregate global information through a series of block discarding attention modules and deformable attention modules, extracting local features again and combining shallow feature information through jump connections, extracting feature information of each spatial block vector in the time dimension using time trajectories and completing feature alignment, fusing the obtained forward and backward super-resolution results and outputting the final high-resolution frame results through a reconstruction module. However, this method does not fully consider the relationship between local features and global features, and it is necessary to discard frames in the video, which will lead to incomplete consideration of the video.

[0004] Most existing methods for judging the authenticity of videos rely on manually designed features, or only use the spatiotemporal information of videos to detect deep fake face videos. Therefore, their performance is often limited by the accuracy of feature extraction, and it is difficult to accurately identify abnormal spatiotemporal features in fine fake videos, and they lack generalization capabilities. The existing methods usually only focus on local features or global features of the video, while ignoring the correlation and interaction between them, resulting in inaccurate judgment results. Summary of the invention

[0005] To address the deficiencies of the prior art, the present invention proposes a video authenticity judgment method, system, electronic device and computer-readable storage medium based on global-local spatiotemporal feature alignment.

[0006] To achieve the above objectives, the technical solutions adopted by the present invention include:

[0007] A video authenticity judgment method based on global-local spatiotemporal feature alignment, characterized by comprising:

[0008] S1. Video sampling: obtaining a target video and sampling the target video to obtain a preset number of video groups, wherein the video groups contain continuous frames of a corresponding video length;

[0009] S2, local spatiotemporal feature extraction: performing frame-level spatial feature extraction on each of the continuous frames, connecting the frame-level spatial features in the time dimension to obtain continuous features, and using a time learning network to derive local spatiotemporal features based on the continuous features;

[0010] S3, extracting global spatiotemporal features: fusing the local spatiotemporal features corresponding to the video groups to form global spatiotemporal features;

[0011] S4, feature alignment: domain-dividing the global spatiotemporal features, extracting domain features, and calculating the distribution difference of the domain features using the maximum average difference metric, and aligning the global spatiotemporal features by adjusting the distribution of the domain features to reduce the distribution difference;

[0012] S5. Video authenticity judgment: Map the aligned global spatiotemporal features to a binary classification space, use a function to perform probability conversion, and input them into an authenticity judgment model for authenticity judgment, wherein the authenticity judgment model is trained using video data with real or fake labels.

[0013] Furthermore, in step S5, mapping the aligned global spatiotemporal features to a binary classification space specifically includes: mapping the aligned global spatiotemporal features to a binary classification space using a fully connected layer, and the number of output nodes of the fully connected layer is set to 2.

[0014] Furthermore, the use of a function for probability conversion in step S5 specifically includes: using a softmax function for probability conversion, and making the sum of the converted probability values ​​​​1, and performing authenticity judgment based on the probability values.

[0015] Furthermore, the authenticity judgment model in step S5 is trained using video data with real or fake labels and also includes: during the training process, the weights of the fully connected layer are adjusted by an optimization algorithm, and iterated to minimize the loss function between the judgment result of the authenticity judgment model and the real labeled video data.

[0016] Optionally, the fusing of the local spatiotemporal features corresponding to the video group in step S3 specifically includes:

[0017] A global-local conversion model is used to fuse the local spatiotemporal features corresponding to the video group. The global-local conversion model includes a local conversion model and a global conversion model. The local conversion model is used to mine the short-term interaction between the local spatiotemporal features, and the global conversion model is used to mine the long-term interaction between the local spatiotemporal features and the global spatiotemporal cues.

[0018] Furthermore, the local transformation model includes a multi-head attention mechanism, a feedforward layer and a normalization layer.

[0019] In addition, the present invention also relates to a video authenticity judgment system based on global-local spatiotemporal feature alignment, which is characterized by comprising:

[0020] A video sampling module is used to obtain a target video and sample the target video to obtain a preset number of video groups, wherein the video groups contain continuous frames of a corresponding video length;

[0021] A local spatiotemporal feature extraction module, configured to extract frame-level spatial features from each of the continuous frames, connect the frame-level spatial features in the time dimension to obtain continuous features, and use a temporal learning network to derive local spatiotemporal features based on the continuous features;

[0022] A global spatiotemporal feature extraction module, used for fusing the local spatiotemporal features corresponding to the video group to form a global spatiotemporal feature;

[0023] A feature alignment module, used to divide the global spatiotemporal features into domains, extract domain features, calculate the distribution difference of the domain features using a maximum average difference metric, and align the global spatiotemporal features by adjusting the distribution of the domain features to reduce the distribution difference;

[0024] The video authenticity judgment module is used to map the global spatiotemporal features after alignment processing to a binary classification space, and use a function to perform probability conversion, and input it into an authenticity judgment model for authenticity judgment, wherein the authenticity judgment model is trained using video data with real or fake labels.

[0025] Furthermore, the video authenticity judgment module further includes:

[0026] A mapping unit is used to map the global spatiotemporal features after the alignment process to a binary classification space using a fully connected layer, and the number of output nodes of the fully connected layer is set to 2.

[0027] Furthermore, the video authenticity judgment module further includes:

[0028] The probability conversion unit is used to perform probability conversion using a softmax function, and make the sum of the converted probability values ​​​​1, and perform authenticity judgment based on the probability values.

[0029] Furthermore, the video authenticity judgment module further includes:

[0030] The optimization unit is used to adjust the weight of the fully connected layer through an optimization algorithm during the training process, and iterate to minimize the loss function between the judgment result of the authenticity judgment model and the real label video data.

[0031] Optionally, the global spatiotemporal feature extraction module further includes:

[0032] A fusion unit adopts a global-local conversion model to fuse the local spatiotemporal features corresponding to the video group, wherein the global-local conversion model includes a local conversion model and a global conversion model, and the local conversion model is used to mine the short-term interaction between the local spatiotemporal features, and the global conversion model is used to mine the long-term interaction between the local spatiotemporal features and the global spatiotemporal clues.

[0033] Furthermore, the local conversion model also includes: a multi-head attention mechanism, a feedforward layer and a normalization layer.

[0034] In addition, the present invention also relates to an electronic device, characterized in that it comprises a processor and a memory;

[0035] The memory is used to store operation instructions;

[0036] The processor is used to execute the above method by calling the operation instruction.

[0037] In addition, the present invention also relates to a computer-readable storage medium, characterized in that a computer program is stored on the storage medium, and the computer program implements the above method when executed by a processor.

[0038] By adopting the scheme provided in the present application, the target video is sampled to obtain a video group containing continuous frames, and the local spatiotemporal features of each video group are extracted and then fused to form global spatiotemporal features. The global spatiotemporal features are domain divided and distribution difference calculated, and the distribution of domain features is adjusted to achieve feature alignment. Finally, the aligned global spatiotemporal features are mapped to a binary classification space, and the authenticity of the video is judged using a authenticity judgment model. Therefore, by adopting the above technical scheme, while taking local features into consideration, the global features of the entire video group are also integrated, which can accurately identify abnormal spatiotemporal features and improve the accuracy and generalization ability of video authenticity judgment. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1a A schematic diagram of a forged face video with inconsistent local consecutive frames provided in an embodiment of the present application;

[0040] Figure 1b A schematic diagram of a forged face video with inconsistent global non-continuous frames provided in an embodiment of the present application;

[0041] Figure 2a A schematic flow chart of a method for determining video authenticity based on global-local spatiotemporal feature alignment provided in an embodiment of the present application;

[0042] Figure 2b A schematic diagram of a video authenticity determination process provided in an embodiment of the present application;

[0043] Figure 3 A schematic diagram of the architecture of the global-local conversion model provided in an embodiment of the present application;

[0044] Figure 4 A schematic diagram of the structure of a video authenticity judgment system based on global-local spatiotemporal feature alignment provided in an embodiment of the present application;

[0045] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present invention.

[0047] Those skilled in the art can understand that, unless otherwise stated, the singular forms "one", "an", "said" and "the" used herein may also include plural forms, and the "first", "second", etc., are only used to distinguish and define objects for the purpose of introducing the scheme clearly, and do not limit the objects themselves. Of course, the objects defined by "first" and "second" may be the same terminal, device, user, etc., or the same type of terminal, device and user. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0048] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following embodiments may be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0049] In deep fake face videos, local continuous frames can often present highly realistic spatial details, such as Figure 1a However, when we look deeper from the perspective of time and space, we can find that there are subtle time and space inconsistencies in these local consecutive frames. In particular, the shape of the eyebrows in the second frame is significantly different from that in the subsequent consecutive frames, and then gradually becomes less significant in the consecutive frames.

[0050] Moreover, even between non-consecutive frames, obvious dynamic inconsistencies can be observed. Figure 1b As shown in Figure 1, the mole on the face shows a gradual change in the global non-continuous frame sequence, and this change forms an inconsistent dynamic pattern on the continuous timeline. Therefore, in order to detect deep fake face videos more comprehensively and accurately, we need to not only pay attention to the spatiotemporal inconsistency in local continuous frames, but also comprehensively consider the global spatiotemporal perspective to dig out more detailed and comprehensive spatiotemporal inconsistency clues.

[0051] Figure 2a , 2b The flowchart of the video authenticity judgment method based on global-local spatiotemporal feature alignment and the schematic diagram of the video authenticity judgment architecture provided by the embodiment of the present application are shown, which specifically include:

[0052] S1, video sampling: obtain the target video, and uniformly sample the target video to obtain L video groups {G1, ..., G1} as the input sequence, 1∈[1:L], and the video group contains T consecutive frames;

[0053] S2. Local spatiotemporal feature extraction: For each of the consecutive frames, a ResNet-50 network is used to extract frame-level spatial features ft t∈[1:T], and the frame-level spatial features are extracted. Connect in the time dimension to get continuous features F 1 , and using a temporal learning network composed of a standard Transformer (TRF) module to derive local spatiotemporal features based on the continuous features, wherein the standard TRF module includes a multi-head self-attention module;

[0054] By extracting local spatiotemporal features in the above manner, it is possible to understand and focus on the important parts of the input data, thereby extracting information that is more important to the current task.

[0055] S3, extracting global spatiotemporal features: fusing the local spatiotemporal features corresponding to the video groups to form global spatiotemporal features;

[0056] In order to more clearly introduce the global spatiotemporal feature formation scheme disclosed in this application, according to Figure 3 To illustrate by way of example:

[0057] The local spatiotemporal features corresponding to the video group are fused by using a global-local transformation model (Transformer). The global-local transformation model is responsible for mining the short-term interaction between local spatiotemporal features and exploring the long-term interaction between local and global spatiotemporal features, aiming to deeply mine more refined and comprehensive spatiotemporal features. The global-local transformation model includes a local transformation model (TRF L0cal ) and the global transformation model (TRF Global ), mining the short-term interactions between the local spatiotemporal features through the local transformation model, and mining the long-term interactions between the local spatiotemporal features and the global spatiotemporal cues through the global transformation model, wherein the structure of the local transformation model includes a multi-head attention mechanism, a feedforward layer, and a normalization layer;

[0058] In order to enhance the effect of time series data processing, sine and cosine functions of different frequencies are used to add position embedding (PE), which is expressed as:

[0059]

[0060]

[0061] Where 2j and 2j+1 are the even and odd indices of the position embedding, PEt represents the position embedding of the tth position, The dimension of is PEt. Based on this strategy, the output of TRFLocal is expressed as:

[0062]

[0063] By dynamically combining local spatiotemporal cues, the algorithm can focus more on key local spatiotemporal feature information.

[0064] Global spatiotemporal characteristics It is obtained by integrating the features of all frames in all video groups, where the fusion calculation can be expressed as:

[0065]

[0066] In the formula It is TRF Global The output of TRF GlobalAbility to generate global spatiotemporal features based on local spatiotemporal features:

[0067]

[0068] By combining local spatiotemporal features And global spatiotemporal characteristics Concatenate to generate the final spatiotemporal features, which can be expressed as:

[0069]

[0070] Based on multi-view spatiotemporal features F v Making judgments can enhance the understanding and recognition of complex spatiotemporal patterns and improve the accuracy of detecting deep fake videos.

[0071] S4, feature alignment: domain-dividing the global spatiotemporal features, extracting domain features, and calculating the distribution difference of the domain features using the maximum average difference metric, and aligning the global spatiotemporal features by adjusting the distribution of the domain features to reduce the distribution difference;

[0072] In order to more clearly introduce the feature alignment solution disclosed in this application, an exemplary description is given:

[0073] The training data is divided into domains. According to different forgery techniques, face images are divided into different domains. There are K domains involved in the training. The face samples in domain d are expressed as: Where, domain d∈{1,...,K}, n d is the number of samples in the d domain. The label corresponding to each sample is expressed as There are two categories (real / fake). After mining the inherent features, the input features of the mth fully connected layer from K training domains are expressed as:

[0074]

[0075] In the formula, Represents the input features of the mth fully connected layer (FC) from domain d. In order to eliminate the feature distribution deviation between the forgery operation techniques, a cross-domain alignment loss is proposed as:

[0076]

[0077] In the formula, Represents the distance between different counterfeit technical features, using the maximum mean difference MMD metric expressed as:

[0078]

[0079] In the formula, Indicates from The input features of the t-th sample in the cross-domain alignment loss In the above example, we need to minimize the MMD distance between domains and use the cross entropy classification loss and cross-domain alignment loss Training detection framework, the network parameter learning process can be expressed as:

[0080]

[0081] In the formula, λ is the weight parameter of the bilateral cross-domain feature alignment loss, which plays a role in balancing the importance of different loss components in the algorithm. After experiments and tuning, the weight λ of the cross-domain alignment loss is set to 0.1;

[0082] The domain alignment module is used to simultaneously disperse the feature distribution within the same forgery technology and reduce the difference in feature distribution between different forgery technologies, ultimately obtaining a more generalizable feature space.

[0083] S5. Video authenticity judgment: The global spatiotemporal features after alignment are mapped to a binary classification space using a fully connected layer, the number of output nodes of the fully connected layer is set to 2, and the softmax function is used for probability conversion, and the sum of the converted probability values ​​is made to be 1. Authenticity judgment is performed based on the probability values, and the probability values ​​are input into an authenticity judgment model for authenticity judgment, wherein the authenticity judgment model is trained with video data with real or fake labels. During the training process, the weights of the fully connected layer are adjusted through an optimization algorithm, and iterated to minimize the loss function between the judgment result of the authenticity judgment model and the real labeled video data.

[0084] By adopting the above scheme, the target video is sampled to obtain a video group containing continuous frames. The local spatiotemporal features of each video group are extracted and then fused to form global spatiotemporal features. The global spatiotemporal features are domain divided and distribution differences are calculated. The distribution of domain features is adjusted to achieve feature alignment. Finally, the aligned global spatiotemporal features are mapped to the binary classification space, and the authenticity of the video is judged using the authenticity judgment model. Therefore, while considering the local features, the global features of the entire video group are integrated, abnormal spatiotemporal features are accurately identified, and the accuracy of video authenticity judgment is improved.

[0085] based on Figure 2a The video authenticity judgment method based on global-local spatiotemporal feature alignment is shown in FIG. 1 , and on the other hand, it also relates to a video authenticity judgment system based on global-local spatiotemporal feature alignment, whose structure is as follows: Figure 4 As shown, including:

[0086] The video sampling module 401 is used to obtain a target video and sample the target video to obtain a preset number of video groups, wherein the video groups contain continuous frames of a corresponding video length;

[0087] A local spatiotemporal feature extraction module 402 is used to extract frame-level spatial features from each of the continuous frames, connect the frame-level spatial features in the time dimension to obtain continuous features, and use a time learning network to derive local spatiotemporal features based on the continuous features;

[0088] A global spatiotemporal feature extraction module 403, configured to fuse the local spatiotemporal features corresponding to the video group to form a global spatiotemporal feature;

[0089] A feature alignment module 404 is used to divide the global spatiotemporal features into domains, extract domain features, calculate the distribution difference of the domain features using a maximum average difference metric, and align the global spatiotemporal features by adjusting the distribution of the domain features to reduce the distribution difference;

[0090] The video authenticity judgment module 405 is used to map the global spatiotemporal features after alignment processing to a binary classification space, and use a function to perform probability conversion, and input it into the authenticity judgment model for authenticity judgment, wherein the authenticity judgment model is trained using video data with real or fake labels.

[0091] Furthermore, the video authenticity judgment module 405 further includes:

[0092] The mapping unit 4051 is used to map the global spatiotemporal features after the alignment process to the binary classification space by using a fully connected layer, and the number of output nodes of the fully connected layer is set to 2.

[0093] Furthermore, the video authenticity judgment module 405 further includes:

[0094] The probability conversion unit 4052 is used to perform probability conversion using a softmax function, and make the sum of the converted probability values ​​​​to be 1, and perform authenticity judgment based on the probability values.

[0095] Furthermore, the video authenticity judgment module 405 further includes:

[0096] The optimization unit 4053 is used to adjust the weight of the fully connected layer through an optimization algorithm during the training process, and iterate to minimize the loss function between the judgment result of the authenticity judgment model and the real label video data.

[0097] Optionally, the global spatiotemporal feature extraction module 403 further includes:

[0098] The fusion unit 4031 adopts a global-local conversion model to fuse the local spatiotemporal features corresponding to the video group. The global-local conversion model includes a local conversion model and a global conversion model. The local conversion model is used to mine the short-term interaction between the local spatiotemporal features, and the global conversion model is used to mine the long-term interaction between the local spatiotemporal features and the global spatiotemporal clues.

[0099] Furthermore, the local conversion model also includes: a multi-head attention mechanism, a feedforward layer and a normalization layer.

[0100] By using this system, the above method can be executed and the corresponding technical effects can be achieved.

[0101] An embodiment of the present invention further provides an electronic device for executing the above method, as an implementation device of the method, comprising a processor and a memory;

[0102] A memory, used for storing operation instructions;

[0103] The processor is used to execute the video authenticity determination method provided in any embodiment of the present application by calling an operation instruction.

[0104] As an example, Figure 5 A schematic diagram of the structure of an electronic device applicable to the embodiment of the present application is shown. Figure 5 As shown, the electronic device 500 includes: a processor 501 and a memory 503. The processor 501 and the memory 503 are connected, such as being connected via a bus 502. Optionally, the electronic device 500 may further include a transceiver 504. It should be noted that in actual applications, the transceiver 504 is not limited to one. It is understandable that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the specific structure of the electronic device 500. In other embodiments of the present application, the electronic device 500 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware. Optionally, the electronic device may further include a display screen 505 for displaying images, or receiving user operation instructions when necessary.

[0105] The processor 501 is applied in the embodiment of the present application to implement the method shown in the above method embodiment. The transceiver 504 may include a receiver and a transmitter. The transceiver 504 is applied in the embodiment of the present application to implement the function of the electronic device of the embodiment of the present application communicating with other devices when executed.

[0106] Processor 501 may be a CPU (Central Processing Unit), a general purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 501 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0107] The processor 501 may also include one or more processing units, for example: the processor 501 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a memory, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural network processor (Neural-network Processing Unit, NPU), etc. Different processing units may be independent devices or integrated into one or more processors. Among them, the controller may be the nerve center and command center of the electronic device 500. The controller may generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions. A memory may also be set in the processor 501 for storing instructions and data. In some embodiments, the memory in the processor 501 is a high-speed cache memory. The memory may store instructions or data that the processor 501 has just used or circulated. If the processor 501 needs to use the instruction or data again, it can be directly called from the memory. Repeated access is avoided, the waiting time of the processor 501 is reduced, and the efficiency of the system is improved.

[0108] The processor 501 can run the video authenticity determination method provided in the embodiment of the present application to reduce the user's operation complexity, improve the intelligence of the terminal device, and enhance the user experience. The processor 501 can include different devices. For example, when the CPU and GPU are integrated, the CPU and GPU can cooperate to execute the video authenticity determination method provided in the embodiment of the present application, with part of the algorithm executed by the CPU and the other part of the algorithm executed by the GPU to obtain faster processing efficiency.

[0109] The bus 502 may include a path to transmit information between the above components. The bus 502 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 502 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0110] The memory 503 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory), or may include a high-speed random access memory, or may include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this.

[0111] Optionally, the memory 503 is used to store application code for executing the solution of the present application, and the execution is controlled by the processor 501. The processor 501 is used to execute the application code stored in the memory 503 to implement the video authenticity determination method provided in any embodiment of the present application.

[0112] The memory 503 can be used to store computer executable program codes, which include instructions. The processor 501 executes various functional applications and data processing of the electronic device 500 by running the instructions stored in the memory 503. The memory 503 may include a program storage area and a data storage area. Among them, the program storage area can store the code of the operating system, application program, etc. The data storage area can store data created during the use of the electronic device 500 (such as images, videos, etc. collected by the camera application), etc.

[0113] The memory 503 may also store one or more computer programs corresponding to the video authenticity determination method provided in the embodiment of the present application. The one or more computer programs are stored in the above-mentioned memory 503 and are configured to be executed by the one or more processors 501, and the one or more computer programs include instructions, and the above-mentioned instructions can be used to execute each step in the above-mentioned corresponding embodiment.

[0114] Of course, the code of the video authenticity judgment method provided in the embodiment of the present application can also be stored in an external memory.

[0115] The display screen 505 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 500 may include 1 or N display screens 505, where N is a positive integer greater than 1. The display screen 505 can be used to display information input by a user or information provided to a user and various graphical user interfaces (GUIs). For example, the display screen 505 can display photos, videos, web pages, or files, etc.

[0116] The electronic device provided in the embodiment of the present application is applicable to any embodiment of the above method. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, which will not be repeated here.

[0117] An embodiment of the present invention also provides a computer-readable storage medium capable of implementing all the steps in the method in the above embodiment, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, all the steps in the method in the above embodiment are implemented.

[0118] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0119] The present invention is described with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the process and / or box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a system for implementing the functions specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0120] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction system that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0121] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operating steps are performed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes of the flowchart and / or one or more blocks of the block diagram. Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0122] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A video authenticity judgment method based on global-local spatiotemporal feature alignment, characterized in that: include: S1. Video sampling: obtaining a target video and sampling the target video to obtain a preset number of video groups, wherein the video groups contain continuous frames of a corresponding video length; S2, local spatiotemporal feature extraction: performing frame-level spatial feature extraction on each of the continuous frames, connecting the frame-level spatial features in the time dimension to obtain continuous features, and using a time learning network to derive local spatiotemporal features based on the continuous features; S3, extracting global spatiotemporal features: fusing the local spatiotemporal features corresponding to the video groups to form global spatiotemporal features; S4, feature alignment: domain-dividing the global spatiotemporal features, extracting domain features, and calculating the distribution difference of the domain features using the maximum average difference metric, and aligning the global spatiotemporal features by adjusting the distribution of the domain features to reduce the distribution difference; S5. Video authenticity judgment: Map the aligned global spatiotemporal features to a binary classification space, use a function to perform probability conversion, and input them into an authenticity judgment model for authenticity judgment, wherein the authenticity judgment model is trained using video data with real or fake labels.

2. The method according to claim 1, characterized in that: Mapping the aligned global spatiotemporal features to the binary classification space in step S5 specifically includes: mapping the aligned global spatiotemporal features to the binary classification space using a fully connected layer, and the number of output nodes of the fully connected layer is set to 2.

3. The method according to claim 2, characterized in that The use of a function to perform probability conversion in step S5 specifically includes: using a softmax function to perform probability conversion, and making the sum of the converted probability values ​​​​1, and performing authenticity judgment based on the probability values.

4. The method according to claim 3, characterized in that The authenticity judgment model in step S5 is trained using video data with real or fake labels and also includes: during the training process, adjusting the weight of the fully connected layer through an optimization algorithm and iterating to minimize the loss function between the judgment result of the authenticity judgment model and the real labeled video data.

5. The method according to claim 1, characterized in that The step S3 of fusing the local spatiotemporal features corresponding to the video group specifically includes: A global-local conversion model is used to fuse the local spatiotemporal features corresponding to the video group. The global-local conversion model includes a local conversion model and a global conversion model. The local conversion model is used to mine the short-term interaction between the local spatiotemporal features, and the global conversion model is used to mine the long-term interaction between the local spatiotemporal features and the global spatiotemporal cues.

6. The method according to claim 5, characterized in that The local transformation model includes a multi-head attention mechanism, a feed-forward layer and a normalization layer.

7. A video authenticity judgment system based on global-local spatiotemporal feature alignment, characterized in that: include: A video sampling module is used to obtain a target video and sample the target video to obtain a preset number of video groups, wherein the video groups contain continuous frames of a corresponding video length; A local spatiotemporal feature extraction module, configured to extract frame-level spatial features from each of the continuous frames, connect the frame-level spatial features in the time dimension to obtain continuous features, and use a temporal learning network to derive local spatiotemporal features based on the continuous features; A global spatiotemporal feature extraction module, used for fusing the local spatiotemporal features corresponding to the video group to form a global spatiotemporal feature; A feature alignment module, used to divide the global spatiotemporal features into domains, extract domain features, calculate the distribution difference of the domain features using a maximum average difference metric, and align the global spatiotemporal features by adjusting the distribution of the domain features to reduce the distribution difference; The video authenticity judgment module is used to map the global spatiotemporal features after alignment processing to a binary classification space, and use a function to perform probability conversion, and input it into an authenticity judgment model for authenticity judgment, wherein the authenticity judgment model is trained using video data with real or fake labels.

8. The system according to claim 7, characterized in that The video authenticity judgment module also includes: A mapping unit, used to map the global spatiotemporal features after the alignment process to a binary classification space using a fully connected layer, wherein the number of output nodes of the fully connected layer is set to 2; A probability conversion unit, used to perform probability conversion using a softmax function, so that the sum of the converted probability values ​​is 1, and to perform authenticity judgment based on the probability values; The optimization unit is used to adjust the weight of the fully connected layer through an optimization algorithm during the training process, and iterate to minimize the loss function between the judgment result of the authenticity judgment model and the real label video data.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store operation instructions; The processor is used to execute the method according to any one of claims 1 to 6 by calling the operation instruction.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Citation Information

Cited By

  • Artificial intelligence identification method for discovering abnormal state in audio and video playing content

    CN120877191A