Video processing method, related equipment and computer readable medium

Abstract

The embodiment of the invention discloses a video processing method, a video processing device, electronic equipment, a computer readable medium and a computer program product. The method comprises the following steps: dividing an initial video containing a specified object into a plurality of video segments; then, determining a key frame image in each video segment based on the object display quality of the image in each video segment about the specified object; then, performing video trueness evaluation processing on each video segment to obtain the object trueness of the specified object contained in each video segment; performing image trueness evaluation processing on each key frame image to obtain the object trueness of the specified object contained in each key frame image; and finally, based on the object trueness of each video segment and the object trueness of each key frame image, identifying whether the initial video is an imaginary video so as to obtain a video detection result of the initial video. According to the technical scheme of the embodiment of the invention, the detection accuracy of the imaginary video can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer and communication technology, and in particular, to a video processing method, a video processing device, an electronic device, a computer-readable medium, and a computer program product. Background Art

[0002] With the development of artificial intelligence technology, it is now possible to merge and superimpose other images or videos onto source images or source videos through machine learning models to generate fictional videos. The abuse of artificial intelligence technology for generating fictional videos will bring great social harm. Therefore, methods for detecting fictional videos have emerged. In related technologies, the method for detecting fictional videos mainly extracts the video features of the entire original video through a trained machine learning model, and then determines whether the original video is a fictional video by analyzing the video features. However, the current method for detecting fictional videos is easily affected by incomplete and unclear image frames in the video, resulting in low detection accuracy. Therefore, how to improve the detection accuracy of fictional videos is a problem that needs to be solved urgently. Summary of the invention

[0003] The embodiments of the present application provide a video processing method, a video processing device, an electronic device, a computer-readable medium, and a computer program product, which can improve the detection accuracy of fictitious videos.

[0004] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0005] In a first aspect, an embodiment of the present application provides a video processing method, including:

[0006] Segmenting an initial video containing a specified object to obtain a plurality of video segments;

[0007] Determining key frame images in each video segment based on the object display quality of the image in each video segment with respect to the designated object;

[0008] Performing video authenticity evaluation processing on each of the video segments to obtain object authenticity of a specified object contained in each of the video segments;

[0009] Performing image authenticity evaluation processing on each key frame image to obtain the object authenticity of a specified object contained in each key frame image;

[0010] Based on the object authenticity of each video segment and the object authenticity of each key frame image, it is identified whether the initial video is a fictitious video to obtain a video detection result of the initial video.

[0011] In a second aspect, an embodiment of the present application provides a video processing device, the device comprising an acquisition unit and a processing unit, wherein:

[0012] The acquisition unit is used to segment the initial video containing the specified object to obtain multiple video segments;

[0013] The acquisition unit is further used to determine the key frame images in each video segment based on the object display quality of the image in each video segment with respect to the designated object;

[0014] The processing unit is used to perform video authenticity evaluation processing on each video segment to obtain the object authenticity of the specified object contained in each video segment;

[0015] The processing unit is further used to perform image authenticity evaluation processing on each key frame image to obtain the object authenticity of the specified object contained in each key frame image;

[0016] The processing unit is further configured to identify whether the initial video is a fictitious video based on the object authenticity of each video segment and the object authenticity of each key frame image, so as to obtain a video detection result of the initial video.

[0017] In a third aspect, an embodiment of the present application provides an electronic device, comprising one or more processors; a storage device for storing one or more computer programs, so that when the one or more computer programs are executed by the one or more processors, the electronic device implements the video processing method as described above.

[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor of an electronic device, the electronic device executes the video processing method as described above.

[0019] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, wherein the computer program is stored in a computer-readable storage medium, and a processor of an electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device performs the video processing method as described above.

[0020] In the technical solution provided in the embodiment of the present application, by dividing the video into multiple video segments to perform video authenticity evaluation processing separately, the fictitious video clue information can be accurately integrated in different time periods to evaluate the object authenticity of the video from multiple time periods, and to explore the time inconsistency of the object authenticity, thereby making the object authenticity of the video segment more diverse and rich, which is conducive to the subsequent accurate detection of fictitious videos. At the same time, the embodiment of the present application can avoid the interference caused by low-quality images in the video segment (such as images where the specified object is blocked or unclear) to a certain extent by performing image authenticity evaluation processing on the key frame images in each video segment, reduce the false detection of fictitious videos, and thus improve the detection accuracy of fictitious videos.

[0021] It can be seen that the embodiment of the present application can integrate more diverse and rich object realities and avoid the interference of low-quality images by identifying whether a video is a fictional video from two aspects: video segments and key frame images, which is conducive to the accurate detection of fictional videos and achieves the purpose of improving the accuracy of fictional video detection.

[0022] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0024] Figure 1 is a structural diagram of a video processing system provided in an embodiment of the present application;

[0025] Figure 2 It is a flowchart of a video processing method provided by an embodiment of the present application;

[0026] Figure 3 is a flowchart of another video processing method provided in an embodiment of the present application;

[0027] Figure 4 It is a flowchart of another video processing method provided in an embodiment of the present application;

[0028] Figure 5 is a schematic diagram of the network structure of the authenticity assessment model provided in the embodiment of the present application;

[0029] Figure 6It is a flowchart of another video processing method provided in an embodiment of the present application;

[0030] Figure 7 is a schematic diagram of a detection process of a fictitious face video provided by an embodiment of the present application;

[0031] Figure 8 is a schematic diagram of a video detection framework provided in an embodiment of the present application;

[0032] Fig. 9 is a structural block diagram of a video processing device provided in an embodiment of the present application;

[0033] Fig.10 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0034] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0035] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0036] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all contents and operations, nor must they be executed in the order described. For example, some operations may be decomposed, while some operations may be combined or partially combined, so the actual execution order may change according to actual conditions.

[0037] It should also be noted that the "multiple" mentioned in this application refers to two or more than two. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship.

[0038] With the continuous development of Internet technology, artificial intelligence (AI) technology has also been better developed. The so-called artificial intelligence technology refers to the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science; it mainly produces a new intelligent machine that can respond in a similar way to human intelligence by understanding the essence of intelligence, so that the intelligent machine has multiple functions such as perception, reasoning and decision-making. Correspondingly, AI technology is a comprehensive discipline, which mainly includes several major directions such as computer vision technology (Computer Vision, CV), speech technology, natural language processing technology and machine learning (Machine Learning, ML) / deep learning.

[0039] Among them, machine learning is a multi-disciplinary cross-discipline involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of AI and the fundamental way to make computer devices intelligent; the so-called machine learning is a multi-disciplinary cross-discipline involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines; it specializes in studying how computer devices simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structures to continuously improve their performance. Deep learning is a technology that uses deep neural network systems to perform machine learning; machine learning / deep learning can usually include artificial neural networks, reinforcement learning (RL), supervised learning, unsupervised learning and other technologies; the so-called supervised learning refers to the processing method of using training samples with known categories (with labeled categories) for model training, and unsupervised learning refers to the processing method of using training samples with unknown categories (not labeled) for model training.

[0040] Based on artificial intelligence technology and machine learning technology, deepfake technology has been derived. Deepfake technology is a technology that can merge and superimpose images or videos onto source images or source videos through machine learning models such as generative adversarial networks (GANs) to generate fictional images or fictional videos. Among them, face fiction technology is one of the most common deepfake technologies. Face fiction technology is an artificial intelligence technology that uses machine learning models to perform sample learning and can splice a person's voice, facial expressions and body movements into fake content.

[0041] The abuse of deep fiction technology will bring great social harm, so the detection method for fictional videos has emerged. In related technologies, the video features of the video are extracted through a trained machine learning model, and then the video features are analyzed to determine whether the original video is a fictional video. Among them, the machine learning model used for fictional video detection focuses on using complex network structures to extract robust video features; however, the video features extracted by such models are sometimes not robust enough due to being too complex. At the same time, such models are easily affected by incomplete and unclear image frames in the video when extracting features, resulting in low detection accuracy.

[0042] Based on this, an embodiment of the present application provides a video processing solution, which first divides a video containing a specified object into multiple video segments, and determines the key frame images in each video segment; then, performs video authenticity evaluation processing on each video segment, and performs image authenticity evaluation processing on each key frame image, so as to obtain the object authenticity of each video segment and the object authenticity of each key frame image; finally, based on the object authenticity of each video segment and the object authenticity of each key frame image, it is identified whether the video is a fictional video, so as to obtain a video detection result.

[0043] The specified object may be any object or part of an object such as a face, a vehicle, a flower, etc., and is not limited here. The object authenticity of each video segment refers to the object authenticity of the specified object contained in each video segment, and the object authenticity of each key frame image refers to the object authenticity of the specified object contained in each key frame image.

[0044] At the same time, the key frame images in each video segment can be determined based on the object display quality of the images in each video segment about the specified object. The object display quality of any image can be evaluated from aspects such as the integrity and clarity of the specified object contained in the image. Among them, the key frame images of each video segment can be one or more frames of images with the highest or higher object display quality in each video segment.

[0045] In addition, the video authenticity assessment process can be implemented based on the features of each image in the video segment and the associated features between the images in the video segment; and the image authenticity assessment process can be implemented based on the image features of the key frame image.

[0046] Since the integrity and clarity of a specified object in a video image may change over time, the richness of the fictional video clue information (such as the motion trajectory of the specified object, time and space artifacts, etc.) contained in each frame of a video may be different. Therefore, the object authenticity of a complete fictional video may be different at different times.

[0047] Therefore, this solution can accurately integrate the fictitious video clue information by time period by dividing the video into multiple video segments to evaluate the video authenticity separately, so as to evaluate the object authenticity of the video from multiple time periods, thereby making the object authenticity of the video segment more diverse and rich, which is conducive to the subsequent accurate detection of fictitious videos. At the same time, this solution can avoid the interference caused by low-quality images in the video segment (such as images where the specified object is blocked or unclear) to a certain extent by evaluating the image authenticity of the key frame images in each video segment, reduce the false detection of fictitious videos, and thus improve the detection accuracy of fictitious videos.

[0048] It can be seen that this scheme can integrate more diverse and rich object realities by identifying whether a video is a fictional video from two aspects: video segments and key frame images, avoid the interference of low-quality images, and is conducive to the accurate detection of fictional videos, thereby achieving the purpose of improving the accuracy of fictional video detection.

[0049] Based on the above video processing solution, the present application embodiment provides a video processing system, which can be seen in Figure 1 , Figure 1 The video processing system shown may include multiple terminal devices 101 and multiple servers 102. Among them, a communication connection is established between any terminal device and any server. For example, the terminal device 101 may include any one or more of a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart car, a medical imaging device, and a smart wearable device. A multimedia playback client, an image display client, a browser client, an information stream client, etc. can be run in the terminal device 101. The server 102 can be a server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. Any terminal device 101 and any server 102 can be directly or indirectly connected in communication via wired or wireless communication, and this application is not limited here.

[0050] In some embodiments, the above video processing method can be performed by only Figure 1The terminal device 101 in the video processing system shown in the figure executes, and the specific execution process is: the terminal device 101 can perform segmentation processing on the initial video containing the specified object to obtain multiple video segments. Then, the terminal device 101 can determine the key frame images in each video segment based on the object display quality of the image in each video segment about the specified object. After that, the terminal device 101 can perform video authenticity assessment processing on each video segment to obtain the object authenticity of the specified object contained in each video segment. At the same time, the terminal device 101 can perform image authenticity assessment processing on each key frame image to obtain the object authenticity of the specified object contained in each key frame image. Finally, the terminal device 101 can identify whether the initial video is a fictitious video based on the object authenticity of each video segment and the object authenticity of each key frame image, so as to obtain the video detection result of the initial video.

[0051] Optionally, the terminal device 101 may also output a video detection result of the initial video. Optionally, if the video detection result of the initial video indicates that the initial video is a fictitious video, the terminal device 101 may output prompt information that the initial video is a fictitious video.

[0052] Optionally, the above video processing method may also be performed only by Figure 1 The server 102 in the video processing system shown is executed. The specific execution process can refer to the specific execution process of the terminal device 101, which will not be repeated here.

[0053] In other embodiments, the video processing method may be run in a video processing system, which may include a terminal device and a server. Figure 1 The terminal device 101 and the server 102 included in the video processing system shown are jointly completed, and the specific execution process is: after the terminal device 101 obtains the initial video containing the specified object, the initial video can be sent to the server 102; after receiving the initial video, the server 102 can segment the initial video to obtain multiple video segments. Then, the server 102 can determine the key frame images in each video segment based on the object display quality of the image in each video segment about the specified object. After that, the server 102 can perform video authenticity evaluation processing on each video segment to obtain the object authenticity of the specified object contained in each video segment. At the same time, the server 102 can perform image authenticity evaluation processing on each key frame image to obtain the object authenticity of the specified object contained in each key frame image. Finally, the server 102 can identify whether the initial video is a fictitious video based on the object authenticity of each video segment and the object authenticity of each key frame image, so as to obtain the video detection result of the initial video.

[0054] Optionally, the server 102 may send the video detection result of the initial video to the terminal device 101; further, the terminal device 101 may output the video detection result of the initial video. Optionally, if the video detection result of the initial video indicates that the initial video is a fictitious video, the server 102 may send prompt information about the initial video being a fictitious video to the terminal device 101, so that the terminal device 101 outputs the prompt information.

[0055] It should be noted that the embodiments of the present application can process any video and can be applied to various scenarios, including but not limited to cloud technology, AI (Artificial Intelligence), smart medical care, smart home, smart driving and other scenarios. It can also be used in medical applications, conference applications, social applications, shopping applications and other applications that can transmit or view videos, without limitation.

[0056] Specifically, the technical solution of the embodiment of the present application can be applied to the intelligent driving scene, such as setting the designated object as an unmanned vehicle, the initial video can be a test drive video of the unmanned vehicle, and the server can provide background services for unmanned driving. In order to avoid the falsification of the test drive video, after the intelligent driving server receives the test drive video uploaded by the R&D personnel, the test drive video can be segmented to obtain multiple test drive video segments; then, the server can determine the key test drive images in each test drive video segment based on the vehicle display quality of the image in each test drive video segment about the unmanned vehicle. After that, the server can perform video authenticity evaluation processing on each test drive video segment to obtain the object authenticity of the unmanned vehicle contained in each test drive video segment; and the server can also perform image authenticity evaluation processing on each key test drive image to obtain the object authenticity of the unmanned vehicle contained in each key test drive image. Finally, the server can identify whether the test drive video is a fictitious video based on the object authenticity of each test drive video segment and the object authenticity of each key test drive image, so as to obtain the video detection result of the test drive video. Optionally, the server can also send the video detection result of the test drive video to the vehicle terminal in the unmanned vehicle.

[0057] For another example, the technical solution of the embodiment of the present application is applied to a social application, the terminal device can be a terminal device running a social application, the designated object can be a face, and the server can be a server providing video processing services for the social application. After detecting the video session with the conversation object in the social application using the terminal device, the terminal device can obtain the face video of the conversation object displayed in the video session; then, the terminal device can send the face video to the server, and the server can segment the face video to obtain multiple face video segments; then, the server can determine the key face image in each face video segment based on the face display quality of the image in each face video segment. After that, the server can perform video authenticity evaluation processing on each face video segment to obtain the face video authenticity of each face video segment; and the server can also perform image authenticity evaluation processing on each key face image to obtain the face image authenticity of each key face image. Finally, the server can identify whether the face video is a fictitious video based on the face video authenticity of each face video segment and the face image authenticity of each key face image, so as to obtain the video detection result of the face video.

[0058] Optionally, the server may also send the video detection result of the face video to the terminal device, so that the terminal device displays the video detection result of the face video in the video session. Optionally, if the video detection result of the face video indicates that the face video is a fictitious video, the server may generate prompt information that the face video is a fictitious video and the conversation object is a fictitious object, and send the prompt information to the terminal device, so that the terminal device outputs the prompt information; thereby preventing the object using the terminal device from being deceived.

[0059] It should be noted that in the specific implementation of this application, if relevant data or information such as videos and images are related to objects, when the embodiments of this application are applied to specific products or technologies, it is necessary to obtain the permission or consent of the object, and the collection, use and processing of relevant data or information need to comply with the relevant laws, regulations and standards of relevant countries and regions.

[0060] The following is a detailed description of various implementation details of the technical solution of the embodiment of the present application:

[0061] like Figure 2 As shown, Figure 2 is a flow chart of a video processing method according to an embodiment of the present application, which can be applied to Figure 1 The video processing system shown in the figure, the method can be executed by the terminal device or the server, or can be executed by the terminal device and the server together. In the embodiment of the present application, the method is executed by the terminal device as an example for explanation, wherein the video processing method may include S201 to S205, which are described in detail as follows:

[0062] S201: Segment an initial video containing a specified object to obtain multiple video segments.

[0063] In the embodiment of the present application, the specified object may be any specified object or any type of object. For example, the specified object may be a face, a vehicle, an airplane, etc., without limitation.

[0064] In one embodiment, the specific method of segmentation processing may be: obtaining the total duration of the initial video and the number of preset video segments; determining the target duration of each video segment according to the total duration and the number of preset video segments; and dividing the initial video into multiple video segments with the target duration. The number of preset video segments may be manually set or may be set by the terminal device or server in the above video processing system, which is not limited thereto.

[0065] In one embodiment, the specific method of segmentation processing can also be: obtaining the total number of image frames contained in the initial video and the preset number of video segments; determining the target number of frames for each video segment based on the total number of frames and the preset number of video segments; and dividing the initial video into multiple video segments each containing the target number of image frames.

[0066] Optionally, a preset number of video segment frames may be obtained first; then, the initial video may be divided into a plurality of video segments each containing a preset number of image frames. The preset number of video segment frames may be manually set or may be set by a terminal device or server in the above-mentioned video processing system, and this is not limited. For example, the preset number of video segment frames may be set to 16, and the total number of frames of a video may be 64, then the video may be divided into 4 video segments, each of which contains 16 consecutive frames of images.

[0067] Optionally, the initial video may be segmented in other ways, which are not limited here.

[0068] S202: Determine a key frame image in each video segment based on the object display quality of the image in each video segment with respect to a designated object.

[0069] In an embodiment of the present application, the object display quality of an image with respect to a specified object can be evaluated from aspects such as the integrity and clarity of the specified object presented in the image, or the object quality evaluation of each image in each video segment can be completed by training an evaluation model for the object display quality; optionally, other methods can also be used to evaluate the object display quality of an image, which are not limited here.

[0070] Specifically, the object display quality can be a specific value, such as 85, 20, 100, etc.; or a corresponding quality level, such as low, medium, high, etc. For example, when the specified object is a face, since a complete face generally needs to include two ears, two eyes, two eyebrows, a nose and a mouth, etc.; therefore, the initial object display quality of a certain image can be set to 100. If the image lacks any of the above face areas, the object display quality is reduced by 10; if the image lacks part of any of the above face areas, the object display quality is reduced by 5. For example, if the face in the image is missing an ear and half of the mouth, the object display quality of the image will be updated to 85. At the same time, the object display quality is also related to the clarity of the specified object in the image; if the clarity of the specified object in the image is low, the object display quality can be reduced by 15, and the object display quality of the image will be updated to 75.

[0071] In one embodiment, the number of key frame images may be one or more. Specifically, the key frame image may be an image with the highest object display quality in the video segment; then, the specific method of determining the key frame image in each video segment may be: using the image with the highest object display quality in each video segment as the key frame image of the corresponding video segment.

[0072] Optionally, the key frame images may be some images with higher object display quality in the video segment; the specific method for determining the key frame images in each video segment may be: for any video segment, sort the images in any video segment in descending order of object display quality to obtain a plurality of sorted images; then, the first m% of the images in the sorted plurality of images may be selected as key frame images. Wherein, m may be set manually or by the terminal device or server in the above video processing system, and this is not limited.

[0073] Optionally, if the object display quality is divided into levels, the images in each video segment whose object display quality is greater than a preset level can be used as key frame images of the corresponding video segment. The preset level can be set manually or by a terminal device or server in the above-mentioned video processing system, and there is no limitation on this. For example, the object display quality includes five levels: low, relatively low, medium, relatively high, and high. The preset level can be set to medium; therefore, the images in the video segment whose object display quality is greater than medium can be determined as key frame images.

[0074] Optionally, the key frame image may be determined based on the display quality of the object in other ways, which are not limited here.

[0075] S203: Perform video authenticity evaluation processing on each video segment to obtain the object authenticity of the specified object contained in each video segment.

[0076] In the embodiments of the present application, the object authenticity can be a specific numerical value, such as 0, 1, 0.8, 60%, etc.; it can also be a character or field. For example, the object authenticity can include two fields, real and unreal, and A can be set to represent real and B to represent unreal, etc., which is not limited here.

[0077] Since a video is composed of multiple frames of images, the video authenticity evaluation process for a video segment can be decomposed into the authenticity evaluation of the images in the video segment. Specifically, the video authenticity evaluation process can be implemented based on the features of each image in the video segment and the associated features between each image in the video segment.

[0078] In one embodiment, since there may be some unnatural motion and artifacts of the specified object in the fictional video of the specified object, the specific method of video authenticity assessment processing may be: for each image in any video segment, feature extraction processing is performed on each image to obtain texture features and motion features of each image; then, based on the texture features and motion features of each image, the predicted authenticity of each image is predicted; finally, the object authenticity of any video segment can be obtained based on the predicted authenticity of each image.

[0079] Among them, texture features refer to features that reflect the surface texture of objects in images. Specifically, texture features may include grayscale change features, grayscale co-occurrence matrices, and other features used to characterize image texture. In addition, motion features refer to features that reflect the degree of motion of the image content of each frame of an image in a video. Specifically, motion features may include motion estimation (i.e., the displacement of corresponding pixels in adjacent images), image signal-to-noise ratio, and the like. At the same time, the prediction authenticity of each image can be obtained by training a corresponding neural network or machine learning model based on the texture features and motion features of each image. Since the model training process is a technical means commonly used by those skilled in the art, it will not be described in detail here.

[0080] Optionally, the object authenticity of each video segment may be determined by using other algorithms or models for evaluating video authenticity, which are not limited here.

[0081] S204: Perform image authenticity evaluation processing on each key frame image to obtain the object authenticity of the designated object contained in each key frame image.

[0082] In the embodiment of the present application, the image authenticity evaluation process can be implemented based on the image features of the key frame images. Specifically, the image authenticity evaluation process can be: performing image feature extraction processing on each key frame image to obtain the image features of each key frame image; performing classification processing on the image features of each key frame image to obtain the object authenticity of each key frame image.

[0083] In one embodiment, the number of classification categories for classification processing may be two or more than two. For example, when there are two classification categories, 0 may be set to represent untrue and 1 may represent true; when there are more than two classification categories, the object authenticity may be set to include 10 categories from 1 to 10, where 1 represents the lowest degree of object authenticity and 10 represents the highest degree of object authenticity.

[0084] S205 , identifying whether the initial video is a fictitious video based on the object authenticity of each video segment and the object authenticity of each key frame image, so as to obtain a video detection result of the initial video.

[0085] In an embodiment of the present application, the video detection result of the initial video may include at least one or more of the following: a recognition result and an object authenticity of the initial video. The recognition result is used to indicate whether the initial video is a fictional video. At the same time, the object authenticity of the initial video may be determined based on the object authenticity of each video segment and the object authenticity of each key frame image.

[0086] In one embodiment, the object authenticity of each video segment and the average value of the object authenticity of each key frame image can be calculated, and the calculated average value is used as the object authenticity of the initial video. Optionally, the object authenticity of the initial video can also be obtained by calculating the weighted sum of the object authenticity of each video segment and the object authenticity of each key frame image. Among them, the weights of each video segment and each key frame image can be set to be the same, while the weights of the video segment and the key frame image are different; the weights of each video segment and each key frame image can also be set to be different, and there is no limitation on this.

[0087] In one embodiment, if the object authenticity of the initial video is less than a preset authenticity, a video detection result indicating that the initial video is a fictitious video may be generated; if the object authenticity of the initial video is greater than or equal to the preset authenticity, a video detection result indicating that the initial video is a real video may be generated. The preset authenticity may be manually set or may be set by a terminal device or server in the above video processing system, which is not limited thereto.

[0088] Optionally, after obtaining the video detection result of the initial video, the terminal device may also output the video detection result of the initial video.

[0089] Since the integrity and clarity of a specified object in a video image may change over time, the richness of the fictional video clue information (such as the motion trajectory of a specified object, time and space artifacts, etc.) contained in each frame of a video may be different. Therefore, the object authenticity of a complete fictional video may be different at different times; therefore, if only the overall authenticity of the video is evaluated, the inconsistency of video segments at different times will be ignored, resulting in inaccurate authenticity of the final video.

[0090] Therefore, in the embodiment of the present application, by dividing the video into multiple video segments to perform video authenticity evaluation processing separately, the fictitious video clue information can be accurately integrated in different time periods to evaluate the object authenticity of the video from multiple time periods, and to explore the time inconsistency of the object authenticity, thereby making the object authenticity of the video segment more diverse and rich, which is conducive to the subsequent accurate detection of fictitious videos. At the same time, the embodiment of the present application can avoid the interference caused by low-quality images in the video segment (such as images where the specified object is blocked or unclear) to a certain extent by performing image authenticity evaluation processing on the key frame images in each video segment, reduce the false detection of fictitious videos, and thus improve the detection accuracy and robustness of fictitious videos.

[0091] It can be seen that the embodiment of the present application can integrate more diverse and rich object realities and avoid the interference of low-quality images by identifying whether a video is a fictional video from two aspects: video segments and key frame images, which is conducive to the accurate detection of fictional videos and achieves the purpose of improving the accuracy of fictional video detection.

[0092] In one embodiment of the present application, another video processing method is provided. The video processing method can be applied to Figure 1 The video processing system shown in the figure, the method can be executed by the terminal device or the server, or can be executed by the terminal device and the server together. In the embodiment of the present application, the method is executed by the server as an example for explanation. Figure 3 FIG. 4 is a flow chart showing another video processing method. Figure 2 This paper expands on the method shown in .

[0093] Among them, S301 to S307 are described in detail as follows:

[0094] S301: Segment an initial video containing a specified object to obtain multiple video segments.

[0095] In the embodiment of the present application, the specific implementation of step S301 can refer to the specific implementation of step S201 in the above embodiment, which will not be repeated here.

[0096] S302: Determine a key frame image in each video segment based on the object display quality of the image in each video segment with respect to a designated object.

[0097] In the embodiment of the present application, the specific implementation of step S302 can refer to the specific implementation of step S202 in the above embodiment, which will not be repeated here.

[0098] S303: Perform recognition feature extraction processing on each image in each video segment to obtain object recognition features of each image.

[0099] In the embodiment of the present application, the specific method of the recognition feature extraction process can be: calling the trained object recognition network to extract the object recognition features of each image in each video segment. The trained object recognition network is a neural network used to identify whether the specified object is a fictional object or a real object.

[0100] Specifically, the training process of the object recognition network can be: obtaining a training image containing a specified object and an object label of the training image; calling the object recognition network to be trained, performing recognition feature extraction processing on the training image, and obtaining the object recognition feature of the training image; and performing object classification processing on the object recognition feature of the training image to obtain a predicted label of the training image; finally, based on the predicted label and object label of the training image, the object recognition network to be trained is trained to obtain a trained object recognition network.

[0101] The number of training images may include multiple ones. The object label is used to indicate whether the specified object contained in the training image is a fictitious object or a real object; and the prediction label is used to indicate whether the specified object contained in the training image is a fictitious object or a real object. Object classification processing can be implemented by a classification function, and the classification function can be sigmoid (a loss function for binary classification), cross entropy loss function, etc., without limitation. In addition, the object recognition network to be trained can be trained in the direction of reducing the gap between the prediction label and the object label of the training image.

[0102] In a specific implementation, the network structure of the object recognition network can be a convolutional neural network (CNN), a recurrent convolutional neural network (RNN), a generative adversarial network, etc., without limitation.

[0103] S304 , based on the previous frame image of each image, classify and predict the object recognition features of each image to obtain the object authenticity of each image.

[0104] In the embodiment of the present application, the video segment can be regarded as an image sequence, and the previous frame image of each image refers to one or more frames of images that are arranged before each image in the image sequence.

[0105] In one embodiment, the specific process of classification prediction processing can be: in the process of traversing each image, the object recognition features of the current traversal image are subjected to state extraction processing to obtain the hidden state features of the current traversal image; based on the hidden state features of the previous frame image of the current traversal image, the hidden state features of the current traversal image are subjected to feature classification processing to obtain the object authenticity of the current traversal image.

[0106] Specifically, the classification prediction network can be called to complete the state extraction processing and feature classification processing. In the specific implementation, the classification prediction network can be a gated recurrent neural network (GRU), a recurrent neural network (RNN), a Long Short Term Memory (LSTM, a neural network with the ability to remember long-term and short-term information) and other neural networks with time series data processing capabilities, without limitation.

[0107] S305: Determine the object authenticity of the corresponding video segment based on the object authenticity of each image.

[0108] In the embodiment of the present application, based on the object realities of each image, the specific method of determining the object realities of the corresponding video segment may be: weighted addition of the object realities of each image, and using the result of the addition as the object realities of the corresponding video segment. Since each image refers to each image in each video segment, the corresponding video segment of each image refers to the video segment including the aforementioned each image.

[0109] Optionally, the weight of the object realism of each image can be pre-set; for example, the weight of the object realism of each image can be set to the inverse of the total number of frames of the image in the video segment, and then the weighted addition is equivalent to calculating the average value of the object realism of each image.

[0110] Optionally, the weight of the object authenticity of each image may also be set according to the object display quality of each image, and the object display quality may be positively correlated with the weight of the object authenticity of each image. Optionally, the weight of the object authenticity of each image may also be set in other ways, which are not limited here.

[0111] S306: Perform image authenticity evaluation processing on each key frame image to obtain the object authenticity of the designated object contained in each key frame image.

[0112] In the embodiment of the present application, the specific implementation of step S306 can refer to the specific implementation of step S204 in the above embodiment, which will not be repeated here.

[0113] S307 , identifying whether the initial video is a fictitious video based on the object authenticity of each video segment and the object authenticity of each key frame image, so as to obtain a video detection result of the initial video.

[0114] In an embodiment of the present application, the object authenticity of the initial video can be determined based on the object authenticity of each video segment and the object authenticity of each key frame image; then, the video detection result of the initial video is obtained according to the object authenticity of the initial video.

[0115] In one embodiment, the multiple video segments can form a video sequence, and the multiple key frame images can form a key image sequence; then, based on the object realism of each video segment and the object realism of each key frame image, the specific method of determining the object realism of the initial video can be: based on the object realism of each video segment, determine the object realism of the video sequence; based on the object realism of each key frame image, determine the object realism of the key image sequence; calculate the weighted sum of the object realism of the video sequence and the object realism of the key image sequence to obtain the object realism of the initial video.

[0116] Among them, the weight of the object authenticity of the video sequence and the weight of the object authenticity of the key image sequence can be set by a person, a terminal device or a server in the above-mentioned video processing system, or can be obtained by training in the process of training the neural network used for authenticity fusion, and are not limited here.

[0117] Optionally, based on the object realities of each video segment, the specific method of determining the object realities of the video sequence may be: weighted addition of the object realities of each video segment to obtain the object realities of the video sequence; or the average value of the object realities of each video segment may be calculated to obtain the object realities of the video sequence.

[0118] Optionally, based on the object realities of each key frame image, the specific method of determining the object realities of the key image sequence may be: weighted addition of the object realities of each key frame image to obtain the object realities of the key image sequence; or the average value of the object realities of each key frame image may be calculated to obtain the object realities of the key image sequence.

[0119] It should be noted that the specific method of determining the object authenticity of the initial video based on the object authenticity of each video segment and the object authenticity of each key frame image can also be found in the specific implementation of step S205, which will not be repeated here.

[0120] In an embodiment of the present application, after the object recognition features of each image in the video segment are extracted, the object recognition features of each image are classified and predicted based on the previous frame image of each image, so that part of the information of the previous frame image can be continuously transmitted to the subsequent image, and while establishing a long-term dependency relationship between image frames, it is also convenient to mine the temporal inconsistency of object authenticity, so that the subsequent images can be more accurately classified and predicted with the help of relevant information of the previous frame image, which is beneficial to improving the accuracy of the object authenticity of each image and the accuracy of the object authenticity of the corresponding video segment, and further helps to improve the detection accuracy of fictional videos.

[0121] In one embodiment of the present application, another video processing method is provided. The video processing method can be applied to Figure 1 The video processing system shown in the figure, the method can be executed by the terminal device or the server, or can be executed by the terminal device and the server together. In the embodiment of the present application, the method is executed by the server as an example for explanation. Figure 4 As shown, a flow chart of another video processing method is shown. Figure 2 and Figure 3 This paper expands on the method shown in .

[0122] Among them, S401 to S407 are described in detail as follows:

[0123] S401: Segment an initial video containing a specified object to obtain multiple video segments.

[0124] In the embodiment of the present application, the specific implementation of step S401 can refer to the specific implementation of step S201 in the above embodiment, which will not be repeated here.

[0125] S402: Determine a key frame image in each video segment based on the object display quality of the image in each video segment with respect to a designated object.

[0126] In the embodiment of the present application, the specific implementation of step S402 can refer to the specific implementation of step S202 in the above embodiment, which will not be repeated here.

[0127] S403: Perform video authenticity evaluation processing on each video segment to obtain the object authenticity of the specified object contained in each video segment.

[0128] In the embodiment of the present application, the specific implementation of step S402 can refer to the specific implementation of step S203 and steps S303 to S305 in the above embodiment, which will not be repeated here.

[0129] S404: Perform image feature extraction processing on each key frame image to obtain image features of each key frame image.

[0130] In the embodiment of the present application, the specific method of image feature extraction processing can be: calling a feature extraction network, processing each key frame image, and obtaining the image features of the key frame image. The feature extraction network can be a convolutional neural network, a recurrent convolutional neural network, or other neural network that can be used to extract image features, which is not limited here.

[0131] S405 , performing pre-classification processing on the image features of each key frame image to obtain the probability that the object truth of each key frame image is each estimated truth among multiple estimated truths.

[0132] In the embodiment of the present application, the multiple estimated realities may be continuous values ​​or fields, such as 1 to 10, 0.1 to 0.8, etc. The estimated realities may be manually set, or may be set by a terminal device or server in the above video processing system, which is not limited.

[0133] In addition, the pre-classification process is a multi-classification process; therefore, the specific method of the pre-classification process can be: calling a preset multi-classification function, processing the image features of each key frame image, and obtaining the probability that the authenticity of each key frame image is each estimated authenticity among multiple estimated authenticity. Among them, the preset multi-classification function can be EarthMover's Distance (EMD loss function); it can also be a cross entropy loss function, a normalized exponential function (i.e., a softmax function), etc., which is not limited here.

[0134] S406 , processing each estimated realism based on the probability of each estimated realism to obtain the object realism of each key frame image.

[0135] In an embodiment of the present application, based on the probability of each estimated realism, the specific method of processing each estimated realism can be: multiplying the probability of each estimated realism with the corresponding estimated realism to obtain the realism contribution value corresponding to each estimated realism; adding the realism contribution values ​​corresponding to each estimated realism to obtain the object realism of each key frame image.

[0136] For example, the estimated authenticity can be set to 1, 2, 3, 4, and 5; if after pre-classification of the image features of a key frame image, the probability that the object authenticity of the key frame image is 1 is 0.3, the probability of 2 is 0.5, the probability of 3 is 0.2, the probability of 4 is 0, and the probability of 5 is 0; then, the object authenticity of the key frame image can be finally calculated through the estimated authenticity and its probability, which is 1x0.3+2x0.5+3x0.2+4x0+5x0=1.9.

[0137] S407 , identifying whether the initial video is a fictitious video based on the object authenticity of each video segment and the object authenticity of each key frame image, so as to obtain a video detection result of the initial video.

[0138] In the embodiment of the present application, the specific implementation of step S406 can refer to the specific implementation of steps S205 and S308 in the above embodiment, which will not be repeated here.

[0139] In the specific implementation, the video authenticity assessment process, the image authenticity assessment process, and the object authenticity of the initial video can be completed through the authenticity assessment model. Figure 5 , which shows a schematic diagram of the network structure of a truth evaluation model. Figure 5 As shown, the reality assessment model 501 may include a key frame assessment module 502 (the key frame assessment module may also be referred to as a key frame reality assessment network, i.e., Keyframe image Realism Assessment, KFA), a video assessment module 503 (the video assessment module may also be referred to as a video reality assessment network, i.e., Video Realism Assessment, VFA) and a reality fusion module 504. Among them, the key frame assessment module 502 includes a feature extraction network and a multi-classification function layer connected in sequence; in practical applications, the key frame assessment module may be a convolutional neural network in which a fully connected layer is modified to an EMD function layer. The video assessment module 503 includes an object recognition network and a classification prediction network connected in sequence; among them, the object recognition network in the video assessment module 503 may be a partial network used for recognition feature extraction processing in the trained object recognition network mentioned in step S303; and the classification prediction network may be a GRU network. In addition, the reality fusion module 504 may specifically be a fully connected layer that may be used for weighted addition.

[0140] The specific training process of the authenticity evaluation model can be: after obtaining the training video and the authenticity label of the training video, the training video can be segmented to obtain multiple training video segments and determine the training key frame images of each training video segment; then, Figure 5As shown, each training key frame image can be input into the key frame evaluation module 502, and the key frame evaluation module 502 processes each training key frame image to obtain the object authenticity of each training key frame image; and each training video segment can be input into the video evaluation module 503, and the video evaluation module 503 processes each training video segment to obtain the object authenticity of each training video segment.

[0141] Afterwards, if Figure 5 As shown, the object realism of each training key frame image and the object realism of each training video segment are processed by the realism fusion module 504 to output the training realism of the training video.

[0142] Finally, the network parameters of the key frame evaluation module 502, the authenticity fusion module 504 and the classification prediction network in the video evaluation module 503 of the authenticity evaluation model 501 are trained according to the training authenticity and authenticity labels of the training videos, and finally a trained authenticity evaluation model 501 is obtained.

[0143] In the embodiment of the present application, since the pre-classification process is a multi-classification process, and the multiple estimated realities can be ordered values ​​or fields, it can be reflected that the neural network for image realism assessment processing of key frame images is an ordered class framework. The ordered class multi-classification network has better performance than the regression model in evaluation learning such as quality assessment. Therefore, the object realism of the key frame image that can be accurately obtained through the embodiment of the present application is conducive to further improving the detection accuracy of fictional videos.

[0144] In one embodiment of the present application, another video processing method is provided. The video processing method can be applied to Figure 1 The video processing system shown in the figure, the method can be executed by the terminal device or the server, or can be executed by the terminal device and the server together. In the embodiment of the present application, the method is executed by the server as an example for explanation. Figure 6 As shown, a flow chart of another video processing method is shown. Figure 2 , Figure 3 and Figure 4 This paper expands on the method shown in .

[0145] Among them, S601 to S607 are described in detail as follows:

[0146] S601: Acquire an original video containing a specified object.

[0147] In the embodiment of the present application, the size of each image in the original video is larger than the size of each image in the initial video. Specifically, compared with the initial video, each image in the original video contains more background information other than the specified object.

[0148] S602: Crop each image in the original video according to the image area containing the specified object in each image in the original video to obtain an initial video.

[0149] In the embodiment of the present application, the specific method of the cropping process may be: cropping the image area containing the specified object from each image of the original video to obtain the initial video.

[0150] Since whether the initial video is a fictional video is mainly detected by the degree of authenticity of the specified object contained in the video; and each image in the original video, in addition to the image area containing the specified object, also contains a large number of background image areas that may cause interference; therefore, video detection by using the initial video obtained after cropping can reduce background interference, which is conducive to further improving the detection accuracy of fictional videos.

[0151] Optionally, since the deep fiction technology will inevitably leave artifacts and other information about the fictional video clues in the surrounding image area of ​​the specified object, if the image area containing the specified object and the surrounding image area of ​​the image area are retained during cropping, the information about the specified object in the initial video will be richer, which is conducive to further improving the detection accuracy of the fictional video. At the same time, the size of the image area containing the specified object in each image may be different, so the image area containing the specified object can be aligned by retaining part of the surrounding image area during cropping, thereby forming the initial video.

[0152] Then, the specific method of cropping processing can be: determine the image area of ​​each image in the original video that contains the specified object; perform region expansion processing on the image area of ​​each image that contains the specified object to obtain the target image area of ​​each image; finally, crop the target image area of ​​each image from each image in the original video to obtain the initial video.

[0153] The specific method of the region expansion process may be: expanding a preset number of pixels to the image region containing the specified object in each image to obtain the target image region. The preset number may be set manually or by the terminal device or server in the above video processing system, and is not limited thereto.

[0154] S603: Segment the initial video containing the specified object to obtain multiple video segments.

[0155] In the embodiment of the present application, the specific implementation of step S603 can refer to the specific implementation of step S201 in the above embodiment, which will not be repeated here.

[0156] S604: Determine a key frame image in each video segment based on the object display quality of the image in each video segment with respect to the designated object.

[0157] In the embodiment of the present application, the specific implementation of step S604 can refer to the specific implementation of step S202 in the above embodiment, which will not be repeated here.

[0158] S605: Perform video authenticity evaluation processing on each video segment to obtain the object authenticity of the specified object contained in each video segment.

[0159] In the embodiment of the present application, the specific implementation of step S605 can refer to the specific implementation of step S203 and steps S303 to S305 in the above embodiment, which will not be repeated here.

[0160] S606: Perform image authenticity evaluation processing on each key frame image to obtain the object authenticity of the designated object contained in each key frame image.

[0161] In the embodiment of the present application, the specific implementation of step S606 can refer to the specific implementation of step S204 and steps S404 to S406 in the above embodiment, which will not be repeated here.

[0162] S607: Identify whether the initial video is a fictitious video based on the object authenticity of each video segment and the object authenticity of each key frame image to obtain a video detection result of the initial video.

[0163] In the embodiment of the present application, the specific implementation of step S607 can refer to the specific implementation of step S205 and step S307 in the above embodiment, which will not be repeated here.

[0164] In the specific implementation, the specified object can be set as a human face. Figure 7 , shows a schematic diagram of the detection process of a fictitious face video. Figure 7 As shown in FIG. 7 , after acquiring the face video 701, the face video 701 may be trimmed and segmented to obtain a video segment sequence 702 including multiple video segments. Then, as shown in FIG. Figure 7 As shown, the key frame image of each video segment in the video segment sequence 702 can be extracted to obtain a key image sequence 703.

[0165] Afterwards, each video segment in the video segment sequence 702 can be input into the VFA network in the real assessment model, and each video segment is processed by the object recognition network 704 (specifically, it can be CNN) and the classification prediction network 705 (specifically, it can be GRU) in the VFA network to obtain the object authenticity of each video segment.

[0166] Specifically, for a video segment v having m frames in the video segment sequence 702, i , the jth frame can be input into the object recognition network 704 (CNN) to obtain the object recognition feature Among them, m, i, j are positive integers, m is the video segment v i The total number of image frames in , i is less than or equal to the total number N of video segments in the video segment sequence 702, and j is less than or equal to m.

[0167] Then, Input the classification prediction network 705 (GRU) to obtain the video segment v i The object authenticity of each frame image F(f i j Finally, the object authenticity {F(v1), ..., F(v2)} of each video segment in the video segment sequence 702 can be obtained. N )}.

[0168] At the same time, each key frame image in the key image sequence 703 can be input into the KFA network 706 in the real assessment model, and the KFA network 706 (specifically, it can be a convolutional neural network in which the fully connected layer is modified to the EMD function layer) processes each key frame image to obtain the object authenticity of each key frame image.

[0169] Specifically, each key frame image kf in the key image sequence 703 is quantized by the KFA network 706. i By processing, each key frame image kf in the key image sequence 703 can be obtained. i (i=1, ..., N) object truth {F(kf1), ..., F(kf N )}.

[0170] like Figure 7As shown, after obtaining the object authenticity of each video segment and the object authenticity of each key frame image, the average value F(V) of the object authenticity of the video segments in the video sequence can be calculated as the object authenticity of the video sequence; and the average value F(KF) of the object authenticity of the key frame images in the key image sequence can be calculated as the object authenticity of the key image sequence. Finally, the object authenticity of the video sequence and the object authenticity of the key image sequence are weighted and added to obtain the final output target object authenticity F (i.e., the authenticity of the face video 701).

[0171] Specifically, the target object authenticity F is the weighted average of F(V) and F(KF), and its calculation formula is F = (1-λ)F(V) + λF(KF). After obtaining the target object authenticity, we can judge the authenticity of the face video 701 according to the preset face authenticity threshold t; wherein, if the target object authenticity F is greater than or equal to t, it can be determined that the face video 701 is real; if the target object authenticity F is less than t, it can be determined that the face video 701 is forged. Among them, the weight λ and the threshold t are hyperparameters, which are determined during the training process of the authenticity evaluation model.

[0172] In practical applications, when the specified object is a face, you can obtain face training videos and authenticity labels of face training videos from datasets such as FaceForensics++ (FF++ for short, a well-known face forgery detection dataset, containing 1,000 real videos and 4,000 corresponding forged videos generated by four deep forgery methods: DeepFakes, Face2Face, FaceSwap, and NeuralTextures), Celeb-DF (consisting of 590 real videos collected from video websites and 5,639 corresponding forged videos, with relatively high forgery quality and relatively fewer visual artifacts), and WildDeepfake (a more challenging real-world dataset, containing 3,805 real face sequences and 3,509 forged face sequences collected from the Internet. The WildDeepfake dataset is more diverse in forgery methods, video length, video quality, and video content) to serve as training data for the authenticity evaluation model. Among them, in order to facilitate training, the authenticity labels of face training videos can also be normalized to the range of 0 to 1.

[0173] After obtaining the face training video and the authenticity label of the face training video, considering that the fictional video about the face is just an edit of the facial position in the face image, in order to avoid excessive interference from other information in the image and further improve the detection accuracy, Dlib (a modern programming language toolbox that contains machine learning algorithms and tools for creating complex software in C++ to solve practical problems) can be used as a face detection and alignment module to detect the face area of ​​each frame in the face training video, and cut out the face image of each frame from the face training video according to the face area of ​​each frame; then Dlib can align multiple frames of face images to form a target training video.

[0174] At the same time, since there will be some visual artifacts around the faces in fictional videos, and the feature information about the visual artifacts can assist in detecting whether the video is a fictional video; therefore, in order to include certain background information, Dlib can expand the image area of ​​20 pixels around the face area as the final face area after detecting the face area of ​​each frame image; then, according to the final face area, the face image is cropped from the corresponding image of the face training video, and the resolution of the cropped face image is reset to 224*224 to obtain the final face image.

[0175] Then, the face video authenticity assessment model to be trained is trained using the face training video and its authenticity label to obtain the face video authenticity assessment model. The training process of the face video authenticity assessment model can be found in Figure 5 The specific implementation process in the example will not be described here.

[0176] Among them, through experiments, we can obtain the detection comparison of the face video authenticity evaluation model with MesoNet (a method of detecting forged faces using deep learning), Face X-ray (face X-ray detection, a method of detecting forged faces), RFM (Representative Forgery Mining for Fake Face Detection), Add-Net (Deep Neural Networks Using FPGA-Optimized Multipliers), F3-Net (a fictitious video detection model proposed by SenseTime), and other fictitious video detection methods, as shown in Table 1:

[0177] Table 1

[0178]

[0179] Among them, Ours represents the face video authenticity evaluation model in this solution. ACC is an important indicator for evaluating the performance of machine learning algorithms, indicating the accuracy of machine learning models. AUC, namely Area Under Curve, the closer the AUC is to 1, the higher the authenticity of the detection method; when AUC is equal to 0.5, it means that the detection method has the lowest authenticity and has no application value. In addition, the '-' in the table means that the result is unavailable, and the one in bold is the best result.

[0180] As can be seen from Table 1, the face video authenticity assessment model trained by this scheme has the highest ACC and AUC on the FF++, Celeb-DF and WildDeepfake datasets, and has a certain degree of improvement in ACC and AUC compared with other fictional video detection methods.

[0181] In addition, in order to verify the generalization performance of this scheme, cross-dataset experiments can be further conducted, using the same fictional video detection method to perform training and testing on different datasets.

[0182] Specifically, we can train on the FF++ dataset and then test on the WildDeepfake dataset, and obtain the AUC of MLDG (Meta-learning for domain generalization), F3-Net (a fictitious video detection model proposed by SenseTime), Local-relation (a fictitious video detection algorithm using frequency graphs), and the face video authenticity assessment model of this solution on the FF++ dataset and the AUC on the WildDeepfake dataset. The experimental results are shown in Table 2:

[0183] Table 2

[0184]

[0185] It can be seen from Table 2 that the AUC of the face video authenticity assessment model of this scheme is higher than that of other fictitious video detection methods, whether in the training stage or in the testing stage; it can be seen that the face video authenticity assessment model of this scheme has good generalization performance.

[0186] Furthermore, in order to verify the effectiveness of each module in the face video authenticity assessment model proposed in this scheme, corresponding ablation experiments can also be designed.

[0187] First, we can design a w / o frame solution, that is, only retain the VFA network in the face video authenticity assessment model, and keep other experimental settings unchanged; and a w / o video solution, that is, only retain the KFA network in the video detection model, and keep other experimental settings unchanged. The experimental results of the above two solutions and the complete face video authenticity assessment model are shown in Table 3, which shows the comparison of VFA and KFA on ACC:

[0188] Table 3

[0189]

[0190] Among them, the fictional videos in the FF++(df) dataset are generated using the DeepFakes method, the fictional videos in the FF++(f2f) dataset are generated using the Face2Face method, the fictional videos in the FF++(fs) dataset are generated using the FaceSwap method, and the fictional videos in the FF++(nt) dataset are generated using the NeuralTextures method.

[0191] It can be seen from Table 3 that if only the VFA network in the face video authenticity assessment model is retained to detect fictitious videos, the ACC will decrease to varying degrees; if only the KFA network in the face video authenticity assessment model is retained, the ACC will remain unchanged or slightly decrease; therefore, including both VFA and KFA in the face video authenticity assessment model can effectively improve the detection accuracy of fictitious face videos.

[0192] Secondly, two key frame image selection strategies can be designed. One is the randomframe scheme, which randomly extracts images from the video segment as key frame images, and other experimental settings remain unchanged; the other is the worstframe scheme, which selects the image with the worst face quality in the video segment as the key frame image, and other experimental settings remain unchanged. The experimental results of the face video authenticity assessment model of the above two schemes and this scheme are shown in Table 4, which shows the comparison of the key frame image selection strategies on ACC:

[0193] Table 4

[0194]

[0195] As can be seen from Table 4, compared with this scheme, the ACC of the randomframe scheme and the worstframe scheme mostly show a downward trend. In other words, whether it is randomly extracting images in the video segment as key frame images, or selecting the image with the worst face quality in the video segment as the key frame image, the effect is not as good as selecting the image with better face quality as the key frame image in this scheme.

[0196] In a possible implementation, the trained face detection and alignment module, the key frame extraction module for determining the key frame image, and the face video authenticity assessment model can also be integrated into a large video detection framework. Figure 8 , shows a schematic diagram of a video detection framework. Figure 8 As shown, the video detection framework includes an input end 801, a detection end 802 and an output end 803; wherein the detection end 802 includes a face detection and alignment module, a video evaluation module, a key frame extraction module, a key frame evaluation module and a reality fusion module.

[0197] The detection personnel can upload the face video to the input terminal 801, and then the input terminal 801 will send the face video to the detection terminal 802. The face detection and alignment module in the detection terminal 802 will crop and segment the face video to obtain a sequence of face video segments.

[0198] Afterwards, the video evaluation module in the detection end 802 processes the video segments in the face video segment sequence to obtain the face authenticity of each video segment. At the same time, the key frame extraction module in the detection end 802 determines the key frame images of each video segment in the face video segment sequence to form a key image sequence; and the key frame evaluation module in the detection end 802 processes the key frame images in the key image sequence to obtain the face authenticity of each key frame image. Finally, the authenticity fusion module in the detection end 802 processes the face authenticity of each video segment and the face authenticity of each key frame image to obtain the target face authenticity.

[0199] After obtaining the target face authenticity, the detection end 802 may send the target face authenticity to the output end 803, and the target face authenticity is output by the output end 803. Optionally, the output end 803 may also compare the target face authenticity with a preset face authenticity threshold, and if the target face authenticity is less than the preset face authenticity threshold, the output end 803 may generate a video detection result indicating that the face video is a fictitious video; if the target face authenticity is greater than or equal to the preset face authenticity threshold, the output end 803 may generate a video detection result indicating that the face video is a real video.

[0200] In an embodiment of the present application, by cropping the original video, interference caused by background areas other than the specified object in the video screen can be avoided to a certain extent; subsequent authenticity evaluations of the video segment and the authenticity evaluations of the key frame images are focused on the specified object, which is beneficial to improving the accuracy of the object authenticity of the video segment and the object authenticity of the key frame images, thereby helping to further improve the accuracy of detecting fictional videos.

[0201] Here, the device embodiment of the present application is introduced, which can be used to execute the video processing method in the above embodiment of the present application. For details not disclosed in the device embodiment of the present application, please refer to the above embodiment of the video processing method of the present application.

[0202] The present application embodiment provides a video processing device, such as Fig. 9 As shown, the device includes an acquisition unit 901 and a processing unit 902, wherein:

[0203] An acquisition unit 901 is used to segment an initial video containing a specified object to obtain multiple video segments;

[0204] The acquisition unit 901 is further configured to determine a key frame image in each video segment based on the object display quality of the image in each video segment with respect to the designated object;

[0205] The processing unit 902 is used to perform video authenticity evaluation processing on each video segment to obtain the object authenticity of the specified object contained in each video segment;

[0206] The processing unit 902 is further used to perform image authenticity evaluation processing on each key frame image to obtain the object authenticity of the specified object contained in each key frame image;

[0207] The processing unit 902 is further configured to identify whether the initial video is a fictitious video based on the object authenticity of each video segment and the object authenticity of each key frame image, so as to obtain a video detection result of the initial video.

[0208] In one embodiment of the present application, based on the aforementioned scheme, when the processing unit 902 performs video authenticity evaluation processing on each video segment to obtain the object authenticity of the specified object contained in each video segment, it can be specifically used to: perform recognition feature extraction processing on each image in each video segment to obtain the object recognition feature of each image; based on the previous frame image of each image, perform classification prediction processing on the object recognition feature of each image to obtain the object authenticity of each image; based on the object authenticity of each image, determine the object authenticity of the corresponding video segment.

[0209] In one embodiment of the present application, based on the aforementioned scheme, when the processing unit 902 performs classification and prediction processing on the object recognition features of each image based on the previous frame image of each image to obtain the object authenticity of each image, it can be specifically used for: in the process of traversing each image, performing state extraction processing on the object recognition features of the current traversed image to obtain the hidden state features of the current traversed image; based on the hidden state features of the previous frame image of the current traversed image, performing feature classification processing on the hidden state features of the current traversed image to obtain the object authenticity of the current traversed image.

[0210] In one embodiment of the present application, based on the aforementioned scheme, when the processing unit 902 performs image authenticity evaluation processing on each key frame image to obtain the object authenticity of a specified object contained in each key frame image, it can be specifically used to: perform image feature extraction processing on each key frame image to obtain the image features of each key frame image; perform classification processing on the image features of each key frame image to obtain the object authenticity of each key frame image.

[0211] In one embodiment of the present application, based on the aforementioned scheme, when the processing unit 902 classifies the image features of each key frame image to obtain the object authenticity of each key frame image, it can be specifically used to: pre-classify the image features of each key frame image to obtain the probability that the object authenticity of each key frame image is each estimated realities among multiple estimated realities; based on the probability of each estimated realities, process each estimated realities to obtain the object authenticity of each key frame image.

[0212] In one embodiment of the present application, based on the aforementioned scheme, the acquisition unit 901 can also be used to: acquire an original video containing a specified object; wherein the size of each image in the original video is larger than the size of each image in the initial video; and according to the image area of ​​each image in the original video containing the specified object, crop each image in the original video to obtain the initial video.

[0213] In one embodiment of the present application, based on the aforementioned scheme, when the processing unit 902 identifies whether the initial video is a fictional video based on the object realism of each video segment and the object realism of each key frame image to obtain a video detection result of the initial video, it can be specifically used to: determine the object realism of the initial video based on the object realism of each video segment and the object realism of each key frame image; if the object realism of the initial video is less than the preset realism, generate a video detection result indicating that the initial video is a fictional video.

[0214] In one embodiment of the present application, multiple video segments constitute a video sequence, and multiple key frame images constitute a key image sequence; based on the aforementioned scheme, when the processing unit 902 determines the object authenticity of the initial video based on the object authenticity of each video segment and the object authenticity of each key frame image, it can be specifically used to: determine the object authenticity of the video sequence based on the object authenticity of each video segment; determine the object authenticity of the key image sequence based on the object authenticity of each key frame image; calculate the weighted sum of the object authenticity of the video sequence and the object authenticity of the key image sequence to obtain the object authenticity of the initial video.

[0215] It should be noted that the device provided in the above embodiment and the method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment and will not be repeated here.

[0216] The device provided in the above embodiment can be arranged in a terminal device or in a server. The device provided in the embodiment of the present application can accurately integrate the fictitious video clue information in different time periods by dividing the video into multiple video segments to perform video authenticity evaluation processing respectively, so as to evaluate the object authenticity of the video from multiple time periods, and mine the time inconsistency of the object authenticity, thereby making the object authenticity of the video segment more diverse and rich, which is conducive to the subsequent accurate detection of fictitious videos. At the same time, the embodiment of the present application can avoid the interference caused by low-quality images in the video segment (such as images where the specified object is blocked or unclear) to a certain extent by performing image authenticity evaluation processing on the key frame images in each video segment, reduce the false detection of fictitious videos, and thus improve the detection accuracy of fictitious videos.

[0217] An embodiment of the present application also provides an electronic device, comprising one or more processors and a storage device, wherein the storage device is used to store one or more computer programs, and when the one or more computer programs are executed by one or more processors, the electronic device implements the above video processing method.

[0218] Fig.10 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.

[0219] It should be noted that Fig.10 The computer system 1000 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0220] like Fig.10 As shown, the computer system 1000 includes a processor (Central Processing Unit, CPU) 1001, which can perform various appropriate actions and processes according to the program stored in the read-only memory (Read-Only Memory, ROM) 1002 or the program loaded from the storage part 1008 to the random access memory (Random Access Memory, RAM) 1003, such as executing the method in the above embodiment. In RAM 1003, various programs and data required for system operation are also stored. CPU 1001, ROM 1002 and RAM 1003 are connected to each other through bus 1004. Input / output (Input / Output, I / O) interface 1005 is also connected to bus 1004.

[0221] In some embodiments, the following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0222] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part 1009, and / or installed from a removable medium 1011. When the computer program is executed by a processor (CPU) 1001, various functions defined in the system of the present application are executed.

[0223] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (Erasable Programmable Read Only Memory), a flash memory, an optical fiber, a portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal can take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0224] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the devices, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and a computer program.

[0225] The units or modules involved in the embodiments described in this application may be implemented by software or hardware, and the units or modules described may also be set in a processor. The names of these units or modules do not, in some cases, constitute limitations on the units or modules themselves.

[0226] Another aspect of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the video processing method as described above is implemented. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0227] Another aspect of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of an electronic device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the electronic device executes the video processing method as described above in the above embodiments.

[0228] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0229] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0230] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person of ordinary skill in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. A video processing method, characterized in that: include: Segmenting an initial video containing a specified object to obtain a plurality of video segments; Determining key frame images in each video segment based on the object display quality of the image in each video segment with respect to the designated object; Performing video authenticity evaluation processing on each of the video segments to obtain object authenticity of a specified object contained in each of the video segments; Performing image authenticity evaluation processing on each key frame image to obtain the object authenticity of a specified object contained in each key frame image; Based on the object authenticity of each video segment and the object authenticity of each key frame image, it is identified whether the initial video is a fictitious video to obtain a video detection result of the initial video.

2. The method according to claim 1, characterized in that The performing video authenticity evaluation processing on each video segment to obtain the object authenticity of a specified object contained in each video segment includes: Performing recognition feature extraction processing on each image in each video segment to obtain object recognition features of each image; Based on the previous frame image of each image, classify and predict the object recognition features of each image to obtain the object authenticity of each image; Based on the object realities of the respective images, the object realities of the corresponding video segments are determined.

3. The method according to claim 2, characterized in that The performing classification prediction processing on the object recognition features of each image based on the previous frame image of each image to obtain the object authenticity of each image includes: In the process of traversing each of the images, performing state extraction processing on the object recognition features of the currently traversed image to obtain the hidden state features of the currently traversed image; Based on the hidden state features of the previous frame image of the current traversal image, feature classification processing is performed on the hidden state features of the current traversal image to obtain the object authenticity of the current traversal image.

4. The method according to claim 1, characterized in that: The performing image authenticity evaluation processing on each key frame image to obtain the object authenticity of the specified object contained in each key frame image includes: Performing image feature extraction processing on each of the key frame images to obtain image features of each of the key frame images; The image features of each key frame image are classified to obtain the object authenticity of each key frame image.

5. The method according to claim 4, characterized in that The classifying and processing the image features of each key frame image to obtain the object authenticity of each key frame image includes: Pre-classifying the image features of each key frame image to obtain the probability that the object authenticity of each key frame image is each estimated authenticity among multiple estimated authenticity; Based on the probabilities of the respective estimated realities, the respective estimated realities are processed to obtain the object realities of the respective key frame images.

6. The method according to claim 1, characterized in that The method further comprises: Acquire an original video containing the specified object; wherein the size of each image in the original video is larger than the size of each image in the initial video; According to the image area of ​​each image in the original video containing the designated object, each image in the original video is cropped to obtain the initial video.

7. The method according to any one of claims 1 to 6, characterized in that The identifying whether the initial video is a fictitious video based on the object authenticity of each video segment and the object authenticity of each key frame image to obtain a video detection result of the initial video includes: Determining the object authenticity of the initial video based on the object authenticity of each video segment and the object authenticity of each key frame image; If the object authenticity of the initial video is less than a preset authenticity, a video detection result indicating that the initial video is a fictitious video is generated.

8. The method according to claim 7, characterized in that The multiple video segments form a video sequence, and the multiple key frame images form a key image sequence; and determining the object authenticity of the initial video based on the object authenticity of each video segment and the object authenticity of each key frame image comprises: Determining the object realism of the video sequence based on the object realism of each video segment; Determining the object authenticity of the key image sequence based on the object authenticity of each key frame image; The weighted sum of the object authenticity of the video sequence and the object authenticity of the key image sequence is calculated to obtain the object authenticity of the initial video.

9. A video processing device, characterized in that: It includes an acquisition unit and a processing unit, wherein: The acquisition unit is used to segment the initial video containing the specified object to obtain multiple video segments; The acquisition unit is further used to determine the key frame images in each video segment based on the object display quality of the image in each video segment with respect to the designated object; The processing unit is used to perform video authenticity evaluation processing on each video segment to obtain the object authenticity of the specified object contained in each video segment; The processing unit is further used to perform image authenticity evaluation processing on each key frame image to obtain the object authenticity of the specified object contained in each key frame image; The processing unit is further configured to identify whether the initial video is a fictitious video based on the object authenticity of each video segment and the object authenticity of each key frame image, so as to obtain a video detection result of the initial video.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the video processing method according to any one of claims 1 to 8 is implemented.

11. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the video processing method according to any one of claims 1 to 8.

Images