Deepfake analysis system and method using face and behavior pattern analysis based on artificial intelligence model

Abstract

Disclosed are a deepfake analysis system and method using face and behavior pattern analysis based on an artificial intelligence model. A deepfake analysis system according to an embodiment of the present invention comprises: a target selection unit for determining an analysis section in an input video on the basis of a specific time period or a specific person according to preconfigured criteria, extracting feature points of an analysis person selected from the analysis section of the input video, and then generating and providing an analysis target video that is obtained by processing a part to be analyzed to a preconfigured size; a deepfake analysis unit for performing tampering texture analysis according to a preconfigured first analysis criterion, including analysis of spatial tampering information and temporal tampering information, on the basis of the analysis target video, and performing behavior pattern analysis according to a preconfigured second analysis criterion on the basis of the analysis target video, thereby identifying whether the person is a real person; and a deepfake result processing unit for inferring a comprehensive score by adjusting weights of tampering texture analysis and behavior pattern analysis results according to preconfigured comprehensive analysis criteria, and acquiring authenticity and analysis results of the analysis target video on the basis of the inferred comprehensive score and a threshold.

Description

Deepfake Analysis System and Method Using Artificial Intelligence Model-Based Facial and Behavioral Pattern Analysis

[0001] The disclosed embodiments relate to deepfake analysis systems and methods.

[0002] Generative AI models are a field that is experiencing relatively rapid growth across various AI-based fields. These generative AI models can not only converse like real people, but also create images that are difficult to distinguish from real ones. These generative AI models can also implement deepfake technology, which imitates specific individuals.

[0003] Unlike the crude imitations of certain celebrities in the past, deepfake technology has gradually developed to a level of perfection that makes it difficult to distinguish them from real people. Consequently, blackmail sexual crimes using fake creations are occurring in everyday life, and the scope of these crimes is gradually expanding to target ordinary people. Furthermore, videos imitating world-famous leaders are expanding into areas such as election-related fake news.

[0004] The disclosed embodiments provide a deepfake analysis system and method using an artificial intelligence model-based facial and behavioral pattern analysis to determine whether a video is forged by analyzing characteristics including behavioral patterns of the analyzed person in the video.

[0005] According to one embodiment, a deepfake analysis system includes: a target selection unit for determining an analysis section based on a specific time section or a specific person in an input video according to preset criteria, extracting feature points of a selected analysis person in the analysis section of the input video, and generating and providing an analysis target video in which a part to be analyzed is processed to a preset size; a deepfake analysis unit for performing forgery texture analysis based on the analysis target video according to preset first analysis criteria, including analysis of spatial forgery information and temporal forgery information, and performing behavioral pattern analysis based on the analysis target video according to preset second analysis criteria to determine whether the video is a real person; and a deepfake result processing unit for adjusting weights of results of the forgery texture analysis and the behavioral pattern analysis according to preset comprehensive analysis criteria to infer a comprehensive score, and obtaining an analysis result and determining whether the video is authentic or not based on the inferred comprehensive score and a threshold point.

[0006] The above target selection unit, when extracting feature points of the analysis person, extracts feature points by analyzing bounding box coordinates and feature point coordinates for the face or joints of the analysis person using coordinate information for the facial bounding box and body region of the analysis person. If the analysis target video is based on the face of the analysis person, facial feature points can be extracted, and if the analysis target video is based on the upper body of the analysis person, facial feature points and upper body joint feature points can be extracted.

[0007] The above target selection unit processes the shape and size and performs a pooling operation according to the criteria for each artificial intelligence model for deepfake analysis for characteristic analysis to be applied to the analysis target video when processing the part to be analyzed to a preset size, and calculates a trajectory for the center of gravity of the feature points and calculates coordinates of a size that includes all of the feature points to determine a crop size of the analysis target video.

[0008] The above target selection unit can, when processing the analysis target video to the preset size, crop and separately store each preset section of the entire body of the analysis subject.

[0009] The above deepfake analysis unit can obtain the spatial forgery information by analyzing the first synthetic defect criterion, including the spatial feature and frequency feature according to the application of the artificial intelligence model in the video to be analyzed.

[0010] The above deepfake analysis unit can analyze the video to be analyzed based on the second synthetic defect criteria, including discontinuous scenes or exposure defects resulting from the application of an artificial intelligence model, and obtain the temporal forgery information by recognizing differences from the actual video.

[0011] The above deepfake analysis unit, when acquiring the spatial forgery information, can identify correlations through cross-attention for the spatial characteristics and the frequency characteristics, and process them into a form that complements different data formats through correlation map inference.

[0012] The above deepfake analysis unit can extract preset behavioral characteristics (attributions) from the video to be analyzed, and analyze the time series correlation of the extracted characteristics to identify anomalies in the behavioral patterns of the analyzed person.

[0013] According to another embodiment, a deepfake analysis method is provided, which is performed by a deepfake analysis system, comprising: a step in which the deepfake analysis system determines an analysis section based on a specific time section or a specific person according to preset criteria in an input video, extracts feature points of a selected analysis person from the analysis section of the input video, and then generates and provides an analysis target video in which a part to be analyzed is processed to a preset size; a step in which a forgery texture analysis is performed based on the analysis target video according to preset first analysis criteria, including analysis of spatial forgery information and temporal forgery information, and a step in which a behavioral pattern analysis is performed based on the analysis target video according to preset second analysis criteria to determine whether the video is a real person; and a step in which a weight is adjusted based on a preset comprehensive analysis criterion to infer a comprehensive score based on the results of the forgery texture analysis and the behavioral pattern analysis, and a step in which the authenticity of the analysis target video and an analysis result are obtained based on the inferred comprehensive score and a threshold point.

[0014] In the above deepfake analysis method, in the step of generating and providing the analysis target video, when extracting feature points of the analysis subject, feature points are extracted by analyzing bounding box coordinates and feature point coordinates for the face or joints of the analysis subject using coordinate information for the facial bounding box and body region of the analysis subject. If the analysis target video is based on the face of the analysis subject, facial feature points can be extracted, and if the analysis target video is based on the upper body of the analysis subject, facial feature points and upper body joint feature points can be extracted.

[0015] In addition, a computer-readable recording medium recording a computer program for executing a method for implementing the disclosed embodiment may be further provided.

[0016] According to the disclosed embodiments, it is expected that the detection performance of videos to which deepfake technology is applied can be improved by analyzing not only the image texture analysis method for analyzing traces of forgery in videos, but also the behavioral patterns of the analyzed person in parallel to determine how much correlation there is with the behavioral patterns of real people, and by focusing on examining areas that artificial intelligence models cannot imitate.

[0017] Additionally, according to the disclosed embodiments, crimes caused by videos using deepfake technology can be prevented in advance.

[0018] Figure 1 is a block diagram illustrating a deepfake analysis system according to one embodiment.

[0019] Figure 2 is a block diagram for explaining the deepfake analysis system of Figure 1 in more detail.

[0020] Figures 3 to 7 are exemplary diagrams for explaining a deepfake analysis method according to one embodiment.

[0021] Figure 8 is a flowchart for explaining a deepfake analysis method according to one embodiment.

[0022] FIG. 9 is a block diagram illustrating a computing environment including a computing device according to one embodiment.

[0023] Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. The following detailed description is provided to facilitate a comprehensive understanding of the methods, devices, and / or systems described herein. However, these are merely examples and the present invention is not limited thereto.

[0024] In describing embodiments of the present invention, if a detailed description of a known technology related to the present invention is judged to unnecessarily obscure the gist of the present invention, the detailed description will be omitted. In addition, the terms described below are terms defined in consideration of their functions in the present invention, and this may vary depending on the intention or custom of the user or operator. Therefore, the definitions should be made based on the contents throughout this specification. The terminology used in the detailed description is only for the purpose of describing embodiments of the present invention and should not be limited in any way. Unless clearly used otherwise, the singular form includes the plural form. In this description, expressions such as "comprises" or "having" are intended to indicate certain features, numbers, steps, operations, elements, parts or combinations thereof, and should not be construed to exclude the presence or possibility of one or more other features, numbers, steps, operations, elements, parts or combinations thereof other than those described.

[0025] FIG. 1 is a block diagram for explaining a deepfake analysis system according to one embodiment, and FIG. 2 is a block diagram for explaining the deepfake analysis system of FIG. 1 in more detail.

[0026] Hereinafter, a description will be given with reference to FIGS. 3 to 7, which are exemplary diagrams for explaining a deepfake analysis method according to one embodiment.

[0027] The deepfake analysis system (100) may be configured to detect whether a deepfake operation, such as a generative artificial intelligence model, has been applied to an input video using an artificial intelligence model for deepfake analysis. Specifically, the deepfake analysis system (100) can detect whether a person included in the input video is a real person who has not been forged or a fake person who has been forged. To this end, the artificial intelligence model for deepfake analysis applied in the present embodiment may be pre-trained by inputting information about a specific person in advance. The analysis person disclosed in the present embodiment may be a famous person such as a broadcaster, politician, entertainer, YouTuber, etc., but is not limited thereto. If the analysis person is a famous person, the deepfake analysis system (100) may generate an artificial intelligence model for deepfake analysis through pre-training using information about the famous person (e.g., appearance including face, behavioral patterns, etc.). The deepfake analysis system (100) described below may be explained by using an example in which the above-described pre-trained artificial intelligence model for deepfake analysis is applied. The above pre-trained artificial intelligence models for deepfake analysis can be implemented independently of each other or integrated, depending on the role to which they are applied.

[0028] Referring to FIG. 1, the deepfake analysis system (100) includes a target selection unit (110), a deepfake analysis unit (130), and a deepfake result processing unit (150). The components illustrated in FIG. 1 are not essential for implementing the deepfake analysis system (100) according to the present disclosure, and thus the deepfake analysis system (100) described in this specification may have more or fewer components than the components listed above.

[0029] The components illustrated in FIG. 1 may be communicatively connected to one another via a communications network (not shown). In some embodiments, the communications network may include the Internet, one or more local area networks, a wide area network, a cellular network, a mobile network, other types of networks, or a combination of these networks.

[0030] Referring to FIG. 2, the target selection unit (110) can perform the operations of video clipping (①), selecting a person to be analyzed (②), extracting feature points (③), and ROI Pooling (④).

[0031] The target selection unit (110) may determine an analysis section based on a specific time section or a specific person in the input video according to preset criteria, extract characteristic points of the analysis person selected in the analysis section of the input video, and then generate and provide an analysis target video by processing the part to be analyzed to a preset size. The input video may refer to a video input for deepfake analysis.

[0032] Specifically, the target selection unit (110) can determine a time section to be analyzed primarily within the input video when clipping the video (① in FIG. 2).

[0033] For example, the target selection unit (110) may determine a specific time interval, with a start time and end time selected within an input video provided by the user, as a deepfake analysis interval. This method may be applied to detecting and selecting an interval to be intensively analyzed within an input video when the user clearly recognizes, based on their own judgment, an interval suspected of being altered within the input video. However, the present invention is not limited thereto.

[0034] As another example, the target selection unit (110) may implement an automatic selection method that recommends an analysis time interval based on preset automatic time interval setting criteria based on the analysis subject selected by the user. This method is applicable when detection is desired regardless of a specific time interval within the input video, and may enable detection across the entire time interval, but is not limited thereto.

[0035] Specifically, when applying the automatic selection method, the target selection unit (110) can detect a video section that is determined to be the same action and the same scene by calculating the Intersection Over Union (IOU) of the previous frame and the current frame in the input video using an object detection model or a face detection model for the face of the analysis subject selected by the user. At this time, the target selection unit (110) can determine whether a scene has changed because the IOU of the bound box decreases when the scene changes or moves rapidly.

[0036] The above IOU can be calculated by dividing the intersection of the current frame and the previous frame in the input video by the union of the current frame and the previous frame in the current input video. In other words, the IOU is calculated by dividing the Area of ​​Intersection by the Area of ​​Union. ) can be. At this time, the target selection unit (110) can determine that the scene has changed as the intersection becomes smaller, meaning that the overlapping part between the previous bound box and the current bound box has become smaller. In addition, the target selection unit (110) can determine that the bound boxes are in almost the same position if the intersection is above a certain threshold. At this time, if the scene has changed to another scene and the IOU is higher than the threshold, the target selection unit (110) can determine the scene change by calculating the difference between the previous frame and the current frame.

[0037] The difference between the previous frame and the current frame can be calculated through IOU, which is calculated by dividing the intersection of the current frame and the previous frame in the input video by the union of the current frame and the previous frame in the current input video.

[0038] Specifically, the target selection unit (110) can infer bounding box coordinates (rectangular boxes representing the area of ​​a person) representing the area of ​​a person (character) for each frame in the input video using a deep learning model. The target selection unit (110) calculates the degree of overlap between the bounding box of a specific nth frame and the bounding box of the n+1 frame using the bounding box coordinates, and if the difference is less than a threshold value, it is considered to be the same scene, and if the difference is greater than the threshold value, it can be determined that a scene change has occurred. For example, if a video of someone having a discussion is input, and a scene where person A speaks changes to a scene where person B answers, the bounding box positions of A and B become different, so the IOU value between the frames at the time of the scene change becomes greater than the threshold value, and it can be determined that a scene change has occurred. The difference between the bound box positions of A and B can be determined by the difference in the bound box positions due to the different positions of A and B on the screen or the different appearances of the characters.

[0039] Referring to FIG. 2, the process of selecting a person to be analyzed in the target selection unit (110) can be performed when a specific time section is determined by the user during video clipping (② of FIG. 2).

[0040] The target selection unit (110) can determine the analysis person to be analyzed in detail in a specific time period.

[0041] At this time, if there is only one person appearing in a specific time period, the target selection unit (110) can automatically recognize that person as the analysis person by default.

[0042] If there are multiple people appearing in a specific time period, the target selection unit (110) can select an analysis subject from among the multiple people.

[0043] For example, if there are two or more people in the video, the target selection unit (110) can automatically select a person with a relatively larger face as the person to be analyzed and automatically perform the target selection and preprocessing procedures. This may be because, in the case of a deepfake video featuring multiple people, the face of the main person to whom the deepfake is applied occupies a relatively larger area on the screen than the other faces, and thus code logic is applied to select the bounding box with the largest face size.

[0044] As another example, the target selection unit (110) can continuously track and find a person initially designated as an analysis person by comprehensively judging how much the center point of the bounding box found in the previous frame has changed in the IoU analysis described above.

[0045] Once the selection of the analysis subject is completed, the target selection unit (110) can identify the spatial area to be analyzed based on the selected analysis subject using a detection model, including an object detection model and a face detection model, and an IOU analysis method. At this time, if the resolution of the spatial area to be analyzed is greater than a reference value, the target selection unit (110) can identify the face area or body area after reducing the resolution through a pooling operation or a resizing operation. As a result, it is expected that the effect of preventing in advance that it may be computationally inefficient if the resolution of the spatial area to be analyzed is greater than a reference value can be expected.

[0046] Referring to FIGS. 3 and 4, when extracting feature points of an analysis person, the target selection unit (110) can utilize coordinate information for a facial bounding box (e.g., person A, person B) and a body region for the analysis person. The target selection unit (110) can extract feature points by analyzing the bounding box coordinates and feature point coordinates for the face or joints of the analysis person (③ of FIG. 2).

[0047] At this time, the target selection unit (110) can extract facial feature points if the analysis target video is based on the face of the analysis subject, and can extract facial feature points and upper body joint feature points if the analysis target video is based on the upper body of the analysis subject. When determining whether the analysis subject is based on the face or the upper body, the target selection unit (110) can determine based on the existence probability (confidence) of the upper body joint, and if the determination result shows upper body information, the upper body joint feature points can be extracted.

[0048] Referring to FIG. 4, when extracting feature points of the above-described analysis person, considering that if the entire resolution of the input video is scanned and utilized, the convolution operation for feature point extraction is excessively large and inefficient, the target selection unit (110) can crop only the facial area or body area detected from the analysis person when clipping the video and selecting the analysis person, pool the area for the area of ​​interest, and infer feature points from the pooled area of ​​interest.

[0049] Once the bounding box coordinate and feature point coordinate analysis is completed, the target selection unit (110) can calibrate the analysis target video to the original resolution value prior to pooling. Through this, the target selection unit (110) can efficiently obtain accurate analysis values ​​with relatively small errors between analyzing the entire image prior to pooling and analyzing the pooled image.

[0050] Referring to FIG. 5, the target selection unit (110) can process the part to be analyzed from the analysis target video by cutting it into an appropriate size (a preset size reflecting the free area) based on the part through ROI pooling. Using the entire image of the input video as input may cause other textures to be input other than the region of interest, which may act as a type of noise, and this may prevent the original performance of the deepfake analysis artificial intelligence model applied to this embodiment from being fully utilized. Accordingly, the target selection unit (110) of this embodiment can process the input video into an input form corresponding to each deepfake analysis artificial intelligence model for characteristic analysis, so that the optimal analysis performance of the deepfake analysis artificial intelligence model applied can be utilized.

[0051] Specifically, when processing a part to be analyzed into a preset size, the target selection unit (110) processes the shape and size according to the criteria for each artificial intelligence model for deepfake analysis for characteristic analysis to be applied to the video to be analyzed, and performs a pooling operation. In addition, the target video to be analyzed may be cropped to a size by calculating the trajectory for the center of gravity of the feature points and calculating the coordinates of a size that includes all feature points. Processing the shape and size and performing the pooling operation may be an ROI Pooling operation.

[0052] For example, in the case of a face, the target selection unit (110) can calculate a path for the center of gravity of all feature points and calculate the coordinates of an area that can include all the calculated paths. At this time, the target selection unit (110) can calculate a margin using the Left Top and Right Bottom coordinates. In the case of a body, the target selection unit (110) can perform a task of analyzing the centers of gravity of all feature points to calculate the center point in the same way as in the case of the face, and cropping so that a preset margin is reflected by utilizing the maximum activity radius of each body.

[0053] If the entire body is cropped, the range of motion becomes wider than that of a local area, so the margin is set relatively large, which may result in unnecessary information being included in the information to be analyzed, resulting in poor detection performance. In other words, since cropping can be done based on a relatively large spatial area, even areas unnecessary for analysis may be included in the analysis part, and information that would otherwise become noise may be applied during analysis. Accordingly, when the target selection unit (110) processes the analysis target video to a preset size, it may crop and separately store the entire body of the person being analyzed by preset division area (for example, by joint). For example, since the target selection unit (110) separately crops and stores the right arm and the left arm, the deepfake analysis unit (130) can then separately preprocess and analyze the right arm and the left arm, respectively.

[0054] Referring to Figure 2, the deepfake analysis unit (130) can perform forgery texture analysis (⑤) and behavior pattern analysis (⑥).

[0055] Below, a method for detecting artifacts that may occur when performing deepfakes will be described in more detail with reference to the forged texture analysis (⑤) and behavioral pattern analysis (⑥) of Fig. 2.

[0056] The deepfake analysis unit (130) performs forgery texture analysis based on the analysis target video according to a first analysis criterion set in advance, including analysis of spatial forgery information and temporal forgery information, and performs behavioral pattern analysis based on the analysis target video according to a second analysis criterion set in advance, thereby determining whether the subject is a real person. The first analysis criterion may be a criterion arbitrarily set by the operator for forgery texture analysis. The second analysis criterion may be a criterion arbitrarily set by the operator for behavioral pattern analysis.

[0057] The deepfake analysis unit (130) can obtain spatial forgery information by analyzing the spatial features and frequency features according to the first synthetic defect criteria, including the spatial features and frequency features according to the application of an artificial intelligence model (e.g., a generative artificial intelligence model) to the video to be analyzed.

[0058] In the spatial analysis of the disclosed embodiment, spatial forgery can refer to a method of determining that it is a synthesis (deepfake) by comprehensively analyzing the analysis of frequency components generated by operations such as upsampling that mainly occur in generative artificial intelligence models, traces of blending generated when artificially synthesizing a face, and other defects (e.g., misalignment, etc.) that occur when the synthesis is not performed properly.

[0059] Specifically, the deepfake analysis unit (130) can obtain spatial forgery information through color analysis, frequency analysis, residual noise analysis, and multi-stream processing according to the analysis area range.

[0060] The deepfake analysis unit (130) can detect deepfakes by identifying the shape transformation of the image of the video to be analyzed based on color analysis criteria including RGB, gray scale, HSV (hue saturation value) (color, saturation, brightness), CMYK (cyan, magenta, yellow, black (key=black)), and YCbCr. This may be an analysis method that takes into account cases where high-quality fake videos that have been processed by reflecting various post-processing in the case of color-based deepfakes are artificially edited, such as chroma key work.

[0061] The deepfake analysis unit (130) can detect deepfakes by identifying an artificial grid in the video to be analyzed based on frequency analysis criteria using DCT (Discrete Cosine Transform) and DFT (Discrete Fourier Transform). Additionally, the deepfake analysis unit (130) can apply LoG (Laplacian of Gaussian) and SRM (Spatial Rich Model) filters, which are easy to analyze texture edges, when detecting deepfakes based on frequency analysis criteria. This may be an analysis method that takes into account that the frequency may be artificially compressed by two times during the upsampling process, and thus, traces of an artificial grid, such as a chessboard, may remain when converted to the frequency domain.

[0062] The deepfake analysis unit (130) can detect deepfakes by identifying noise in the video to be analyzed based on residual noise analysis criteria using auto-correlation operations and power spectra analysis. Additionally, the deepfake analysis unit (130) can also apply phase analysis when detecting deepfakes based on residual noise analysis criteria for cases where noise components are relatively small at high frequencies.

[0063] The deepfake analysis unit (130) can detect deepfakes in the video to be analyzed through multi-stream processing according to the area range of the part to be analyzed in the video to be analyzed. Specifically, the deepfake analysis unit (130) can apply coarse image analysis when analyzing high level features in the video to be analyzed, and can apply local forgery analysis when analyzing low level features. Since both of the above analysis methods have complementary characteristics, the deepfake analysis unit (130) can separate the area of ​​the part to be analyzed in the video to be analyzed into two streams, and analyze macroscopic information in one stream through coarse image analysis, and detect local forgery evidence in the other stream through local forgery analysis.

[0064] When acquiring spatial forgery information, the deepfake analysis unit (130) can identify correlations through cross-attention for spatial characteristics and frequency characteristics, and process them into a form that complements different data formats (e.g., modality) through correlation map inference.

[0065] Specifically, the image may have dimensions of (c, h, w). In this case, if the image is composed of RGB channels, c may be 3. The deepfake analysis unit (130) may perform a DCT transformation to extract frequency information for each channel. If a DCT transformation is performed, the dimension size may be the same as the image, which is (h, w). That is, even if an image of size (h, w) for the R channel is DCT-transformed, it may become (h, w), and even if an image (h, w) for the remaining G and B channels is DCT-transformed, it may become (h, w). That is, even if a DCT transformation is performed for each RGB channel of the image, the dimensions of (c, h, w) may be maintained. That is, the deepfake analysis unit (130) may perform a cross operation on two different modalities between the spatial characteristics of the image itself and the frequency characteristics converted into frequencies to identify the correlation.

[0066] The various forgery texture analysis methods described above can be applied to this embodiment separately or in combination.

[0067] The deepfake analysis unit (130) analyzes the target video for analysis based on the second synthetic defect criteria, including discontinuous scenes or exposure defects (popping defects) resulting from the application of an artificial intelligence model (e.g., a generative artificial intelligence model), and recognizes differences from the actual video to obtain temporal forgery information. The actual video may refer to a video that has not been forged. The discontinuous scenes or exposure defects may be a discontinuous flickering issue, etc., which frequently appears in deepfake videos.

[0068] While real videos smoothly flow with the movements of characters within them, deepfakes utilizing AI models, including generative AI models, synthesize single frames. Therefore, when implemented as videos, unlike real videos, discontinuous scenes or exposure defects (popping artifacts) may appear. These popping artifacts may refer to defects in the analyzed video that can be perceived as unnatural compared to the real video.

[0069] The deepfake analysis unit (130) can detect various defects, including the discontinuous scenes and popping defects described above, during analysis to obtain temporal forgery information, and identify differences from the actual video.

[0070] For example, the deepfake analysis unit (130) can utilize 5 frames to view a section corresponding to approximately 0.2 seconds based on a 30FPS video. The deepfake analysis unit (130) can complete spatial forgery analysis for each frame and use the information for the 5 frames by concatenating them channel-wise. Since the time information of the temporal forgery analysis is concatenated channel-wise, the deepfake analysis unit (130) can design an artificial intelligence model for deepfake analysis with a structure that can focus on time analysis through a 1x1 kernel, and perform temporal forgery information analysis through this.

[0071] Referring to the classifier of ⑤ of FIG. 2 and FIG. 6, the deepfake analysis unit (130) can determine whether the analyzed person in the video to be analyzed is real or fake using the final features. The final features may use DCT (Discrete Cosine Transformed Image) and DFT (Discrete Fourier Transformed Image) related to frequency, but are not limited thereto. In addition, the final features may be images that have undergone SRM (Steganalysis Rich Model) filtering, which is good for extracting artifacts at the boundary.

[0072] When training an artificial intelligence model for deepfake analysis, the deepfake analysis unit (130) may designate real videos as real classes and train them in spatially similar locations, and may configure fake videos to be relatively distant from the boundary of the real class rather than grouped into classes, so that if there is a difference from real data (including real videos), it is detected as a fake. There may be cases where a forgery technique that has not been inevitably seen in the video to be analyzed is treated as an unrecognized class and cannot be responded to. This may be because the classifier has a tendency to embed similar classes in the same location in the hyper dimension. The present embodiment may enable response to unseen data through the analysis of the deepfake analysis unit (130) described above.

[0073] Let's explain in more detail the learning in the spatially similar locations mentioned above. Real data is one cluster, but fake data can be multiple clusters. For example, if data synthesized by method A, data synthesized by method B, and data synthesized by method X are all vectorized and trained, it may be difficult to define the boundary by having them plotted in each fixed location when actually producing a t-SNE graph. However, if you divide the data into real and fake data and train so that only the real data is plotted in the same location, and the rest are trained to move away from this cluster, even if a new fake technique is applied, the fake detection ability for relatively unseen data types can be improved because they are plotted at an external boundary rather than the real boundary.

[0074] The deepfake analysis unit (130) extracts preset behavioral characteristics (attributions) from the video to be analyzed, and analyzes the correlation of the extracted characteristics based on time series to identify behavioral pattern anomalies of the analyzed subject. When extracting preset behavioral characteristics, the deepfake analysis unit (130) can comprehensively extract characteristics related to the subject's head angle, lip utterance, facial muscles, body, and voice.

[0075] Specifically, the deepfake analysis unit (130) can verify whether the image is a real person by analyzing only the character's behavioral patterns, not the texture, when analyzing behavioral patterns. Attribution for various behaviors can be extracted from the images extracted in the ROI Pooling section (④ of FIG. 2) of FIG. 2. The extracted features can be arranged in a time series and then analyzed for correlation. This correlation can include the degree of correlation between two different behavioral patterns. For example, if a specific person has a habit of shrinking their facial muscles when speaking, the correlation may be high. All possible correlations for various behavioral features can be calculated and trained using a support vector machine (SVM). Outliers can be excluded from the trained results and boundaries for conservative behavioral ranges can be determined. An outlier can typically refer to a small or large value that deviates significantly from the observed data range.

[0076] Below, we will explain attribution time series analysis as an example.

[0077] The deepfake analysis unit (130) can perform behavioral pattern analysis by performing analysis based on the head angle, lip expression, and facial muscle changes of the face of the person being analyzed in the video being analyzed.

[0078] In the case of the head angle, the deepfake analysis unit (130) can obtain the angles such as Yaw, Pitch, and Roll by inputting them into a head pose estimation model.

[0079] In the case of lip speech, the deepfake analysis unit (130) can extract facial feature points using keypoint estimation and estimate the lip shape using the horizontal L2 length and vertical L2 length of the lips.

[0080] The above L2 is another term for Euclidean Distance and can mean the straight line length between two points. It is calculated by taking the square root of the difference (x1 - x2) between the coordinates of two features, and among the feature points, the horizontal and vertical straight line lengths of the lips can be calculated and used as a measure for calculating the movement of the lips.

[0081] Referring to A of FIG. 7, the deepfake analysis unit (130) can express the L2 distance between the neck and the left shoulder as feature points 1-2, the L2 distance between the neck and the right shoulder as feature points 1-5, the L2 distance between the right shoulder and the wrist as feature points 2-3, the L2 distance between the left shoulder and the wrist as feature points 5-6, the L2 distance between the right wrist and the wrist as feature points 3-4, the L2 distance between the left wrist and the wrist as feature points 6-7, the left arm angle as the wrist angle created by joints 2-3 and 3-4, and the right arm angle as the wrist angle created by joints 5-6 and 6-7.

[0082] Meanwhile, in the case of the body, since some parts may not be displayed in the video to be analyzed, the deepfake analysis unit (130) may apply a method of excluding them from attribution.

[0083] Referring to B of FIG. 7, the deepfake analysis unit (130) can use AU1, 2, 4, 6, 7, 9, 10, 12, 14, 15, 17, 23, and 25 as characteristics that can determine how muscles of a specific facial part move in the case of facial muscle changes. In particular, the 45th case related to eye blinking can be excluded because the frequency varies excessively depending on the given environment.

[0084] At this time, the deepfake analysis unit (130) can define facial expressions using AU (action units). The AU is a basic element of expressions that can be expressed by changes in facial muscles, and all human expressions can be expressed by a combination of one or more AUs. The above-mentioned AU1 may mean Inner Brow Raiser, AU2 may mean Outer Brow Raiser, AU4 may mean Brow Lower, AU6 may mean Cheek Raiser, AU7 may mean Lid Tightener, AU9 may mean Nose Wrinkler, AU10 may mean Upper Lid Raiser, AU12 may mean Lid Corner Puller, AU14 may mean Dimpler, AU15 may mean Lip Corner Depressor, AU17 may mean Chin Raiser, AU23 may mean Lip Tightener, and AU25 may mean Lip Part. Additionally, AU5 can mean Upper Lid Raiser, AU16 can mean Lower Lip Depressor, AU20 can mean Lip Stretcher, and AU26 can mean Jaw Drop.

[0085] Referring to C of FIG. 7, the deepfake analysis unit (130) can analyze the voice using Mel constants (8 used) converted to DCT type 3, Sample Rate 16,000, nfft 2048, and window size 128ms. For example, in FIG. 7, the first column of C is a time-amplitude graph (Sample rate = 16,000), the second column can be a graph converted to a Mel Spectrogram, and the third column can be a graph converted to 8 Mel constants, nfft = 2048, and window size 128ms.

[0086] Referring to Figure 2, the deepfake result processing unit (150) can perform the operations of comprehensive analysis and result generation (⑦) and determining whether it is a specific person (⑧).

[0087] The deepfake result processing unit (150) adjusts the weights of the results of the forged texture analysis and the behavioral pattern analysis according to preset comprehensive analysis criteria to infer a comprehensive score, and can obtain the authenticity of the video to be analyzed and the analysis result based on the inferred comprehensive score and the critical point.

[0088] Specifically, the deepfake result processing unit (150) can perform a comprehensive analysis procedure by considering the comprehensive results of the forgery texture analysis part and the behavioral pattern analysis pattern.

[0089] Since the above-mentioned forgery texture analysis and behavioral pattern analysis are learned separately, the results of the two modules' judgment of authenticity may differ. Since the present embodiment recognizes the authenticity of the learning data in advance, it is possible to identify cases where the accuracy is higher between the results of the forgery texture analysis and the results of the behavioral pattern analysis. Accordingly, the deepfake result processing unit (150) adjusts the weighting of the forgery score, which is the result of the forgery texture analysis, and the forgery score, which is the result of the behavioral pattern analysis, according to a preset adjustment criterion when performing a comprehensive analysis and producing a result (⑦), so as to find the point with the highest score in the learning data set and infer the comprehensive score. At this time, the preset adjustment criterion may be set based on the accuracy of the forgery texture analysis and the behavioral pattern analysis identified when the deepfake result processing unit (150) learns the deepfake analysis artificial intelligence model for processing the comprehensive analysis, or at the discretion of the operator.

[0090] The deepfake result processing unit (150) can determine whether the analyzed person is true (1.O in FIG. 2) or false (2.X in FIG. 2) based on the comprehensive score and critical point obtained by adjusting the weights of the results of the forged texture analysis and the behavioral pattern analysis according to the preset comprehensive analysis criteria. At this time, since the deepfake result processing unit (150) manages the results, including the scores for the forged texture analysis and the behavioral pattern analysis, separately from each other, it can obtain detailed analysis results for the deepfake results for the analysis target video, including the analyzed person.

[0091] The deepfake analysis service provided through the deepfake analysis system (100) may be provided, for example, in the form of a web page or through an application installed on a user terminal (not shown).

[0092] According to one embodiment, the target selection unit (110), the deepfake analysis unit (130), and the deepfake result processing unit (150) may be implemented using one or more physically separate devices, or may be implemented by one or more hardware processors or a combination of one or more hardware processors and software, and may not be clearly distinguished in specific operations, unlike the illustrated example.

[0093] The deepfake analysis system (100) of the present disclosure may be composed of one or more cores, and may include a processor for data analysis and deep learning, such as a central processing unit, a general purpose graphics processing unit, and a tensor processing unit of a computing device. The processor (not shown) may read a computer program stored in a memory (not shown) and perform data processing for machine learning according to the present disclosure. According to the present disclosure, the processor (not shown) may perform operations for learning a neural network. The processor (not shown) may perform calculations for learning a neural network, such as processing input data for learning in deep learning, extracting features from input data, calculating errors, and updating weights of a neural network using backpropagation.

[0094] The above neural network model may be a deep neural network. In the present disclosure, neural network, network function, and neural network may be used interchangeably. A deep neural network (DNN) may refer to a neural network that includes multiple hidden layers in addition to an input layer and an output layer. Using a deep neural network, it is possible to identify latent structures of data. That is, it is possible to identify latent structures of photos, text, videos, voices, and music (e.g., what objects are in the photo, what the content and emotion of the text are, what the content and emotion of the voice are, etc.). A deep neural network may include a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a Q network, a U network, a Siamese network, etc.

[0095] Convolutional neural networks (CNNs) are a type of deep neural network that include neural networks containing convolutional layers. CNNs are a type of multilayer perceptron designed to use minimal preprocessing. CNNs can be composed of one or more convolutional layers and artificial neural network layers combined with them. CNNs can additionally utilize weight and pooling layers. This structure allows CNNs to fully utilize two-dimensional input data. CNNs can be used to recognize objects in images. CNNs can process image data by representing it as a matrix with dimensions. For example, in the case of image data encoded in RGB (red-green-blue), each of the R, G, and B colors can be represented as a two-dimensional (for example, in a two-dimensional image) matrix. That is, the color value of each pixel of the image data can be an element of a matrix, and the size of the matrix can be the same as the size of the image. Therefore, the image data can be represented as three two-dimensional matrices (a three-dimensional data array).

[0096] In a convolutional neural network, a convolutional process (input and output of a convolutional layer) can be performed by moving the convolutional filter and multiplying the matrix elements at each location of the image with the convolutional filter. The convolutional filter can be composed of an n*n matrix. The convolutional filter can generally be composed of a fixed-shape filter that is smaller than the total number of pixels in the image. That is, when an m*m image is input to a convolutional layer (for example, a convolutional layer whose convolutional filter has a size of n*n), a matrix representing n*n pixels containing each pixel of the image can be component-wise multiplied with the convolutional filter (i.e., each element of the matrix is ​​multiplied). By multiplying with the convolutional filter, a component matching the convolutional filter can be extracted from the image. For example, a 3*3 convolutional filter for extracting up and down straight line components from an image can be configured as [[0,1,0], [0,1,0], [0,1,0]]. When a 3*3 convolutional filter for extracting up and down straight line components from an image is applied to an input image, up and down straight line components matching the convolutional filter from the image can be extracted and output. A convolutional layer can apply a convolutional filter to each matrix for each channel representing an image (i.e., R, G, B colors in the case of an R, G, B coded image). A convolutional layer can extract features matching the convolutional filter from the input image by applying a convolutional filter to the input image. The filter value of the convolutional filter (i.e., the value of each element of the matrix) can be updated by backpropagation during the learning process of a convolutional neural network.

[0097] A subsampling layer can be connected to the output of a convolutional layer to simplify the output of the convolutional layer, thereby reducing memory usage and computational complexity. For example, when the output of a convolutional layer is input to a pooling layer having a 2*2 max pooling filter, the image can be compressed by outputting the maximum value contained in each 2*2 patch from each pixel of the image. The aforementioned pooling may be a method of outputting the minimum value from a patch or the average value of the patch, and any pooling method may be included in the present disclosure.

[0098] A convolutional neural network may include one or more convolutional layers and subsampling layers. A convolutional neural network can extract features from an image by repeatedly performing convolutional and subsampling processes (e.g., the aforementioned max pooling). Through repeated convolutional and subsampling processes, the neural network can extract global features of the image.

[0099] The output of a convolutional layer or a subsampling layer can be input to a fully connected layer. A fully connected layer is a layer in which all neurons in one layer are connected to all neurons in the neighboring layer. A fully connected layer can refer to a structure in a neural network in which all nodes in each layer are connected to all nodes in other layers.

[0100] At least one of a CPU, GPGPU, and TPU of a processor (not shown) can process network function learning. For example, the CPU and GPGPU can jointly process network function learning and data classification using the network function. Furthermore, in one embodiment of the present disclosure, processors of multiple computing devices can be used together to process network function learning and data classification using the network function. Furthermore, a computer program executed on a computing device according to one embodiment of the present disclosure may be a CPU, GPGPU, or TPU executable program.

[0101] FIG. 8 is a flowchart illustrating a deepfake analysis method according to one embodiment. The method illustrated in FIG. 8 may be performed, for example, by the aforementioned deepfake analysis system (100). While the illustrated flowchart divides the method into multiple steps, at least some of the steps may be performed in a different order, combined with other steps and performed together, omitted, divided into substeps, or performed with one or more additional steps not illustrated.

[0102] The deepfake analysis method disclosed below can be applied in the same manner to the operation of the deepfake analysis system (100) disclosed in the descriptions of FIGS. 1 to 7 described above, but for the convenience of explanation, redundant detailed descriptions will be omitted.

[0103] At step 1100, the deepfake analysis system (100) determines an analysis section based on a specific time section or a specific person in the input video according to preset criteria through the target selection unit (110), extracts characteristic points of the selected analysis person in the analysis section of the input video, and then generates and provides an analysis target video in which the part to be analyzed is processed to a preset size.

[0104] At this time, when extracting feature points of the analysis subject, the target selection unit (110) extracts feature points by analyzing the bounding box coordinates and feature point coordinates for the face or joints of the analysis subject using coordinate information for the facial bounding box and body region of the analysis subject. If the analysis subject video is based on the face of the analysis subject, facial feature points can be extracted, and if the analysis subject video is based on the upper body of the analysis subject, facial feature points and upper body joint feature points can be extracted.

[0105] At step 1200, the deepfake analysis system (100) performs a forgery texture analysis based on the analysis target video through the deepfake analysis unit (130) according to a first analysis criterion that is preset, including analysis of spatial forgery information and temporal forgery information, and performs a behavioral pattern analysis based on the analysis target video according to a second analysis criterion that is preset, thereby determining whether the video is a real person.

[0106] At step 1300, the deepfake analysis system (100) adjusts the weights of the results of the forgery texture analysis and the behavioral pattern analysis through the deepfake result processing unit (150) according to preset comprehensive analysis criteria to infer a comprehensive score, and can obtain the authenticity of the video to be analyzed and the analysis result based on the inferred comprehensive score and the threshold.

[0107] FIG. 9 is a block diagram illustrating a computing environment including a computing device according to one embodiment. In the illustrated embodiment, each component may have different functions and capabilities other than those described below, and may include additional components other than those described below.

[0108] The illustrated computing environment (10) includes a computing device (12). The computing device (12) may be one or more components included in a deepfake analysis system (100) according to one embodiment.

[0109] A computing device (12) includes at least one processor (14), a computer-readable storage medium (16), and a communication bus (18). The processor (14) may cause the computing device (12) to operate according to the exemplary embodiments mentioned above. For example, the processor (14) may execute one or more programs stored in the computer-readable storage medium (16). The one or more programs may include one or more computer-executable instructions, which, when executed by the processor (14), may be configured to cause the computing device (12) to perform operations according to the exemplary embodiments.

[0110] A computer-readable storage medium (16) is configured to store computer-executable instructions or program code, program data, and / or other suitable forms of information. A program (20) stored in the computer-readable storage medium (16) includes a set of instructions executable by the processor (14). In one embodiment, the computer-readable storage medium (16) may be a memory (volatile memory such as random access memory, non-volatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, any other form of storage medium that can be accessed by the computing device (12) and store desired information, or a suitable combination thereof.

[0111] A communication bus (18) interconnects various other components of the computing device (12), including the processor (14) and computer-readable storage media (16).

[0112] The computing device (12) may also include one or more input / output interfaces (22) that provide interfaces for one or more input / output devices (24) and one or more network communication interfaces (26). The input / output interfaces (22) and the network communication interfaces (26) are connected to the communication bus (18). The input / output devices (24) may be connected to other components of the computing device (12) via the input / output interfaces (22). Exemplary input / output devices (24) may include input devices such as pointing devices (such as a mouse or a trackpad), a keyboard, a touch input device (such as a touchpad or a touchscreen), a voice or sound input device, various types of sensor devices and / or photographing devices, and / or output devices such as display devices, printers, speakers and / or network cards. The exemplary input / output devices (24) may be included within the computing device (12) as a component constituting the computing device (12), or may be connected to the computing device (12) as a separate device distinct from the computing device (12).

[0113] Because each generative AI model has its own unique structure and operations, when applied to a video, it may leave traces of specific frequencies being emphasized. Considering that frequency patterns can represent traces of forgery, the present embodiments can implement deepfake detection by detecting traces of forgery. This approach can be effective for generative AI models such as StyleGAN.

[0114] In particular, since many upsampling operations are performed from latent vectors with relatively small dimensions, the accumulated frequency distortion traces can be very noticeable. Recently, in the case of generative AI models such as the diffusion model, since images are generated stepwise from the noise region, the number of upsampling operations is relatively reduced, and as a result, even the frequency traces are gradually disappearing. Since there is no guarantee that advanced models to be introduced in the future will entirely rely on these operations, it may be somewhat difficult to create an infinite number of defensive AI models that respond to fake images created individually by generative AI models.

[0115] This embodiment maintains the existing, orthodox method of detecting signs of forgery in fake works. However, it proposes an advanced AI model capable of determining the authenticity of videos depicting a specific individual by fundamentally learning information such as the behavioral patterns of the person being synthesized. In other words, by comprehensively considering even the behavioral patterns of a specific individual, which cannot be represented, the model can overcome the fundamental limitations of generative AI models by determining authenticity. Consequently, the present embodiments can detect fake videos of famous politicians, broadcasters, and leaders.

[0116] The disclosed embodiments may be implemented in the form of a recording medium storing computer-executable instructions. The instructions may be stored in the form of program code, and when executed by a processor, may generate program modules to perform the operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

[0117] While representative embodiments of the present invention have been described in detail above, those skilled in the art will appreciate that various modifications to the above-described embodiments are possible without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined not only by the claims set forth below but also by equivalents thereof.

Claims

1. A target selection unit that determines an analysis section based on a specific time section or a specific person according to preset criteria in an input video, extracts characteristic points of a selected analysis person in the analysis section of the input video, and then generates and provides an analysis target video by processing the part to be analyzed to a preset size; A deepfake analysis unit that performs forgery texture analysis based on the first analysis criterion, including spatial forgery information analysis and temporal forgery information analysis, based on the above analysis target video, and performs behavior pattern analysis based on the second analysis criterion, based on the above analysis target video, to determine whether the video is a real person; and A deepfake analysis system, comprising a deepfake result processing unit that adjusts weights according to preset comprehensive analysis criteria for the results of the forged texture analysis and the behavioral pattern analysis to infer a comprehensive score, and obtains the authenticity of the analysis target video and the analysis result based on the inferred comprehensive score and a threshold point.

2. In claim 1, The above target selection section is, A deepfake analysis system, wherein when extracting feature points of the above-mentioned analyzed person, feature points are extracted by analyzing the bounding box coordinates and feature point coordinates for the face or joints of the analyzed person using coordinate information for the facial bounding box and body region of the analyzed person, and when the analysis target video is based on the face of the analyzed person, facial feature points are extracted, and when the analysis target video is based on the upper body of the analyzed person, facial feature points and upper body joint feature points are extracted.

3. In claim 2, The above target selection section is, A deepfake analysis system that processes the shape and size and performs a pooling operation according to the criteria for each artificial intelligence model for deepfake analysis for characteristic analysis to be applied to the video to be analyzed when processing the part to be analyzed to a preset size, and calculates the trajectory for the center of gravity of the feature points and calculates the coordinates of the size that includes all of the feature points to determine the crop size of the video to be analyzed.

4. In claim 3, The above target selection section is, A deepfake analysis system that, when processing the above analysis target video to the above preset size, crops and separately stores each preset section of the entire body of the person being analyzed.

5. In claim 1, The above deepfake analysis unit, A deepfake analysis system that obtains spatial forgery information by analyzing the spatial features and frequency features of the video to be analyzed based on the first synthetic defect criterion, including the spatial features and frequency features according to the application of the artificial intelligence model.

6. In claim 5, The above deepfake analysis unit, A deepfake analysis system that analyzes the above-mentioned analysis target video according to the second synthetic defect criteria, including discontinuous scenes or exposure defects due to application of an artificial intelligence model, and obtains the above-mentioned temporal forgery information by recognizing differences from the actual video.

7. In claim 5, The above deepfake analysis unit, A deepfake analysis system that, when acquiring the above spatial forgery information, identifies correlations through cross-attention for the spatial characteristics and the frequency characteristics, and processes them into a form that complements different data formats through correlation map inference.

8. In claim 1, The above deepfake analysis unit, A deepfake analysis system that extracts preset behavioral characteristics (attributions) from the video to be analyzed, analyzes the time series correlation of the extracted characteristics, and identifies abnormalities in the behavioral patterns of the analyzed person.

9. In a method performed by a deepfake analysis system, A step in which the deepfake analysis system determines an analysis section based on a specific time section or a specific person according to preset criteria in an input video, extracts characteristic points of the selected analysis person in the analysis section of the input video, and then generates and provides an analysis target video in which the part to be analyzed is processed to a preset size; A step of performing a forgery texture analysis based on the above analysis target video according to a preset first analysis criterion, including analysis of spatial forgery information and temporal forgery information, and performing a behavior pattern analysis based on the above analysis target video according to a preset second analysis criterion to determine whether the video is a real person; and A deepfake analysis method, comprising a step of inferring a comprehensive score by adjusting weights based on preset comprehensive analysis criteria based on the results of the forged texture analysis and the behavioral pattern analysis, and obtaining the authenticity of the video to be analyzed and the analysis result based on the inferred comprehensive score and a threshold point.

10. In claim 9, In the step of creating and providing the video to be analyzed above, A deepfake analysis method in which, when extracting feature points of the above-mentioned analyzed person, feature points are extracted by analyzing the bounding box coordinates and feature point coordinates for the face or joints of the analyzed person using coordinate information for the facial bounding box and body region of the analyzed person, and when the analysis target video is based on the face of the analyzed person, facial feature points are extracted, and when the analysis target video is based on the upper body of the analyzed person, facial feature points and upper body joint feature points are extracted.

Images