Image processing method and device, electronic equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses an image processing method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: obtaining multi-modal detection information which corresponds to a to-be-detected image and comprises image information, equipment parameter information and behavior track information; inputting each piece of feature information in the multi-modal detection information into a feature extraction network, and determining corresponding feature information; fusing the feature information, and determining fusion features with time sequence consistency; counterfeit features in the fusion features are detected, a detection result is determined, and the detection result is used for representing whether the to-be-detected image is a counterfeit image or not. Therefore, according to the embodiment of the invention, image counterfeiting detection is carried out by using the multi-modal detection information, so that the robustness of counterfeited face detection and the security of face verification can be improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically to an image processing method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] With the rapid development of artificial intelligence and deep learning technologies, deepfake technology has made significant progress in the field of visual content generation and editing. However, the misuse of this technology has brought serious social problems, especially privacy violations, the spread of misinformation, and identity fraud; it also poses challenges to the security of facial recognition. Therefore, how to effectively detect and prevent deepfakes during facial verification has become an important research topic in the field. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide an image processing method, apparatus, electronic device, and computer-readable storage medium to improve the robustness of fake face detection and the security of face verification.

[0004] In a first aspect, embodiments of the present invention aim to provide an image processing method, the method comprising:

[0005] Obtain multimodal detection information corresponding to the image to be detected, wherein the multimodal detection information includes image information, device parameter information, and behavior trajectory information;

[0006] Each feature information in the multimodal detection information is input into the feature extraction network to determine the corresponding feature information;

[0007] The aforementioned feature information is fused to determine a fused feature with temporal consistency;

[0008] The forgery features in the fused features are detected, and the detection result is determined. The detection result is used to characterize whether the image to be detected is a forgery image.

[0009] Furthermore, the method also includes:

[0010] In response to the detection result indicating that the image to be detected is a forged image, a detection report of the image to be detected is generated, the detection report including the detection result and the reason for forgery.

[0011] Furthermore, the method also includes:

[0012] The multimodal detection information is analyzed to determine the cause of forgery.

[0013] Furthermore, the analysis of the multimodal detection information to determine the cause of forgery includes:

[0014] The permission settings information in the device parameters are checked;

[0015] In response to the detection that the permission settings information has been modified, the corresponding reason for forgery is determined to be that the device permissions have been modified.

[0016] Furthermore, the analysis of the multimodal detection information to determine the cause of forgery includes:

[0017] The face-crossing trajectory information in the behavioral trajectory information is subjected to trajectory continuity analysis, time consistency analysis, and / or location consistency analysis in order to determine the corresponding forgery reason based on the analysis results.

[0018] Furthermore, the behavioral trajectory information includes face-crossing trajectory information, and the acquisition of multimodal detection information corresponding to the image to be detected includes:

[0019] Obtain consecutive frame images corresponding to the image to be detected, wherein the consecutive frame images include the image to be detected and adjacent images of adjacent frames of the image to be detected;

[0020] The image information in the image to be detected and the adjacent images is identified to determine the face trajectory information corresponding to the image information.

[0021] Furthermore, the feature extraction network includes a visual encoder, a first conversion neural network, and a second conversion neural network. The visual encoder is used to extract features from the image information to determine the corresponding visual features; the first conversion neural network is used to extract features from the device parameter information to determine the corresponding device parameter features; and the second conversion neural network is used to extract features from the behavior trajectory information to determine the corresponding trajectory features.

[0022] Furthermore, the multimodal detection information also includes audio information and / or text information, wherein the text information is generated based on the image information.

[0023] Furthermore, the feature extraction network includes an audio encoder and / or a text encoder, wherein the audio encoder is used to extract features from the audio information to determine the corresponding audio features; and the text encoder is used to extract features from the text information to determine the corresponding text features.

[0024] Furthermore, the step of inputting each feature information from the multimodal detection information into the feature extraction network to determine the corresponding feature information includes:

[0025] The image information, device parameter information, and behavior trajectory information are preprocessed respectively to determine the corresponding encoding representation;

[0026] Each of the aforementioned encoded representations is input into the feature extraction network to determine the corresponding feature information.

[0027] Secondly, embodiments of the present invention aim to provide an image processing apparatus, the apparatus comprising:

[0028] The information acquisition unit is used to acquire multimodal detection information corresponding to the image to be detected, wherein the multimodal detection information includes image information, device parameter information and behavior trajectory information;

[0029] The feature extraction unit is used to input the feature information from the multimodal detection information into the feature extraction network to determine the corresponding feature information;

[0030] The forgery detection unit is used to fuse the aforementioned feature information to determine fused features with temporal consistency; to detect forgery features in the fused features and to determine the detection result, wherein the detection result is used to characterize whether the image to be detected is a forged image.

[0031] Thirdly, embodiments of the present invention aim to provide a computer program product, the computer program product including a computer program / instruction, which, when executed by a processor, implements the method described in any of the preceding claims.

[0032] Fourthly, embodiments of the present invention aim to provide an electronic device, including a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in any of the preceding claims.

[0033] Fifthly, embodiments of the present invention aim to provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any of the preceding claims.

[0034] The technical solution of this invention obtains multimodal detection information corresponding to the image to be detected, including image information, device parameter information, and behavioral trajectory information. It then inputs each feature information from the multimodal detection information into a feature extraction network to determine the corresponding feature information. The feature information is fused to determine a fused feature with temporal consistency. Finally, it detects forgery features in the fused feature and determines the detection result. The detection result is used to characterize whether the image to be detected is a forged image. This allows for image forgery detection using multimodal detection information, thereby improving the robustness of forged face detection and the security of face verification. Attached Figure Description

[0035] The above and other objects, features and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0036] Figure 1 This is a flowchart of an image processing method according to an embodiment of the present invention;

[0037] Figure 2 This is a flowchart illustrating the process of obtaining face trajectory information according to an embodiment of the present invention;

[0038] Figure 3 This is a schematic diagram of a face forgery detection system according to an embodiment of the present invention;

[0039] Figure 4 This is a schematic diagram of an image processing apparatus according to an embodiment of the present invention;

[0040] Figure 5 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0041] The present application is described below based on embodiments, but it is not limited to these embodiments. In the detailed description of the present application below, certain specific details are described in detail. Those skilled in the art can fully understand the present application without these details. To avoid obscuring the substance of the present application, well-known methods, processes, flows, elements, and circuits are not described in detail.

[0042] Furthermore, those skilled in the art should understand that the accompanying drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0043] Unless otherwise expressly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise expressly limited. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0044] Unless the context explicitly requires it, words such as "including" or "contains" throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, meaning "including but not limited to".

[0045] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0046] The solutions described in this specification and embodiments, if involving information acquisition, will collect data under legal and compliant conditions, ensuring the legality of the data source, and will take appropriate technical and management measures to ensure data security. If involving personal information processing, processing will be carried out under legal grounds (e.g., obtaining the consent of the personal information subject, or being necessary for contract performance), and will only be conducted within the prescribed or agreed scope. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.

[0047] Currently, facial recognition is used in many scenarios to verify user identity. For example, ride-hailing services need to verify whether the driver accepting the order is a registered driver and whether they are the actual driver. However, the development of facial spoofing technology has made it possible to use fake faces for facial verification. To prevent the abuse of facial spoofing technology and improve the security and reliability of facial recognition, facial spoofing detection is necessary.

[0048] Current face forgery detection technologies mostly rely on visual signals, using image analysis tools to capture subtle visual differences in forged videos (such as pixel-level discontinuities or forged facial movements). However, this method, which utilizes information from a single modality, can only identify simple forgeries and cannot handle complex ones. Furthermore, faced with increasingly sophisticated face forgery techniques, especially those targeting forgeries based on different angles, lighting conditions, or facial expression details, existing detection technologies show a significant decline in accuracy and robustness.

[0049] In view of this, an image processing method is proposed in this embodiment of the invention to improve the robustness of face forgery detection and the security of face verification.

[0050] Figure 1 This is a flowchart of an image processing method according to an embodiment of the present invention. Figure 1 As shown, in this embodiment, the following method is used to detect face forgery in the image to be detected.

[0051] In step S110, multimodal detection information corresponding to the image to be detected is obtained.

[0052] In this embodiment, the multimodal detection information includes at least image information, device parameter information, and behavior trajectory information. It may also include one or more of the feature information from different modalities related to image forgery, such as audio information and text information. The specific selection can be made according to the actual application scenario. Generally, the more types of feature information in the multimodal detection information, the more accurate the corresponding forgery detection result.

[0053] Optionally, the image information in this embodiment includes texture information, edge and contour information, color information, shape information, etc., of the image to be detected. Texture information is used to characterize the detailed textures of key facial features, such as the eyes, mouth, and nose. Edge and contour information is used to characterize the outer contour of the face and the structure of key facial features. Color information includes the color information of the face region and the surrounding area, which can be represented by attributes such as hue, saturation, and brightness of the image. Shape information is used to characterize the shape features of the face, including aspect ratio, distribution of facial features, etc.

[0054] Device parameter information refers to the parameter information of the device (such as a smartphone or other terminal device) that acquires the image to be detected. This includes device permission settings (such as whether it is rooted) and device environment information (such as whether it is a virtual machine).

[0055] Behavioral information includes location information, camera parameter information, and face-crossing trajectory information collected after user authorization. Location information represents the location of the device acquiring the image to be detected; this location can be represented by latitude and longitude. Camera parameters are those that affect the shooting effect and image quality of the image to be detected, including exposure-related parameters (such as aperture, shutter speed, ISO sensitivity, exposure compensation, etc.), color and diagonal-related parameters (such as white balance, focus mode, etc.), lens-related parameters (such as focal length, the relationship between aperture and focal length), and other parameters (resolution, frame rate, pixel size, etc.). Face-crossing trajectory information represents the movement trajectory of key points in the face image.

[0056] Optionally, in this embodiment, the location information and camera parameter information are attribute information inherent to the image to be detected. The location information and camera parameter information in the behavior information can be determined by viewing the attribute information of the image to be detected. However, since the face trajectory information reflects the movement trajectory of the face image, it needs to be determined by other images related to the image to be detected.

[0057] Furthermore, in this embodiment, by means of... Figure 2 The method shown obtains face trajectory information.

[0058] In step S210, the continuous frame images corresponding to the image to be detected are obtained. The continuous frame images include the image to be detected and the adjacent images of the adjacent frames of the image to be detected.

[0059] In this embodiment, considering that face verification is usually a dynamic verification process, continuous frame images corresponding to the image to be detected can be obtained to acquire the face trajectory information of the face in the image to be detected. Furthermore, the continuous frame images are the image sequence acquired during face verification, the image to be detected is one frame in the continuous frame images, and the images adjacent to the frame number of the image to be detected are the adjacent images of the image to be detected.

[0060] Optionally, the adjacent images of the image to be detected can be the frame image of the previous frame, the frame image of the image to be detected, or both the frame image of the previous frame and the frame image of the image to be detected. Furthermore, in scenarios with higher processing time requirements, only the frame image of the previous or next frame of the image to be detected can be selected as adjacent images to determine the face trajectory information of the face in the image to be detected; while in scenarios with higher processing accuracy requirements, both the frame images of the previous and next frames of the image to be detected can be selected as adjacent images to determine the face trajectory information of the face in the image to be detected.

[0061] In step S220, image information in the image to be detected and neighboring images is identified to determine the face trajectory information corresponding to the image information.

[0062] In this embodiment, since the face trajectory information reflects the movement trajectory of a face image, when determining the face trajectory information using the image to be detected and adjacent images, it is necessary to first identify the image information in the image to be detected and adjacent images to locate the face region in each image. Then, feature extraction is performed on the face region in each image, extracting the shape, texture, color, and other information of key parts (such as eyes, nose, mouth, etc.). This information extracted from different images is then matched, and if a match is successful, it is recorded. Finally, the recorded information is used to generate the movement trajectory of the face, i.e., the face trajectory information.

[0063] Optionally, in this embodiment, existing face recognition methods can be used to locate the face region in the image to be detected and the corresponding adjacent images, and the information extracted from the image to be detected and the corresponding adjacent images can be matched by matching feature points of key parts or by matching based on face templates, so as to determine the complete face trajectory information.

[0064] Furthermore, in this embodiment, after determining the face-crossing trajectory information based on the image information in the image to be detected and its corresponding neighboring images, the face-crossing trajectory information is further optimized and post-processed, including occlusion processing, trajectory smoothing, and trajectory analysis. Occlusion processing can be implemented using methods such as background subtraction, motion estimation, and multi-feature fusion to eliminate trajectory recognition errors caused by face occlusion, thereby improving the robustness of trajectory tracking under occlusion conditions. Trajectory smoothing can employ algorithms such as Kalman filtering and particle filtering to smooth the obtained face-crossing trajectory, eliminating the influence of noise and errors in the trajectory generation process. Trajectory analysis includes extracting motion information such as the speed and direction of the face in the image, as well as the time when the face appears and disappears in the image, to enrich the content of the face-crossing trajectory information and improve the accuracy and reliability of subsequent face forgery detection based on the face-crossing trajectory information.

[0065] Optionally, the multimodal information in this embodiment also includes audio information and / or text information. The audio information can be audio information acquired simultaneously with the image to be detected, including audio features such as the speech rhythm, intonation, and timbre of the task in the image. The text information includes image metadata, tags, or annotation information describing the image to be detected. The text information can be generated based on the image information. Specifically, it can be generated by processing the image information using existing technologies or future image-to-text methods, and this text information is then used as the text information of the image to be detected for subsequent forgery detection.

[0066] Furthermore, in this embodiment, all the aforementioned feature information in the multimodal detection information is generated and collected on the terminal device. When face forgery detection is required, either the terminal with forgery detection function or the remote device can acquire and use this feature information for face forgery detection with user authorization. When face forgery detection is processed remotely, the remote device acquires the multimodal detection information collected on the terminal device through a communication connection with the terminal or receives the collected multimodal detection information uploaded by the terminal. Specifically, in this embodiment, the multimodal detection information on the terminal is collected in real time through the SDK (Software Development Kit) embedded in the terminal device application, so as to use this multimodal detection information for face forgery detection.

[0067] In step S120, the feature information of each item in the multimodal detection information is input into the feature extraction network to determine the corresponding feature information.

[0068] Optionally, since different feature information in multimodal detection information is usually represented in different ways, in this embodiment, the feature extraction network is set according to the type and quantity of feature information in multimodal detection information, and each feature information has a corresponding feature extraction module.

[0069] Figure 3 This is a schematic diagram of a face forgery detection system according to an embodiment of the present invention. Figure 3As shown, the face forgery detection system in this embodiment includes a feature extraction network 31. When the multimodal detection information includes image information, device parameter information, behavioral trajectory information, audio information, and text information, the feature extraction network 31 in this embodiment includes a visual encoder 311, a first conversion neural network 312, a second conversion neural network 313, an audio encoder 314, and a text encoder 315. The visual encoder is used to extract features from the image information to determine the corresponding visual features. The first conversion neural network is used to extract features from the device parameter information to determine the corresponding device parameter features. The second conversion neural network is used to extract features from the behavioral trajectory information to determine the corresponding trajectory features. The audio encoder is used to extract features from the audio information to determine the corresponding audio features. The text encoder is used to extract features from the text information to determine the corresponding text features.

[0070] Optionally, in this embodiment, the visual encoder, the first conversion neural network, the second conversion neural network, the audio encoder, and the text encoder all use pre-trained models. The model types used by the visual encoder, the first conversion neural network, the second conversion neural network, the audio encoder, and the text encoder can be selected according to the actual use scenario. For example, the model type can be a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory network (LSTM), or a transformer model.

[0071] It should also be understood that when the multimodal detection information only includes a portion of the feature information mentioned above, the feature extraction network in this embodiment only includes the feature extraction module corresponding to this portion of feature information. For example, when the multimodal detection information includes image information, device parameter information, and behavior trajectory information, the feature extraction network includes a visual encoder, a first conversion neural network, and a second conversion neural network.

[0072] Optionally, to improve the efficiency and accuracy of feature extraction, in this embodiment, before extracting features from each feature information, each feature information is preprocessed to determine the corresponding encoding representation, and then each encoding representation is input into the feature extraction network to determine the corresponding feature information.

[0073] Furthermore, when preprocessing image information, audio information, and text information, this embodiment can use a preset encoding method corresponding to the feature information for encoding. For example, image information can use lossy encoding, lossless encoding, or layered encoding; audio information can use pulse code modulation (PCM) or adaptive differential pulse code modulation (ADPCM); and text information can use adaptive differential pulse code modulation (ADPCM), Unicode encoding, UTF-8 encoding, etc., to generate image information, audio information, and text information in a target format that is convenient for subsequent feature extraction.

[0074] When preprocessing device parameter information and behavior trajectory information, standardization or normalization can be used to ensure that these features are on the same scale, thereby avoiding errors introduced when using these features for forgery detection. Specifically, in this embodiment, normalization is used to preprocess the device parameter information and behavior trajectory information to convert the data values ​​in the device parameter information and behavior trajectory information into values ​​within the range of 0-1.

[0075] Furthermore, in this embodiment, after preprocessing various feature information based on the above method, an encoded representation in the target format corresponding to each feature information is generated. Then, by inputting the encoded representation corresponding to each feature information into the corresponding feature extraction module for feature extraction, the feature information corresponding to each feature information can be obtained. This allows for the subsequent use of the feature information corresponding to each feature information in the multimodal detection information to achieve face forgery detection, thereby improving the robustness of face forgery detection and the security of face verification.

[0076] In step S130, the feature information is fused to determine the fused features with temporal consistency.

[0077] In this embodiment, by fusing various feature information to determine fused features with temporal consistency, it is possible to simultaneously utilize feature information from multiple modalities to achieve face forgery detection, thereby improving the robustness of face forgery detection and the security of face verification.

[0078] Optionally, in this embodiment, fusion is achieved by concatenating the feature vectors corresponding to each feature information, thereby determining a fused feature with temporal consistency. Temporal consistency characterizes the ability of the fused feature to maintain the continuity and relevance of the original feature information over time. Furthermore, when fusing the feature information, not only the spatial relationships between the feature information are considered, but also the temporal order and interdependencies between them.

[0079] Furthermore, before splicing the feature information, this embodiment first performs time alignment on the feature information, that is, the feature value at each time point corresponds to the same timestamp, to ensure that all feature information is aligned in the time series.

[0080] In this embodiment, a simple splicing method can be used to directly splice multiple feature information into a longer feature vector on the time series. This method can simply and intuitively achieve the splicing of each feature information, thereby fusing the feature information.

[0081] Alternatively, to further utilize the temporal relationship between feature information, this embodiment can also use a weighted concatenation method to fuse the feature information. First, each feature is weighted according to its importance or relevance, and then the features are concatenated. This can be achieved through feature selection algorithms or model-based feature importance assessment.

[0082] Furthermore, in this embodiment, convolutional neural networks or attention mechanism networks can be used to fuse various feature information. When using convolutional neural networks for fusion, multiple feature information is convolved using models such as convolutional neural networks (CNNs) to capture the local correlations and temporal patterns between feature information, thereby achieving convolutional fusion of different feature information.

[0083] When using attention-based networks for fusion, the attention mechanism is used to dynamically allocate weights based on the importance of the input data, thereby achieving weighted concatenation of different features. This method can enhance the model's attention to important features and improve the temporal consistency of fused features.

[0084] Furthermore, in this embodiment, a suitable network model can be selected for feature information fusion based on the specific usage requirements and data characteristics of the application scenario. For example, for time series prediction problems, models such as Recurrent Neural Networks (RNNs), Long Short-Term Memory Networks (LSTMs), or Gated Recurrent Units (GRUs) can be selected. Simultaneously, during the training and optimization process, the network model can be trained using training data, and its performance can be evaluated through methods such as cross-validation. During training, the quality of the fused features can be evaluated by comparing the model's prediction results with the true values, and the model's parameters and feature concatenation methods can be continuously adjusted to optimize the temporal consistency of the fused features and the model's predictive ability.

[0085] In step S140, the forgery features in the fused features are detected, and the detection result is determined. The detection result is used to characterize whether the image to be detected is a forgery image.

[0086] In this embodiment, forgery features in the fusion features are detected by analyzing the information consistency between different modalities in the fusion features, thereby determining whether the image to be detected is a forgery image.

[0087] Optionally, the fusion features used in this embodiment to detect forgery features can be post-processed fusion features. Post-processing includes operations such as dimensionality reduction and feature selection. This enables the use of more refined fusion features to detect forgery features, thereby determining the detection results and improving the efficiency and accuracy of the detection results.

[0088] Furthermore, in this embodiment, the extracted feature information can be fused using the same neural network model to determine fused features with temporal consistency, and forged features in the fused features can be detected to determine the detection result. Specifically, as follows... Figure 3 As shown in the figure, in this embodiment, each feature information is input into a pre-trained feature detection network 32, and the feature detection network 32 fuses the feature information after feature extraction to determine the fused features with temporal consistency, and detects the fake features in the fused features to determine the detection result, thereby determining whether the image to be detected is a fake image.

[0089] Optionally, the feature detection network 32 in this embodiment adopts a large language model, but a suitable network model can also be selected according to the actual use scenario. There is no limitation on this.

[0090] Furthermore, in the face forgery detection system of this embodiment, the feature extraction network 31 and the feature detection network 32 can be trained independently based on different training samples, and then fine-tuned based on the same training samples (different from the training samples during independent training), and the fine-tuned network can be used for face forgery detection; or, the feature extraction network 31 and the feature detection network 32 can be trained simultaneously based on the same training samples, and the trained feature extraction network 31 and the feature detection network 32 can be applied to face forgery detection.

[0091] Optionally, in this embodiment, after determining the detection result of the object to be detected, in order to intuitively display the face forgery detection result, a detection report of the image to be detected will be generated in response to the detection result indicating that the image to be detected is a forged image. The detection report includes the detection result and the reason for forgery.

[0092] Furthermore, the image processing method in this embodiment also includes analyzing multimodal detection information, determining the cause of forgery, and displaying the detection result and the cause of forgery on the terminal device after determining the face forgery detection result and the cause of forgery, thereby realizing end-to-end display of the detection result and the cause of forgery on both the cloud and the terminal.

[0093] Optionally, since each feature in the multimodal detection information will affect the detection result, each feature can be analyzed separately in this embodiment to determine the forgery reason corresponding to each feature.

[0094] Furthermore, since forged facial images differ from real images in texture, edge contours, color, and shape, this embodiment can determine the cause of forgery by analyzing image information, thereby expressing the cause of forgery from the perspective of image representation.

[0095] When determining the cause of forgery by analyzing image information, this embodiment can use liveness detection technology to identify the image information of the image to be detected. By analyzing various features in the face image or video, such as image quality, texture details, and face movements, it can be determined whether the face in the image to be detected is real, and if the face is not real, the corresponding cause of forgery can be determined.

[0096] Specifically, when performing image quality analysis to determine the cause of forgery, if the image to be detected contains color loss, screen moiré patterns, or facial distortion, the corresponding forgery cause is output as poor image quality. When performing texture detail analysis, if the face in the image to be detected lacks skin texture, pores, or other detailed textures, or if the identified detailed textures differ significantly from those in a real face, it indicates insufficient texture detail in the image to be detected, and the corresponding forgery cause is output as texture anomaly. When performing facial motion analysis, if the facial motions (e.g., blinking, opening the mouth) in the image to be detected deviate significantly from the same type of facial motion in a real face, it indicates that the facial motions in the image to be detected are unnatural or impossible to complete, and the corresponding forgery cause is output as motion anomaly.

[0097] Furthermore, under normal circumstances, when users perform facial verification using real facial images, the device being detected is typically a general-purpose device with standard permission settings. However, if the device permission settings are intentionally modified or the device environment information is not authentic, the probability of the facial image uploaded by that device being a forged image is higher. Therefore, by analyzing the device parameter information of the device that collected the image under detection, the reasons for forgery can be expressed from a device perspective.

[0098] When determining the cause of forgery by analyzing the device parameters corresponding to the image to be detected, this embodiment can utilize the permission setting information and / or device environment information in the device parameters. When using permission setting information to determine the cause of forgery, the method in this embodiment includes: detecting the permission setting information in the device parameters; and, in response to detecting that the permission setting information has been modified, determining that the corresponding cause of forgery is that the device permissions have been modified.

[0099] Specifically, when the terminal device acquiring the image to be detected is a mobile phone, considering that rooting a mobile phone is usually a way to obtain the highest administrator privileges of the mobile phone operating system, root privileges allow users to access and modify system files, install uncertified applications, delete pre-installed software, and perform system-level optimizations, this embodiment can detect the reason for forgery by detecting whether the device has been rooted. When root modification information or permission setting information indicating that the device has been rooted is detected in the device's permission settings, it indicates that the device's permission settings have been modified, and the corresponding reason for forgery is output as "device permissions have been modified".

[0100] When using device environment information to determine the cause of forgery, the method for determining the cause of forgery in this embodiment includes: detecting the device environment information in the device parameters, and in response to detecting that the device environment information is a virtual environment, determining that the corresponding cause of forgery is that the device is a virtual device.

[0101] Furthermore, when a user performs face verification using a real face image, the location information in the behavioral information corresponding to the image to be detected is the real location, and when the image to be detected includes an environmental image, the location information also matches the environmental information in the image to be detected. Simultaneously, the face-crossing trajectory corresponding to the image to be detected is continuous; face-crossing images at different times should have temporal consistency, and face-crossing images at different locations should have location consistency. Therefore, in this embodiment, the reasons for forgery can be expressed from a behavioral perspective by analyzing the behavioral information corresponding to the image to be detected.

[0102] In determining the cause of forgery by analyzing the behavioral information corresponding to the image to be detected, this embodiment can utilize the location information, camera parameter information, and / or face trajectory information in the behavioral information. When using location information to determine the cause of forgery, the authenticity of the location information and the location represented by the location information are matched with the environmental image in the image to be detected. If the location information in the behavioral information is detected to be a non-real location or does not match the environmental image (such as landmarks, buildings, etc. in the background) in the image to be detected, the corresponding cause of forgery is output as location anomaly.

[0103] When using camera parameter information to determine the cause of forgery, the values ​​in the camera parameters are checked. If abnormal values ​​are detected in the camera parameters (such as values ​​that are beyond the actual capabilities of the camera or are uncommon camera parameter values), or if the values ​​are inconsistent with the image content in the image to be detected, or if there are obvious logical errors, the corresponding cause of forgery is output as abnormal camera parameters.

[0104] When using face-crossing trajectory information to determine the cause of forgery, this embodiment performs trajectory continuity analysis, time consistency analysis, and / or location consistency analysis on the face-crossing trajectory information to determine the corresponding cause of forgery based on the analysis results. Specifically, when the analysis results indicate that the trajectory is discontinuous, the time is inconsistent, or the location is inconsistent, it indicates that the face-crossing trajectory is unnatural or lacks necessary features, and the corresponding cause of forgery is output as face-crossing trajectory abnormality.

[0105] Furthermore, when users perform face verification using real facial images, the audio information corresponding to the image to be detected is associated with the image information. By analyzing the consistency between the voice content and the visual content of the image, more details and features of the forgery behavior can be captured, so as to express the reasons for forgery from the audio dimension.

[0106] In determining the cause of forgery by analyzing the audio information of the image to be detected, this embodiment can extract audio features such as speech rhythm, intonation, and timbre from the audio information, and compare the extracted audio features with the audio features of a real person to determine the authenticity of the audio information. Specifically, when the audio features extracted from the audio information differ significantly from the audio features of a real person, it is determined that the face in the image is forged, and the corresponding cause of forgery is output as audio anomaly.

[0107] The technical solution of this embodiment combines multimodal information for face forgery detection, which can comprehensively analyze various situations in the face verification process, identify cross-modal inconsistencies in the forgery process, and effectively deal with complex high-quality forgery content. In particular, it can maintain high-efficiency detection capability when facing new and multi-layered forgery attacks, overcome the limitations of single-modal detection, and improve the accuracy of face forgery detection. At the same time, multimodal fusion can capture more subtle forgery traces, thereby dealing with high-precision forgery content and improving the robustness of forgery detection. Especially when facing realistic and new forgeries, it can more accurately and quickly capture forgery features, achieve the interception of deepfake attacks, and thus improve the security of face verification.

[0108] Figure 4 This is a schematic diagram of an image processing apparatus according to an embodiment of the present invention. Figure 4As shown, the image processing device in this embodiment includes an information acquisition unit 1, a feature extraction unit 2, and a forgery detection unit 3. The information acquisition unit 1 acquires multimodal detection information corresponding to the image to be detected, including image information, device parameter information, and behavioral trajectory information. The feature extraction unit 2 inputs various feature information from the multimodal detection information into a feature extraction network to determine the corresponding feature information. The forgery detection unit 3 fuses the feature information to determine fused features with temporal consistency; and detects forgery features in the fused features to determine the detection result, which is used to characterize whether the image to be detected is a forged image.

[0109] Optionally, the forgery detection unit 3 in this embodiment is further configured to analyze the multimodal detection information to determine the cause of forgery; and in response to the detection result indicating that the image to be detected is a forged image, generate a detection report of the image to be detected, the detection report including the detection result and the cause of forgery.

[0110] Furthermore, in this embodiment, when analyzing multimodal detection information to determine the cause of forgery, the forgery detection unit 3 is specifically used to detect the permission setting information in the device parameters; in response to detecting that the permission setting information has been modified, it determines that the corresponding cause of forgery is that the device permissions have been modified; and to perform trajectory continuity analysis, time consistency analysis and / or location consistency analysis on the face-crossing trajectory information in the behavior trajectory information, so as to determine the corresponding cause of forgery based on the analysis results.

[0111] Optionally, the behavioral trajectory information in this embodiment includes face-crossing trajectory information. When acquiring the multimodal detection information corresponding to the image to be detected, the information acquisition unit 1 is also used to acquire the continuous frame images corresponding to the image to be detected. The continuous frame images include the image to be detected and the adjacent images of the adjacent frames of the image to be detected. The image information in the image to be detected and the adjacent images are identified to determine the face-crossing trajectory information corresponding to the image information.

[0112] Figure 5 This is a schematic diagram of an electronic device according to an embodiment of the present invention. (For example...) Figure 5The illustrated electronic device is a general-purpose data processing device, comprising a general-purpose computer hardware architecture, including at least a processor 41 and a memory 42. The processor 41 and memory 42 are connected via a bus 43. The memory 42 is adapted to store instructions or programs executable by the processor 41. The processor 41 can be a standalone microprocessor or a collection of one or more microprocessors. Thus, the processor 41 executes the instructions stored in the memory 42, thereby performing the method flow of the embodiments of the present invention as described above to process data and control other devices. The bus 43 connects the aforementioned components together, and also connects these components to a display controller 44, a display device, and an input / output (I / O) device 45. The input / output (I / O) device 45 can be a mouse, keyboard, modem, network interface, touch input device, motion-sensing input device, printer, and other devices known in the art. Typically, the input / output device 45 is connected to the system via an input / output (I / O) controller 46.

[0113] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus (devices), or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0114] This application is described with reference to flowchart illustrations of methods, apparatus (devices), and computer program products according to embodiments of this application. It should be understood that each step in the flowchart can be implemented by computer program instructions.

[0115] These computer program instructions may be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction means, the implementation process of which is described in the instruction means. Figure 1 The function specified in one or more processes.

[0116] These computer program instructions may also be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, produce instructions for implementing processes. Figure 1 A device for a function specified in one or more processes.

[0117] Another embodiment of the present invention relates to a non-volatile storage medium for storing a computer-readable program for use by a computer to execute some or all of the above-described method embodiments.

[0118] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program specifying the relevant hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0119] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image processing method, characterized in that, The method includes: Obtain multimodal detection information corresponding to the image to be detected, wherein the multimodal detection information includes image information, device parameter information, and behavior trajectory information; Each feature information in the multimodal detection information is input into the feature extraction network to determine the corresponding feature information; The aforementioned feature information is fused to determine a fused feature with temporal consistency; The forgery features in the fused features are detected, and the detection result is determined. The detection result is used to characterize whether the image to be detected is a forgery image.

2. The method according to claim 1, characterized in that, The method further includes: In response to the detection result indicating that the image to be detected is a forged image, a detection report of the image to be detected is generated, the detection report including the detection result and the reason for forgery.

3. The method according to claim 2, characterized in that, The method further includes: The multimodal detection information is analyzed to determine the cause of forgery.

4. The method according to claim 3, characterized in that, The analysis of the multimodal detection information to determine the cause of forgery includes: The permission settings information in the device parameters are checked; In response to the detection that the permission settings information has been modified, the corresponding reason for forgery is determined to be that the device permissions have been modified.

5. The method according to claim 3, characterized in that, The analysis of the multimodal detection information to determine the cause of forgery includes: The face-crossing trajectory information in the behavioral trajectory information is subjected to trajectory continuity analysis, time consistency analysis, and / or location consistency analysis in order to determine the corresponding forgery reason based on the analysis results.

6. The method according to claim 1, characterized in that, The behavioral trajectory information includes face-crossing trajectory information, and the acquisition of multimodal detection information corresponding to the image to be detected includes: Obtain consecutive frame images corresponding to the image to be detected, wherein the consecutive frame images include the image to be detected and adjacent images of adjacent frames of the image to be detected; The image information in the image to be detected and the adjacent images is identified to determine the face trajectory information corresponding to the image information.

7. The method according to claim 1, characterized in that, The feature extraction network includes a visual encoder, a first conversion neural network, and a second conversion neural network. The visual encoder is used to extract features from the image information to determine the corresponding visual features; the first conversion neural network is used to extract features from the device parameter information to determine the corresponding device parameter features. The second conversion neural network is used to extract features from the behavioral trajectory information and determine the corresponding trajectory features.

8. The method according to claim 1, characterized in that, The multimodal detection information also includes audio information and / or text information, wherein the text information is generated based on the image information.

9. The method according to claim 8, characterized in that, The feature extraction network includes an audio encoder and / or a text encoder. The audio encoder is used to extract features from the audio information to determine the corresponding audio features; the text encoder is used to extract features from the text information to determine the corresponding text features.

10. The method according to claim 1, characterized in that, The step of inputting each feature information from the multimodal detection information into the feature extraction network to determine the corresponding feature information includes: The image information, device parameter information, and behavior trajectory information are preprocessed respectively to determine the corresponding encoding representation; Each of the aforementioned encoded representations is input into the feature extraction network to determine the corresponding feature information.

11. An image processing apparatus, characterized in that, The device includes: The information acquisition unit is used to acquire multimodal detection information corresponding to the image to be detected, wherein the multimodal detection information includes image information, device parameter information and behavior trajectory information; The feature extraction unit is used to input the feature information from the multimodal detection information into the feature extraction network to determine the corresponding feature information; The forgery detection unit is used to fuse the aforementioned feature information to determine fused features with temporal consistency; to detect forgery features in the fused features and to determine the detection result, wherein the detection result is used to characterize whether the image to be detected is a forged image.

12. A computer program product, characterized in that, The computer program product includes a computer program / instruction that, when executed by a processor, implements the method of any one of claims 1-10.

13. An electronic device comprising a memory and a processor, characterized in that, The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method of any one of claims 1-10.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1-10.

Images