Multi-modal risk data synthesis method for anti-fraud and risk identification

Abstract

The invention relates to the technical field of artificial intelligence, and provides a multi-modal risk data synthesis method for anti-fraud and risk identification. The method comprises the following steps: acquiring a target face image, target sound data and a video containing a face and sound; in the acquired video or image, replacing an original face with a target face through a deep counterfeiting technology, and replacing original sound in the video with target sound; marking images and audios in the generated multi-modal data; and inputting the forged and synthesized multi-modal data and the labels thereof into a multi-modal large model, identifying and analyzing the multi-modal data, and generating a risk abstract. According to the method, a systematic multi-modal risk content generation process is constructed, the requirement of a multi-modal large model for high-quality training data is met, the advantages of synthetic data in risk analysis are brought into full play through combined use of forged data and the multi-modal large model, and the risk analysis efficiency is improved. And the identification and coping capability of potential risks in a complex multi-modal scene is obviously improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multimodal risk data synthesis method for anti-fraud and risk identification. Background Art

[0002] With the rapid development of communication and digital technologies, various fraud methods and the ways of spreading false information have become increasingly complex and diverse. Among them, new multimodal risk content such as fake faces, fake voices, and fake videos are emerging in an endless stream, seriously threatening national security, economic development, and social stability. Existing content recognition technologies face many challenges in dealing with these complex and diverse risk contents, and it is difficult to provide comprehensive and accurate detection methods.

[0003] Current anti-fraud technology mainly relies on single-modal data analysis and recognition when dealing with fake faces, sounds, and videos. However, this approach shows obvious limitations when faced with complex risk content that includes multimodal data. Single image or voice processing technology cannot effectively deal with forgeries involving multiple modalities, such as video content that combines fake faces with synthetic audio. With the continuous advancement of deepfake technology, the authenticity and complexity of this type of multimodal forged content has significantly increased, further increasing the challenges of traditional detection technology.

[0004] In recent years, with the continuous advancement of artificial intelligence technology, multimodal large models have gradually become an important tool in the field of content recognition. These models can simultaneously accept information input from multiple modes such as video and audio, and integrate information from different modes to achieve more accurate risk content analysis and identification. However, although multimodal large models have powerful analytical capabilities, existing multimodal risk data is extremely scarce and cannot provide sufficient training and testing support for these models. This lack of data directly limits the actual application effect of multimodal large models in anti-fraud and risk identification.

[0005] Therefore, there is an urgent need for a synthesis method that can generate high-quality multimodal risk data, especially for the synthesis of forged faces, sounds, and videos, to meet the needs of large multimodal models and improve the overall effectiveness of anti-fraud and risk identification systems.

[0006] In view of this, overcoming the defects of the prior art is an urgent problem to be solved in the field of this technology. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a multimodal risk data synthesis method for anti-fraud and risk identification.

[0008] The present invention adopts the following technical solution:

[0009] In a first aspect, the present invention provides a multimodal risk data synthesis method for anti-fraud and risk identification, comprising:

[0010] In step 201, a target face image, target voice data, and a video containing the face and voice are obtained as raw materials for multimodal data synthesis;

[0011] In step 202, in the acquired video or image, the original face is replaced with the target face through deep fake technology, ensuring that the posture and expression of the target face in the image or video are consistent with the original content. At the same time, the original sound in the video is replaced with the target sound using audio synthesis technology, so that the replaced sound is synchronized with the face movement, thereby generating multimodal data; wherein the target face is taken from the target face image, and the target sound is taken from the target sound data;

[0012] In step 203, the generated multimodal data is annotated, including the annotation of images and audios: the forged face area is segmented at the pixel level using a mask segmentation model to generate an accurate mask; the forged sound data is transcribed into text and the corresponding time period is annotated to mark the starting and ending positions of the forged audio;

[0013] In step 204, the forged synthesized multimodal data and its annotations are input into the multimodal big model, and the input multimodal data is identified and analyzed using prompt word engineering and multi-step thinking chain technology to generate a risk summary containing risk points.

[0014] Preferably, the step of obtaining the target face image, target sound data, and a video containing the face and sound specifically includes:

[0015] Get the target face data from the FFHQ dataset, get the target sound data from the MAGICDATA dataset, and get the video from the ChineseFaces dataset.

[0016] Preferably, the step of acquiring the target face image, target sound data, and a video containing the face and sound further includes:

[0017] Preprocessing the target face image, target sound data and video; wherein the preprocessing includes resizing and / or denoising.

[0018] Preferably, in the acquired video or image, replacing the original human face with the target human face by deep fake technology specifically includes:

[0019] Perform frame extraction on each video to obtain the face image in each frame;

[0020] Use face recognition technology to accurately locate the position of the face, randomly extract multiple different target face images from the face dataset, and use the face forgery model to replace the face in each frame;

[0021] Among them, the face dataset is a dataset formed by multiple target face images, the face forgery model includes a DeepFaceLab model or an inswapper_128 model, and the inswapper_128 model includes one or more of a Roop model, a FaceSwapLab model and a Reactor model.

[0022] Preferably, the method of replacing the original sound in the video with the target sound by using audio synthesis technology specifically includes:

[0023] Separate the sound of the video into separate wav files to obtain the audio of the video; and perform vocal separation on these audios to extract clear voice clips;

[0024] The audio of each video is sliced ​​and a plurality of different sound models are randomly selected for speech conversion; wherein the sound model is trained using an open source database or provided by an already trained open source model; a timestamp is randomly generated for each audio clip, and then the human voice is converted within the range of these timestamps.

[0025] Preferably, the use of a mask segmentation model to perform pixel-level segmentation on the forged face area to generate an accurate mask specifically includes:

[0026] Use the Grounding DINO model to perform preliminary face recognition and locate the forged face.

[0027] After identifying the forged face position, the SAM model is used to perform pixel-level segmentation on the area where the forged face position is located to generate a segmentation mask.

[0028] Preferably, the forged sound data is transcribed into text and the corresponding time period is marked to mark the starting position and the ending position of the forged audio, specifically including:

[0029] The timestamp added during the process of replacing the original sound in the video with the target sound is used as the annotation information;

[0030] The voice data is transcribed into text. For Chinese content, the Dharma ASR model is used for voice transcription. For voice transcription in other languages, the Whisper model is used for processing to ensure the transcription accuracy and consistency of multilingual voice data.

[0031] Preferably, the step of inputting the forged synthesized multimodal data and its annotations into the multimodal macro model specifically includes:

[0032] Input the annotated faces and audios in the forged and synthesized multimodal data into the multimodal large model; the video is processed by frame extraction, and the extracted frame images, corresponding forged images and forged labels are input into the model together; for the audio part, the text labels and timestamp labels are used as input;

[0033] The multimodal large model processes images and audio simultaneously, and by combining information from different modalities, that is, analyzing the association between image and audio data through multiple iterations, it can understand and analyze forged content and generate a summary of risk points.

[0034] Preferably, the use of prompt word engineering and multi-step thinking chain technology to identify and analyze the input multimodal data and generate a risk summary containing risk points specifically includes:

[0035] Set up multiple roles for the reasoning process of multimodal large models;

[0036] In the first role, the model generates a detailed description of the original data, including content analysis of images, audio, and video that have not been manipulated;

[0037] In the second role, the model describes the falsified data and identifies anomalies or changes compared to the original data;

[0038] In the third role, the model determines the potential risk points in the data based on the forged annotation information and identifies the forged areas and time periods;

[0039] In the summary role, the model synthesizes and analyzes the information generated in the previous steps to generate a final risk summary.

[0040] Preferably, the multimodal large model is a closed source model or an open source model; wherein the closed source model includes one or more of GPT-4o, Claude3.5-Sonnet, and Gemini-1.5-Pro; the open source model is one or more of InternVL2-Llama3-76B and MiniCPM-V-2.6.

[0041] In a second aspect, the present invention further provides a multimodal risk data synthesis device for anti-fraud and risk identification, which is used to implement the multimodal risk data synthesis method for anti-fraud and risk identification described in the first aspect, and the device includes:

[0042] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the multimodal risk data synthesis method for anti-fraud and risk identification described in the first aspect.

[0043] In a third aspect, the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors to complete the method described in the first aspect.

[0044] In a fourth aspect, a chip is provided, comprising: a processor and an interface, for calling and running a computer program stored in the memory from a memory, and executing the method of the first aspect.

[0045] According to a fifth aspect, a computer program product comprising instructions is provided. When the instructions are executed on a computer or a processor, the computer or the processor executes the method according to the first aspect.

[0046] The present invention constructs a systematic multimodal risk content generation process, which meets the needs of multimodal large models for high-quality training data. By combining forged data with multimodal large models, the advantages of synthetic data in risk analysis can be fully utilized, significantly improving the ability to identify and respond to potential risks in complex multimodal scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 It is a flow chart of a multimodal risk data synthesis method for anti-fraud and risk identification provided by an embodiment of the present invention;

[0049] Figure 2 It is a flow chart of a multimodal risk data synthesis method for anti-fraud and risk identification provided by an embodiment of the present invention;

[0050] Figure 3 It is a flow chart of a multimodal risk data synthesis method for anti-fraud and risk identification provided by an embodiment of the present invention;

[0051] Figure 4 is a schematic diagram of a multimodal risk data synthesis method for anti-fraud and risk identification provided by an embodiment of the present invention;

[0052] Figure 5 is a schematic diagram of a multimodal risk data synthesis method for anti-fraud and risk identification provided by an embodiment of the present invention;

[0053] Figure 6 It is a schematic diagram of the architecture of a multimodal risk data synthesis device for anti-fraud and risk identification provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0055] Unless the context requires otherwise, throughout the specification and claims, the term "including" is to be interpreted as open inclusion, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "examples", "specific examples" or "some examples" and the like are intended to indicate that specific features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any appropriate manner, that is, although they may be carried in the embodiments or examples of the above terms due to reasons such as the order and position of appearance, it is not limited to that they can be carried in combination by one embodiment or example.

[0056] In the description of the present invention, the terms "first" and "second" are used only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, the meaning of "multiple" is two or more. In addition, for example, the same type of nouns may be described as two independent individuals by adding "A" and "B" at the end. In this case, the corresponding features defined as "A" and "B" are only used to distinguish the same type of individuals for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features.

[0057] In the description of the present invention, the expression "A and / or B" (where A and B are used to formally represent specific characteristic contents) will be involved, and the corresponding expressions include the following three combinations: only A, only B, and a combination of A and B.

[0058] As used herein, "about," "substantially," or "approximately" includes the stated value and an average value that is within an acceptable range of deviation from the particular value as determined by one of ordinary skill in the art taking into account the measurements in question and the errors associated with the measurement of the particular quantity (i.e., the limitations of the measurement system).

[0059] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0060] Embodiment 1:

[0061] In view of the problem in the prior art that the lack of multimodal risk data makes it impossible to effectively support the application of multimodal large models in anti-fraud and risk identification, the present invention provides a multimodal data synthesis method for anti-fraud and risk identification, which aims to provide an efficient synthetic data technical means through the generation of multimodal risk data to improve the overall performance of the anti-fraud and risk identification system, that is, to generate high-quality multimodal forged data (including faces, audio and video) to meet the needs of multimodal large models for diversified data during training and testing, thereby solving the technical problem in the prior art that the performance of the anti-fraud and risk identification system is insufficient due to the lack of sufficient multimodal risk data. The multimodal risk data synthesis method for anti-fraud and risk identification provided in Example 1 of the present invention is as follows: Figure 1 As shown, including:

[0062] In step 201, a target face image, target voice data, and a video containing a face and voice are obtained as raw materials for multimodal data synthesis; wherein the obtaining of the target face image, target voice data, and a video containing a face and voice specifically includes: obtaining the target face data from the FFHQ dataset, obtaining the target voice data from the MAGICDATA dataset, and obtaining the video from the ChineseFaces dataset.

[0063] In step 202, in the acquired video or image, the original face is replaced with the target face by deep fake technology, ensuring that the posture and expression of the target face in the image or video are consistent with the original content. At the same time, the original sound in the video is replaced with the target sound by audio synthesis technology, so that the replaced sound is synchronized with the face movement, thereby generating multimodal data (i.e., the video or image after face replacement and sound replacement); wherein the target face is taken from the target face image, and the target sound is taken from the target sound data.

[0064] The target face refers to the facial image that will be used to replace the original face in the image or video during the synthesis process. These images must have high clarity and diversity to ensure that the generated fake data can truly reflect different facial features. Our method can use a variety of open source face image datasets, such as FFHQ (Flickr-Faces-HQ), which contains 70,000 high-quality and diverse face images, covering a wide range of samples of different ages, genders, and races. The FFHQ dataset is known for its high resolution and diversity of images, providing a solid foundation for synthesizing high-quality fake faces.

[0065] The target voice refers to the speech data that will be used to replace the original audio during the synthesis process. These speech samples should cover a wide range of speaking speeds, intonations, and accents to ensure that the generated forged audio can realistically simulate different speaking styles. Our method can use a variety of open source audio datasets, such as MAGICDATA (Mandarin Chinese ReadSpeech Corpus), which is an open source dataset containing 755 hours of Mandarin speech data recorded by 1,080 speakers. The richness and diversity of the MAGICDATA dataset ensures the authenticity and wide applicability of the forged voices, allowing the synthesized audio to adapt to various application scenarios.

[0066] Regarding videos containing faces and voices, these data need to have clear close-ups of faces and synchronized audio to ensure accurate matching and replacement during the synthesis process, so as to truly simulate possible fraud scenarios in reality. Taking the ChineseFaces dataset as an example, the dataset contains 300 (150 females and 150 males) real face video data, each of which is 15 seconds long and covers a wide range of scenes and character features. ChineseFaces not only contains real video data, but also provides three different types of forged data, including identity replacement, expression-driven, and attribute editing (such as age editing), each with 300 videos, providing rich material for multimodal data synthesis.

[0067] In addition, specific real-world data can be obtained using crawler technology or other data sources based on specific needs. For example, for the dialect of a certain region, we can capture relevant voice data; similarly, if facial data of a specific region or group of people is needed, we can use crawler technology or specific data sets to obtain it. This flexible data preparation method not only makes the generated data more suitable for actual application scenarios, but also provides more accurate support for risk identification and fraud prevention. In this way, we can take advantage of data synthesis and more effectively generate targeted data to meet specific needs in complex scenarios, thereby improving the overall system's defense capabilities.

[0068] In step 203, the generated multimodal data is annotated, including the annotation of images and audios: the forged face area is segmented at the pixel level using a mask segmentation model to generate an accurate mask (the process of generating a mask is the process of annotating the image); the forged sound data is transcribed into text and the corresponding time period is annotated, marking the starting and ending positions of the forged audio (i.e., the process of annotating the audio).

[0069] In step 204, the forged synthesized multimodal data and its annotations are input into the multimodal big model, and the input multimodal data is identified and analyzed using prompt word engineering and multi-step thinking chain technology to generate a risk summary containing risk points.

[0070] In actual use, the step of obtaining the target face image, target voice data, and a video containing the face and voice also includes:

[0071] Preprocess the target face image, target sound data and video; wherein the preprocessing includes resizing and / or denoising. And after the above data is preprocessed (such as resizing, denoising, etc.), it is divided into a test set and a validation set to ensure the diversity and authenticity of the data and the effectiveness of the model training.

[0072] In actual application scenarios, in the acquired video or image, the original face is replaced with the target face through deep fake technology, specifically including:

[0073] In step 301, each video is subjected to frame extraction processing to obtain a face image in each frame.

[0074] In step 302, face recognition technology is used to accurately locate the position of the face, and multiple different target face images are randomly extracted from the face data set, and the face in each frame is replaced using the face forgery model. The number of extracted target face images is obtained by those skilled in the art based on empirical analysis. For example, in an optional embodiment, 30 different target face images are extracted.

[0075] Among them, the face dataset is a dataset formed by multiple target face images, the face forgery model includes a DeepFaceLab model or an inswapper_128 model, and the inswapper_128 model includes one or more of a Roop model, a FaceSwapLab model and a Reactor model.

[0076] The method of replacing the original sound in the video with the target sound by using the audio synthesis technology specifically includes:

[0077] In step 401, the sound of the video is separated into separate wav files to obtain the audio of the video; and the human voice is separated from the audio to extract clear voice segments.

[0078] In step 402, the audio of each video is sliced ​​and a plurality of different sound models are randomly selected for voice conversion; wherein the sound model is obtained by training using an open source database or provided by an already trained open source model; a timestamp is randomly generated for each audio segment, and then the human voice is converted within the range of these timestamps. The number of selected sound models is obtained by those skilled in the art based on empirical analysis, such as in an optional embodiment, 30 different sound models are selected.

[0079] In actual use, the above steps 301-302 and 401-402 can adopt a modular design, allowing flexible selection of different deep fake methods. Both face fakes and audio fakes can be implemented by selecting different technical modules according to needs, including but not limited to face replacement after large-scale training using DeepFaceLab, or face fakes through popular open source technologies such as Roop, FaceSwapLab, Face Fusion, Reactor, etc.; the audio fake part can achieve audio conversion through trained voice models or open source trained models (such as RVC or so-vits-svc). In addition, the most appropriate fake methods and modules can be flexibly selected according to the final use of the data and the specific demand scenarios to better meet the requirements of specific applications, thereby ensuring the flexibility and adaptability of the method.

[0080] In an optional implementation, the use of a mask segmentation model to perform pixel-level segmentation on the forged face area to generate an accurate mask specifically includes: using a Grounding DINO model to perform preliminary face recognition to locate the forged face position; after identifying the forged face position, using a SAM model (Segment Anything Model) to perform pixel-level segmentation on the area where the forged face position is located, thereby generating a segmentation mask (i.e., the above-mentioned accurate mask).

[0081] The forged sound data is transcribed into text and the corresponding time periods are marked, and the starting and ending positions of the forged audio are marked, specifically including: using the timestamp added in the process of replacing the original sound in the video with the target sound (i.e., the timestamp generated in the above step 402) as the marking information; and transcribing the speech data into text; wherein, for Chinese content, the Dharma ASR model is used for speech transcription; for speech transcription in other languages, the Whisper model is used for processing to ensure the transcription accuracy and consistency of multilingual speech data.

[0082] That is, the generated multimodal data is accurately annotated to ensure the quality of the data and the effectiveness of subsequent training. Data annotation includes two parts: pixel-level segmentation of the face area and timestamp and text transcription annotation of the voice data. First, for the generated face forgery data, the Grounding DINO model is used for preliminary face recognition to accurately locate the forged face position. The Grounding DINO model can accurately identify the face area in the image, laying the foundation for subsequent segmentation processing. After locating the face area, the Segment Anything Model (SAM) is used for pixel-level segmentation. The SAM model uses efficient segmentation technology to assign labels to each pixel to clearly distinguish the forged part from the original background, thereby generating high-precision segmentation masks. These masks will be used as annotation data for subsequent training to ensure that the model recognizes forged faces more accurately during the training process. Secondly, for the generated forged voice data, the audio data is first annotated based on the timestamps recorded during the forgery. These timestamps record the start and end time of the forged voice segment and are important reference data in the annotation process. In addition, the voice data needs to be transcribed into text. For Chinese speech, the Dharma ASR model is used for automatic transcription to ensure efficient and accurate transcription of speech; for speech in other languages, the Whisper model is used for transcription processing. In this way, the generated text annotations not only improve the availability of speech data, but also provide richer training data for the model to identify forged speech. Through strict annotation processing of face and voice data, the present invention ensures the high quality and consistency of the generated data, so that the model can more effectively learn and identify multimodal forged content during training.

[0083] In some embodiments, inputting the forged synthesized multimodal data and its annotations into the multimodal macro model specifically includes:

[0084] The labeled faces and audios in the forged synthesized multimodal data are input into the multimodal large model; the video is processed by frame extraction, and the extracted frame images, corresponding forged images and forged labels are input into the model together; for the audio part, text labels and timestamp labels are used as input.

[0085] The multimodal large model processes images and audio simultaneously, and by combining information from different modalities, that is, analyzing the association between image and audio data through multiple iterations, it can understand and analyze forged content and generate a summary of risk points.

[0086] Specifically: The forged synthetic data and its annotation information are input into the multimodal large model, and the input risk data is identified and analyzed using the prompt word engineering and multi-step thinking chain technology, and finally a risk summary containing risk points is generated. When inputting the multimodal large model, the video content is firstly processed by frame extraction, and the extracted frame image, the corresponding forged image and the forged mark are input into the model together; for the audio part, the text mark and the timestamp mark are used as input. The multimodal large model can process image and audio data at the same time. By combining information from different modalities, the model can comprehensively analyze and understand the forged content. To further improve the recognition accuracy of the model, the prompt word engineering technology is used when inputting data. By designing and inputting appropriate prompt words, the model is guided to focus on specific risk features in the forged data, such as unnatural facial expressions or abnormal voice, thereby enhancing the recognition ability of the model. Next, the model applies the multi-step thinking chain technology in the processing process. This technology simulates the multi-step reasoning process of humans, enabling the model to gradually deduce potential risk points in the forged data. Specifically, the model analyzes the association between image and audio data through multiple iterations, gradually identifies the forged content, and generates a more accurate summary of risk points. The model will be set up in different roles, first generating a description of the original data, and then describing the forged data, focusing on identifying anomalies or changes compared to the original data. Then, the model determines the potential risk points in the data based on the forged annotation information (such as forged mask segmentation and voice timestamp marking), and identifies the areas and time periods that may be forged. Finally, the model conducts a comprehensive analysis of the information generated in the previous steps to generate a final risk point summary. The risk summary output by the model not only contains the identification results of the forged content, but also describes in detail the potential risks that these forged data may bring. These summary information will serve as an important input to the anti-fraud and risk identification system, helping to improve the overall defense capabilities of the system and effectively respond to a variety of threat scenarios.

[0087] The multimodal large model is a closed source model or an open source model; wherein the closed source model includes one or more of GPT-4o, Claude3.5-Sonnet, and Gemini-1.5-Pro; the open source model is one or more of InternVL2-Llama3-76B and MiniCPM-V-2.6. The selection of the multimodal large model can be flexibly adjusted according to the complexity of the data and the specific application scenario. The model includes a multimodal large model with strong capabilities that can complete complex recognition and reasoning tasks: that is, closed source models such as GPT-4o, Claude3.5-Sonnet, and Gemini-1.5-Pro, which are used to process in-depth analysis of complex multimodal data; or open source models such as InternVL2-Llama3-76B, MiniCPM-V-2.6, etc., which are used to provide flexible support for multimodal information processing in different application scenarios.

[0088] In an optional embodiment, the use of prompt word engineering and multi-step thinking chain technology to identify and analyze the input multimodal data and generate a risk summary containing risk points, specifically including: setting multiple roles for the reasoning process of the multimodal large model; the model generates a detailed description of the original data in the first role, including content analysis of images, audio and video that have not been forged; the model describes the forged data in the second role, and identifies anomalies or changes compared with the original data; the model determines potential risk points in the data based on the forged annotation information in the third role, and identifies the forged areas and time periods; the model comprehensively analyzes the information generated in the previous steps in the summary role to generate a final risk summary.

[0089] This embodiment constructs a systematic multimodal risk content generation process to meet the needs of multimodal large models for high-quality training data. And by combining forged data with multimodal large models, it is possible to give full play to the advantages of synthetic data in risk analysis and significantly improve the ability to identify and respond to potential risks in complex multimodal scenarios. And it is possible to efficiently synthesize high-quality multimodal data, achieve accurate correspondence between forged faces and voices, and fill the problem of scarcity of such data in the prior art. At the same time, the present invention performs well in generating high-precision annotated data, including pixel-level mask segmentation of images and accurate timestamp annotation of sounds. In addition, through prompt word engineering and multi-step thinking chain technology, the present invention not only generates original multimodal risk data and its corresponding annotations, but also further generates a corrected risk point summary. These generated data provide strong support for the training of multimodal large models, which can significantly improve the accuracy and generalization ability of risk identification. The model trained with data generated by this method has improved the accuracy of video forgery detection by 11.5% compared with the most advanced model Google TruFor, further proving the practicality and reliability of the present invention in complex risk identification scenarios.

[0090] Embodiment 2:

[0091] The present invention is based on the method described in Example 1, combined with specific application scenarios, and uses technical descriptions in related scenarios to illustrate the implementation process of the present invention in characteristic scenarios. Figure 1 As shown, the present invention proposes a multimodal risk data synthesis method for anti-fraud and risk identification, comprising the following steps:

[0092] Step 1: Data preparation: obtain the target face image, target sound data, and video data containing face and sound as the original materials for multimodal data synthesis.

[0093] Specifically, in the data acquisition process, we use a variety of open source resources and specific datasets to build the foundation for synthesizing multimodal risk data. The target face images can come from open source datasets such as FFHQ (Flickr-Faces-HQ), which contains diverse face images of different ages, genders, and races to ensure the diversity and authenticity of the forged data. The target voice data can come from MAGICDATA (Mandarin Chinese Read Speech Corpus), which contains 755 hours of Mandarin voice recordings of 1,080 speakers, providing rich voice samples covering different speaking speeds, intonations, and accents. In addition, video data can be obtained from the ChineseFaces dataset, which contains 300 real videos and covers a variety of forgery types (such as identity replacement, expression-driven, and attribute editing). These data are pre-processed, such as resizing and denoising, to ensure the quality of the data, laying a solid foundation for subsequent multimodal forgery synthesis.

[0094] Step 2: In the acquired video or image, replace the original face with the target face through deep fake technology to ensure that the posture and expression of the target face in the image or video are consistent with the original content. For the ChineseFaces dataset, first extract the frame of the video to obtain the face image of each frame. After accurately locating the face position using face recognition technology, 30 different face images are randomly extracted from the face dataset, and the inswapper_128_fp16.onnx model is used to replace the original face in each frame. The replaced face image is enhanced using the GFPGANv1.4 model to make it more realistic. At the same time, the original sound in the video is replaced with the target sound using audio synthesis technology, so that the replaced sound is synchronized with the face movement, thereby generating multimodal risk content with high authenticity. The specific operation includes using the open source RVC model, which provides 24,218 different sound models for selection. During the operation, the audio of the video is first separated into a wav file and the human voice is separated. Then the audio is sliced ​​and 30 different sound models are randomly selected and applied to the audio data within the timestamp of the audio slice to ensure that the sound is synchronized with the facial movements. In this way, multimodal forged data with a high degree of authenticity is generated.

[0095] Specifically, since our data synthesis method is modular, we can flexibly choose different forgery methods to generate data to deal with various types of forgery methods.

[0096] In terms of face forgery, most of the fake face data in the past was generated through DeepFaceLab, a very popular open source tool that can easily embed the target face into the video to generate highly realistic fake faces. It is worth noting that using DeepFaceLab usually requires a large number of target face images to train a face-swapping model, which enables this method to generate highly realistic face replacement effects in complex scenes. Recently, some methods based on the inswapper_128 model have also emerged, such as Roop, FaceSwapLab, and Reactor. A significant advantage of these methods is that they only require one target face image to achieve the face-swapping effect and do not require additional model training, which greatly reduces the operational threshold. These technologies are powerful and easy to use, but they are also more likely to be used by criminals to commit fraud. We can use these forgery techniques to generate corresponding data for training models to enhance the ability to recognize forged faces.

[0097] Specifically, taking the ChineseFaces dataset as an example, we first extract frames from each video to obtain the face image in each frame. Then, we use face recognition technology to accurately locate the face position, and randomly extract 30 different face images from the face dataset, and use the above face forgery model to replace the face in each frame. With this method, we are able to generate a large amount of diverse forged video data, which can be used to train our model to deal with complex forgery scenarios.

[0098] In terms of voice forgery, existing technologies such as so-vits-svc and RVC have demonstrated highly realistic voice conversion capabilities. These tools can convert one person's voice into another person's voice, and the converted audio is extremely realistic in terms of intonation, speaking speed, and sound quality. The ease of use and efficiency of these technologies make them potential tools for criminals to commit voice fraud. We can use these voice forgery technologies to generate a large amount of forged audio data as part of the training set to improve the model's ability to recognize forged voices.

[0099] In specific operations, we can achieve voice forgery in two ways: one is to use open source datasets to train some specific human voice models for voice conversion; the other is to directly use open source trained models for voice conversion. For example, the RVC model open sourced by applio provides 24,218 different voice models that can be used for various voice conversion needs. Taking the ChineseFaces dataset as an example, first, we separate the sound of the video into separate wav files and perform human voice separation on these audios to extract clear voice fragments. Next, the audio of each video is sliced ​​and 30 different voice models are randomly selected for voice conversion. In order to ensure the authenticity and diversity of the forgery, we randomly generate a timestamp for each audio clip and then convert the human voice within these timestamps. This method can effectively generate diverse forged audio and provide rich materials for training models to identify forged voices.

[0100] Through this modular forgery replacement method, we can generate corresponding adversarial data for different types of forgery methods, providing more accurate training materials for the multimodal risk identification system, thereby significantly improving the system's defense capabilities and practicality.

[0101] Step 3: Label the generated multimodal data, including: using a mask segmentation model to perform pixel-level segmentation on the forged face area to generate an accurate mask; transcribing the forged sound data into text and labeling the corresponding time periods, clearly marking the start and end positions of the forged audio.

[0102] Specifically, we accurately annotate the generated multimodal data to ensure the quality of the data and the effectiveness of subsequent training. The data annotation includes two parts: pixel-level segmentation of the face area and forged timestamps and text transcription annotations of the voice data.

[0103] First, for the generated forged face data, we use the Grounding DINO model to perform preliminary face recognition and locate the forged face position. The Grounding DINO model can accurately identify the face area in the image, laying the foundation for subsequent precise segmentation. After identifying the face position, we use the SAM model to perform pixel-level segmentation of the area. The SAM model can efficiently segment the face area and assign labels to each pixel to distinguish the forged part from the original background. In this way, we are able to generate high-precision segmentation masks and provide clear and reliable labeled data for subsequent training.

[0104] Secondly, for the generated forged voice data, we annotated it according to the timestamps generated during the forgery. Specifically, we recorded the start and end time of each audio segment during the forgery process and used these timestamps as annotation information. In addition, to further enhance the usability of the data, we transcribed the voice data into text. For Chinese content, we used the Dharma ASR model for voice transcription, which can efficiently and accurately transcribe voice into text. For voice transcription in other languages, we used the Whisper model for processing to ensure the transcription accuracy and consistency of multilingual voice data. This precise timestamp and text annotation can help the model better identify and distinguish different voice segments during training, thereby improving the ability to recognize forged voices.

[0105] By rigorously annotating the generated data, we ensure the high quality and consistency of the data, enabling the model to more effectively learn and identify multimodal forged content during training.

[0106] In the above step 3, the face area in the forged data is accurately segmented at the pixel level, including generating a high-precision segmentation mask by identifying and locating the forged face position; at the same time, the forged audio data is annotated, specifically including segmenting the audio based on the timestamp information during the forgery process, and transcribing the voice content into text to ensure the accuracy and consistency of the annotation. Furthermore, the forged annotation not only includes the distinction between positive and negative samples, but also accurately marks which pixels or audio segments have been tampered with, so that the model can identify the most subtle traces of forgery during training, thereby improving the recognition accuracy of forged data and the overall reliability of the system. The pixel-level segmentation of the face area is performed by using the GroundingDINO model for preliminary face recognition and positioning, and then the Segment Anything Model (SAM) is used to segment the identified face area at the pixel level to generate a high-precision segmentation mask; audio annotation is performed based on the timestamp generated during forgery, and the voice data is transcribed into text through automatic speech recognition (ASR) technology, where Chinese voice is transcribed using the Dharma ASR model and other language voices are transcribed using the Whisper model to ensure the accuracy of voice transcription and the consistency of annotation in a multilingual environment.

[0107] Step 4: Generate risk analysis data. Input the forged synthetic data and its annotations into the multimodal large model, use prompt word engineering and multi-step thinking chain technology to identify and analyze the input risk data, and generate a risk summary containing risk points.

[0108] Specifically, we input the forged synthetic data and its annotations into the multimodal large model, use the prompt word engineering and multi-step thinking chain technology to identify and analyze the input risk data, and finally generate a risk summary containing risk points. This process aims to generate high-quality and diverse forged analysis data to improve the model's ability to identify and process complex risk content. That is, the annotated forged data is input into the multimodal large model, and the prompt word engineering technology is used to guide the model to focus on the key features in the forged data, and the multi-step thinking chain technology is used to gradually deduce and identify potential risk points in the data, and finally generate a risk summary containing risk points to improve the accuracy and reliability of the anti-fraud and risk identification system. The choice of the multimodal large model can be flexibly adjusted according to the complexity of the data and the specific application scenario. The models include large multimodal models with strong capabilities that can complete complex recognition and reasoning tasks: closed-source models, such as GPT-4o, Claude3.5-Sonnet, Gemini-1.5-Pro, which are used to process in-depth analysis of complex multimodal data; or open-source models, such as InternVL2-Llama3-76B, MiniCPM-V-2.6, etc., which are used to provide flexible support for multimodal information processing in different application scenarios.

[0109] When inputting the multimodal large model, the prompt word engineering technology extracts frames from the video content, and inputs the extracted frame images, corresponding forged images, and forged marks into the model together; for the audio part, the text mark and timestamp mark are used as input. After receiving these inputs, the model guides the model to focus on the key features in the forged data through carefully designed prompt words, and identifies the differences between the original data and the forged data. Through the multi-step thinking chain technology, the model generates descriptions of the original data and the forged data in turn, and determines potential risk points based on the forged annotation information. Finally, all information is summarized to generate a risk summary containing risk points, thereby effectively improving the ability to identify complex multimodal forged data.

[0110] First, the labeled face and voice data are input into the multimodal large model. The video content is processed by frame extraction, and the extracted frame image, the corresponding forged image, and the forged mark are input into the model together; for the audio part, the text mark and the timestamp mark are used as input. The multimodal large model can process image and audio data at the same time. By combining information from different modalities, the model can more comprehensively understand and analyze forged content.

[0111] To improve the model's analysis accuracy, we use prompt word engineering technology when inputting data. By designing and inputting appropriate prompt words, we guide the model to focus on specific risk features in forged data, such as unnatural facial expressions or voice anomalies, thereby enhancing the model's recognition ability. Next, the model uses multi-step thinking chain technology during the analysis process. This technology simulates the multi-step reasoning process of human thinking, enabling the model to gradually deduce and mine potential risk points in forged data. The model will analyze the association between image and audio data through multiple iterations, gradually identify forged content, and generate a more accurate summary of risk points.

[0112] During the reasoning process, the multimodal large model is set to different roles. The model first generates a detailed description of the original data, and then generates a description of the forged data, focusing on identifying anomalies or changes compared to the original data. Then, the model determines potential risk points based on the forged annotation information (such as forged mask segmentation and voice timestamp marking), and identifies the areas and time periods that may be forged. Finally, the model conducts a comprehensive analysis of the information generated in the previous steps to generate a final summary data containing risk points. Specifically: The multimodal large model is set to different roles during the reasoning process. The first role of the model is responsible for generating a detailed description of the original data, including content analysis of images, audio, and video that have not been forged. Next, the model plays the second role and describes the forged data, focusing on identifying anomalies or changes compared to the original data. Then, in the third role, the model determines potential risk points in the data based on the forged annotation information (such as forged mask segmentation and voice timestamp marking), and identifies the areas and time periods that may be forged. Finally, in the summary role, the model conducts a comprehensive analysis of the information generated in the previous steps to generate a final summary data of risk points. This multi-step reasoning approach enables the model to gradually and deeply analyze various aspects of forged data, thereby more accurately identifying and describing potential risks in multimodal data, and providing more efficient support for risk identification and anti-fraud systems.

[0113] Finally, the risk summary output by the model not only contains the identification results of the forged content, but also describes in detail the potential risks that these forged data may bring. These summaries will serve as important inputs to the anti-fraud and risk identification system, helping to improve the overall defense capabilities of the system and effectively respond to diverse threat scenarios. Through this step, we combine forged data with multimodal large models to give full play to the role of synthetic data in risk analysis, significantly improving the ability to identify and respond to potential risks in complex multimodal scenarios.

[0114] Through this step, we combined the generated data with the multimodal large model, giving full play to the role of forged data in risk analysis, thereby significantly improving the ability to identify and respond to potential risks in complex multimodal scenarios.

[0115] Embodiment 3:

[0116] like Figure 6 , is a schematic diagram of the architecture of a multimodal risk data synthesis device for anti-fraud and risk identification according to an embodiment of the present invention. The multimodal risk data synthesis device for anti-fraud and risk identification according to this embodiment includes one or more processors 21 and a memory 22. Figure 6 A processor 21 is taken as an example.

[0117] The processor 21 and the memory 22 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0118] The memory 22 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs and non-volatile computer executable programs, such as the multimodal risk data synthesis method for anti-fraud and risk identification in Example 1. The processor 21 executes the multimodal risk data synthesis method for anti-fraud and risk identification by running the non-volatile software programs and instructions stored in the memory 22.

[0119] The memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 22 may optionally include a memory remotely arranged relative to the processor 21, and these remote memories may be connected to the processor 21 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0120] The program instructions / modules are stored in the memory 22, and when executed by the one or more processors 21, the multimodal risk data synthesis method for anti-fraud and risk identification in the above-mentioned embodiment 1 is executed.

[0121] It is worth noting that the information interaction, execution process, etc. between the modules and units within the above-mentioned devices and systems are based on the same concept as the processing method embodiment of the present invention. The specific contents can be found in the description of the method embodiment of the present invention and will not be repeated here.

[0122] A person skilled in the art may understand that all or part of the steps in the various methods of the embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0123] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multimodal risk data synthesis method for anti-fraud and risk identification, characterized in that: include: Obtaining a target face image, target voice data, and a video containing the face and voice as raw materials for multimodal data synthesis; In the acquired video or image, the original face is replaced with the target face through deep fake technology, ensuring that the posture and expression of the target face in the image or video are consistent with the original content; using audio synthesis technology, the original sound in the video is replaced with the target sound, so that the replaced sound is synchronized with the face movement, thereby generating multimodal data; wherein the target face is taken from the target face image, and the target sound is taken from the target sound data; The generated multimodal data is annotated, including the annotation of images and audio: the mask segmentation model is used to perform pixel-level segmentation of the forged face area to generate an accurate mask; the forged sound data is transcribed into text and the corresponding time period is annotated, marking the start and end positions of the forged audio; The forged and synthesized multimodal data and their annotations are input into the multimodal big model. The prompt word engineering and multi-step thinking chain technology are used to identify and analyze the input multimodal data to generate a risk summary containing risk points.

2. The multimodal risk data synthesis method for anti-fraud and risk identification according to claim 1 is characterized in that: The obtaining of the target face image, the target sound data and the video containing the face and the sound specifically includes: Get the target face data from the FFHQ dataset, get the target sound data from the MAGICDATA dataset, and get the video from the ChineseFaces dataset.

3. The multimodal risk data synthesis method for anti-fraud and risk identification according to claim 1 is characterized in that: The step of obtaining the target face image, target sound data, and a video containing the face and sound also includes: Preprocess the target face image, target sound data and video; wherein the preprocessing includes resizing and / or denoising.

4. The multimodal risk data synthesis method for anti-fraud and risk identification according to claim 1, characterized in that: In the obtained video or image, the original human face is replaced with the target human face by deep fake technology, specifically including: Perform frame extraction on each video to obtain the face image in each frame; Use face recognition technology to accurately locate the position of the face, randomly extract multiple different target face images from the face dataset, and use the face forgery model to replace the face in each frame; Among them, the face dataset is a dataset formed by multiple target face images, the face forgery model includes a DeepFaceLab model or an inswapper_128 model, and the inswapper_128 model includes one or more of a Roop model, a FaceSwapLab model and a Reactor model.

5. The multimodal risk data synthesis method for anti-fraud and risk identification according to claim 1 is characterized in that: The method of replacing the original sound in the video with the target sound by using the audio synthesis technology specifically includes: Separate the sound of the video into separate wav files to obtain the audio of the video; and perform vocal separation on these audios to extract clear voice clips; The audio of each video is sliced ​​and a plurality of different sound models are randomly selected for speech conversion; wherein the sound model is trained using an open source database or provided by an already trained open source model; a timestamp is randomly generated for each audio clip, and then the human voice is converted within the range of these timestamps.

6. The multimodal risk data synthesis method for anti-fraud and risk identification according to claim 1, characterized in that: The method of using the mask segmentation model to perform pixel-level segmentation on the forged face area to generate an accurate mask specifically includes: Use the Grounding DINO model to perform preliminary face recognition and locate the forged face. After identifying the forged face position, the SAM model is used to perform pixel-level segmentation on the area where the forged face position is located to generate a segmentation mask.

7. The multimodal risk data synthesis method for anti-fraud and risk identification according to claim 1 is characterized in that: The forged sound data is transcribed into text and the corresponding time period is marked to mark the starting position and the ending position of the forged audio, specifically including: The timestamp added during the process of replacing the original sound in the video with the target sound is used as the annotation information; The voice data is transcribed into text. For Chinese content, the Dharma ASR model is used for voice transcription. For voice transcription in other languages, the Whisper model is used for processing to ensure the transcription accuracy and consistency of multilingual voice data.

8. The multimodal risk data synthesis method for anti-fraud and risk identification according to claim 1, characterized in that: The step of inputting the forged synthesized multimodal data and its annotations into the multimodal macro model specifically includes: Input the annotated faces and audios in the forged and synthesized multimodal data into the multimodal large model; the video is processed by frame extraction, and the extracted frame images, corresponding forged images and forged labels are input into the model together; for the audio part, the text labels and timestamp labels are used as input; The multimodal large model processes images and audio simultaneously, and by combining information from different modalities, that is, analyzing the association between image and audio data through multiple iterations, it can understand and analyze forged content and generate a summary of risk points.

9. The multimodal risk data synthesis method for anti-fraud and risk identification according to claim 1, characterized in that: The use of prompt word engineering and multi-step thinking chain technology to identify and analyze the input multimodal data and generate a risk summary containing risk points, specifically including: Set up multiple roles for the reasoning process of multimodal large models; In the first role, the model generates a detailed description of the original data, including content analysis of images, audio, and video that have not been manipulated; In the second role, the model describes the falsified data and identifies anomalies or changes compared to the original data; In the third role, the model determines the potential risk points in the data based on the forged annotation information and identifies the forged areas and time periods; In the summary role, the model synthesizes and analyzes the information generated in the previous steps to generate a final risk summary.

10. The multimodal risk data synthesis method for anti-fraud and risk identification according to any one of claims 1 to 9, characterized in that: The multimodal large model is a closed source model or an open source model; wherein the closed source model includes one or more of GPT-4o, Claude3.5-Sonnet, and Gemini-1.5-Pro; the open source model is one or more of InternVL2-Llama3-76B and MiniCPM-V-2.6.

Images