Risk summary generation method based on multi-modal data

Abstract

This application discloses a risk summary generation method based on multimodal data, belonging to the field of secure content recognition technology. The method includes: acquiring multimodal data, the data type of which includes at least one of video image data, audio data, and text data; inputting the multimodal data into a risk identification and detection model to obtain multiple first semantic information and first detection results, the risk identification and detection model including a trained video image forgery detection sub-model, an audio forgery detection sub-model, a text forgery detection sub-model, an image-text summarization sub-model, and a speech transcription sub-model; associating the multiple first semantic information and the multiple first detection results to obtain second semantic information; and generating a risk summary based on the second semantic information and a risk knowledge base. This method improves the accuracy and effectiveness of risk content recognition and risk summary generation.

Description

Technical Field

[0001] This application belongs to the field of secure content recognition technology, and in particular relates to a risk summary generation method based on multimodal data. Background Technology

[0002] With the continuous development and popularization of generative artificial intelligence, deepfake tools and algorithms are becoming increasingly easy to obtain and use. The availability and popularity of deepfake technology further exacerbate the challenges to multimodal content security.

[0003] Traditional detection methods can only extract features from a single modality, resulting in incomplete and redundant information extraction, which affects the generalization ability of the model. In addition, most current research focuses on using only video image modalities for forgery detection, ignoring text modal information that is closely related to the image.

[0004] Related technologies for multimodal data identification are incomplete, failing to recognize multimodal data and establish correlations between modalities. They suffer from low accuracy in single-modal data identification, feature redundancy, insufficient generalization, inability to identify data generated using unknown methods, and high learning costs. Summary of the Invention

[0005] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a risk summary generation method based on multimodal data, which improves the accuracy and effectiveness of risk content identification and risk summary generation.

[0006] In a first aspect, this application provides a risk summary generation method based on multimodal data, the method comprising:

[0007] Acquire multimodal data, wherein the data type of the multimodal data includes at least one of video image data, audio data, and text data;

[0008] The multimodal data is input into the risk identification and detection model to obtain multiple first semantic information and first detection results. The risk identification and detection model includes a trained video image forgery detection sub-model, an audio forgery detection sub-model, a text forgery detection sub-model, an image and text summarization sub-model, and a speech transcription sub-model.

[0009] The first semantic information and the first detection results are correlated to obtain the second semantic information;

[0010] Based on the second semantic information and the risk knowledge base, a risk summary is generated.

[0011] According to one embodiment of this application, the step of inputting the multimodal data into a risk identification and detection model to obtain multiple first semantic information and first detection results includes:

[0012] Obtain the mapping relationship between the multimodal data and each sub-model, as follows:

[0013]

[0014] in, This represents the mapping relationship between multimodal data and sub-models. Represents a multimodal data set. This indicates how the sub-model processes multimodal data. This represents a string concatenation operation. Data type representing multimodal data;

[0015] Based on the mapping relationship, the corresponding sub-model is invoked to process the multimodal data to obtain the multiple first semantic information and the first detection result.

[0016] According to one embodiment of this application, the training process of the video image forgery detection sub-model includes:

[0017] A preset video image forgery detection sub-model is constructed, which includes a text encoder and an image encoder;

[0018] A cue vector is constructed based on the video image data, and the cue vector can be represented by the following formula:

[0019]

[0020] Each of them It is a learnable vector with the same dimensions as the category embedding. The hyperparameter for the number of learnable cues. The class tag in the prompt;

[0021] The cue vector is input into the text encoder to obtain a semantic feature vector, and the video image data is input into the image encoder to extract image features to obtain an embedding vector;

[0022] A video image dataset is obtained based on the embedding vector and the semantic feature vector;

[0023] Based on the first cross-entropy loss function, the preset video image forgery detection sub-model is trained on the video image dataset to obtain the video image forgery detection sub-model.

[0024] According to one embodiment of this application, the training process of the audio forgery detection sub-model includes:

[0025] A preset audio forgery detection sub-model is constructed, wherein the preset audio forgery detection sub-model is a time-frequency channel attention residual network model;

[0026] The spectral features of the audio data are extracted based on short-time Fourier transform, Mel frequency cepstral coefficients, and linear frequency cepstral coefficients, respectively, to obtain the first spectral feature, the second spectral feature, and the third spectral feature;

[0027] Obtain the audio dataset of the preset audio forgery detection sub-model, wherein the audio dataset is obtained by concatenating the first spectral feature, the second spectral feature and the third spectral feature;

[0028] The audio forgery detection sub-model is trained based on the second cross-entropy loss function and the audio dataset to obtain the audio forgery detection sub-model.

[0029] According to one embodiment of this application, the text forgery detection sub-model is an unsupervised training-classifier model, the image and text summarization sub-model is a pre-trained BLIP-2 model, and the speech transcription sub-model is a whisper-large-v2 model.

[0030] According to one embodiment of this application, the second semantic information obtained by associating the plurality of first semantic information and the plurality of first detection results can be obtained by the following formula:

[0031]

[0032] in For second semantic information, The first semantic information and the first detection result are the output of the i-th sub-model on the multimodal data.

[0033] According to one embodiment of this application, generating a risk summary based on the second semantic information and the risk knowledge base includes:

[0034] Based on the second semantic information and the prompt information, an analysis result is generated, and keywords are extracted from the analysis result.

[0035] Based on the acquired fraud case data, a risk knowledge base is constructed, and the fraud case data is segmented to obtain multiple risk data modules;

[0036] Based on the keywords and the multiple risk data modules, the contextual information of the risk summary is obtained;

[0037] The risk summary is generated based on the context information and the second semantic information.

[0038] Secondly, this application provides a risk summary generation apparatus based on multimodal data, the apparatus comprising:

[0039] The acquisition module is used to acquire multimodal data, wherein the data type of the multimodal data includes at least one of video image data, audio data, and text data;

[0040] The processing module is used to input the multimodal data into the risk identification and detection model to obtain the first semantic information. The risk identification and detection model includes a trained video image forgery detection sub-model, an audio forgery detection sub-model, a text forgery detection sub-model, an image and text summarization sub-model, and a speech transcription sub-model.

[0041] The association module is used to associate the plurality of first semantic information and the plurality of first detection results to obtain second semantic information;

[0042] The generation module is used to generate a risk summary based on the second semantic information and the risk knowledge base.

[0043] Thirdly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the risk summary generation method based on multimodal data as described in the first aspect above.

[0044] Fourthly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the risk summary generation method based on multimodal data as described in the first aspect above.

[0045] Fifthly, this application provides a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the risk summary generation method based on multimodal data as described in the first aspect.

[0046] In a sixth aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the risk summary generation method based on multimodal data as described in the first aspect above.

[0047] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application.

[0048] The risk summary generation method based on multimodal data provided by this invention has the following advantages over existing technologies:

[0049] (1) By inputting multimodal data into the risk identification and detection model, multiple first semantic information and first detection results are obtained. The risk identification and detection model includes a trained video image forgery detection sub-model, an audio forgery detection sub-model, a text forgery detection sub-model, a graphic summary sub-model, and a speech transcription sub-model. It can comprehensively identify multimodal data. By calling each sub-model to provide additional knowledge to the risk identification and detection model, the risk content identification and risk summary generation tasks are decomposed into multiple sub-tasks. Based on the idea of ​​the risk identification and detection model with sub-models (multi-model hybrid architecture), each sub-model is assigned a separate sub-task, which greatly reduces the learning cost of the risk identification and detection model. The semantic information between different modalities is analyzed for correlation. Finally, the risk knowledge base is combined with information retrieval to assist the risk identification and detection model in generating risk summaries, which improves the accuracy and generalization of risk identification and risk summary generation.

[0050] (2) By obtaining the mapping relationship between multimodal data and each sub-model, and based on the mapping relationship, calling the corresponding sub-model to process the multimodal data, multiple first semantic information and first detection results can be obtained, which can reduce the learning cost of the risk identification and detection model and improve the efficiency of risk identification and risk summary generation.

[0051] (3) Through the analysis results generated based on the second semantic information and prompt information, as well as the risk knowledge base constructed, risk data can be analyzed in detail and extracted accurately, which improves the processing efficiency of risk information. Furthermore, by separating and extracting keywords, potential risk factors can be accurately captured, providing richer contextual information for risk decision-making and prevention, thereby improving the accuracy and generalization of risk identification and risk summary generation. Attached Figure Description

[0052] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0053] Figure 1 This is one of the flowcharts illustrating the risk summary generation method based on multimodal data provided in this application embodiment;

[0054] Figure 2 This is the second flowchart illustrating the risk summary generation method based on multimodal data provided in this application embodiment;

[0055] Figure 3 This is a schematic diagram of the structure of the risk summary generation device based on multimodal data provided in the embodiments of this application;

[0056] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0057] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0058] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0059] The risk summary generation method, device, electronic device, and readable storage medium based on multimodal data provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0060] Among them, the risk summary generation method based on multimodal data can be applied to the terminal, specifically executed by the hardware or software in the terminal.

[0061] The terminal includes, but is not limited to, portable communication devices such as mobile phones or tablets with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that, in some embodiments, the terminal may not be a portable communication device, but rather a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).

[0062] The following embodiments describe a terminal including a display and a touch-sensitive surface. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, mouse, and joystick.

[0063] The risk summary generation method based on multimodal data provided in this application embodiment can be executed by an electronic device or a functional module or entity in an electronic device that can implement the risk summary generation method based on multimodal data. The electronic devices mentioned in this application embodiment include, but are not limited to, mobile phones, tablets, computers, cameras and wearable devices. The risk summary generation method based on multimodal data provided in this application embodiment will be described below using an electronic device as the execution subject.

[0064] With the rapid development and application of generative artificial intelligence technology, unprecedented possibilities have been brought to people's lives, entertainment, and artistic creation. However, this has also led to new security risks. Some criminals are using generative artificial intelligence technology to forge, tamper with, and synthesize audio, video, and text, posing a new challenge to social governance and cyberspace security. The relevant technical solutions have the following problems:

[0065] (1) Multimodal data recognition is incomplete, that is, the model recognition content is all single modal data, and it is unable to recognize multimodal data and make correlations between modalities.

[0066] The availability and widespread use of deepfake technology further exacerbates the challenges of multimodal content security. With the continuous development and popularization of generative artificial intelligence, deepfake tools and algorithms are becoming increasingly accessible and usable. This lowers the barrier to entry for using these tools to forge audio and video content, increasing the associated application security risks. The continuous development and innovation of deepfake technology also brings new security challenges. As technology advances, the quality and realism of forged audio and video are constantly improving, posing new threats to audio and video security detection. Simultaneously, with continuous technological innovation, the methods and means of forging audio and video are constantly evolving, making audio and video security face more complex and diverse challenges. However, existing content security technologies are proving inadequate in dealing with increasingly complex multimodal content risks. When faced with sophisticated fraud methods or forged information, traditional single-modal recognition methods often struggle to comprehensively capture all key information. For example, forged videos may deceive auditory and visual senses, with fraudulent content potentially containing various misleading textual information. These different modalities of information intertwine, forming multi-dimensional deception strategies, making detection and prevention extremely difficult, urgently requiring multimodal detection and analysis methods. Furthermore, current deepfake detection algorithms suffer from insufficient robustness and generalization, often proving inaccurate when detecting audio and video manipulated by unknown deepfake algorithms.

[0067] (2) Single-modal data recognition is inaccurate, features are redundant, generalization is insufficient, and it is unable to identify data forged by unknown generation methods.

[0068] Traditional methods for identifying risky content include classification-based, statistical, and deep learning-based approaches. However, these methods still have some drawbacks in identifying multimodal content. When faced with complex fraud schemes or forged information, traditional single-modal identification methods often struggle to capture all key information comprehensively. Furthermore, current popular deepfake detection algorithms suffer from insufficient robustness and generalization, often lacking accuracy when detecting audio and video manipulated by unknown deepfake algorithms.

[0069] (3) Large model learning costs are high. When a single large model completes the recognition task of full modality data, it requires a large amount of data and computing resources.

[0070] With the development of multimodal large models, their powerful data analysis capabilities have made risk identification methods based on multimodal large models possible. However, using multimodal large models for full-modal risk content identification requires a large amount of data and computing resources, which greatly increases the cost. Furthermore, under limited resources, the model may not achieve its optimal performance. To address the complexity, dynamism, and diversity of risk content, and moreover, most inventions in multimedia risk content identification and risk summary generation have not considered the cost of large model learning; that is, they use only a single large model to complete multimodal risk content identification and risk summary generation. This requires a more powerful large model, indirectly increasing the training and inference resources required for large models, thus increasing costs.

[0071] Figure 1 This is one of the flowcharts illustrating the risk summary generation method based on multimodal data provided in this application embodiment, such as... Figure 1 As shown, the risk summary generation method based on multimodal data includes steps 110, 120, 130 and 140.

[0072] Step 110: Obtain multimodal data, wherein the data type of the multimodal data includes at least one of video image data, audio data, and text data;

[0073] It is easy to understand that electronic devices can acquire multimodal data through video image acquisition devices, speech recognition devices, text acquisition devices, etc. The data types of multimodal data include at least one of video image data, audio data, and text data.

[0074] Optionally, video image data refers to images and video content obtained through camera equipment, monitoring equipment, or other image acquisition equipment. Video image data includes rich visual information, involving changes in the scene, the movement of objects, and the behavior of people.

[0075] For example, video image data consists of continuous image frames, which can be stored using a video encoding format. Video image data can be used to extract image features of moving objects, identify forgery or tampering in videos, or perform scene analysis based on image content.

[0076] Optionally, audio data is sound information acquired through audio acquisition devices such as microphones and voice recognition devices. Audio data includes dialogue, background noise, music, and other environmental sounds. Audio data is usually stored and processed in the form of waveforms or spectra to analyze speech content, audio signal quality, forgery elements in the sound (such as audio editing, voice changing, etc.), and to identify specific voice commands or danger signals.

[0077] For example, audio data can help determine changes in tone of voice during a call, whether there is voice spoofing, and potential risks in the audio, such as malicious voice commands or information leaks.

[0078] Optionally, text data refers to information in written form, which can originate from news reports, social media, emails, forum posts, documents, etc. Text data can be analyzed and processed using natural language processing techniques to extract key information, sentiment, themes, or potential risk information, thereby identifying misinformation, fraud, or other potentially risky content online.

[0079] Step 120: Input the multimodal data into the risk identification and detection model to obtain multiple first semantic information and first detection results. The risk identification and detection model includes a trained video image forgery detection sub-model, an audio forgery detection sub-model, a text forgery detection sub-model, an image and text summarization sub-model, and a speech transcription sub-model.

[0080] Furthermore, a risk identification and detection model is constructed, which includes sub-models for video image forgery detection, audio forgery detection, text forgery detection, image-text summarization, and speech transcription. The risk identification and detection model can be represented in the following form:

[0081]

[0082] in, For risk identification and detection models, For video image forgery detection sub-model, For audio forgery detection sub-model, This is a sub-model for text forgery detection. For the image and text summarization sub-model, This is a speech transcription sub-model.

[0083] Optionally, the risk identification and detection model can be a pre-trained Transformer neural network model with more than 1 billion weight parameters.

[0084] It should be noted that in the process of building the risk identification and detection model, it is first necessary to train the video image forgery detection sub-model, audio forgery detection sub-model, text forgery detection sub-model, image and text summarization sub-model, and speech transcription sub-model.

[0085] Optionally, for the video image forgery detection sub-model, the video image data synthesized by the Stable Diffusion (SD) model integrates multimodal information of images and text. The video image forgery detection sub-model can be constructed using a cross-modal cue fine-tuning method, which can achieve the correlation and fusion of cross-modal information.

[0086] Optionally, for the audio forgery detection sub-model, since deepfake audio data contains only a small number of artifacts (deepfake features) in most frames and also contains artifacts in the non-resonant frequency domain, a time-frequency channel attention residual network can be used to construct the audio forgery detection sub-model, which can highlight audio key frame information and extract non-resonant frequency domain information.

[0087] Optionally, for the text forgery detection sub-model, the pre-trained text feature extraction model BERT can be used to extract text features, or other machine learning models can be used for text feature extraction; this application does not impose any restrictions.

[0088] The electronic device inputs the acquired multimodal data into the risk identification and detection model. The risk identification and detection model processes data of different data types into corresponding sub-models according to the data type of the multimodal data, and obtains multiple first semantic information and first detection results.

[0089] For example, the multimodal data includes video image data, audio data, and text data. The video image data can be input into the video image forgery detection sub-model and the image-text summarization sub-model to obtain two first semantic information and two first detection results. The audio data can be input into the audio forgery detection sub-model and the speech transcription sub-model to obtain two first semantic information and two first detection results. The text data can be input into the text forgery detection sub-model to obtain one first semantic information and one first detection result. The multimodal data input into the risk identification detection model yields a total of five first semantic information and five first detection results.

[0090] Step 130: Associate the multiple first semantic information and the multiple first detection results to obtain second semantic information;

[0091] Furthermore, semantic information between different data modalities can be correlated by designing thought chains and small sample prompts, automatically associating multiple first semantic information and multiple first detection results, and combining multiple first detection results and multiple first semantic information to obtain second semantic information.

[0092] Step 140: Generate a risk summary based on the second semantic information and the risk knowledge base.

[0093] Finally, based on the second semantic information and the information retrieved from the risk knowledge base, a complete risk summary is generated. The risk knowledge base is a knowledge base system built based on existing risk data.

[0094] According to the risk summary generation method based on multimodal data provided in this application, by inputting multimodal data into a risk identification and detection model, multiple first semantic information and first detection results are obtained. The risk identification and detection model includes a trained video image forgery detection sub-model, an audio forgery detection sub-model, a text forgery detection sub-model, an image and text summarization sub-model, and a speech transcription sub-model, which can comprehensively identify multimodal data. By calling each sub-model to provide additional knowledge to the risk identification and detection model, the risk content identification and risk summary generation tasks are decomposed into multiple sub-tasks. Based on the idea of ​​the risk identification and detection model with sub-models (multi-model hybrid architecture), separate sub-tasks are assigned to each sub-model, which greatly reduces the learning cost of the risk identification and detection model. Furthermore, the semantic information between different modalities is analyzed for correlation. Finally, information retrieved from the risk knowledge base is combined to assist the risk identification and detection model in generating risk summaries, thereby improving the accuracy and generalization of risk identification and risk summary generation.

[0095] In some embodiments, inputting the multimodal data into a risk identification and detection model to obtain multiple first semantic information and first detection results includes:

[0096] Obtain the mapping relationship between the multimodal data and each sub-model, as follows:

[0097]

[0098] in, This represents the mapping relationship between multimodal data and sub-models. Represents a multimodal data set. This indicates how the sub-model processes multimodal data. This represents a string concatenation operation. Data type representing multimodal data;

[0099] Based on the mapping relationship, the corresponding sub-model is invoked to process the multimodal data to obtain the multiple first semantic information and the first detection result.

[0100] It is easy to understand that, since multimodal data has different data types, different data types require different processing methods. By constructing a mapping function between data types and each sub-model, and representing the mapping relationship semantically, the risk identification and detection model can understand the function of each sub-model and the correspondence between them and the data types. This enables the risk identification and detection model to automatically select and call the corresponding sub-model to process the corresponding type of data based on the input multimodal data type.

[0101] The mapping relationship between multimodal data and each sub-model can be constructed in the following form:

[0102]

[0103] in, This represents the mapping relationship between multimodal data and sub-models. Represents a multimodal data set. This indicates how the sub-model processes multimodal data. This represents a string concatenation operation. This represents a data type for multimodal data.

[0104] For example, when the data type of multimodal data is video image data, the video image data can be input into the video image forgery detection sub-model and the image and text summarization sub-model, and a mapping relationship between the video image data and the video image forgery detection sub-model and the image and text summarization sub-model can be constructed.

[0105] For example, when the data type of multimodal data is audio data, the audio data can be input into the audio forgery detection sub-model and the speech transcription sub-model, and a mapping relationship between the audio data and the audio forgery detection sub-model and the speech transcription sub-model can be constructed.

[0106] For example, when the data type of multimodal data is text data, the text data can be input into the text forgery detection sub-model, and a mapping relationship between the text data and the text forgery detection sub-model can be constructed.

[0107] In this embodiment, by obtaining the mapping relationship between multimodal data and each sub-model, and based on the mapping relationship, calling the corresponding sub-model to process the multimodal data, multiple first semantic information and first detection results can be obtained, which can reduce the learning cost of the risk identification and detection model and improve the efficiency of risk identification and risk summary generation.

[0108] In some embodiments, the training process of the video image forgery detection sub-model includes:

[0109] A preset video image forgery detection sub-model is constructed, which includes a text encoder and an image encoder;

[0110] A cue vector is constructed based on the video image data, and the cue vector can be represented by the following formula:

[0111]

[0112] Each of them It is a learnable vector with the same dimensions as the category embedding. The hyperparameter for the number of learnable cues. The class tag in the prompt;

[0113] The cue vector is input into the text encoder to obtain a semantic feature vector, and the video image data is input into the image encoder to extract image features to obtain an embedding vector;

[0114] A video image dataset is obtained based on the embedding vector and the semantic feature vector;

[0115] Based on the first cross-entropy loss function, the preset video image forgery detection sub-model is trained on the video image dataset to obtain the video image forgery detection sub-model.

[0116] The process is straightforward: first, a pre-defined video image forgery detection sub-model is constructed. This sub-model can be a fine-tuned contrastive language-image pre-trained model, including a text encoder and an image encoder. The text prompt vector input from the text encoder and sentences describing categories relevant to deep forgery detection of video images are compared with image features generated by the image encoder to complete the downstream task of deep forgery detection. The specific training process of the pre-defined video image forgery detection sub-model is as follows:

[0117] (1) A learnable text prompt vector in a continuous space is constructed based on video image data. The vector is learned end-to-end from the data. The parameters of the text encoder and the image encoder are kept frozen. The learnable prompt design formula input into the text encoder is as follows:

[0118]

[0119] Each of them It is a learnable vector with the same dimensions as the category embedding. (For example, 16) is a hyperparameter representing the number of learnable cues. The class tag in the prompt.

[0120] For example, The class labels in the prompt are represented as true and false, respectively.

[0121] (2) Learnable prompts The input is fed into a text encoder to obtain a semantic feature vector that embeds the visual concept of the category. Then, an image encoder extracts image features. By comparing the image with the semantic features, inference and classification are performed. The category prediction probability is calculated as shown in the following formula:

[0122]

[0123] in, Indicates category, This represents the input image. Indicates a prompt Embedded vectors generated by the text encoder The class markers in the text are replaced with word embedding vectors corresponding to the class names. Representing an image Embedding vectors generated by the image encoder =2 represents the number of categories.

[0124] (3) During the training of the learnable cue vectors, the vectors are trained based on cross-entropy to minimize the standard classification loss, and the loss is calculated as shown in the following formula:

[0125]

[0126] in For the true class probability, To predict the probability of the category, the gradient calculated from the loss can be backpropagated through the text encoder. This leverages the rich knowledge within the text encoder to optimize the cue vector, thereby achieving adaptation for downstream tasks such as deepfake detection in video images.

[0127] Finally, the video image dataset is obtained based on the embedding vector and semantic feature vector. Based on the above loss function, the preset video image forgery detection sub-model is trained on the video image dataset to obtain the video image forgery detection sub-model.

[0128] In this embodiment, different types of data are correlated and fused through a text encoder and an image encoder. Based on the first cross-entropy loss function, a preset video image forgery detection sub-model is trained on the video image dataset to obtain a video image forgery detection sub-model. This achieves cross-modal information correlation and fusion, which can extract more comprehensive data features, reduce redundant information, and improve the generalization and accuracy of the video image forgery detection sub-model.

[0129] In some embodiments, the training process of the audio forgery detection sub-model includes:

[0130] A preset audio forgery detection sub-model is constructed, wherein the preset audio forgery detection sub-model is a time-frequency channel attention residual network model;

[0131] The spectral features of the audio data are extracted based on short-time Fourier transform, Mel frequency cepstral coefficients, and linear frequency cepstral coefficients, respectively, to obtain the first spectral feature, the second spectral feature, and the third spectral feature;

[0132] Obtain the audio dataset of the preset audio forgery detection sub-model, wherein the audio dataset is obtained by concatenating the first spectral feature, the second spectral feature and the third spectral feature;

[0133] The audio forgery detection sub-model is trained based on the second cross-entropy loss function and the audio dataset to obtain the audio forgery detection sub-model.

[0134] It is easy to understand that, firstly, a preset audio forgery detection sub-model is constructed, and a time-frequency channel attention residual network is designed. The spectral features of the audio data are extracted based on short-time Fourier transform, Mel frequency cepstral coefficients and linear frequency cepstral coefficients, respectively, to obtain the first spectral feature, the second spectral feature and the third spectral feature, which comprehensively capture the features of the audio signal. Finally, the three features are concatenated in the channel dimension and used as the model input to train the preset audio forgery detection sub-model.

[0135] The training process for the preset audio forgery detection sub-model is as follows:

[0136] (1) Feature extraction of Short-Time Fourier Transform, Mel-frequency cepstral coefficients, and linear frequency cepstral system. The process of extracting Short-Time Fourier Transform (STFT) features is as follows: For a given audio signal, the signal is first framed and windowed to divide the continuous signal into overlapping frames. Each frame contains a fixed number of sample points to ensure data continuity and reduce edge effects.

[0137] To reduce the impact of discontinuities at the edges of each frame on the transformation result, each frame's data is multiplied by a window function, calculated as shown in the following formula:

[0138]

[0139] in, It is the window length. It is the sample index in the window.

[0140] Then, a Fast Fourier Transform is performed on the windowed signal for each frame to transform the signal from the time domain to the frequency domain. The calculation formula is shown below:

[0141]

[0142] in, It is the first The signal value at each sampling point It is a window function. It is the number of points in the Fast Fourier Transform. It is a frequency index.

[0143] Finally, the modulus of the Fast Fourier Transform output is taken to calculate the amplitude of each frequency component. The calculation formula is shown below:

[0144]

[0145] in, and These are the real and imaginary parts of the Fast Fourier Transform result, respectively. The process is repeated for all frames of the entire signal to obtain a two-dimensional time-frequency representation, which serves as the feature of the Short Time Fourier Transform.

[0146] The extraction process of Mel Frequency Cepstral Coefficient (MFCC) features is as follows: For a given audio signal, the Fast Fourier Transform (FFT) result is first passed through a set of Mel-scale triangular filters. The calculation of filtering the audio using all Mel-scale triangular filters is shown in the following formula:

[0147]

[0148] in, It is the amplitude of the Fast Fourier Transform. It is the first One filter.

[0149] Then, the energy output of each filter is calculated, and the logarithm is taken to obtain the Mel frequency cepstral coefficient characteristics.

[0150] Linear Frequency Cepstral Coefficients (LFCC) characteristics are similar to MFCC, but LFCC uses linear frequency scale filters. These filters are evenly spaced to capture the characteristics of synthesized audio at non-resonant frequencies. As shown in the following formula:

[0151]

[0152] in, It is the first The frequency response of each filter.

[0153] It should be noted that the above three types of spectra can reflect audio characteristics from different perspectives. The short-time Fourier transform converts the time-domain signal into a frequency-domain signal. Its real and imaginary parts reflect the amplitude and phase information of the signal in the frequency domain, respectively. This helps the model capture the inconsistencies in the local phase of the synthesized audio and can reveal the instantaneous spectral information of the audio signal, which is helpful in detecting short-term anomalies in the synthesized audio. The Mel spectrum is based on the auditory characteristics of the human ear. Through Mel filter banks and logarithmic energy calculation, it simulates the human ear's perception of different frequencies. It can capture the characteristics of non-resonant frequencies in the human voice frequency range, such as pronunciation and pauses between words. It has a significant effect on the detection of synthesized audio in the human voice frequency range. The linear spectrum is similar to Mel, but uses a linear frequency scale filter, which can directly reflect the energy distribution of the sound signal at different frequencies.

[0154] (2) The short-time Fourier transform features, Mel-frequency cepstral coefficient features, and linear-frequency cepstral coefficient features are concatenated along the channel dimension and used as the model input. The feature map is first passed through a residual block with a 3×3 pooling layer to adjust the size of the input feature map. After that, two core residual blocks are designed, one for compressing the dimension of the feature map, called Compress Residual. The calculation process of this module is shown in the following formula:

[0155]

[0156] in, This indicates that the input has a number of channels. The size is The tensor. Indicates the number of output channels is Convolution calculation, This represents Batch Normalization.

[0157] Another module used to extract more diverse feature maps through convolution operations is called Deep Residual. Its computation process is shown in the following formula:

[0158]

[0159] in, This indicates that the input has a number of channels. The size is The tensor. Indicates the number of output channels is Convolution calculation, This represents Batch Normalization.

[0160] Both Compress Residual and Deep Residual activation functions can alleviate gradient vanishing in deep neural connections and are used alternately in the network. Furthermore, since the Mish activation function is continuously differentiable across the entire real domain and possesses smooth nonlinear characteristics, it is used within residual blocks to ensure the stability of gradient calculation. The Mish activation function is shown in the following formula:

[0161]

[0162] (3) Calculate different attention masks for the channel, temporal, and frequency correlations of the feature maps, and calculate the comprehensive frame-level attention score accordingly. First, adaptive average pooling applied to the channel dimension compresses the feature map to one dimension, and the attention level of the network for each feature map is calculated using MLP and normalized to the (0,1) interval to obtain the network's attention mask for the feature map channels. The calculation formula is shown below:

[0163]

[0164]

[0165] in, It is the input to the attention module. Indicates to Adaptive pooling, It is the attention weight of each channel.

[0166] When calculating the attention for temporal correlation of the feature map, the channel attention output is used as input. A convolution operation compresses the feature map dimension to two dimensions, yielding the degree of attention to each frequency's sound features at different time steps. This is then normalized to the (0,1) interval to obtain the network's attention mask for the temporal-frequency correlation of the feature map. The calculation formula is as follows:

[0167]

[0168]

[0169] in, This represents the calculation of the maximum and average matrices of the eigenmap. Indicates to convolution, The output representing channel attention, This represents the attention weights in the time and frequency domains.

[0170] (4) During the training phase of the preset audio forgery detection sub-model, a method with... The cross-entropy loss of the regularization term is used as the loss function, and the parameters are optimized using the gradient descent algorithm, as shown in the following formula:

[0171]

[0172] in For the true class probability, To predict the probability of a category, For network parameters, To balance the hyperparameters.

[0173] Finally, the audio dataset for the preset audio forgery detection sub-model is obtained by concatenating the first, second, and third spectral features. Based on the above loss function, the preset audio forgery detection sub-model is trained on the audio dataset to obtain the audio forgery detection sub-model.

[0174] In this embodiment, the spectral features of the audio data are extracted by using short-time Fourier transform, Mel frequency cepstral coefficients, and linear frequency cepstral coefficients to obtain first spectral features, second spectral features, and third spectral features. This allows for comprehensive feature extraction, enriches feature representation, and reduces information redundancy. The use of a time-frequency channel attention residual network model can highlight key frame information and non-resonant frequency domain information extraction, thereby improving the accuracy and generalization of the audio forgery detection sub-model.

[0175] In some embodiments, the text forgery detection sub-model is an unsupervised training-classifier model, the image and text summarization sub-model is a pre-trained BLIP-2 model, and the speech transcription sub-model is a whisper-large-v2 model.

[0176] It's easy to understand that the text forgery detection sub-model can adopt an unsupervised training model + classifier model structure. The unsupervised pre-trained model uses the BERT encoder model, and the classifier model accurately classifies the output features of the pre-trained BERT encoder model.

[0177] The image-text summarization model uses a pre-trained BLIP-2 model for semantic recognition of images, that is, to describe the content of images in text (image-to-text).

[0178] The speech transcription model uses the whisper-large-v2 model for semantic recognition of speech, that is, to describe the audio content into text (speech-to-text), and combines speculative decoding technology based on the whisper-tiny model to accelerate inference.

[0179] In this embodiment, the accuracy and generalization of the text forgery detection sub-model can be improved by using an unsupervised training-classifier model. The pre-trained BLIP-2 model and whisper-large-v2 model can further process video image data and voice data, making feature extraction more comprehensive, reducing redundant information, and improving the accuracy and generalization of risk identification and risk summary generation.

[0180] In some embodiments, the association between the plurality of first semantic information and the plurality of first detection results to obtain second semantic information can be obtained by the following formula:

[0181]

[0182] in For second semantic information, The first semantic information and the first detection result are the output of the i-th sub-model on the multimodal data.

[0183] Furthermore, the risk identification and detection model semantically associates multiple first semantic information and multiple first detection results, integrating data from different modalities such as video images, text, and audio to extract and associate their semantic information, thus obtaining second semantic information. This second semantic information can be constructed in the following form:

[0184]

[0185] in For second semantic information, The first semantic information and the first detection result are the output of the i-th sub-model on the multimodal data.

[0186] In this embodiment, by associating multiple first semantic information and multiple first detection results, second semantic information is obtained, which enables the fusion of data from different modalities to form more comprehensive and accurate second semantic information, thereby improving the accuracy and effectiveness of risk identification and risk summary generation.

[0187] In some embodiments, generating a risk summary based on the second semantic information and the risk knowledge base includes:

[0188] Based on the second semantic information and the prompt information, an analysis result is generated, and keywords are extracted from the analysis result.

[0189] Based on the acquired fraud case data, a risk knowledge base is constructed, and the fraud case data is segmented to obtain multiple risk data modules;

[0190] Based on the keywords and the multiple risk data modules, the contextual information of the risk summary is obtained;

[0191] The risk summary is generated based on the context information and the second semantic information.

[0192] It is easy to understand that after acquiring the second semantic information, the risk identification and detection model will conduct a comprehensive analysis. By designing content risk analysis prompts, it will comprehensively analyze the second semantic information and generate analysis results. The generation process is as follows:

[0193]

[0194] in The analysis results include the data identification results and a description of the data. This represents a risk identification and detection model. These are prompt words indicating that semantic synthesis analysis is being performed.

[0195] Meanwhile, electronic devices acquire existing fraud case data to build a risk knowledge base. The fraud case data covers a variety of fraud and risk event data, which serves as a knowledge supplement for the risk identification and detection model to generate risk summaries. The collected fraud case data is preprocessed, including cleaning, standardization and word segmentation, and the processed text data is divided into smaller, more manageable blocks to facilitate subsequent parallel processing.

[0196] Furthermore, the risk identification and detection model further analyzes the obtained results, separates relevant instructions and information from the results, and derives the keyword set in the query. The separation process is shown in the following formula:

[0197]

[0198] in This indicates the extracted instructions. This indicates the extracted keywords. R represents the prompt words for performing instructions and keyword extraction, and R represents the analysis results.

[0199] After obtaining the keywords, in order to improve subsequent retrieval efficiency and reduce the omission of highly relevant content, each processed text block in the risk knowledge base is submitted in parallel to the risk identification and detection model. The risk identification and detection model evaluates the relevance of each block to the analysis results and filters out irrelevant information. The filtering process is as follows:

[0200]

[0201] in This indicates the correlation between each text block and the analysis results. Prompt words indicating that a correlation analysis is being performed. R represents a single text block, and R represents the analysis result.

[0202] Furthermore, the BM25 retrieval algorithm can be used to search for text blocks containing highly relevant keywords, thereby locating the most relevant text blocks as context for subsequent risk summaries. The process of obtaining the context is as follows:

[0203]

[0204] in This represents a search function that searches based on keywords. In the relevant text block The most relevant text blocks are retrieved from the database.

[0205] Finally, by utilizing the contextual information retrieved from the risk knowledge base and combining it with the analysis results, the risk identification and detection model's ability to identify content risk patterns is further enhanced, generating the final risk summary. The generation process is shown below:

[0206]

[0207] in These are prompts that guide the risk identification and detection model in generating summaries, including definitions of the model's role, objectives, and the generated template. Indicates context.

[0208] In this embodiment, the contextual information of the risk summary is obtained by using keywords derived from the analysis results and multiple risk data modules obtained from the risk knowledge base. This enables in-depth analysis of risk information at different levels and dimensions, improving data processing efficiency. The final risk summary is generated based on the contextual information and second semantic information, which can effectively summarize key information and automatically extract key information to generate the risk summary, thereby improving the accuracy and effectiveness of risk identification and risk summary generation.

[0209] Figure 2 This is the second flowchart illustrating the risk summary generation method based on multimodal data provided in this application embodiment, as shown below. Figure 2 As shown, after obtaining the multimodal data, the multimodal data is input into the risk identification and detection model. The risk identification and detection model calls the sub-model corresponding to the multimodal data based on the thought chain prompt words to obtain multiple first semantic information and multiple first detection results. Then, the risk identification and detection model associates the multiple first semantic information and multiple first detection results through semantic association analysis to obtain second semantic information. Finally, based on the second semantic information and the relevant context retrieved from the risk knowledge base, a risk summary is generated.

[0210] The risk summary generation method based on multimodal data provided in this application can be executed by a risk summary generation device based on multimodal data. This application uses an example of a risk summary generation device based on multimodal data executing the risk summary generation method based on multimodal data to illustrate the risk summary generation device based on multimodal data provided in this application.

[0211] This application also provides a risk summary generation device based on multimodal data, such as... Figure 3 As shown, the risk summary generation device based on multimodal data includes: an acquisition module 310, a processing module 320, an association module 330, and a generation module 340.

[0212] The acquisition module 310 is used to acquire multimodal data, wherein the data type of the multimodal data includes at least one of video image data, audio data, and text data;

[0213] Processing module 320 is used to input the multimodal data into the risk identification and detection model to obtain multiple first semantic information and multiple first detection results. The risk identification and detection model includes a trained video image forgery detection sub-model, an audio forgery detection sub-model, a text forgery detection sub-model, an image and text summarization sub-model, and a speech transcription sub-model.

[0214] The association module 330 is used to associate the plurality of first semantic information and the plurality of first detection results to obtain second semantic information;

[0215] The generation module 340 is used to generate a risk summary based on the second semantic information and the risk knowledge base.

[0216] According to the risk summary generation device based on multimodal data provided in the embodiments of this application, by inputting multimodal data into a risk identification and detection model, multiple first semantic information and first detection results are obtained. The risk identification and detection model includes a trained video image forgery detection sub-model, an audio forgery detection sub-model, a text forgery detection sub-model, an image and text summarization sub-model, and a speech transcription sub-model. It can comprehensively identify multimodal data. By calling each sub-model to provide additional knowledge to the risk identification and detection model, the risk content identification and risk summary generation tasks are decomposed into multiple sub-tasks. Based on the idea of ​​the risk identification and detection model with sub-models (multi-model hybrid architecture), separate sub-tasks are assigned to each sub-model, which greatly reduces the learning cost of the risk identification and detection model. Furthermore, the semantic information between different modalities is analyzed for correlation. Finally, information retrieved from the risk knowledge base is combined to assist the risk identification and detection model in generating risk summaries, thereby improving the accuracy and generalization of risk identification and risk summary generation.

[0217] The risk summary generation device based on multimodal data provided in this application embodiment can achieve... Figures 1 to 2 The various processes implemented in the risk summary generation method based on multimodal data will not be described again here to avoid repetition.

[0218] In some embodiments, such as Figure 4 As shown, this application embodiment also provides an electronic device 400, including a processor 401, a memory 402, and a computer program stored in the memory 402 and executable on the processor 401. When the program is executed by the processor 401, it implements the various processes of the above-described risk summary generation method embodiment based on multimodal data and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0219] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0220] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described risk summary generation method embodiment based on multimodal data and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0221] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0222] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described risk summary generation method based on multimodal data.

[0223] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0224] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described risk summary generation method embodiments based on multimodal data, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0225] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0226] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0227] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the risk summary generation method based on multimodal data of the various embodiments of this application.

[0228] In the description of this application, "first feature" and "second feature" may include one or more of the features.

[0229] In the description of this application, "multiple" means two or more.

[0230] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0231] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0232] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for risk summary generation based on multi-modal data, characterized in that, The method includes: Acquire multimodal data, wherein the data type of the multimodal data includes at least one of video image data, audio data, and text data; The multimodal data is input into the risk identification and detection model to obtain multiple first semantic information and first detection results. The risk identification and detection model includes a trained video image forgery detection sub-model, an audio forgery detection sub-model, a text forgery detection sub-model, an image and text summarization sub-model, and a speech transcription sub-model. The first semantic information and the first detection results are correlated to obtain the second semantic information; Based on the second semantic information and the risk knowledge base, a risk summary is generated; The step of generating a risk summary based on the second semantic information and the risk knowledge base includes: Based on the second semantic information and the prompt information, by designing content risk analysis prompt words, analysis results are generated, and keywords are extracted from the analysis results; Based on the acquired fraud case data, a risk knowledge base is constructed, and the fraud case data is segmented to obtain multiple risk data modules; Based on the keywords and the multiple risk data modules, the contextual information of the risk summary is obtained; The risk summary is generated based on the context information and the second semantic information.

2. The method for risk summary generation based on multi-modal data according to claim 1, characterized in that, The process involves inputting the multimodal data into a risk identification and detection model to obtain multiple first semantic information items and first detection results, including: Obtain the mapping relationship between the multimodal data and each sub-model, as follows: ； in, This represents the mapping relationship between multimodal data and sub-models. Represents a multimodal data set. This indicates how the sub-model processes multimodal data. This represents a string concatenation operation. Data type representing multimodal data; Based on the mapping relationship, the corresponding sub-model is invoked to process the multimodal data to obtain the multiple first semantic information and the first detection result.

3. The risk summary generation method based on multimodal data according to claim 1, characterized in that, The training process of the video image forgery detection sub-model includes: A preset video image forgery detection sub-model is constructed, which includes a text encoder and an image encoder; A cue vector is constructed based on the video image data, and the cue vector can be represented by the following formula: ； Each of them It is a learnable vector with the same dimensions as the category embedding. The hyperparameter for the number of learnable cues. The class tag in the prompt; The cue vector is input into the text encoder to obtain a semantic feature vector, and the video image data is input into the image encoder to extract image features to obtain an embedding vector; A video image dataset is obtained based on the embedding vector and the semantic feature vector; Based on the first cross-entropy loss function, the preset video image forgery detection sub-model is trained on the video image dataset to obtain the video image forgery detection sub-model.

4. The risk summary generation method based on multimodal data according to claim 1, characterized in that, The training process of the audio forgery detection sub-model includes: A preset audio forgery detection sub-model is constructed, wherein the preset audio forgery detection sub-model is a time-frequency channel attention residual network model; The spectral features of the audio data are extracted based on short-time Fourier transform, Mel frequency cepstral coefficients, and linear frequency cepstral coefficients, respectively, to obtain the first spectral feature, the second spectral feature, and the third spectral feature; Obtain the audio dataset of the preset audio forgery detection sub-model, wherein the audio dataset is obtained by concatenating the first spectral feature, the second spectral feature and the third spectral feature; The audio forgery detection sub-model is trained based on the second cross-entropy loss function and the audio dataset to obtain the audio forgery detection sub-model.

5. The risk summary generation method based on multimodal data according to claim 1, characterized in that, The text forgery detection sub-model is an unsupervised training-classifier model, the image and text summarization sub-model is a pre-trained BLIP-2 model, and the speech transcription sub-model is a whisper-large-v2 model.

6. The risk summary generation method based on multimodal data according to claim 1, characterized in that, The second semantic information obtained by associating the plurality of first semantic information and the plurality of first detection results can be obtained by the following formula: ； in For second semantic information, The first semantic information and the first detection result are the output of the i-th sub-model on the multimodal data.

7. A risk summary generation device based on multimodal data, implemented using the risk summary generation method based on multimodal data as described in claim 1, characterized in that, The device includes: The acquisition module is used to acquire multimodal data, wherein the data type of the multimodal data includes at least one of video image data, audio data, and text data; The processing module is used to input the multimodal data into the risk identification and detection model to obtain multiple first semantic information and first detection results. The risk identification and detection model includes a trained video image forgery detection sub-model, an audio forgery detection sub-model, a text forgery detection sub-model, an image and text summarization sub-model, and a speech transcription sub-model. The association module is used to associate the plurality of first semantic information and the plurality of first detection results to obtain second semantic information; The generation module is used to generate a risk summary based on the second semantic information and the risk knowledge base.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the risk summary generation method based on multimodal data as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the risk summary generation method based on multimodal data as described in any one of claims 1 to 6.

Images