A financial fraud dialogue summary and keyword joint generation method based on multi-modal fusion

Abstract

The application relates to the technical field of natural language processing, in particular to a financial fraud dialogue summary and keyword joint generation method based on multi-modal fusion. Chat records in a mobile phone of a criminal suspect are used, a text summary and keyword joint generation model of information in a financial fraud field is fused, and a dialogue summary and keywords of the chat records of the criminal suspect are obtained. The method has strong generalization ability, solves the problems that existing methods have single data type, are difficult to process short text of chat records and financial fraud colloquial information, and the like; all-around collection and arrangement of the chat records of the criminal suspect are realized, a contact person engaged in criminal activities is studied as a focus by a case handling personnel, key dialogue information of a criminal accomplice is extracted, key evidence is located, and criminal facts are mastered; the application prospect is wide, and the method can be applied to the need for financial fraud dialogue summary and keyword generation in various real scenes, reduces labor cost, and improves work efficiency.

Description

Technical Field

[0001] This application belongs to the field of natural language processing technology, and more specifically, it relates to a method for jointly generating financial fraud dialogue summaries and keywords based on multimodal fusion. Background Technology

[0002] When the police investigate the criminal activities of suspects, the sheer volume of data on their mobile phones, including information such as companies, locations, organizations, individuals, contact information, and amounts involved, is overwhelming and difficult to retrieve manually. Furthermore, the data on the suspects' phones contains various types of information, including text messages, images, videos, and voice recordings, which significantly complicates the investigation.

[0003] There are some precedents for automatic summarization methods related to finance. Existing methods involve inputting financial text and images, building an encoder-decoder model, and finally generating a financial text summary model. However, such methods have the following drawbacks: First, the data types they can handle are relatively limited, processing only text or image data. In reality, the data on a suspect's phone contains multiple types, such as text, images, videos, and voice recordings, making it impossible to obtain all the information comprehensively, resulting in a lack of crucial evidence. Second, these methods primarily handle long text information such as financial articles, with limited application and less effective use for short text information in financial fraud chat logs. They fail to address issues such as short text, wide topic ranges, incoherent semantic information, and excessive irrelevant information in chat logs. Third, these methods fail to consider relevant knowledge in the field of financial fraud, making it difficult to identify some colloquial language used by suspects. Summary of the Invention

[0004] To achieve the above objectives, the technical solution adopted in this application is: to provide a method for jointly generating financial fraud dialogue summaries and keywords based on multimodal fusion, which utilizes chat records on the suspect's mobile phone and obtains dialogue summaries and keywords from the suspect's chat records by combining text summaries and keywords that integrate information in the field of financial fraud through a joint generation model.

[0005] Optionally, the following steps are included:

[0006] Step 1: Extract and fuse features from various modalities using a multimodal model to obtain a multimodal feature vector with dialogue temporal information;

[0007] Step 2: Train and generate an ERNIE model that incorporates knowledge of the financial fraud domain. Input the multimodal feature vector with dialogue temporal information into the ERNIE model that incorporates knowledge of the financial fraud domain to obtain the latent feature vector.

[0008] Step 3: Input the latent feature vector into the pointer-dual generator network, which then jointly generates dialogue summaries and keywords.

[0009] Optionally, a multimodal model can be used to extract and fuse features from each modality, including the following steps:

[0010] Step S101: Obtain chat history, classify chat history data types, and extract feature vectors for each modality;

[0011] Step S102, assign weights;

[0012] Step S103: Feature fusion is performed using an attention mechanism;

[0013] Step S104 involves role encoding and location encoding, ultimately forming a multimodal feature vector with dialogue temporal information.

[0014] Optionally, in step S101, the chat history information is categorized according to data type:

[0015] Image information is input into a ResNet network to obtain a feature map. The obtained feature map is then used as input to a fully connected neural network to generate a feature vector of the image information.

[0016] The speech information is input into the Transformer architecture, and the latent features output by the Transformer architecture are the feature vectors of the speech information.

[0017] The text information is input into the Embedding module of the ERNIE model, which incorporates knowledge of financial fraud, to generate dynamic word vectors, thus obtaining the feature vector of the text information.

[0018] Optionally, in step S103, an attention mechanism is used to fuse the feature vectors of image information, speech information, and text information to obtain a multimodal feature vector.

[0019] Optionally, in step S104, based on the multimodal feature vector, the roles are encoded and embedded using {0, 1, ..., i}, that is, the information of the first role is encoded as 0, and then embedded into the multimodal feature vector;

[0020] A positional encoding method based on sine and cosine functions is used to process multimodal feature vectors, ultimately forming multimodal feature vectors with dialogue temporal information.

[0021] Optionally, an ERNIE model that incorporates knowledge from the financial fraud domain is trained, including the following steps:

[0022] Step S201: Collect a large amount of textual data and dictionaries related to financial fraud;

[0023] Step S202: Train the ERNIE model using two pre-training tasks;

[0024] Step S203: Input the multimodal feature vector with dialogue temporal information from Step 1 into the ERNIE model combined with financial fraud domain knowledge in Step S202 for training to obtain the latent feature vector.

[0025] Optionally, in step S202, the two pre-training tasks are a word-level pre-training task and a task-level pre-training task, respectively.

[0026] The word-level pre-training task includes two sub-tasks: the whole-word mask task and the next-sentence prediction task. The whole-word mask task performs a whole-word mask on the corpus text of financial fraud and words in the dictionary to infer the corresponding financial fraud words. The next-sentence prediction task sorts the corpus text of financial fraud in a certain order and judges the relationship between the context by predicting the next sentence. The training generates an ERNIE model with word-level capabilities in the field of financial fraud.

[0027] The task-level pre-training tasks include the financial fraud classification sub-task and the financial fraud named entity recognition sub-task. These sub-tasks are used to enable the model to learn semantic-level knowledge of the financial fraud domain and the feature distribution of words and phrases in the financial fraud domain, thereby training and generating an ERNIE model with both word-level and task-level capabilities in the financial fraud domain.

[0028] Through training on word-level pre-training tasks and task-level pre-training tasks, an ERNIE model incorporating knowledge from the financial fraud domain is obtained.

[0029] Optionally, a pointer-dual generator network is used to jointly generate dialogue summaries and keywords, including step S301, in which the latent feature vector obtained in step two is input into a bidirectional long short-term memory network (BiLSTM), and the output of the forward LSTM and the output of the backward LSTM are concatenated to obtain a feature vector that simultaneously contains latent features from left to right and latent features from right to left.

[0030] The attention network inputs feature vectors containing both left-to-right and right-to-left latent features, and outputs feature vectors with both global and local information.

[0031] Optionally, the joint generation of dialogue summaries and keywords using a pointer-dual generator network further includes step S302, where the attention network outputs a feature vector with global and local information and inputs it into the pointer-dual generator network. First, the generation probability is calculated by multiplying the feature vector with global and local information with the vector generated by the pointer-dual generator network. Then, the final distribution is calculated using the word distribution and attention distribution. Finally, the dialogue summary and keywords of the suspect's chat history are generated.

[0032] This application provides a method for jointly generating financial fraud dialogue summaries and keywords based on multimodal fusion. This method has strong generalization ability and solves the problems of existing methods in handling single data types, short text chat logs, and colloquial information in financial fraud. This method can utilize chat logs on the suspect's mobile phone, and through a model that integrates text summaries and keywords from the financial fraud field, obtain dialogue summaries and keywords from the suspect's chat logs. This enables comprehensive collection and organization of the suspect's chat logs, assisting investigators in focusing on contacts who are also engaged in criminal activities, extracting key dialogue information from accomplices' chats, and grasping the facts of the crime. This method has broad application prospects and can be used in various real-world scenarios to meet the needs of generating financial fraud dialogue summaries and keywords. It can assist investigators in grasping the main content of the suspect's chats, quickly locating key information, timely obtaining facts of the crime, reducing manpower costs, and improving work efficiency. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a flowchart of the method for jointly generating financial fraud dialogue summaries and keywords based on multimodal fusion, as described in this invention.

[0035] Figure 2 This is a flowchart of the feature extraction and fusion method based on multimodal chat records of the present invention;

[0036] Figure 3 This is a flowchart of the text classification method based on the ERNIE model, which incorporates knowledge from the field of financial fraud, as described in this invention.

[0037] Figure 4 This is a flowchart of the dialogue summary and keyword joint generation method based on pointer-dual generator network of the present invention. Detailed Implementation

[0038] To make the technical problems, technical solutions, and beneficial effects to be solved by this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this application.

[0039] This application now describes a method for jointly generating financial fraud dialogue summaries and keywords based on multimodal fusion. This method utilizes chat logs from a suspect's mobile phone and, through a text summarization and keyword generation model that integrates information from the financial fraud field, obtains dialogue summaries and keywords from the suspect's chat logs.

[0040] Specifically, it includes the following steps:

[0041] See Figure 1 Step 1: Use a multimodal model to extract and fuse features from each modality.

[0042] See Figure 2 The multimodal model is used to extract and fuse features from various modalities, including the following steps:

[0043] Step S101: Obtain chat history, classify chat history data types, and extract feature vectors for each modality.

[0044] The chat history information is classified according to data type. The image information is input into the ResNet network to obtain the feature map. The obtained feature map is used as the input of the fully connected neural network to generate the feature vector of the image information.

[0045] The speech information is input into the Transformer architecture, and the latent features output by the Transformer architecture are the feature vectors of the speech information.

[0046] The text information is input into the Embedding module of the ERNIE model, which incorporates knowledge of financial fraud, to generate dynamic word vectors and obtain the feature vector of the text information.

[0047] Thus, we have obtained the feature vectors for image information, speech information, and text information.

[0048] In step S101, the feature map is a feature map.

[0049] Step S102: Assign weights. Unlike other text, text and voice information in chat logs are more important than image information. Therefore, in the multimodal feature fusion process, different weights are assigned to the three feature vectors, with higher weights given to text and voice information.

[0050] Step S103: Feature fusion is performed using an attention mechanism. The attention mechanism is used to fuse the feature vectors of image information, speech information, and text information to obtain a multimodal feature vector.

[0051] Step S104: Role Encoding and Position Encoding. Unlike long text articles, the chat logs of criminal suspects are mostly dialogues between two or more people. Therefore, based on the current multimodal feature vector, role encoding is performed. Different role information is encoded and embedded using {0, 1, ..., i}, that is, the information of the first role is encoded as 0, and then embedded into the multimodal feature vector. At the same time, the chat logs of criminal suspects have a temporal sequence, so a position encoding method based on sine and cosine functions is used for processing, finally forming a multimodal feature vector with dialogue temporal information.

[0052] See Figure 1 Step two: Train and generate an ERNIE model that incorporates knowledge from the field of financial fraud.

[0053] See Figure 3 The training steps for the ERNIE model, which incorporates knowledge from the field of financial fraud, are as follows:

[0054] Step S201: Collect a large amount of textual data and dictionaries related to financial fraud.

[0055] Step S202: Train the ERNIE model using two pre-training tasks.

[0056] The first training task is a word-level pre-training task, which includes two sub-tasks: a whole-word mask task and a next-sentence prediction task. The whole-word mask task performs a whole-word mask on the corpus text of financial fraud and words in the dictionary to infer the corresponding financial fraud terms. The next-sentence prediction task sorts the corpus text of financial fraud in a certain order and determines the context relationship by predicting the next sentence. This training generates an ERNIE model with word-level capabilities in the financial fraud domain.

[0057] The second training task is a task-level pre-training task, which includes two sub-tasks: financial fraud classification and financial fraud named entity recognition. These two sub-tasks allow the model to learn semantic-level knowledge of the financial fraud domain and the feature distribution of words and phrases in the financial fraud domain. The training generates an ERNIE model with both word-level and task-level capabilities in the financial fraud domain.

[0058] First, a word-level pre-training task is performed, followed by a task-level pre-training task. Through these two pre-training tasks, an ERNIE model incorporating knowledge from the financial fraud domain is obtained.

[0059] In step S203, the multimodal feature vector containing dialogue temporal information from step one is input into the ERNIE model, which incorporates knowledge of financial fraud, in step S202 for training, to obtain a latent feature vector. This latent feature vector serves as input to a fully connected neural network, which then outputs a classification result indicating whether the chat log involves criminal activity, thereby eliminating a large number of irrelevant chat logs.

[0060] See Figure 1 Step 3: Use a pointer-dual generator network to jointly generate dialogue summaries and keywords.

[0061] See Figure 4 The following method uses a pointer-dual generator network for joint generation of dialogue summaries and keywords:

[0062] Step S301: Input the latent feature vector obtained in step two into the Bidirectional Long Short-Term Memory (BiLSTM) network. The BiLSTM network selectively forgets internal information and remembers new information through gating units, retaining information useful for the next time step. Finally, the BiLSTM network splices the outputs of the forward LSTM and the backward LSTM to obtain a feature vector that contains latent features from left to right and from right to left.

[0063] Bidirectional Long Short-Term Memory (BiLSTM) avoids the problem of missing context information caused by the forward-to-back transmission of unidirectional LSTM.

[0064] The feature vector containing both left-to-right and right-to-left hidden features is the output of the hidden layer of the Bidirectional Long Short-Term Memory (BiLSTM) network.

[0065] The attention network inputs feature vectors containing both left-to-right and right-to-left latent features. This allows the attention network to encompass global information, avoiding the problem of focusing only on local information. The attention network outputs feature vectors that contain both global and local information.

[0066] Step S302: Use a pointer-dual generator network to jointly generate dialogue summaries and keywords.

[0067] To generate both summaries and keywords simultaneously, the traditional pointer-generator network is improved into a pointer-dual-generator network. Two unidirectional LSTM networks are used as the generator part of the pointer-dual-generator network; one LSTM network generates the dialogue summary, and the other generates the keywords. To achieve joint generation, a multi-task learning approach is adopted, summing the training function losses for both the dialogue summary and keyword generation tasks to jointly train the pointer-dual-generator network model.

[0068] The attention network outputs a feature vector containing global and local information, which is then input into the pointer-dual generator network. First, the feature vector containing global and local information is multiplied with the vector generated by the pointer-dual generator network to calculate the generation probability. Then, the final distribution is calculated using the word distribution and attention distribution. Finally, the dialogue summary and keywords of the suspect's chat history are generated.

[0069] After extracting the chat logs of a criminal suspect, this application first performs data preprocessing, dividing the data into text, image, and voice modalities, which are then input into a multimodal model. The multimodal model is used to extract and fuse features from each modality, resulting in a multimodal feature vector containing dialogue temporal information. Using financial fraud-related corpus as input, an enhanced ERNIE model incorporating knowledge of the financial fraud domain is trained using two pre-training methods. The obtained multimodal feature vector is then used as input to the enhanced ERNIE model incorporating knowledge of the financial fraud domain to extract latent feature vectors. A fully connected neural network is then used for classification to determine whether the chat log involves criminal activity. If criminal activity is involved, the feature vector is used as input to a pointer-dual generator network to ultimately generate a key dialogue summary and keywords related to the criminal suspect's involvement in criminal activity.

[0070] This application provides a method for jointly generating financial fraud dialogue summaries and keywords based on multimodal fusion. This method utilizes chat logs from a suspect's mobile phone, combining text summarization and keyword generation models that integrate information from the financial fraud field to obtain dialogue summaries and keywords from the suspect's chat logs. The synthesis method of this application has strong generalization ability, overcoming the shortcomings of existing methods in handling single data types, difficulty in processing short text chat logs, and colloquial information related to financial fraud. It can be applied to various real-world scenarios requiring the generation of financial fraud dialogue summaries and keywords; it can assist investigators in grasping the main content of the suspect's chats, quickly locating key information, timely obtaining facts of the crime, reducing manpower costs, and improving work efficiency.

[0071] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for jointly generating financial fraud dialogue summaries and keywords based on multimodal fusion, characterized in that, By utilizing the chat logs on the suspect's mobile phone, a text summarization and keyword joint generation model that integrates information from the financial fraud field is used to obtain the dialogue summary and keywords in the suspect's chat logs. Step 1: Extract and fuse features from various modalities using a multimodal model to obtain a multimodal feature vector with dialogue temporal information; Step 2: Train and generate an ERNIE model that incorporates knowledge of the financial fraud domain. Input the multimodal feature vector with dialogue temporal information into the ERNIE model that incorporates knowledge of the financial fraud domain to obtain the latent feature vector. Step 3: Input the latent feature vector into the pointer-dual generator network, which then jointly generates the dialogue summary and keywords. The process of extracting and fusing features from various modalities using a multimodal model includes the following steps: Step S101: Obtain chat history, classify chat history data types, and extract feature vectors for each modality; Step S102, assign weights; Step S103: Feature fusion is performed using an attention mechanism; Step S104 involves role encoding and location encoding, ultimately forming a multimodal feature vector with dialogue temporal information.

2. The method for jointly generating financial fraud dialogue summaries and keywords based on multimodal fusion as described in claim 1, characterized in that, In step S101, the chat history information is categorized according to data type: Image information is input into a ResNet network to obtain a feature map. The obtained feature map is then used as input to a fully connected neural network to generate a feature vector of the image information. The speech information is input into the Transformer architecture, and the latent features output by the Transformer architecture are the feature vectors of the speech information. The text information is input into the Embedding module of the ERNIE model, which incorporates knowledge of financial fraud, to generate dynamic word vectors, thus obtaining the feature vector of the text information.

3. The method for jointly generating financial fraud dialogue summaries and keywords based on multimodal fusion as described in claim 2, characterized in that, In step S103, an attention mechanism is used to fuse the feature vectors of image information, speech information, and text information to obtain a multimodal feature vector.

4. The method for jointly generating financial fraud dialogue summaries and keywords based on multimodal fusion as described in claim 3, characterized in that, In step S104, based on the multimodal feature vector, the roles are encoded, and different role information is used... Encoding embedding is performed, that is, the information of the first role is encoded as 0, and then it is embedded into the multimodal feature vector; A positional encoding method based on sine and cosine functions is used to process multimodal feature vectors, ultimately forming multimodal feature vectors with dialogue temporal information.

5. The method for jointly generating financial fraud dialogue summaries and keywords based on multimodal fusion as described in claim 1, characterized in that, Training and generating an ERNIE model that incorporates knowledge from the field of financial fraud includes the following steps: Step S201: Collect a large amount of textual data and dictionaries related to financial fraud; Step S202: Train the ERNIE model using two pre-training tasks; Step S203: Input the multimodal feature vector with dialogue temporal information from Step 1 into the ERNIE model combined with financial fraud domain knowledge in Step S202 for training to obtain the latent feature vector.

6. The method for jointly generating financial fraud dialogue summaries and keywords based on multimodal fusion as described in claim 5, characterized in that, In step S202, the two pre-training tasks are a word-level pre-training task and a task-level pre-training task; The word-level pre-training task includes two sub-tasks: the whole-word mask task and the next-sentence prediction task. The whole-word mask task performs a whole-word mask on the corpus text of financial fraud and words in the dictionary to infer the corresponding financial fraud words. The next-sentence prediction task sorts the corpus text of financial fraud in a certain order and judges the relationship between the context by predicting the next sentence. The training generates an ERNIE model with word-level capabilities in the field of financial fraud. The task-level pre-training tasks include the financial fraud classification sub-task and the financial fraud named entity recognition sub-task. These sub-tasks are used to enable the model to learn semantic-level knowledge of the financial fraud domain and the feature distribution of words and phrases in the financial fraud domain, thereby training and generating an ERNIE model with both word-level and task-level capabilities in the financial fraud domain. Through training on word-level pre-training tasks and task-level pre-training tasks, an ERNIE model incorporating knowledge from the financial fraud domain is obtained.

7. The method for jointly generating financial fraud dialogue summaries and keywords based on multimodal fusion as described in claim 1, characterized in that, The dialog summary and keyword joint generation is performed using a pointer-dual generator network, including step S301, where the latent feature vector obtained in step two is input into a bidirectional long short-term memory network (BiLSTM). The BiLSTM network concatenates the outputs of the forward LSTM and the backward LSTM to obtain a feature vector that simultaneously contains latent features from left to right and from right to left. The attention network inputs feature vectors containing both left-to-right and right-to-left latent features, and outputs feature vectors with both global and local information.

8. The method for jointly generating financial fraud dialogue summaries and keywords based on multimodal fusion as described in claim 7, characterized in that, The method of using a pointer-dual generator network to jointly generate dialogue summaries and keywords also includes step S302, where the attention network outputs a feature vector with global and local information and inputs it into the pointer-dual generator network. First, the generation probability is calculated by multiplying the feature vector with global and local information with the vector generated by the pointer-dual generator network. Then, the final distribution is calculated using the word distribution and attention distribution. Finally, the dialogue summary and keywords of the suspect's chat history are generated.

Images