Financial fraud dialogue abstract and keyword joint generation method based on multi-modal fusion
Through the ERNIE model and pointer-dual generator network of multimodal fusion technology and knowledge in the field of financial fraud, dialogue summary and keywords are generated, and the problem of single data type and colloquial information processing in the existing technology is solved, and criminal suspicion is realized The comprehensive information sorting and key information positioning of human chat records have improved case handling efficiency.
Patent Information
- Application Number
- CN202310413475.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-13
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2043-04-13
AI Technical Summary
The existing automatic summary method of financial fraud processing is single data type, and it is unable to effectively process multiple types of criminal suspect mobile data, especially short texts of chat records and spoken information, and fails to effectively identify relevant knowledge in the field of financial fraud, resulting in information Incoherent and irrelevant information.
Using a joint generation method of financial fraud dialogue summary and keywords based on multimodal fusion, the dialogue summary and keywords are generated through multimodal model extraction and fusion features, combined with the knowledge of the financial fraud field, and the ERNIE model and pointer-dual generator network are used to generate dialogue summary and keywords. , process various data types such as text, pictures, and voice, and use attention mechanism and position encoding to form a multimodal feature vector with dialogue timing information.
It has realized the comprehensive collection and sorting of the chat records of criminal suspects, assisted case handlers to quickly locate key information, reduce labor costs, improve work efficiency, and be able to effectively process short texts of chat records and spoken information of financial fraud.
Smart Images

Figure CN118981531A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of natural language processing technology, and more specifically, to a method for jointly generating financial fraud dialogue summaries and keywords based on multimodal fusion. Background Art
[0002] When the public security department investigates the criminal behavior of the suspect, it is difficult to search manually because the amount of data in the suspect's mobile phone is huge, and the information on the company, location, organization, person, contact information, amount involved, etc. is also scattered. At the same time, the data in the suspect's mobile phone includes conversation text, pictures, videos and voice, etc. The complex types of information also bring great difficulties to the public security department in handling the case.
[0003] At present, there are certain precedents for the application of automatic summarization methods related to finance. The existing method is to input financial images and texts, establish an encoder-decoder model, and finally generate a financial text summary generation model. However, this type of method has the following defects: First, the data type that this method can process is relatively single, and can only process text type or image type data, but in reality, the data in the suspect's mobile phone contains many types, such as text, pictures, videos, voice, etc., so it is impossible to get all the information in all aspects, resulting in a lack of key evidence; second, this method is more about processing long text information such as financial articles, and is less used for short text information in financial fraud chat records, and the effect is not ideal. It does not solve the problems of short text, large topic span, incoherent semantic information, and more irrelevant information in the chat; third, this method fails to take into account the relevant knowledge in the field of financial fraud, and it is difficult to identify some colloquial information of the suspect. Summary of the invention
[0004] To achieve the above-mentioned purpose, the technical solution adopted in this application is: to provide a method for jointly generating financial fraud conversation summaries and keywords based on multimodal fusion, using the chat records in the suspect's mobile phone, and through a text summary and keyword joint generation model that integrates information in the field of financial fraud, to obtain the conversation summary and keywords in the suspect's chat records.
[0005] Optionally, the following steps are included:
[0006] Step 1: Use the multimodal model to extract and fuse the features of each modality, and generate a multimodal feature vector with dialogue timing information;
[0007] Step 2: Train and generate an ERNIE model that combines knowledge in the field of financial fraud, and input the multimodal feature vector with conversation time sequence information into the ERNIE model that combines knowledge in the field of financial fraud to obtain a latent feature vector;
[0008] Step 3: Input the latent feature vector into the pointer-dual generator network, and the pointer-dual generator network jointly generates the conversation summary and keywords.
[0009] Optionally, the multimodal model is used to extract and fuse features of each modality, including the following steps:
[0010] Step S101, obtaining chat records, classifying chat record data types, and extracting feature vectors of each modality;
[0011] Step S102, assigning weights;
[0012] Step S103, using the attention mechanism to perform feature fusion;
[0013] Step S104: role coding and position coding, and finally forming a multimodal feature vector with dialogue timing information.
[0014] Optionally, in step S101, the chat record information is classified according to data type:
[0015] Input the image information into the ResNet network to obtain a feature map, and use the obtained feature map as the input of the fully connected neural network to generate a feature vector of the image information;
[0016] The speech information is input into the Transformer architecture, and the hidden features output by the Transformer architecture are the feature vectors of the speech information;
[0017] The text information is input into the Embedding module based on the ERNIE model combined with the knowledge in the field of financial fraud to generate a dynamic word vector and obtain the feature vector of the text information.
[0018] Optionally, in step S103, an attention mechanism is used to perform feature fusion on the feature vector of the image information, the feature vector of the speech information and the feature vector of the text information to obtain a multimodal feature vector.
[0019] Optionally, in step S104, based on the multimodal feature vector, the roles are encoded, and different role information is encoded and embedded using {0, 1, ..., i}, that is, the information of the first role is encoded as 0, and then embedded into the multimodal feature vector;
[0020] The multimodal feature vector is processed by a position encoding method based on sine and cosine functions, and finally a multimodal feature vector with dialogue timing information is formed.
[0021] Optionally, training and generating an ERNIE model that combines knowledge in the field of financial fraud includes the following steps:
[0022] Step S201, collecting a large amount of corpus texts and dictionaries related to financial fraud;
[0023] Step S202, using two pre-training tasks to train the ERNIE model;
[0024] Step S203, input the multimodal feature vector with the conversation timing information in step 1 into the ERNIE model combined with the financial fraud domain knowledge in step S202 for training to obtain a latent feature vector.
[0025] Optionally, in step S202, the two pre-training tasks are a word level pre-training task and a task level pre-training task;
[0026] The word-level pre-training task includes two sub-tasks: the full-word mask task and the next sentence prediction task. The full-word mask task performs full-word masking on the corpus text of financial fraud and the words in the dictionary to infer the corresponding financial fraud words. The next sentence prediction task sorts the corpus text of financial fraud in a certain order, judges the relationship of the context through the next sentence prediction, and trains and generates an ERNIE model with word-level capabilities in the field of financial fraud.
[0027] The task-level pre-training tasks include the financial fraud classification subtask and the financial fraud named entity recognition subtask. The financial fraud classification subtask and the financial fraud named entity recognition subtask are used to allow the model to learn the semantic-level financial fraud domain knowledge and the characteristic distribution of words and sentences in the financial fraud domain, and train and generate an ERNIE model with word-level and task-level capabilities in the financial fraud domain.
[0028] Through training of word-level pre-training tasks and task-level pre-training tasks, the ERNIE model combined with knowledge in the field of financial fraud is obtained.
[0029] Optionally, a pointer-dual generator network is used to jointly generate a conversation summary and keywords, including step S301, inputting the latent feature vector obtained in step 2 into a bidirectional long short-term memory network BiLSTM, and the bidirectional long short-term memory network BiLSTM concatenates the output of the forward LSTM and the output of the backward LSTM to obtain a feature vector containing both left-to-right latent features and right-to-left latent features;
[0030] The feature vector containing both left-to-right latent features and right-to-left latent features is input into the attention network, and the attention network outputs a feature vector with global and local information.
[0031] Optionally, the pointer-dual generator network is used to jointly generate conversation summaries and keywords, and also includes step S302, in which the attention network outputs a feature vector with global and local information and inputs it into the pointer-dual generator network. First, the feature vector with global and local information is multiplied by the vector generated by the pointer-dual generator network to calculate the generation probability, and then the final distribution is calculated using the vocabulary distribution and attention distribution. Finally, the conversation summary and keywords of the suspect's chat record are generated.
[0032] The present application provides a method for jointly generating financial fraud conversation summaries and keywords based on multimodal fusion. The method has strong generalization ability and solves the problems of existing methods that the data type is single and it is difficult to process short texts in chat records and colloquial information of financial fraud. The method can use the chat records in the mobile phone of the suspect to obtain the conversation summaries and keywords in the chat records of the suspect by integrating the text summaries of the information in the field of financial fraud and the model jointly generated by keywords, so as to realize the comprehensive collection and collation of the chat records of the suspect, assist the investigators in focusing on the contacts who are also engaged in criminal activities, extract the key conversation information of the criminal accomplices in the chat, and grasp the facts of the crime. The method has broad application prospects and can be used for the needs of generating financial fraud conversation summaries and keywords in various real-life scenarios. It can assist the investigators in grasping the main content of the suspect's chat, quickly locating key information, timely obtaining the facts of the crime, reducing labor costs, and improving work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0034] Figure 1 This is a flow chart of the method for jointly generating financial fraud dialogue summaries and keywords based on multimodal fusion of the present invention;
[0035] Figure 2 This is a flow chart of a feature extraction and fusion method based on multimodal chat records of the present invention;
[0036] Figure 3 This is a flow chart of a text classification method based on the ERNIE model combined with knowledge in the field of financial fraud in the present invention;
[0037] Figure 4 The present invention is a flow chart of the method for jointly generating conversation summaries and keywords based on the pointer-dual generator network. DETAILED DESCRIPTION
[0038] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0039] A method for jointly generating a summary and keywords of a financial fraud conversation based on multimodal fusion provided by an embodiment of the present application is now described. A method for jointly generating a summary and keywords of a financial fraud conversation based on multimodal fusion uses the chat records in the suspect's mobile phone and obtains the conversation summary and keywords in the suspect's chat records by integrating a text summary and keyword joint generation model of information in the field of financial fraud.
[0040] Specifically, the following steps are included:
[0041] See also Figure 1 ,Step 1: Use the multimodal model to extract and fuse the features of each modality.
[0042] See also Figure 2 , using the multimodal model to extract and fuse the features of each modality, including the following steps:
[0043] Step S101, obtain chat records, classify chat record data types, and extract feature vectors of each modality.
[0044] Classify the chat record information according to data type, input the image information into the ResNet network to obtain a feature map, and use the obtained feature map as the input of the fully connected neural network to generate a feature vector of the image information;
[0045] The speech information is input into the Transformer architecture, and the hidden features output by the Transformer architecture are the feature vectors of the speech information;
[0046] Input the text information into the Embedding module based on the ERNIE model combined with the knowledge of financial fraud to generate dynamic word vectors and obtain the feature vector of the text information;
[0047] At this point, the feature vector of the image information, the feature vector of the voice information, and the feature vector of the text information are obtained.
[0048] The feature map in step S101 is a feature map.
[0049] Step S102, assigning weights. Different from other texts, text information and voice information in chat records are more important than image information. Therefore, in the multimodal feature fusion process, different weights are assigned to the three feature vectors, and higher weights are assigned to text information and voice information.
[0050] Step S103, using the attention mechanism to perform feature fusion. The attention mechanism is used to perform feature fusion on the feature vectors of the image information, the feature vectors of the voice information, and the feature vectors of the text information to obtain a multimodal feature vector.
[0051] Step S104, role coding and position coding. Unlike long text articles, the chat records of criminal suspects are mostly conversation information between two or more people. Therefore, based on the current multimodal feature vector, the roles are coded and different role information is coded and embedded using {0, 1, ..., i}, that is, the information of the first role is coded as 0, and then embedded into the multimodal feature vector. At the same time, the chat records of criminal suspects have time sequence, and are processed using a position coding method based on sine and cosine functions, ultimately forming a multimodal feature vector with conversation time sequence information.
[0052] See also Figure 1 ,Step 2, train and generate the ERNIE model that combines ,the knowledge in the field of financial fraud.
[0053] See also Figure 3 ,The training steps of the ERNIE model combined with the knowledge of financial fraud are as follows:
[0054] Step S201, collect a large amount of corpus text and dictionaries related to financial fraud.
[0055] Step S202: Two pre-training tasks are used to train the ERNIE model.
[0056] The first training task is a word-level pre-training task, which includes two subtasks: the full-word mask task and the next sentence prediction task. The full-word mask task performs full-word masking on the corpus text of financial fraud and the words in the dictionary to infer the corresponding financial fraud words. The next sentence prediction task sorts the corpus text of financial fraud in a certain order and judges the relationship of the context through the next sentence prediction. The training generates an ERNIE model with word-level capabilities in the field of financial fraud.
[0057] The second training task is a task-level pre-training task, which includes two sub-tasks: financial fraud classification and financial fraud named entity recognition. These two sub-tasks allow the model to learn the semantic knowledge of financial fraud and the characteristic distribution of words and sentences in the financial fraud field. The training generates an ERNIE model with word-level and task-level capabilities in the financial fraud field.
[0058] First, the word-level pre-training task is performed, and then the task-level pre-training task is performed. Through the training of the above two pre-training tasks, an ERNIE model combining the knowledge of the financial fraud field is obtained.
[0059] In step S203, the multimodal feature vector with the conversation time sequence information in step 1 is input into the ERNIE model combined with the knowledge of financial fraud in step S202 for training to obtain a latent feature vector. The latent feature vector is used as the input of the fully connected neural network, and finally the fully connected neural network outputs the classification result of whether the chat record involves criminal facts, thereby eliminating a large number of irrelevant chat records.
[0060] See also Figure 1 ,Step 3, use the pointer-dual generator network to jointly generate dialogue summaries and keywords.
[0061] See also Figure 4 , the method of using the pointer-dual generator network to jointly generate dialogue summaries and keywords is as follows:
[0062] Step S301, input the latent feature vector obtained in step 2 into the bidirectional long short-term memory network BiLSTM. The bidirectional long short-term memory network BiLSTM selectively forgets internal information and memorizes new information through the gating unit, retains information that is useful for the next moment, and finally concatenates the output of the forward LSTM and the output of the backward LSTM to obtain a feature vector that contains both left-to-right latent features and right-to-left latent features.
[0063] The bidirectional long short-term memory network BiLSTM avoids the problem of missing context information caused by the unidirectional LSTM that only transmits from the front to the back.
[0064] The feature vector containing both left-to-right latent features and right-to-left latent features is the output of the hidden layer of the bidirectional long short-term memory network BiLSTM.
[0065] The feature vector containing both left-to-right latent features and right-to-left latent features is input into the attention network, which can cover global information and avoid the problem of focusing only on local information. The attention network outputs a feature vector with global and local information.
[0066] Step S302: Use the pointer-dual generator network to jointly generate the conversation summary and keywords.
[0067] In order to generate summaries and keywords at the same time, the traditional pointer-generator network is improved to a pointer-dual generator network. Two unidirectional LSTM networks are used as the generator part of the pointer-dual generator network, one LSTM network is used to generate dialogue summaries, and the other LSTM network is used to generate keywords. To achieve joint generation, a multi-task learning method is used to add the training function losses of the two tasks of generating dialogue summaries and keywords, and jointly train the pointer-dual generator network model.
[0068] The feature vector with global and local information output by the attention network is input into the pointer-double generator network. First, the generation probability is calculated by multiplying the feature vector with global and local information with the vector generated by the pointer-double generator network. Then, the final distribution is calculated by the vocabulary distribution and attention distribution. Finally, the conversation summary and keywords of the suspect's chat record are generated.
[0069] After extracting the chat records of the suspect, this application first performs data preprocessing, divides the data into various modalities such as text, image, and voice, and inputs them into the multimodal model. The multimodal model is used to extract and fuse the features of each modality to obtain a multimodal feature vector with conversation timing information; using financial fraud-related corpus as input, two pre-training methods are used to train an enhanced ERNIE model that combines knowledge in the field of financial fraud. The obtained multimodal feature vector is used as the input of the enhanced ERNIE model that combines knowledge in the field of financial fraud, and the latent feature vector is extracted. The fully connected neural network is used for classification to determine whether the chat record involves criminal facts. If it involves criminal facts, the feature vector is used as the input of the pointer-dual generator network, and finally the key conversation summary and keywords involving the criminal suspect's criminal facts are generated.
[0070] The present application provides a method for jointly generating financial fraud dialogue summaries and keywords based on multimodal fusion. The method can utilize the chat records in the suspect's mobile phone, and obtain the dialogue summaries and keywords in the suspect's chat records by integrating the text summary and keyword joint generation model of the information in the field of financial fraud. The synthesis method of the present application has a strong generalization ability. It overcomes the defects of the existing methods in processing single data types, difficulty in processing short texts of chat records and colloquial information of financial fraud, etc. It can be used for the needs of generating financial fraud dialogue summaries and keywords in various real-life scenarios; it can assist case handlers in grasping the main content of the suspect's chat, quickly locating key information, obtaining criminal facts in a timely manner, reducing labor costs, and improving work efficiency.
[0071] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.
Claims
1. A method for jointly generating financial fraud dialogue summaries and keywords based on multimodal fusion, characterized in that: By using the chat records in the suspect's mobile phone and integrating the text summary and keyword joint generation model of financial fraud information, the conversation summary and keywords in the suspect's chat records are obtained.
2. The method for jointly generating financial fraud dialogue summaries and keywords based on multimodal fusion as claimed in claim 1, characterized in that: The following steps are involved: Step 1: Use the multimodal model to extract and fuse the features of each modality, and generate a multimodal feature vector with dialogue timing information; Step 2: Train and generate an ERNIE model that combines knowledge in the field of financial fraud, and input the multimodal feature vector with conversation time sequence information into the ERNIE model that combines knowledge in the field of financial fraud to obtain a latent feature vector; Step 3: Input the latent feature vector into the pointer-dual generator network, and the pointer-dual generator network jointly generates the conversation summary and keywords.
3. The method for jointly generating financial fraud dialogue summaries and keywords based on multimodal fusion as claimed in claim 2, characterized in that: The multimodal model is used to extract and fuse the features of each modality, including the following steps: Step S101, obtaining chat records, classifying chat record data types, and extracting feature vectors of each modality; Step S102, assigning weights; Step S103, using the attention mechanism to perform feature fusion; Step S104: role coding and position coding, and finally forming a multimodal feature vector with dialogue timing information.
4. The method for jointly generating financial fraud dialogue summaries and keywords based on multimodal fusion as claimed in claim 3, characterized in that: In step S101, the chat record information is classified according to the data type: Input the image information into the ResNet network to obtain a feature map, and use the obtained feature map as the input of the fully connected neural network to generate a feature vector of the image information; The speech information is input into the Transformer architecture, and the hidden features output by the Transformer architecture are the feature vectors of the speech information; The text information is input into the Embedding module based on the ERNIE model combined with the knowledge in the field of financial fraud to generate a dynamic word vector and obtain the feature vector of the text information.
5. The method for jointly generating financial fraud dialogue summaries and keywords based on multimodal fusion as claimed in claim 4, characterized in that: In step S103, an attention mechanism is used to perform feature fusion on the feature vector of the image information, the feature vector of the speech information and the feature vector of the text information to obtain a multimodal feature vector.
6. The method for jointly generating financial fraud dialogue summaries and keywords based on multimodal fusion as claimed in claim 5, characterized in that: In step S104, based on the multimodal feature vector, the roles are encoded, and different role information is encoded and embedded using {0, 1, ..., i}, that is, the information of the first role is encoded as 0, and then embedded into the multimodal feature vector; The multimodal feature vector is processed by a position encoding method based on sine and cosine functions, and finally a multimodal feature vector with dialogue timing information is formed.
7. The method for jointly generating financial fraud dialogue summaries and keywords based on multimodal fusion as claimed in claim 2, characterized in that: Training and generating the ERNIE model that combines knowledge in the field of financial fraud includes the following steps: Step S201, collecting a large amount of corpus texts and dictionaries related to financial fraud; Step S202, using two pre-training tasks to train the ERNIE model; Step S203, input the multimodal feature vector with the conversation timing information in step 1 into the ERNIE model combined with the financial fraud domain knowledge in step S202 for training to obtain a latent feature vector.
8. The method for jointly generating financial fraud dialogue summaries and keywords based on multimodal fusion as claimed in claim 7, characterized in that: In step S202, the two pre-training tasks are a word level pre-training task and a task level pre-training task; The word-level pre-training task includes two sub-tasks: the full-word mask task and the next sentence prediction task. The full-word mask task performs full-word masking on the corpus text of financial fraud and the words in the dictionary to infer the corresponding financial fraud words. The next sentence prediction task sorts the corpus text of financial fraud in a certain order, judges the relationship of the context through the next sentence prediction, and trains and generates an ERNIE model with word-level capabilities in the field of financial fraud. The task-level pre-training tasks include the financial fraud classification subtask and the financial fraud named entity recognition subtask. The financial fraud classification subtask and the financial fraud named entity recognition subtask are used to allow the model to learn the semantic-level financial fraud domain knowledge and the characteristic distribution of words and sentences in the financial fraud domain, and train and generate an ERNIE model with word-level and task-level capabilities in the financial fraud domain. Through training of word-level pre-training tasks and task-level pre-training tasks, the ERNIE model combined with knowledge in the field of financial fraud is obtained.
9. The method for jointly generating financial fraud dialogue summaries and keywords based on multimodal fusion as claimed in claim 2, characterized in that: The conversation summary and keywords are jointly generated using the pointer-dual generator network, including step S301, inputting the latent feature vector obtained in step 2 into a bidirectional long short-term memory network BiLSTM, and the bidirectional long short-term memory network BiLSTM concatenates the output of the forward LSTM and the output of the backward LSTM to obtain a feature vector containing both left-to-right latent features and right-to-left latent features; The feature vector containing both left-to-right latent features and right-to-left latent features is input into the attention network, and the attention network outputs a feature vector with global and local information.
10. The method for jointly generating financial fraud dialogue summaries and keywords based on multimodal fusion as claimed in claim 9, characterized in that: The method uses a pointer-dual generator network to jointly generate conversation summaries and keywords, and also includes step S302, in which the attention network outputs a feature vector with global and local information and inputs it into the pointer-dual generator network. First, the feature vector with global and local information is multiplied by the vector generated by the pointer-dual generator network to calculate the generation probability, and then the final distribution is calculated using the vocabulary distribution and the attention distribution. Finally, the conversation summary and keywords of the suspect's chat record are generated.
Citation Information
Patent Citations
Text abstract automatic generation method and system fused with pre-training model
CN112765345A
BERT SUM and PGN fused supply chain ecological region long document abstract generation method
CN115062140A
Speech abstract generation model training method, speech abstract generation method and device
CN115827854A
Cited By
Anti-fraud method based on ASR speech recognition
CN119170018A
Multi-modal dialogue abstract method based on multi-level visual guidance
CN119918545A
Deep fraud content detection system based on multi-modal data fusion
CN121543041A
