Fraud phone recognition method and system based on multi-modal data fusion

Abstract

The invention discloses a fraud phone recognition method and system based on multi-modal data fusion. The method comprises the following steps: acquiring multi-modal data of a plurality of fraud numbers from historical fraud data; constructing a multi-modal fraud data graph, taking each piece of multi-modal data of the fraud number as a node, and constructing a connection edge between the nodes according to an association relationship between every two pieces of multi-modal data; constructing and finely adjusting a large language model; multi-modal data of a to-be-recognized fraud number is obtained, a plurality of nodes are selected from the multi-modal fraud data graph, then the multi-modal data of the to-be-recognized fraud number and the multi-modal data of the selected nodes form cue words, the cue words are input into the large language model, and whether the obtained to-be-recognized fraud number relates to fraud or not and reason analysis for judging that the to-be-recognized fraud number is fraud are output. The invention relates to the field of communication, can improve the fraud recognition accuracy and system adaptability through deep fusion and calculation of multi-mode data such as call audio, short message text and call ticket data, and provides powerful technical support for fighting against novel telecommunication network fraud.

Description

Technical Field

[0001] The present invention relates to a fraudulent call identification method and system using multi-modal data fusion, and relates to the field of communications. Background Art

[0002] Traditional fraud call recognition technology is mainly based on single-mode data processing, such as audio analysis or text analysis. However, when dealing with complex fraud scenarios, this technology often has low recognition accuracy due to incomplete information and lack of context. In addition, traditional technology is difficult to adapt to the rapid development of new fraud methods and has weak generalization capabilities.

[0003] Chinese patent CN 116320139A relates to a method, device, electronic device and storage medium for analyzing call risk management, wherein the method for analyzing call risk management includes steps S1, extracting call signaling, obtaining terminal location information, call recording and SMS content; S2, extracting call text information from call recording; S3, extracting NLP from SMS content, extracting feature values ​​in text context; S4, inputting feature values ​​and extracted information into a big data storage module for storage; S5, a risk rating module evaluates a risk model based on the feature values ​​extracted from the upstream, and outputs whether a risk is involved, wherein if a risk model is triggered, the event is pushed to a disposal module, and the result is stored in the big data storage; S6, the disposal module receives the push from the risk rating module and performs relevant intervention and disposal actions. Although this technical solution involves risk assessment based on call signaling, call recording and SMS content, in essence, the analysis of related information from different channels such as call audio, SMS text and call bill data is separated, and the deep integration of multimodal data such as call audio, SMS text and call bill data is not achieved.

[0004] Therefore, how to effectively improve the accuracy of fraud identification and system adaptability through deep integration and calculation of multimodal data such as call audio, SMS text and call record data, and provide strong technical support for combating new types of telecommunications network fraud has become a technical issue that technical personnel have focused on. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a fraud call identification method and system with multimodal data fusion, which can effectively improve the accuracy of fraud identification and system adaptability through deep fusion and calculation of multimodal data such as call audio, SMS text and call record data, and provide strong technical support for combating new types of telecommunications network fraud.

[0006] In order to achieve the above object, the present invention provides a fraudulent number identification method based on multimodal data fusion, comprising:

[0007] Step 1: Obtain multimodal data of several fraud numbers from historical fraud data;

[0008] Step 2: Construct a multimodal fraud data graph. The multimodal fraud data graph uses each multimodal data of each fraud number in the historical fraud data as a node, and constructs a connection edge between corresponding nodes based on the correlation between each two multimodal data;

[0009] Step 3: Build and fine-tune a large language model, where the large language model is used to determine and output whether fraud is involved and the reason for determining it as fraud based on the input multimodal data;

[0010] Step 4. Obtain the multimodal data of the fraudulent number to be identified, and select several nodes from the multimodal fraud data graph. Then, the multimodal data of the fraudulent number to be identified and the multimodal data corresponding to the selected nodes are combined into prompt words, which are input into the large language model to obtain the output of the large language model to determine whether the fraudulent number to be identified involves fraud and the reason analysis for being judged as fraud.

[0011] In order to achieve the above object, the present invention also provides a fraudulent number identification system with multimodal data fusion, comprising:

[0012] A multimodal data graph construction device is used to obtain multimodal data of several fraudulent numbers from historical fraud data, and then construct a multimodal fraud data graph, wherein the multimodal fraud data graph uses each multimodal data of each fraudulent number in the historical fraud data as a node, and constructs a connection edge between corresponding nodes according to the association relationship between each two multimodal data;

[0013] A large language model construction device, used to construct and fine-tune a large language model, wherein the large language model is used to determine and output whether fraud is involved and an analysis of the reasons for determining fraud based on input multimodal data;

[0014] A fraud identification device is used to obtain multimodal data of a fraud number to be identified, select several nodes from a multimodal fraud data graph, and then combine the multimodal data of the fraud number to be identified and the multimodal data corresponding to the selected nodes into prompt words, which are input into a large language model, so as to obtain an output of the large language model to determine whether the fraud number to be identified involves fraud and an analysis of the reasons for being determined to be fraud.

[0015] In order to achieve the above object, the present invention further provides a computing device, comprising:

[0016] Memory and processor;

[0017] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the fraud call identification method using multimodal data fusion are implemented.

[0018] In order to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, which stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps of the fraud call identification method using multimodal data fusion are implemented.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention constructs a comprehensive user behavior portrait by deeply integrating call audio data, SMS text data, and call bill data, significantly reduces the recognition blind spots caused by a single data source, and improves data integrity; the present invention converts multimodal data into a unified graph structure representation, and selects the most relevant multiple information from the multimodal fraud data graph as reference information each time identification is performed, and then uses the powerful semantic understanding and reasoning capabilities of the large language model to construct a sustainable and optimized fraud identification system, and improves the recognition accuracy of the large model; the present invention also combines text similarity and graph structure importance, designs a calculation method for correlation scores, and innovatively considers node type specificity, introduces TypeScore scoring, so that the system can perform differentiated scoring according to the characteristics of different types of nodes, and the text similarity (TSS) uses a combination of cosine similarity, TF-IDF, and feature representation vectors of nodes in the multimodal fraud data graph to better solve the semantic matching problem, and the graph structure importance score (GSIS) comprehensively considers PageRank and Betweenness Centra l ity, more comprehensively evaluate the importance of nodes; the present invention also designs an edge-aware graph convolutional network (E-GCN) to calculate the feature representation vector of each node in the multimodal fraud data graph, which has the characteristics of edge attribute encoding, edge weight aggregation and multi-scale feature fusion. The learnable edge attribute encoding improves the model performance and can adapt to different types of relationships. The edge weight aggregation mechanism improves the model's ability to model complex relationships. Multi-scale feature fusion improves the model's ability to perceive local graph structures. The combination of original node features and edge attribute weighted features enhances the model's expressiveness and improves its ability to model relationships. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart of a fraud call identification method using multimodal data fusion, shown as an exemplary embodiment of the present invention.

[0021] Figure 2 An exemplary embodiment of the present invention is shown Figure 1 In step 1 or step 4, a specific process flow chart of obtaining multimodal data of a fraudulent number or a fraudulent number to be identified.

[0022] Figure 3It is a structural diagram of a fraud call identification system using multimodal data fusion, shown as an exemplary embodiment of the present invention.

[0023] Figure 4 It is a schematic diagram of the structure of a computer device according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings.

[0025] like Figure 1 As shown, the present invention provides a fraudulent number identification method using multimodal data fusion, comprising:

[0026] Step 1: Obtain multimodal data of several fraud numbers from historical fraud data;

[0027] Step 2: Construct a multimodal fraud data graph. The multimodal fraud data graph uses each multimodal data of each fraud number in the historical fraud data as a node, and constructs a connection edge between corresponding nodes based on the association relationship between each two multimodal data. The association relationship may include semantic correlation and time series relationship.

[0028] Step 3: Build and fine-tune a large language model, where the large language model is used to determine and output whether fraud is involved and the reason for determining it as fraud based on the input multimodal data;

[0029] Step 4. Obtain the multimodal data of the fraudulent number to be identified, and select several nodes from the multimodal fraud data graph. Then, the multimodal data of the fraudulent number to be identified and the multimodal data corresponding to the selected nodes are combined into prompt words, which are input into the large language model to obtain the output of the large language model to determine whether the fraudulent number to be identified involves fraud and the reason analysis for being judged as fraud.

[0030] The multimodal data in the present invention may include: user number identification, key event description, user behavior description, related fraud knowledge, SMS URL, call behavior indicators, etc. Figure 2 As shown, in step 1 or step 4, obtaining multimodal data of a fraudulent number or a fraudulent number to be identified may further include:

[0031] Step A1: obtaining a call media evidence file of a fraudulent number or a fraudulent number to be identified, and performing voice recognition to obtain a corresponding voice text;

[0032] Step A2: obtaining the SMS text of the fraudulent number or the fraudulent number to be identified, extracting all the URLs in the SMS text, and obtaining the webpage content corresponding to the URLs, and forming the SMS text with the webpage content corresponding to all the URLs;

[0033] Step A3: The voice text and the SMS text are combined into prompt words and input into the large language model. The large language model extracts key event descriptions, user behavior descriptions, and other contents from the voice text and the SMS text. For example, the key event descriptions may include refund inducement, mention of insurance services, and threats of deductions. The user behavior descriptions may include proactively making calls, claiming one's identity, confirming one's identity, continuously guiding topics, using an outbound call system, manipulating the conversation process, attempting to guide users to respond, delaying time, and repeatedly asking questions.

[0034] Step A4: extract relevant fraud knowledge with high similarity to key event descriptions and user behavior descriptions from the fraud knowledge base through vector similarity retrieval;

[0035] Step A5: Calculate and obtain corresponding call behavior indicators, such as caller ratio, called party dispersion, etc., based on the call records of the fraudulent number or the fraudulent number to be identified.

[0036] In the multimodal fraud data graph of step 2, multimodal data such as user number identification, key event description, user behavior description, related fraud knowledge, SMS website, call behavior indicators, etc. can be respectively used as a node in the graph, and then the nodes are connected according to semantic relevance and time series relationship. Semantic relevance refers to extracting keywords from the web page content corresponding to the key event description, user behavior description, related fraud knowledge or SMS website. When the keywords of two multimodal data match each other, it means that there is a semantic correlation between the two multimodal data. Time series relationship refers to the existence of a time series relationship between multimodal data obtained from the same phone number at different time points.

[0037] In step 3, the open source large language model can be used as the basic model to construct a fraud identification question-answer pair dataset, and then the large language model can be fine-tuned, such as using the QLoRA algorithm, so that the large language model can accurately output whether fraud is involved and the reason analysis for fraud. An embodiment of the large language model fine-tuning process can be as follows:

[0038] 1. Data preparation: First, we need to build a question-answer dataset for fraud identification. This dataset should contain the following elements: positive samples, which are multimodal fusion data extracted from known fraud cases and labeled "is fraud"; negative samples, which are similar data extracted from non-fraud communication records and labeled "not fraud".

[0039] 2. Fine-tune strategy:

[0040] (1) Parameter initialization: Load the pre-trained parameters of the large model.

[0041] (2) Model structure adjustment: According to the QLoRA algorithm, some parameters of the model are adjusted, mainly the layers with high task relevance.

[0042] 3. Training process:

[0043] Train the dataset using question answering.

[0044] The cross entropy loss function is used to calculate the error between the predicted label and the actual label.

[0045] Update model parameters via back-propagation, with a special focus on optimizing low-rank parameters.

[0046] Repeat the training iterations until the model performance stabilizes or the preset number of iterations is reached.

[0047] 4. Model evaluation:

[0048] The fine-tuned large model needs to be evaluated on an independent validation set to test its ability to identify unseen fraud cases. Evaluation indicators include accuracy, recall, F1 score, etc.

[0049] The present invention can also construct a graph neural network model, taking the multimodal fraud data graph as input data. After the training of the graph neural network model is completed, the feature representation vector of each node in the multimodal fraud data graph can be obtained. Since these vectors capture the context information of the node and the relationship with other nodes, they can be used for subsequent fraud identification tasks. The present invention also includes:

[0050] Construct a graph neural network model, input the multimodal fraud data graph into the graph neural network model, and obtain the feature representation vectors of all nodes in the multimodal fraud data graph after training.

[0051] The specific training process of the above graph neural network model can be as follows:

[0052] Initialize the graph neural network model, including initialization of model parameters and feature representation vectors of graph nodes;

[0053] During the training process, the feature representation vectors of the graph nodes are updated through forward propagation, and then the loss function (such as the cross entropy loss function) is calculated to evaluate the difference between the model's prediction results and the true labels. Finally, the model parameters are updated through the back-propagation algorithm to minimize the loss function.

[0054] In order to improve the performance of graph neural networks in node classification tasks, the present invention also designs a new graph neural network model, namely, edge-aware graph convolutional network (E-GCN), which can finely capture the complex relationship between nodes by focusing on the attributes and types of edges, thereby improving classification performance. Among them, the calculation formula of the forward propagation process of E-GCN can be as follows: Among them, H l , H l+1 are the node feature matrices of the lth and l+1th layers respectively, A is the adjacency matrix, I is the identity matrix, yes The degree matrix, W l 、V l are the weight matrices of nodes and edges in the lth layer, respectively. Their values ​​can be obtained through training. E is the edge attribute matrix. An initial value can be set at the beginning of training, and the final value can be obtained after learning. is the transformation matrix of E, and σ is the activation function. E-GCN has the following characteristics:

[0055] 1. Edge attribute encoding: We introduce learnable encodings (i.e., E) for each type of edge, which can capture and express the characteristics of the edge.

[0056] 2. Edge weight aggregation: Combine the attributes of the edge with the node features, and use a weighted mechanism to aggregate the features of adjacent nodes to form a new node feature representation.

[0057] 3. Multi-scale feature fusion: The original node features are fused with the adjacent node features weighted by edge attributes to enhance the model's perception of the local graph structure.

[0058] In step 4, selecting a number of nodes from the multimodal fraud data graph may further include:

[0059] User behavior descriptions are extracted from the multimodal data of the fraudulent numbers to be identified, and the correlation scores of the extracted user behavior descriptions and each node in the multimodal fraud data graph are calculated, so as to select several nodes with high correlation scores.

[0060] Taking node n as an example, calculating the correlation score between the extracted user behavior description q and node n in the multimodal fraud data graph may further include:

[0061] Step C1, calculate the graph structure importance value GSIS(n) of node n: GSIS(n) = α*PR(n) + β*BC(n) + γ*TypeScore(n), where GSIS(n) reflects the importance of node n in the entire graph structure, PR(n) and BC(n) are the PageRank (i.e., page ranking) and Betweenness Centrality (i.e., betweenness centrality) values ​​of node n, respectively, which can be calculated using the centrality measure of the node, TypeScore(n) is a predefined type importance score based on the type of the node (i.e., the type of multimodal data corresponding to the node, including user number identification, key event description, behavior description, related fraud knowledge, SMS URL, call behavior indicator, etc.), α, β, γ are adjustment weights, and their values ​​can be set according to the actual application scenario;

[0062] Step C2: Calculate the text similarity TSS(q,n) between the user behavior description q and the node n:

[0063] TSS(q,n)=cos(vec TF-IDF (q),vec G (n)), where vec TF-IDF (q) is the vector that converts q into TF-IDF (term frequency–inverse document frequency) weights, vec G (n) is the feature representation vector of node n in the multimodal fraud data graph, and cos(·) represents the calculated cosine similarity;

[0064] Step C3, calculate the correlation score CRS(q,n) between the user behavior description q and the node n: CRS(q,n)=δ*TSS(q,n)+(1-δ)*GSIS(n), where δ is a parameter between 0 and 1, used to balance the influence of text similarity and graph structure importance.

[0065] like Figure 3 As shown, the present invention provides a fraudulent number identification system with multimodal data fusion, comprising:

[0066] A multimodal data graph construction device is used to obtain multimodal data of several fraudulent numbers from historical fraud data, and then construct a multimodal fraud data graph, wherein the multimodal fraud data graph uses each multimodal data of each fraudulent number in the historical fraud data as a node, and constructs a connection edge between corresponding nodes according to the association relationship between each two multimodal data;

[0067] A large language model construction device, used to construct and fine-tune a large language model, wherein the large language model is used to determine and output whether fraud is involved and an analysis of the reasons for determining fraud based on input multimodal data;

[0068] A fraud identification device is used to obtain multimodal data of a fraud number to be identified, select several nodes from a multimodal fraud data graph, and then combine the multimodal data of the fraud number to be identified and the multimodal data corresponding to the selected nodes into prompt words, which are input into a large language model, so as to obtain an output of the large language model to determine whether the fraud number to be identified involves fraud and an analysis of the reasons for being determined to be fraud.

[0069] See also Figure 4 , Figure 4 4 is a block diagram of a computing device 400 shown in an exemplary embodiment of this specification. The components of the computing device 400 include but are not limited to a memory 410 and a processor 420. The processor 420 is connected to the memory 410 via a bus 430, and the database 450 is used to store data.

[0070] The computing device 400 also includes an access device 440 that enables the computing device 400 to communicate via one or more networks 460. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 440 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and the like.

[0071] In one embodiment of the present specification, the above components of the computing device 400 and Figure 4 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 4The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0072] The computing device 400 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a PC. The computing device 400 may also be a mobile or stationary server or a cloud server, etc.

[0073] The processor 420 is used to execute the following computer executable instructions, which, when executed by the processor, implement the steps of the above-mentioned fraudulent number identification method using multimodal data fusion.

[0074] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the fraudulent number identification method of multimodal data fusion are of the same concept, and the details not described in detail in the technical scheme of the computing device can be found in the description of the technical scheme of the fraudulent number identification method of multimodal data fusion.

[0075] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned fraudulent number identification method using multimodal data fusion.

[0076] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the above-mentioned fraudulent number identification method of multimodal data fusion belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the fraudulent number identification method or system of multimodal data fusion.

[0077] An embodiment of the present specification also provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned fraudulent number identification method using multimodal data fusion.

[0078] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the fraudulent number identification method of multimodal data fusion are of the same concept, and the details not described in detail in the technical scheme of the computer program can be found in the description of the technical scheme of the fraudulent number identification method or system of multimodal data fusion.

[0079] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0080] The computer instructions include computer program codes, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0081] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0082] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A fraud number identification method based on multimodal data fusion, characterized in that: Included are: Step 1: Obtain multimodal data of several fraud numbers from historical fraud data; Step 2: Construct a multimodal fraud data graph. The multimodal fraud data graph uses each multimodal data of each fraud number in the historical fraud data as a node, and constructs a connection edge between corresponding nodes based on the correlation between each two multimodal data; Step 3: Build and fine-tune a large language model, where the large language model is used to determine and output whether fraud is involved and the reason for determining it as fraud based on the input multimodal data; Step 4. Obtain the multimodal data of the fraudulent number to be identified, and select several nodes from the multimodal fraud data graph. Then, the multimodal data of the fraudulent number to be identified and the multimodal data corresponding to the selected nodes are combined into prompt words, which are input into the large language model to obtain the output of the large language model to determine whether the fraudulent number to be identified involves fraud and the reason analysis for being judged as fraud.

2. The method according to claim 1, characterized in that: Multimodal data includes: user number identification, key event description, user behavior description, relevant fraud knowledge, SMS URL, and call behavior indicators.

3. The method according to claim 2, characterized in that In step 1 or step 4, multimodal data of a fraudulent number or a fraudulent number to be identified is obtained, including: Step A1: obtaining a call media evidence file of a fraudulent number or a fraudulent number to be identified, and performing voice recognition to obtain a corresponding voice text; Step A2: obtaining the SMS text of the fraudulent number or the fraudulent number to be identified, extracting all the URLs in the SMS text, and obtaining the webpage content corresponding to the URLs, and forming the SMS text with the webpage content corresponding to all the URLs; Step A3: The voice text and the SMS text are combined into prompt words, which are input into the large language model. The large language model extracts key event descriptions and user behavior descriptions from the voice text and the SMS text. Step A4: extract relevant fraud knowledge with high similarity to key event descriptions and user behavior descriptions from the fraud knowledge base through vector similarity retrieval; Step A5: Calculate and obtain corresponding call behavior indicators based on the call records of the fraudulent number or the fraudulent number to be identified.

4. The method according to claim 1, characterized in that The association relationship between multimodal data includes semantic correlation and time series relationship. Semantic correlation refers to extracting keywords from the description of key events, user behavior descriptions, related fraud knowledge or the web page content corresponding to the SMS URL. When the keywords of two multimodal data match each other, there is semantic correlation between the two multimodal data. Time series relationship refers to the existence of a time series relationship between multimodal data obtained from the same phone number at different time points.

5. The method according to claim 1, characterized in that Also included are: A graph neural network model is constructed, and the multimodal fraud data graph is input into the graph neural network model. After training, the feature representation vectors of all nodes in the multimodal fraud data graph are obtained. The calculation formula of the forward propagation process of the graph neural network model is as follows: Among them, H l , H l+1 are the node feature matrices of the lth and l+1th layers respectively, A is the adjacency matrix, I is the identity matrix, yes The degree matrix, W l 、V l are the weight matrices of nodes and edges in the lth layer, E is the edge attribute matrix, is the transformation matrix of E and σ is the activation function.

6. The method according to claim 1, characterized in that In step 4, a number of nodes are selected from the multimodal fraud data graph, further including: User behavior descriptions are extracted from the multimodal data of the fraudulent numbers to be identified, and the correlation scores of the extracted user behavior descriptions and each node in the multimodal fraud data graph are calculated, so as to select several nodes with high correlation scores.

7. The method according to claim 6, characterized in that Calculating the correlation score between the extracted user behavior description q and the node n in the multimodal fraud data graph further includes: Step C1, calculate the graph structure importance value GSIS(n) of node n: GSIS(n) = α*PR(n) + β*BC(n) + γ*TypeScore(n), where PR(n) and BC(n) are the PageRank and Betweenness Centrality values ​​of node n, respectively, TypeScore(n) is the type importance score predefined according to the type of the node, and α, β, and γ are adjustment weights; Step C2: Calculate the text similarity TSS(q,n) between the user behavior description q and the node n: TSS(q,n)=cos(vec TF-IDF (q),vec G (n)), where vec TF-IDF (q) is the vector that converts q into TF-IDF weights, vec G (n) is the feature representation vector of node n in the multimodal fraud data graph, and cos(·) represents the calculation of cosine similarity; Step C3, calculate the correlation score CRS(q,n) between the user behavior description q and the node n: CRS(q,n)=δ*TSS(q,n)+(1-δ)*GSIS(n), where δ is a parameter between 0 and 1.

8. A fraudulent number identification system based on multimodal data fusion, characterized in that: Included are: A multimodal data graph construction device is used to obtain multimodal data of several fraudulent numbers from historical fraud data, and then construct a multimodal fraud data graph, wherein the multimodal fraud data graph uses each multimodal data of each fraudulent number in the historical fraud data as a node, and constructs a connection edge between corresponding nodes according to the association relationship between each two multimodal data; A large language model construction device, used to construct and fine-tune a large language model, wherein the large language model is used to determine and output whether fraud is involved and an analysis of the reasons for determining fraud based on input multimodal data; A fraud identification device is used to obtain multimodal data of a fraud number to be identified, select several nodes from a multimodal fraud data graph, and then combine the multimodal data of the fraud number to be identified and the multimodal data corresponding to the selected nodes into prompt words, which are input into a large language model, so as to obtain an output of the large language model to determine whether the fraud number to be identified involves fraud and an analysis of the reasons for being determined to be fraud.

9. A computing device, characterized in that include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the fraudulent number identification method using multimodal data fusion as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: It stores computer executable instructions, which, when executed by a processor, implement the steps of the fraudulent number identification method using multimodal data fusion as described in any one of claims 1 to 7.

Images