Multi-modal data-oriented credit large model joint modeling and collaborative reasoning method and device

Abstract

The application discloses a credit large model joint modeling and collaborative reasoning method, a credit large model joint modeling and collaborative reasoning device, a credit large model joint modeling and collaborative reasoning equipment and a computer readable storage medium for multi-modal data, and the method comprises the following steps: acquiring multi-modal credit data from a public credit information center and performing data preprocessing to obtain a multi-modal data set; performing feature vector extraction on the multi-modal data set based on an encoder corresponding to a modal data type to obtain an initial feature representation vector; performing collaborative alignment and fusion processing on a text feature vector, a structured feature vector and a graph feature vector based on a multi-head cross-modal attention layer to obtain a multi-modal representation vector; and inputting the multi-modal representation vector into a credit collaborative reasoning decision model for collaborative reasoning to obtain a reasoning result. Deep fusion of multi-modal credit data is realized, and the accuracy and depth of evaluation are improved.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and more specifically, relates to a method for joint modeling and collaborative reasoning of a large credit model for multimodal data, a device for joint modeling and collaborative reasoning of a large credit model for multimodal data, an equipment for joint modeling and collaborative reasoning of a large credit model, and a computer-readable storage medium. Background Technology

[0002] The public credit information center gathers massive amounts of corporate or individual credit data from various sources. This data typically exhibits multimodal characteristics, including structured business registration information and financial statement data, unstructured administrative penalty decisions, judicial judgments, related news and public opinion text data, as well as graph data composed of investment relationships, guarantee relationships, and senior management appointment relationships between enterprises.

[0003] In related technologies, the comprehensive evaluation of these multimodal credit data typically faces two main problems: Traditional credit assessment systems often employ a divide-and-conquer strategy. For example, they use independent natural language processing modules to analyze textual materials, independent risk control rule engines or machine learning models to process structured financial data, and graph computing engines to analyze relationship networks. Each module works independently, and its analysis results are only simply weighted or summarized in the final stage. This approach ignores the deep correlations between different modalities of data. For instance, the true impact of a negative news article (text modality) may require combining a company's financial situation (structured modality) and its core position in the industry chain (graph modality) to accurately determine. Due to the lack of early and deep cross-modal integration, the assessment model cannot form a comprehensive and three-dimensional understanding of the credit subject, resulting in limited accuracy and depth of the assessment results.

[0004] Therefore, how to break down modal barriers, achieve deep integration of multimodal credit data, and improve the accuracy and depth of assessment is a technical challenge that urgently needs to be solved in the field of public credit assessment. Summary of the Invention

[0005] The purpose of this application is to provide a method for joint modeling and collaborative reasoning of large-scale credit models for multimodal data, a device for joint modeling and collaborative reasoning of large-scale credit models for multimodal data, a computer-readable storage medium for joint modeling and collaborative reasoning of large-scale credit models, in order to break down modal barriers, achieve deep integration of multimodal credit data, and improve the accuracy and depth of assessment.

[0006] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a method for joint modeling and collaborative reasoning of large-scale credit models for multimodal data, comprising:

[0007] Multimodal credit data is obtained from the public credit information center and preprocessed to obtain a multimodal dataset; wherein the multimodal dataset includes: a text corpus, a structured feature table, and a correlation graph; wherein the multimodal credit data includes: text data, structured data, and graph data;

[0008] The encoder corresponding to the modal data type extracts feature vectors from the multimodal data set to obtain an initial feature representation vector; wherein, the initial feature representation vector includes: text feature vector, structured feature vector, and graph feature vector;

[0009] Based on a multi-head cross-modal attention layer, the text feature vector, the structured feature vector, and the graph feature vector are collaboratively aligned and fused to obtain a multimodal representation vector;

[0010] The multimodal representation vector is input into the credit collaborative reasoning decision model for collaborative reasoning to obtain the reasoning result; wherein, the credit collaborative reasoning decision model is an evaluation model trained based on historical credit data; wherein, the reasoning result includes: credit evaluation result and decision basis data.

[0011] Optionally, multimodal credit data can be obtained from a public credit information center and preprocessed to obtain a multimodal dataset, including:

[0012] Unstructured text data is obtained from a first preset path, and the unstructured text data is cleaned to obtain the text corpus; wherein, the first preset path includes: administrative penalty decisions, court judgments, records of dishonest persons subject to enforcement, bidding information, news reports, and operating conditions descriptions in corporate annual reports;

[0013] Structured data is obtained from a second preset path, and the structured data is cleaned to obtain the structured feature table; wherein, the second preset path includes: enterprise business registration information, financial statements, number of social security contributors, and number of intellectual property rights;

[0014] The graph data is obtained from the third preset path, and the graph data is structurally cleaned to obtain the association graph; wherein, the third preset path includes: equity investment chain, external guarantee chain, senior executive cross-appointment network, and supplier customer relationship;

[0015] The text corpus, the structured feature table, and the association graph are integrated to obtain the multimodal data set.

[0016] Optionally, feature vectors are extracted from the multimodal data set based on the encoder corresponding to the modal data type to obtain an initial feature representation vector, including:

[0017] A pre-trained domain-specific large language model is used as the text encoder;

[0018] The text corpus is input into the text encoder for feature vector extraction to obtain the text feature vector;

[0019] A multilayer perceptron is used as a structured data encoder;

[0020] The structured feature table is input into the structured data encoder for feature vector extraction to obtain the structured feature vector;

[0021] A graph neural network is used as the graph encoder;

[0022] The associated graph is input into the graph encoder to obtain the graph feature vector;

[0023] The text feature vector, the structured feature vector, and the graph feature vector are used as the initial feature representation vector.

[0024] Optionally, the text feature vector, the structured feature vector, and the graph feature vector are collaboratively aligned and fused based on a multi-head cross-modal attention layer to obtain a multimodal representation vector, including:

[0025] Based on the multi-head cross-modal attention layer, the text feature vector, the structured feature vector, and the graph feature vector are aligned to obtain updated text feature vectors, updated structured feature vectors, and updated graph feature vectors;

[0026] The updated text feature vector, the updated structured feature vector, and the updated graph feature vector are concatenated to obtain the multimodal representation vector.

[0027] Optionally, the multimodal representation vector is input into the credit collaborative reasoning decision model for collaborative reasoning to obtain the reasoning result, including:

[0028] The multimodal representation vector is input into the credit collaborative reasoning decision model for collaborative reasoning; wherein, the output head of the credit collaborative reasoning decision model includes: a classifier and a text generator;

[0029] When the reasoning is complete, the classifier outputs the credit assessment result, and the text generator outputs the decision basis data.

[0030] This application also provides a joint modeling and collaborative reasoning device for credit large-scale models oriented towards multimodal data, including:

[0031] The data processing module is used to obtain multimodal credit data from the public credit information center and perform data preprocessing to obtain a multimodal dataset; wherein, the multimodal dataset includes: a text corpus, a structured feature table, and a correlation graph; wherein, the multimodal credit data includes: text data, structured data, and graph data;

[0032] The vector extraction module is used to extract feature vectors from the multimodal data set based on the encoder corresponding to the modal data type to obtain an initial feature representation vector; wherein, the initial feature representation vector includes: text feature vector, structured feature vector, and graph feature vector;

[0033] The collaborative fusion module is used to perform collaborative alignment and fusion processing on the text feature vector, the structured feature vector, and the graph feature vector based on a multi-head cross-modal attention layer to obtain a multimodal representation vector;

[0034] The collaborative reasoning module is used to input the multimodal representation vector into the credit collaborative reasoning decision model for collaborative reasoning to obtain the reasoning result; wherein, the credit collaborative reasoning decision model is an evaluation model trained based on historical credit data; wherein, the reasoning result includes: credit evaluation result and decision basis data.

[0035] Optionally, the data processing module is specifically used to obtain unstructured text data from a first preset path and perform data cleaning on the unstructured text data to obtain the text corpus; wherein, the first preset path includes: administrative penalty decisions, court judgments, records of dishonest persons subject to enforcement, bidding information, news reports, and operating status descriptions in enterprise annual reports; obtain structured data from a second preset path and perform structure cleaning on the structured data to obtain the structured feature table; wherein, the second preset path includes: enterprise business registration information, financial statements, number of social security contributors, and number of intellectual property rights; obtain graph data from a third preset path and perform structure cleaning on the graph data to obtain the association graph; wherein, the third preset path includes: equity investment chains, external guarantee chains, cross-appointment networks of senior executives, and supplier-customer relationships; and integrate the text corpus, the structured feature table, and the association graph to obtain the multimodal data set.

[0036] Optionally, the data processing module is specifically configured to: employ a pre-trained domain-specific large language model as a text encoder; input the text corpus into the text encoder for feature vector extraction to obtain the text feature vector; employ a multilayer perceptron as a structured data encoder; input the structured feature table into the structured data encoder for feature vector extraction to obtain the structured feature vector; employ a graph neural network as a graph encoder; input the association graph into the graph encoder to obtain the graph feature vector; and use the text feature vector, the structured feature vector, and the graph feature vector as the initial feature representation vector.

[0037] This application also provides a joint modeling and collaborative reasoning device for a large credit model, including:

[0038] Memory, used to store computer programs;

[0039] A processor is used to execute the computer program to implement the steps of the credit large model joint modeling and collaborative reasoning method as described above.

[0040] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the credit large model joint modeling and collaborative reasoning method described above.

[0041] This application provides a method for joint modeling and collaborative reasoning of a large-scale credit model for multimodal data, comprising: acquiring multimodal credit data from a public credit information center and performing data preprocessing to obtain a multimodal data set; wherein the multimodal data set includes: a text corpus, a structured feature table, and a correlation graph; wherein the multimodal credit data includes: text data, structured data, and graph data; extracting feature vectors from the multimodal data set based on encoders corresponding to the modal data types to obtain initial feature representation vectors; wherein the initial feature representation vectors include: text feature vectors, structured feature vectors, and graph feature vectors; performing collaborative alignment and fusion processing on the text feature vectors, structured feature vectors, and graph feature vectors based on a multi-head cross-modal attention layer to obtain a multimodal representation vector; inputting the multimodal representation vectors into a credit collaborative reasoning decision model for collaborative reasoning to obtain a reasoning result; wherein the credit collaborative reasoning decision model is an evaluation model trained based on historical credit data; wherein the reasoning result includes: credit evaluation results and decision basis data.

[0042] It has the following beneficial effects:

[0043] This invention enhances the comprehensiveness and accuracy of assessments: Through a cross-modal collaborative alignment and fusion module, it achieves deep interaction of heterogeneous data such as text, structured data, and graphs at the feature level, effectively breaking down "modal silos." The model can discover and utilize complementary and corroborative relationships between different modalities to form a comprehensive 360-degree understanding of the credit subject, thereby significantly improving the accuracy and robustness of credit assessment results.

[0044] This invention enhances the depth and breadth of risk identification: Utilizing domain-wide models and deep learning techniques such as graph neural networks, it can automatically learn and extract deep semantic features and complex relational features from massive amounts of data, achieving a leap from "shallow feature statistics" to "deep semantic and structural reasoning." This enables the model to identify hidden risk patterns and transmission paths that are difficult to discover using traditional methods, improving the foresight and accuracy of risk warnings.

[0045] This invention achieves a high degree of interpretability in assessment decisions: the collaborative reasoning decision-making module can simultaneously generate natural language explanations based on key evidence while outputting credit assessment results. This "result + evidence" output mode makes the complex black-box model decision-making process transparent and traceable, which not only enhances the persuasiveness of the assessment conclusions but also greatly meets the compliance requirements of financial regulation, facilitating manual review and auditing. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0047] Figure 1 A flowchart illustrating a joint modeling and collaborative reasoning method for a large credit model oriented towards multimodal data, provided in an embodiment of this application;

[0048] Figure 2 This is a schematic diagram of the structure of a joint modeling and collaborative reasoning device for a large credit model oriented to multimodal data, provided in an embodiment of this application.

[0049] Figure 3 This is a schematic diagram of the structure of the credit large model joint modeling and collaborative reasoning device provided in the embodiments of this application. Detailed Implementation

[0050] The purpose of this application is to provide a method for joint modeling and collaborative reasoning of large-scale credit models for multimodal data, a device for joint modeling and collaborative reasoning of large-scale credit models for multimodal data, a computer-readable storage medium for joint modeling and collaborative reasoning of large-scale credit models, in order to break down modal barriers, achieve deep integration of multimodal credit data, and improve the accuracy and depth of assessment.

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0052] The following example illustrates a joint modeling and collaborative reasoning method for a large credit model oriented towards multimodal data provided in this application.

[0053] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a joint modeling and collaborative reasoning method for a large credit model oriented towards multimodal data, provided in an embodiment of this application.

[0054] In this embodiment, the method may include:

[0055] S101. Obtain multimodal credit data from the public credit information center and perform data preprocessing to obtain a multimodal dataset; wherein, the multimodal dataset includes: a text corpus, a structured feature table, and a correlation graph; wherein, the multimodal credit data includes: text data, structured data, and graph data;

[0056] This step aims to acquire multimodal credit data from the public credit information center and perform data preprocessing to obtain a multimodal dataset. The public credit information center collects massive amounts of heterogeneous raw data about credit subjects. Direct use of this data presents problems such as inconsistent formats, information redundancy, data noise, and structural deficiencies, making it unsuitable for subsequent deep learning models. Therefore, systematic data acquisition and preprocessing are necessary to transform this raw, disorganized data into a structured, standardized, and information-complete multimodal dataset. This step transforms the diverse and disorganized raw credit information into a standardized, high-quality multimodal dataset. This dataset provides a unified, reliable, and information-rich data foundation for all subsequent analytical steps, fundamentally ensuring the data quality of the entire credit assessment method and serving as a prerequisite for improving the accuracy and stability of the final assessment results.

[0057] Furthermore, this step may include:

[0058] First, the system initiates a data acquisition process based on a given unique identifier for a credit entity. This process concurrently accesses databases or file systems in different storage locations within the public credit information center. Specifically, the system retrieves text data from a first preset path (e.g., a database or document server storing unstructured documents such as legal documents, administrative penalties, and news and public opinion). After acquisition, this text data undergoes data cleaning, including removing HTML tags, advertising links, and non-textual special symbols, segmenting into paragraphs and sentences, and normalizing the documents, ultimately forming a clean text corpus.

[0059] Simultaneously, the system retrieves structured data from a second preset path (e.g., a structured database table storing enterprise business registration, financial, and tax information). After retrieval, this data undergoes structure cleaning, which includes handling missing values, identifying and processing outliers, normalizing or standardizing continuous numerical features, and performing one-hot encoding on categorical features, ultimately forming a numerical structured feature table.

[0060] In addition, the system also retrieves graph data from a third pre-defined path (e.g., a graph database or relational database specifically storing enterprise relationships). This data describes the relationships between entities, such as equity investments, external guarantees, and executive appointments. After retrieval, the data undergoes structural cleaning, primarily involving the construction or verification of the graph structure. This ensures that the definitions of nodes (representing enterprises or individuals) and edges (representing relationships) are clear and accurate, and assigns initial attributes to nodes and edges (e.g., the registered capital of nodes, the investment amount of edges), ultimately forming a relationship graph that can be used by the graph computing model.

[0061] Finally, the text corpus, structured feature table, and association graph obtained from the above processing are integrated to establish their association with the same credit subject identifier, forming a complete multimodal dataset.

[0062] S102, based on the encoder corresponding to the modal data type, feature vectors are extracted from the multimodal data set to obtain an initial feature representation vector; wherein, the initial feature representation vector includes: text feature vector, structured feature vector, and graph feature vector;

[0063] Building upon S101, this step aims to extract feature vectors from the multimodal data set based on the encoder corresponding to the modal data type, thereby obtaining an initial feature representation vector.

[0064] Among them, due to the huge differences in structure and connotation among the three data modalities of text, tables and graphs, targeted encoders must be used to capture their respective deep features most effectively.

[0065] For example, the value of textual data lies in its contextual semantics, the value of structured data lies in the numerical relationships between features, and the value of graph data lies in its network topology and the interactions between nodes. Using a single model to process all data inevitably leads to information loss. Therefore, matching the most suitable encoder for each modality of data is crucial for achieving high-precision feature extraction. This step efficiently and losslessly transforms data from different modalities from their original form into a unified mathematical representation in a high-dimensional vector space—the initial feature representation vector. These vectors not only condense the original information but also, through deep learning models, uncover subtle underlying patterns and deep semantics, providing high-quality, high-information-density input for subsequent cross-modal information fusion—a technological prerequisite for breaking down modal silos.

[0066] Furthermore, this step may include:

[0067] After obtaining the multimodal dataset produced in the previous step, the system encodes its three components in parallel. Specifically, for the text corpus, the system employs a pre-trained domain-specific large language model (e.g., a BERT or RoBERTa model fine-tuned on massive amounts of financial and legal text) as the text encoder. In implementation, each key document in the text corpus (such as the full text of a penalty decision) is input into this encoder. The model's internal Transformer structure, through multi-layered self-attention mechanisms, captures complex long-distance dependencies and contextual semantics within the text, not just keywords. After processing, the model outputs a fixed-dimensional text feature vector, which can be considered a highly condensed representation of the entire document's semantics.

[0068] For the structured feature table, the system employs a multilayer perceptron (MLP) as the structured data encoder. In implementation, a row of numerical features (already normalized and encoded) representing the credit subject in the structured feature table is input as a vector into the MLP. Through a series of fully connected layers with non-linear activation functions, the MLP learns the complex non-linear combinations of different numerical features, which is far more powerful than traditional linear models or rule engines. Finally, the output of the last hidden layer of the MLP is the structured feature vector.

[0069] For the relational graph, the system employs a graph neural network as the graph encoder. In implementation, the entire relational graph is input into the graph neural network. The core working mechanism of the graph neural network is message passing; at each layer, each node aggregates information from its neighbors to update its own representation. After multiple iterations, the final representation vector of the target node (i.e., the graph feature vector) not only contains its own attributes but also encodes its position, influence, and features of its surrounding environment within the entire relational network.

[0070] Finally, the system integrates the three separately generated text feature vectors, structured feature vectors, and graph feature vectors to form an initial feature representation vector set, ready for the next step of fusion processing.

[0071] S103, based on a multi-head cross-modal attention layer, performs collaborative alignment and fusion processing on text feature vectors, structured feature vectors, and graph feature vectors to obtain multimodal representation vectors;

[0072] Building upon S102, this step aims to perform collaborative alignment and fusion processing on text feature vectors, structured feature vectors, and graph feature vectors based on a multi-head cross-modal attention layer to obtain a multimodal representation vector.

[0073] It's important to note that when humans evaluate something, they naturally connect information from different senses to make a comprehensive judgment. For example, hearing a negative description (auditory / textual) is combined with seeing financial statements (visual / structured) to determine its severity. This step mathematically achieves this cross-information channel association and calibration by introducing a multi-head cross-modal attention layer.

[0074] As can be seen, the purpose of this step is to break down the independent modal silos between the initial feature representation vectors generated in the previous step, allowing feature information from different modalities to interact, influence, and corroborate each other, thereby generating a unified and higher-order multimodal representation vector that truly integrates information from all modalities. This fusion process is not a simple splicing, but a process of deep interaction and information gain. Through the collaborative alignment and fusion processing in this step, the modal silo problem in traditional methods can be effectively broken, enabling the model to establish connections between information from different sources, perform cross-validation, and make comprehensive judgments, just like human experts. The final generated multimodal representation vector is a digital profile of the credit subject with extremely high information density and rich connotations. It provides an unprecedented comprehensive perspective for subsequent accurate reasoning and decision-making, representing a key technological leap in improving the depth and accuracy of assessment.

[0075] Furthermore, this step may include:

[0076] After receiving the text feature vector (V_text), structured feature vector (V_struct), and graph feature vector (V_graph) produced in the previous step, the system inputs them into one or more multi-head cross-modal attention layers. This process mainly includes two stages: information alignment and fusion.

[0077] During the information alignment phase, taking the enhancement of text feature vectors as an example, the system uses V_text as the query in the attention mechanism, and V_struct and V_graph as the "key" and "value." The attention mechanism calculates the relevance score between each dimension (representing a semantic feature) in V_text and all dimensions in V_struct and V_graph. This score determines how much content should be "borrowed" from structured and graph information when generating new text vectors. For example, if the text mentions "tight cash flow," the attention mechanism might pay close attention to the dimensions representing "cash flow" and "debt ratio" in the structured vector, and the dimension representing "amount of external guarantees" in the graph vector. By weighted summing of the key and value, the system generates an updated text feature vector that now "perceives" information at the financial and network relationship levels.

[0078] Similarly, the system can use V_struct as the query and V_text and V_graph as the key / value pairs to generate updated structured feature vectors; and use V_graph as the query to generate updated graph feature vectors. This alignment process can iterate multiple times to achieve deeper information interaction.

[0079] During the fusion phase, the system concatenates all updated feature vectors (i.e., updated text feature vectors, updated structured feature vectors, and updated graph feature vectors) along the feature dimensions. For example, if the dimensions of the three updated vectors are all 256, the concatenation will result in a 768-dimensional vector. This long vector is the final multimodal representation vector, each part of which has undergone sufficient interaction with other modalities, and its whole contains comprehensive, three-dimensional, and deeply related information about the credit subject.

[0080] S104, input the multimodal representation vector into the credit collaborative reasoning decision model for collaborative reasoning to obtain the reasoning result; wherein, the credit collaborative reasoning decision model is an evaluation model trained based on historical credit data; wherein, the reasoning result includes: credit evaluation result and decision basis data.

[0081] Building upon S103, this step aims to input the multimodal representation vector into the credit collaborative reasoning decision model for collaborative reasoning, and obtain the reasoning results.

[0082] This credit collaborative reasoning decision-making model is essentially a powerful function that learns complex patterns from multidimensional features to credit outcomes, derived from massive amounts of historical data. Simply providing a credit score or rating is insufficient, especially in the highly regulated field of finance, where interpretability is crucial. Therefore, another key principle of this step is multi-task learning: while performing the main assessment task, the model must simultaneously execute an auxiliary task of interpretation generation, ensuring that every assessment result is traceable. Through collaborative reasoning and decision generation in this step, not only can high-precision credit assessment results be produced, but the "black box" problem of deep learning models in credit assessment is also revolutionaryly solved. Its output is no longer cold, hard numbers, but a complete decision scheme containing clear conclusions and detailed, traceable evidence. This greatly enhances the credibility, usability, and compliance of the assessment results, enabling automated assessment systems to be truly applied in core risk control scenarios.

[0083] Furthermore, this step may include:

[0084] First, a pre-trained credit collaborative reasoning decision-making model is required. This model is typically built on the Transformer architecture or other deep neural networks and is obtained through supervised learning on a large amount of historical credit data (each sample contains multimodal data of a credit subject and its known final credit status, such as whether it defaulted, credit rating, etc.). After training, this step is performed as follows: the multimodal representation vector produced in the previous step is used as input and fed into the reasoning decision-making model.

[0085] The inference layers within the model (such as multiple Transformer decoder layers) perform further nonlinear transformations and feature interactions on this high-dimensional vector, simulating a complex logical reasoning process. As defined in claim 5, the output of the model is designed to have two parallel output heads.

[0086] The first output head is the classifier, which is typically a fully connected layer with a softmax activation function. It is responsible for mapping the model's final hidden states to a probability distribution of predefined credit ratings (such as AAA, AA, A, B, C, etc.), and the rating with the highest probability is determined as the credit rating result for this evaluation.

[0087] The second output head is a text generator, which is typically an autoregressive language model decoder. During training, this generator learns to produce a fluent and accurate natural language explanation based on the input features and the final evaluation result.

[0088] During reasoning, to generate a well-founded explanation, the text generator specifically utilizes the attention weights calculated in previous steps. The system identifies raw data fragments that receive the highest attention scores—this could be a key legal provision (from the text modality), an unusual financial ratio (from the structured modality), or a high-risk related party (from the graph modality). The text generator then uses these identified "key pieces of evidence" as core content to organize and generate a logically clear explanatory text, serving as the basis for decision-making. Finally, the system outputs the results from both outputs—the credit assessment result and the decision-making basis data—completing the entire assessment process.

[0089] In summary, this embodiment integrates heterogeneous data from multiple sources, including text corpora, structured feature tables, and association graphs, to construct a comprehensive and multi-dimensional data view of credit subjects, fundamentally overcoming the limitations and biases of traditional assessment methods that rely on a single data source. By employing targeted encoders to extract deep features from each modality and utilizing a multi-head cross-modal attention mechanism to achieve collaborative alignment and deep fusion of features, it effectively breaks down modal silos, enabling information from different dimensions to corroborate and complement each other, thereby generating multi-modal representation vectors with higher information density and richer connotations. Finally, the collaborative reasoning decision-making model not only outputs accurate credit assessment results but also simultaneously generates traceable decision-making evidence, solving the black-box problem of deep learning models and significantly improving the accuracy, transparency, and credibility of credit assessment.

[0090] The following describes a joint modeling and collaborative reasoning device for a large credit model oriented to multimodal data, provided by an embodiment of this application. The joint modeling and collaborative reasoning device for a large credit model oriented to multimodal data and the joint modeling and collaborative reasoning method for a large credit model oriented to multimodal data described below can be referred to in correspondence with each other.

[0091] Please refer to Figure 2 , Figure 2 This is a schematic diagram of the structure of a joint modeling and collaborative reasoning device for a large credit model oriented to multimodal data, provided in an embodiment of this application.

[0092] In this embodiment, the device may include:

[0093] The data processing module 100 is used to obtain multimodal credit data from the public credit information center and perform data preprocessing to obtain a multimodal dataset; wherein, the multimodal dataset includes: a text corpus, a structured feature table, and a correlation graph; wherein, the multimodal credit data includes: text data, structured data, and graph data;

[0094] The vector extraction module 200 is used to extract feature vectors from the multimodal data set based on the encoder corresponding to the modal data type to obtain an initial feature representation vector; wherein, the initial feature representation vector includes: text feature vector, structured feature vector, and graph feature vector;

[0095] The collaborative fusion module 300 is used to perform collaborative alignment and fusion processing on text feature vectors, structured feature vectors, and graph feature vectors based on a multi-head cross-modal attention layer to obtain multimodal representation vectors;

[0096] The collaborative reasoning module 400 is used to input multimodal representation vectors into the credit collaborative reasoning decision model for collaborative reasoning and to obtain reasoning results. The credit collaborative reasoning decision model is an evaluation model trained based on historical credit data. The reasoning results include credit evaluation results and decision basis data.

[0097] This application also provides a device for joint modeling and collaborative inference of large-scale credit models; please refer to [reference needed]. Figure 3 , Figure 3 This is a schematic diagram of the structure of the credit large-scale model joint modeling and collaborative reasoning device provided in the embodiments of this application. The credit large-scale model joint modeling and collaborative reasoning device may include:

[0098] Memory, used to store computer programs;

[0099] A processor, used to execute computer programs, can implement the steps of any of the credit large-scale models for multimodal data joint modeling and collaborative reasoning methods described above.

[0100] like Figure 3 The diagram shows the structural composition of a credit large-scale model joint modeling and collaborative reasoning device. This device may include: a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, memory 11, and communication interface 12 all communicate with each other via the communication bus 13.

[0101] In this embodiment, the processor 10 may be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic devices.

[0102] The processor 10 can call the program stored in the memory 11. Specifically, the processor 10 can execute the operations in the embodiment of the abnormal IP identification method.

[0103] The memory 11 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment, the memory 11 stores at least a program for implementing the following functions:

[0104] Multimodal credit data is obtained from the public credit information center and preprocessed to obtain a multimodal dataset. The multimodal dataset includes: a text corpus, a structured feature table, and a correlation graph. The multimodal credit data includes: text data, structured data, and graph data.

[0105] Based on the encoder corresponding to the modal data type, feature vectors are extracted from the multimodal data set to obtain initial feature representation vectors; among which, the initial feature representation vectors include: text feature vectors, structured feature vectors, and graph feature vectors;

[0106] Based on a multi-head cross-modal attention layer, text feature vectors, structured feature vectors, and graph feature vectors are collaboratively aligned and fused to obtain multimodal representation vectors;

[0107] The multimodal representation vector is input into the credit collaborative reasoning decision model for collaborative reasoning to obtain the reasoning result; wherein, the credit collaborative reasoning decision model is an evaluation model trained based on historical credit data; wherein, the reasoning result includes: credit evaluation result and decision basis data.

[0108] In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; and the data storage area may store data created during use.

[0109] In addition, memory 11 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0110] Communication interface 12 can be an interface for the communication module, used to connect with other devices or systems.

[0111] Of course, it should be noted that, Figure 3 The structure shown does not constitute a limitation on the credit large model joint modeling and collaborative reasoning device in the embodiments of this application. In practical applications, the credit large model joint modeling and collaborative reasoning device may include more than Figure 3 More or fewer components as shown, or combinations of certain components.

[0112] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps of any of the above-described credit large model joint modeling and collaborative reasoning methods for multimodal data.

[0113] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0114] For a description of the computer-readable storage medium provided in this application, please refer to the above method embodiments; further details will not be repeated here.

[0115] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0116] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0117] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0118] The foregoing has provided a detailed description of the following: a method, apparatus, device, and computer-readable storage medium for joint modeling and collaborative reasoning of a large-scale credit model for multimodal data, all provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of these embodiments are merely illustrative of the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the scope of protection of the claims of this application.

Claims

1. A joint modeling and collaborative reasoning method for large-scale credit models oriented towards multimodal data, characterized in that, include: Multimodal credit data is obtained from the public credit information center and preprocessed to obtain a multimodal dataset; wherein the multimodal dataset includes: a text corpus, a structured feature table, and a correlation graph; wherein the multimodal credit data includes: text data, structured data, and graph data; The encoder corresponding to the modal data type extracts feature vectors from the multimodal data set to obtain an initial feature representation vector; wherein, the initial feature representation vector includes: text feature vector, structured feature vector, and graph feature vector; Based on a multi-head cross-modal attention layer, the text feature vector, the structured feature vector, and the graph feature vector are collaboratively aligned and fused to obtain a multimodal representation vector; The multimodal representation vector is input into the credit collaborative reasoning decision model for collaborative reasoning to obtain the reasoning result; wherein, the credit collaborative reasoning decision model is an evaluation model trained based on historical credit data; wherein, the reasoning result includes: credit evaluation result and decision basis data; Specifically, the encoder corresponding to the modal data type extracts feature vectors from the multimodal data set to obtain an initial feature representation vector, including: A pre-trained domain-specific large language model is used as the text encoder; The text corpus is input into the text encoder for feature vector extraction to obtain the text feature vector; A multilayer perceptron is used as a structured data encoder; The structured feature table is input into the structured data encoder for feature vector extraction to obtain the structured feature vector; A graph neural network is used as the graph encoder; The associated graph is input into the graph encoder to obtain the graph feature vector; The text feature vector, the structured feature vector, and the graph feature vector are used as the initial feature representation vector; Specifically, the text feature vector, the structured feature vector, and the graph feature vector are collaboratively aligned and fused based on a multi-head cross-modal attention layer to obtain a multimodal representation vector, including: Based on the multi-head cross-modal attention layer, the text feature vector, the structured feature vector, and the graph feature vector are aligned to obtain updated text feature vectors, updated structured feature vectors, and updated graph feature vectors; The updated text feature vector, the updated structured feature vector, and the updated graph feature vector are concatenated to obtain the multimodal representation vector; Specifically, the multimodal representation vector is input into the credit collaborative reasoning decision model for collaborative reasoning to obtain the reasoning result, including: The multimodal representation vector is input into the credit collaborative reasoning decision model for collaborative reasoning; wherein, the output head of the credit collaborative reasoning decision model includes: a classifier and a text generator; When the reasoning is complete, the classifier outputs the credit assessment result, and the text generator outputs the decision basis data.

2. The credit large-scale model joint modeling and collaborative reasoning method according to claim 1, characterized in that, Multimodal credit data is obtained from the public credit information center and preprocessed to obtain a multimodal dataset, including: Unstructured text data is obtained from a first preset path, and the unstructured text data is cleaned to obtain the text corpus; wherein, the first preset path includes: administrative penalty decisions, court judgments, records of dishonest persons subject to enforcement, bidding information, news reports, and operating conditions descriptions in corporate annual reports; Structured data is obtained from a second preset path, and the structured data is cleaned to obtain the structured feature table; wherein, the second preset path includes: enterprise business registration information, financial statements, number of social security contributors, and number of intellectual property rights; The graph data is obtained from the third preset path, and the graph data is structurally cleaned to obtain the association graph; wherein, the third preset path includes: equity investment chain, external guarantee chain, senior executive cross-appointment network, and supplier customer relationship; The text corpus, the structured feature table, and the association graph are integrated to obtain the multimodal data set.

3. A joint modeling and collaborative reasoning device for credit large-scale models oriented towards multimodal data, characterized in that, The method for implementing the credit large-scale model joint modeling and collaborative reasoning method for multimodal data as described in any one of claims 1-2 includes: The data processing module is used to obtain multimodal credit data from the public credit information center and perform data preprocessing to obtain a multimodal dataset; wherein, the multimodal dataset includes: a text corpus, a structured feature table, and a correlation graph; wherein, the multimodal credit data includes: text data, structured data, and graph data; The vector extraction module is used to extract feature vectors from the multimodal data set based on the encoder corresponding to the modal data type to obtain an initial feature representation vector; wherein, the initial feature representation vector includes: text feature vector, structured feature vector, and graph feature vector; The collaborative fusion module is used to perform collaborative alignment and fusion processing on the text feature vector, the structured feature vector, and the graph feature vector based on a multi-head cross-modal attention layer to obtain a multimodal representation vector; The collaborative reasoning module is used to input the multimodal representation vector into the credit collaborative reasoning decision model for collaborative reasoning to obtain the reasoning result; wherein, the credit collaborative reasoning decision model is an evaluation model trained based on historical credit data; wherein, the reasoning result includes: credit evaluation result and decision basis data; Specifically, the data processing module employs a pre-trained domain-specific large language model as a text encoder; inputs the text corpus into the text encoder for feature vector extraction to obtain the text feature vector; employs a multilayer perceptron as a structured data encoder; inputs the structured feature table into the structured data encoder for feature vector extraction to obtain the structured feature vector; employs a graph neural network as a graph encoder; inputs the associated graph into the graph encoder to obtain the graph feature vector; and uses the text feature vector, the structured feature vector, and the graph feature vector as the initial feature representation vector.

4. The credit large-scale model joint modeling and collaborative reasoning device according to claim 3, characterized in that, The data processing module is specifically used to acquire unstructured text data from a first preset path, and perform data cleaning on the unstructured text data to obtain the text corpus; wherein, the first preset path includes: administrative penalty decisions, court judgments, records of dishonest persons subject to enforcement, bidding information, news reports, and operating status descriptions in corporate annual reports; acquire structured data from a second preset path, and perform structure cleaning on the structured data to obtain the structured feature table; wherein, the second preset path includes: corporate registration information, financial statements, number of social security contributors, and number of intellectual property rights; acquire graph data from a third preset path, and perform structure cleaning on the graph data to obtain the association graph; wherein, the third preset path includes: equity investment chains, external guarantee chains, cross-appointment networks of senior executives, and supplier-customer relationships; and integrate the text corpus, the structured feature table, and the association graph to obtain the multimodal data set.

5. A joint modeling and collaborative reasoning device for a large-scale credit model, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the credit large model joint modeling and collaborative reasoning method as described in any one of claims 1 to 2 when executing the computer program.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the credit large model joint modeling and collaborative reasoning method as described in any one of claims 1 to 2.

Images