Market subject credit loss time sequence knowledge graph representation learning method

Abstract

The application relates to a market subject credit loss time sequence knowledge graph representation learning method. The method extracts entities corresponding to each event, the relationship between the entities and time stamps from preprocessed multi-modal credit loss information, and constructs a time sequence knowledge graph. Multi-modal data describing the entities or the relationship are respectively attached to each entity node and the relationship node associated with the entity node in the time sequence knowledge graph. Encoding and semantic alignment are performed on the multi-modal data corresponding to each entity node and the relationship node to generate unified entity node representation or unified relationship node representation. Then, each time stamp is mapped into a corresponding time vector, and the time interval of continuous time stamps is mapped into a time weight. The time weight is combined with the corresponding node representation after mapping to obtain complete entity node representation or complete relationship node representation. The time sequence knowledge graph representation is generated based on the complete entity node representation, the complete relationship node representation associated with the entity and the time vector.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of temporal knowledge graph representation learning, in particular to a market subject credit risk temporal knowledge graph representation learning method. BACKGROUND

[0002] Market subject credit risk information is a record and disclosure of dishonest behavior of enterprises or individuals in economic activities. Its data form is complex, including structured or semi-structured text such as text announcements and punishment documents, and multi-modal information such as images and audio. As a technology that organizes facts evolving over time into a graph structure, the temporal knowledge graph can depict the dynamic change rule of market subject credit risk behavior. Through representation learning methods, time information can be combined to embed credit risk entities and their relationships into a continuous low-dimensional vector space for downstream retrieval, reasoning, prediction, and other tasks.

[0003] The existing market credit temporal knowledge graph representation learning method mainly has the following shortcomings:

[0004] 1. Insufficient cross-modal semantic modeling, making it difficult to achieve accurate alignment. Market credit data has multi-modal characteristics, but existing methods mostly rely on a single modality and lack effective cross-modal fusion mechanisms. Due to the lack of explicit alignment signals, it is difficult to ensure consistent representation under unlabeled conditions, and the semantic disambiguation effect is poor.

[0005] 2. Insufficient modeling of temporal dynamics, making it difficult to reflect the evolution process of facts. Market subject credit risk behavior has significant time dependence, but existing methods mostly simplify time as position encoding, ignoring the time interval between facts, resulting in knowledge graph representation that cannot reflect the important influence of time difference on fact evolution. For concurrent facts, such as a market subject interacting with multiple objects at the same timestamp, existing methods are difficult to capture their potential semantic associations, limiting the temporal modeling capability.

[0006] 3. Insufficient fusion of semantics and structure, making it difficult to form high-quality representation. Existing methods lack differentiation in the use of domain information, treating different neighbors equally and ignoring their semantic contribution differences. At the same time, the complex dependence of entities and relationships lacks global consistency modeling, limiting the quality of embedding representation.

[0007] 4. Limited context modeling, making it difficult to realize the cooperation of large models and knowledge graphs. When introducing large models for representation learning, existing methods fail to systematically utilize neighbor structure and context dependence, making it difficult to capture global semantics. In the cross-modal scenario, the semantic interpretation and reasoning advantages of large models are not fully utilized, and the learned representation cannot accurately depict the complex semantic relationships in market credit risk information, limiting the effectiveness and accuracy of the representation learning method. SUMMARY

[0008] Based on this, it is necessary to provide a market subject credit loss time sequence knowledge graph representation learning method, comprising:

[0009] S1: Collecting multi-modal credit loss information of market subjects, and preprocessing the multi-modal credit loss information;

[0010] S2: Extracting the entity corresponding to each event, the relationship between entities and the timestamp from the preprocessed multi-modal credit loss information to construct a quadruple corresponding to each event, integrating all quadruples corresponding to events to form a time sequence knowledge graph; each entity node and the relationship node associated therewith in the time sequence knowledge graph is respectively attached with a literal quantity, and the literal quantity includes multi-modal data describing the entity or the relationship;

[0011] S3: The literal quantity attached to each entity node and relationship node is respectively encoded to obtain an entity deep text semantic vector, an entity deep visual semantic vector, a relationship deep text semantic vector and a relationship deep visual semantic vector;

[0012] S4: Based on the cross attention mechanism, the deep text semantic vector and the deep visual semantic vector are semantically aligned, and the aligned entity deep text semantic vector, the entity deep visual semantic vector and the entity node are embedded and combined to generate a unified entity node representation, and the aligned relationship deep text semantic vector, the relationship deep visual semantic vector and the relationship node are embedded and combined to generate a unified relationship node representation;

[0013] S5: Mapping each timestamp in the time sequence knowledge graph to a corresponding time vector, and mapping the time interval of continuous timestamps to a time weight; mapping the time weight to the feature space of the unified entity node representation and the unified relationship node representation, and combining it with the corresponding node representation to obtain a complete entity node representation or a complete relationship node representation; based on the complete entity node representation, the complete relationship node representation associated with the entity, and the time vector, a time sequence knowledge graph representation is generated.

[0014] Beneficial effects: the method extracts the entity corresponding to each event, the relationship between entities and the timestamp from the pre-processed multi-modal untrustworthy information, constructs a time sequence knowledge graph; each entity node and the relationship node associated therewith in the time sequence knowledge graph is respectively attached with multi-modal data describing the entity or the relationship; the multi-modal data corresponding to each entity node and the relationship node is encoded and semantically aligned to generate a unified entity node representation or a unified relationship node representation; then, each timestamp is mapped into a corresponding time vector, and the time interval of continuous timestamps is mapped into a time weight; the time weight is combined with the corresponding node representation after mapping to obtain a complete entity node representation or a complete relationship node representation; based on the complete entity node representation, the complete relationship node representation associated with the entity, and the time vector, a time sequence knowledge graph representation is generated; the method can better capture the dynamic evolution characteristics of untrustworthy entities and relationships in the time dimension, highlight recent key untrustworthy facts, and make the embedding vector more focused on the current important untrustworthy information, providing more time sequence and dynamic feature representation for downstream tasks. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creating any inventive labor.

[0016] Figure 1 The flowchart of the market subject untrustworthy time sequence knowledge graph representation learning method in the embodiments of the present application. DETAILED DESCRIPTION

[0017] In order to make the above-mentioned purposes, features and advantages of the present application more apparent and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings. In the following description, a lot of specific details are set forth in order to provide a sufficient understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the spirit of the present application, therefore the present application is not limited by the specific embodiments disclosed below.

[0018] In addition, the terms "first", "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, such as two, three, etc., unless otherwise explicitly specified.

[0019] As Figure 1As shown, the embodiment provides a market subject credit risk time sequence knowledge graph representation learning method, including:

[0020] S1: Collecting multi-modal credit risk information of market subjects, and preprocessing the multi-modal credit risk information.

[0021] Specifically, the multi-modal credit risk information includes structured data, semi-structured data, and unstructured data; the structured data includes business registration information and penalty records; the semi-structured data includes tables and financial statements; the unstructured data includes news reports, social media texts, social media images, social media audios, and social media videos; the preprocessing of the multi-modal credit risk information includes:

[0022] Noise removal, missing value processing, and duplicate data merging are performed on each type of data in the multi-modal credit risk information.

[0023] The social media audio is converted into a second social media text, the encoding formats of the second social media text and the structured data, semi-structured data, and unstructured data in text form are unified, and each type of text data is subjected to word segmentation and stop word removal.

[0024] Key frames are extracted from the social media videos, and the key frames are converted into second social media images; the social media images and the second social media images in the unstructured data are subjected to cropping, denoising, and size standardization.

[0025] The processed text data and image data are unified into a parseable standard format.

[0026] Image recognition tools are used to extract the occurrence time, publication time, or judgment date corresponding to the image data, obtaining first time data; regular expression is used to extract time information from text data in a standard format, obtaining second time data; and each time data is unified into a standard timestamp format, and an index is established between the standard timestamp and the corresponding text data or image data in a standard format.

[0027] S2: Extracting entities corresponding to each event, relationships between entities, and timestamps from the preprocessed multi-modal credit risk information to construct a quadruple corresponding to each event, integrating all quadruples corresponding to events to form a time sequence knowledge graph; attaching a literal quantity to each entity node and its associated relationship node in the time sequence knowledge graph, the literal quantity including multi-modal data describing the entity or relationship.

[0028] Specifically, the process of constructing the time sequence knowledge graph includes:

[0029] For any one event, the entity recognition method and the relationship extraction method are used to identify the market subject, the regulatory agency and the bad faith behavior from the text data in the standard format;

[0030] The optical character recognition method and the visual recognition method are used to identify the agency identification and the punishment situation of the market subject or the regulatory agency from the image data in the standard format;

[0031] The market subject identified in the event is taken as the head entity, and the regulatory agency is taken as the tail entity. The bad faith behavior and the punishment situation corresponding to the entities are taken as the relationship between the entities;

[0032] The time sequence knowledge graph construction technology is used to construct the quadruple corresponding to the event with the head entity, the relationship between the entities, the tail entity and the corresponding timestamp;

[0033] The quadruples corresponding to each event under the same timestamp are combined to form an event set corresponding to the timestamp, and the regulatory agencies are combined to form an object set. The relationships between the entities are combined to form a bad faith relationship set. Based on the object set, the bad faith relationship set and the event set corresponding to each timestamp, the time sequence knowledge graph is constructed.

[0034] S3: The literal quantity attached to each entity node and relationship node is respectively feature-coded to obtain an entity deep text semantic vector, an entity deep visual semantic vector, a relationship deep text semantic vector and a relationship deep visual semantic vector.

[0035] Specifically, the literal quantity attached to each entity node and relationship node is respectively feature-coded, including feature-coding the text information in the multi-modal data:

[0036] The pre-trained BERT model is used to convert the text information corresponding to the entity node or the relationship node into an entity token sequence or a relationship token sequence, such as an entity token sequence “[CLS] entity text description [SEP]” and a relationship token sequence “[CLS] relationship text description [SEP]”. When the length of the text information is greater than a length threshold, an abstract method is used to make the text information less than the length threshold, so as to ensure that the bad faith key information is not lost;

[0037] Each token in the entity token sequence or the relationship token sequence is embedded to obtain an entity embedding vector sequence or a relationship embedding vector sequence, and the calculation formula is:

[0038] ;

[0039] Wherein, represents the embedding of the i-th token in the entity token sequence or the relationship token sequence; represents the embedding of the i-th token in the entity token sequence or the relationship token sequence; ; a vocabulary corresponding to the token; a position representing the token, used to represent the absolute order of the token in the sequence; a position representing the token, used to represent the absolute order of the token in the sequence; a segment identifier corresponding to the token, used to distinguish the semantic features of different node types; a segment identifier corresponding to the token, used to distinguish the semantic features of different node types; wordPiece tokenizer, which is a wordPiece tokenizer in the pre-trained BERT model, used to map discrete tokens into continuous vectors; sin represents a sine function, a learning matrix, used to encode the semantic features of different fields;

[0040] The entity embedding vector sequence or the relationship embedding vector sequence is linearly transformed into a query matrix, a key matrix, and a value matrix, and then passes through several layers of Transformer modules to obtain an entity deep text semantic vector or a relationship deep text semantic vector.

[0041] In this embodiment, the workflow of the first layer of Transformer modules includes:

[0042] Step 1: Calculate the multi-head self-attention mechanism:

[0043] ;

[0044] ;

[0045] where Q, K, and V are the query matrix, the key matrix, and the value matrix, respectively, obtained by linearly transforming the embedding with hidden layer dimension , the number of attention heads , is a projection matrix, the calculation result of the th attention head, is a learnable parameter matrix to learn the semantic patterns in the untrustworthy text.

[0046] Step 2: Residual connection and layer normalization:

[0047] ;

[0048] where represents the intermediate feature, is the calculation result of the multi-head attention mechanism, which focuses on capturing long-distance dependencies in the description of untrustworthy behavior. is a residual layer normalization operation. Then, it passes through a feedforward neural network and performs a second residual connection:

[0049] ;

[0050] wherein, is a two-layer linear transformation, represents a GELU activation function, , , , are learnable parameters respectively, OT is the first shallow text semantic vector or the second shallow text semantic vector output by the current layer Transformer module, which contains rich dishonesty semantic features.

[0051] The first shallow text semantic vector or the second shallow text semantic vector output by the previous layer is taken as the input of the next layer, and is sequentially passed through 12 Transformer modules for feature transmission, and dishonesty information is gradually abstracted:

[0052] ;

[0053] wherein, represents the hidden state output of the i-th layer, i.e. the entity deep text semantic vector or the relationship deep text semantic vector finally output, the bottom layer captures the lexical features of the dishonesty text, the middle layer extracts the syntactic pattern, and the high layer learns the semantic association. The literal quantity attached to each entity node and relationship node is respectively feature-encoded, including feature-encoding the image information in the multi-modal data:

[0054] The image information corresponding to the entity node or the relationship node is input into the Faster-R-CNN model, and the key visual area is detected, including the enterprise identification area, the business license area, the punishment announcement text area, and the improper behavior area; the calculation formula is:

[0055]

[0056] ; wherein,

[0057] represents the i-th key visual area, represents a rectangular area extraction function, represents image information, , represents the height of the image, represents the width of the image; represents the i-th candidate area detected by the Faster-R-CNN model, represents the number of candidate areas;

[0058] ​​​The key visual areas are respectively encoded by a ResNet encoder to obtain corresponding local features; the calculation formula is:

[0059] ;

[0060] wherein, represents the local feature corresponding to the i-th key visual area, , , represents the dimension of the feature output by the ResNet encoder; represents the ResNet encoder;

[0061] The local features are input into a gated recurrent unit model for aggregation to obtain an aggregated local semantic representation; the calculation formula is:

[0062] ;

[0063] wherein, represents the aggregated local semantic representation, represents the gated recurrent unit model, , , respectively represent the local features corresponding to the 1st, 2nd, ;

[0064] The image information corresponding to the entity node or the relationship node is input into a CLIP-ViT model to output a global feature; the global feature is mapped to the feature space of the aggregated local feature to obtain a global semantic representation; the calculation formula is:

[0065] ;

[0066] ;

[0067] wherein, represents the global semantic representation, represents the weight of the linear layer, represents the global feature, represents the bias of the linear layer, represents the CLIP-ViT model;

[0068] In order to enhance the high-level semantic information, the global semantic representation is concatenated with the aggregated local semantic representation and then subjected to layer normalization to output an entity deep visual semantic vector or a relationship deep visual semantic vector; the calculation formula is:

[0069] ;

[0070] wherein, denote an entity deep visual semantic vector or a relation deep visual semantic vector, which fuses global semantics and local details and contains multi-scale visual features related to the credit event; denote layer normalization, denote a weight coefficient, denote vector splicing, denote a bias coefficient.

[0071] S4: based on the cross-attention mechanism, the deep text semantic vector and the deep visual semantic vector are semantically aligned, and the aligned entity deep text semantic vector, the entity deep visual semantic vector and the entity node are embedded and combined to generate a unified entity node representation, and the aligned relation deep text semantic vector, the relation deep visual semantic vector and the relation node are embedded and combined to generate a unified relation node representation.

[0072] The multi-modal data is encoded by the pre-training language model and the CLIP model, the cross-attention mechanism is used to realize fine-grained semantic alignment, and the unified node representation is generated by residual fusion and graph structure embedding combination, which effectively integrates the complementary features of different modal information.

[0073] Specifically, the semantic alignment of the deep text semantic vector and the deep visual semantic vector based on the cross-attention mechanism comprises:

[0074] Any one of the deep text semantic vectors is taken as a first query, and any one of the deep visual semantic vectors is taken as a first key and a first value, a first attention score is calculated based on the first query, the first key and the first value, the first attention score is added to the corresponding deep text semantic vector, and then the layer normalization is performed to output the aligned entity deep text semantic vector or the relation deep text semantic vector; the calculation formula is:

[0075] ;

[0076] ;

[0077] wherein, denote the aligned entity deep text semantic vector, denote layer normalization, denote the entity deep text semantic vector, denote the first attention score, denote the entity deep visual semantic vector, denote transposition, denote a feature dimension, denote the first query, denote the first key, denote the first value;

[0078] The second attention score is calculated based on the second query, the second key, and the second value, and the second attention score is added to the corresponding deep visual semantic vector after layer normalization, to output an aligned entity deep visual semantic vector or a relationship deep visual semantic vector; the calculation formula is:

[0079] ;

[0080] ;

[0081] wherein, represents the aligned entity deep visual semantic vector, represents the second attention score, represents the second query, represents the second key, represents the second value.

[0082] Further, the unified entity node representation is generated, including:

[0083] The aligned entity deep text semantic vector and the entity deep visual semantic vector are spliced and subjected to layer normalization to obtain a first fusion multi-modal representation; the calculation formula is:

[0084] ;

[0085] wherein, represents the first fusion multi-modal representation, represents layer normalization, represents a fusion weight coefficient, represents the aligned entity deep text semantic vector, represents the aligned entity deep visual semantic vector, represents a learnable bias;

[0086] For an entity node, the aligned entity deep text semantic vectors or the entity deep visual semantic vectors of each multi-modal data attached to the entity node are aggregated, and the aggregation result is subjected to a nonlinear activation function to obtain an entity node representation; the calculation formula is:

[0087] ;

[0088] wherein, represents the entity node representation, represents a nonlinear activation function, represents an aggregation weight, represents an aggregation function, represents the entity node The first an aligned entity deep text semantic vector or an aligned entity deep visual semantic vector of a literal (multimodal data), representing an entity node and a literal , representing an entity node an additional number of literals;

[0089] Based on the first fusion multimodal representation and the entity node representation, a unified entity node representation is calculated. The entity node representation is projected into the same vector space and dimension as the first fusion multimodal representation , residual addition is used to alleviate gradient disappearance in deep networks, and normalization is used to avoid too large differences in numerical ranges of different source features, improving training stability, and the calculation formula is:

[0090] ;

[0091] wherein, represents the unified entity node representation, represents a nonlinear activation function, represents layer normalization, represents the first fusion multimodal representation, represents a mapping weight, represents a mapping bias, represents the entity node representation.

[0092] Similarly, generating a unified relationship node representation includes:

[0093] The aligned relationship deep text semantic vector and the relationship deep visual semantic vector are spliced and subjected to layer normalization to obtain a second fusion multimodal representation;

[0094] For a relationship node, the aligned relationship deep text semantic vector or the relationship deep visual semantic vector of each multimodal data corresponding to the relationship node is aggregated, and the aggregation result is subjected to a nonlinear activation function to obtain a relationship node representation;

[0095] Based on the second fusion multimodal representation and the relationship node representation, a unified relationship node representation is calculated. The relationship node representation is projected into the same vector space and dimension as the second fusion multimodal representation , residual addition is used to alleviate gradient disappearance in deep networks, and normalization is used to avoid too large differences in numerical ranges of different source features, improving training stability, and the calculation formula is:

[0096] ;

[0097] wherein, denote unified relation node representations, denote nonlinear activation functions, denote layer normalization, denote second fused multi-modal representations, denote mapping weights, denote mapping biases, denote relation node representations.

[0098] S5: mapping each timestamp in the time-sequential knowledge graph to a corresponding time vector, and mapping the time interval of continuous timestamps to a time weight; mapping the time weight to the feature space of the unified entity node representation and the unified relation node representation, and combining with the corresponding node representation to obtain a complete entity node representation or a complete relation node representation; generating a time-sequential knowledge graph representation based on the complete entity node representation, the complete relation node representation associated with the entity, and the time vector.

[0099] In the representation learning process, time contains both absolute position information and relative interval information. To represent the relative positions between different timestamps, a time encoding function based on Fourier expansion is introduced to map discrete timestamps to a continuous vector space and capture the periodic patterns of the time sequence. Specifically, mapping each timestamp in the time-sequential knowledge graph to a corresponding time vector includes:

[0100] A time encoding function based on Fourier expansion is introduced to map each timestamp to obtain a corresponding time vector, and the calculation formula is:

[0101] ;

[0102] wherein, denotes the time vector of the timestamp , denotes the embedding dimension (the dimensions of the time vector, the complete entity node representation, and the complete relation node representation are ), denotes the cosine function, denotes the first first learning parameter, denotes the first second learning parameter, denotes the th first learning parameter, denotes the th second learning parameter.

[0103] Further, only time point representation is insufficient to model the dynamic dependence between events, therefore, to capture the influence of time interval on semantic relevance, a time-aware function (hyperbolic tangent function) is introduced to obtain a complete entity node representation or a complete relation node representation:

[0104] The time intervals between timestamps in different events are mapped to time weights, calculated as follows:

[0105] ; ;

[0106] in, Indicates time interval Time weighting Represents the hyperbolic tangent function. This represents the boundary parameter (used to control the attenuation intensity). Represents a timestamp. Represents timestamp Adjacent timestamps, It represents absolute value; the hyperbolic tangent function ensures that the smaller the time interval, the stronger the correlation between events, and the larger the time interval, the more its influence gradually decreases.

[0107] The time weights are mapped to the feature space of the unified entity node representation or the unified relation node representation by the multilayer perceptron, and then added to the corresponding node representation to obtain the complete entity node representation or the complete relation node representation.

[0108] By introducing time coding and time decay functions, discrete timestamps are mapped to vector representations and integrated into node representations. Time smoothing constraints are used to ensure that the representation changes smoothly under continuous timestamps, capturing the dynamic correlation of the evolution of dishonesty events over time.

[0109] The resulting complete entity node representation or complete relation node representation is a high-quality node representation that integrates multimodal, temporal, and structural information. This enables the embedding of dishonest entities and relations to accurately capture semantic associations and dynamic evolution characteristics, providing reliable input for downstream risk prediction, behavior tracking, and decision support.

[0110] In this embodiment, based on the time-aware, multimodal fusion-based complete node representation obtained in step S5, a unified objective function needs to be constructed to ensure the effectiveness and consistency of representation learning. Constraints such as cross-modal semantic alignment, temporal dynamic preservation, and structural consistency are incorporated into the joint optimization framework. Taking entity node x as an example, the optimization process is as follows:

[0111] For each entity node x, obtain the entity's deep text semantic vector. and entity deep visual semantic vector Construct an alignment loss function to encourage different modal representations of the same node to be similar: .right Complete entity node representation or complete relation node representation at any given time. Introduce time smoothing constraints to ensure smooth changes in representation between consecutive timestamps: , which aggregates neighbor information using graph neural networks to ensure that the node representation is consistent with the structure of the knowledge graph: wherein, is a graph neural network aggregation function. The constraints are unified into a joint optimization framework to construct the total loss: . Wherein is a hyperparameter used to balance different constraints, is a regularization term to improve training stability and robustness. Finally, the gradient optimizer Adam is used to iteratively update the parameters: wherein, denotes all updated learnable parameters, denotes all learnable parameters, is the learning rate, denotes the total loss with respect to the gradient of the parameter .

[0112] The market entity discrediting time sequence knowledge graph representation learning method provided by the embodiment has the following beneficial effects:

[0113] The method extracts deep semantic features of text such as discrediting announcements and punishment documents through pre-training language models, extracts visual features of images such as enterprise licenses and punishment screenshots through cross-modal comparison models (CLIP), and ensures that different modal semantic information is represented in their respective optimal spaces. Further, cross-attention mechanisms are used to achieve fine-grained interaction between text and image features, and residual fusion strategies are used to achieve modal alignment, thereby fully utilizing the complementarity of multi-modal information. At the same time, a time coding mechanism is introduced in the market entity discrediting information time sequence knowledge graph representation process, the evolution relationship between historical facts and current facts is comprehensively considered, local and global embedding methods are used to model the dynamic dependence of entities and relationships, and the time sequence perception ability and semantic discrimination of representation learning are improved. To overcome the shortcomings of existing methods in cross-modal semantic alignment, time sequence dynamic modeling, and semantic and structural fusion, a more comprehensive and accurate representation of market entity discrediting behavior is achieved, providing effective support for downstream risk prediction, evolution tracking, and decision support.

[0114] The technical features of the above-described embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described, but as long as the combinations of the technical features do not contradict, they should be considered as part of the scope of the present disclosure.

[0115] The above embodiments only express several implementation ways of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation to the patent scope of the application. It should be pointed out that for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, which all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A method for learning and representing the temporal knowledge graph of market entity default, characterized in that, include: S1: Collect multimodal credit information of market entities and preprocess the multimodal credit information; The multimodal credit information includes structured data, semi-structured data, and unstructured data; structured data includes business registration information and penalty records; semi-structured data includes tables and financial statements; and unstructured data includes news reports, social media text, social media images, social media audio, and social media video. S2: Extract the entity, the relationship between entities, and the timestamp corresponding to each event from the preprocessed multimodal credit information to construct a quadruple corresponding to each event. Integrate the quadruples corresponding to all events to form a time-series knowledge graph. Add literals to each entity node and its associated relationship node in the time-series knowledge graph. The literals include multimodal data describing the entity or relationship. S3: Encode the literal values ​​attached to each entity node and relation node to obtain the entity deep text semantic vector, entity deep visual semantic vector, relation deep text semantic vector, and relation deep visual semantic vector, respectively. S4: Based on a cross-attention mechanism, semantic alignment is performed on deep text semantic vectors and deep visual semantic vectors. The aligned entity deep text semantic vectors, aligned entity deep visual semantic vectors, and entity nodes are then embedded and combined to generate a unified entity node representation, including: The aligned entity deep text semantic vector and the aligned entity deep visual semantic vector are concatenated and then normalized to obtain the first fused multimodal representation. For entity nodes, the aligned entity deep text semantic vector or aligned entity deep visual semantic vector of each multimodal data attached to the entity node is aggregated, and the aggregation result is passed through a non-linear activation function to obtain the entity node representation; Based on the first fused multimodal representation and entity node representation, the unified entity node representation is calculated using the following formula: ； in, Represents a unified entity node representation. Represents a non-linear activation function. Representation layer normalization, This represents the first fused multimodal representation. Indicates the mapping weights. Indicates the mapping bias, Represents entity node representation; The aligned deep textual semantic vectors and aligned deep visual semantic vectors of relationships are embedded and combined with relationship nodes to generate a unified representation of relationship nodes, including: The aligned relational deep text semantic vector and the aligned relational deep visual semantic vector are concatenated and then normalized to obtain the second fused multimodal representation. For relation nodes, the aligned deep textual semantic vectors or aligned deep visual semantic vectors of the relation nodes are aggregated from the multimodal data attached to the relation nodes, and the aggregation results are passed through a non-linear activation function to obtain the relation node representation; Based on the second fused multimodal representation and relation node representation, the unified relation node representation is calculated using the following formula: ； in, This represents a unified relation node representation. Represents a non-linear activation function. Representation layer normalization, This represents the second fused multimodal representation. Indicates the mapping weights. Indicates the mapping bias, Representation of relation nodes; S5: Map each timestamp in the time-series knowledge graph to a corresponding time vector, including: A time encoding function based on Fourier expansion is introduced to map each timestamp separately, resulting in a corresponding time vector. The calculation formula is as follows: ； in, Represents timestamp The time vector, Indicates the embedding dimension. Represents the cosine function. This represents the first learned parameter. This represents the first second learning parameter. Indicates the first The first learning parameter, Indicates the first A second learning parameter; The time intervals between consecutive timestamps are then mapped to time weights, calculated as follows: ； ； in, Indicates time interval Time weighting Represents the hyperbolic tangent function. Represents boundary parameters, Represents a timestamp. Represents timestamp Adjacent timestamps, The absolute value is represented; the time weight is mapped to the feature space of the unified entity node representation and the unified relation node representation, and combined with the corresponding node representation to obtain the complete entity node representation or the complete relation node representation; based on the complete entity node representation, the complete relation node representation associated with the entity, and the time vector, a temporal knowledge graph representation is generated.

2. The method for learning the temporal knowledge graph representation of market entity dishonesty according to claim 1, characterized in that, In S1, the preprocessing of multimodal credit information includes: For each type of data in the multimodal credit information, noise removal, missing value processing, and duplicate data merging are performed; Social media audio is converted into second social media text, and the encoding format of second social media text and structured, semi-structured and unstructured data in text form is unified. Each type of text data is segmented and stop word removed. Extract keyframes from social media videos and convert them into second social media images. Perform cropping, denoising, and size normalization on both the social media images and the second social media images in the unstructured data. The processed text and image data are standardized into a uniform format. Image recognition tools are used to extract the occurrence time, release time, or judgment date corresponding to the image data to obtain the first time data; regularization is used to extract the time information from the standardized text data to obtain the second time data; and all time data are unified into a standard timestamp format, and an index is established between the standard timestamp and the corresponding standardized text data or image data.

3. The method for learning the temporal knowledge graph representation of market entity dishonesty according to claim 2, characterized in that, In S2, the process of constructing a time-series knowledge graph includes: For any given event, entity recognition and relation extraction methods are used to identify market entities, regulatory agencies, and dishonest behaviors from text data in a standardized format. The system uses optical character recognition and visual recognition methods to identify the institutional logos and penalty information of market entities or regulatory agencies from standardized image data. The market entities identified in the incident are designated as head entities, and the regulatory agencies are designated as tail entities. The corresponding dishonest behaviors and penalties of the entities are used as the relationships between the entities. Using temporal knowledge graph construction technology, a quadruple corresponding to an event is constructed with the head entity, the relationship between entities, the tail entity, and the corresponding timestamp. By combining the quadruples corresponding to each event under the same timestamp to form the event set corresponding to the timestamp, and combining the regulatory agencies to form the object set, and combining the relationships between the entities to form the default relationship set, a temporal knowledge graph is constructed based on the object set, the default relationship set, and the event set corresponding to each timestamp.

4. The method for learning the temporal knowledge graph representation of market entity dishonesty according to claim 1, characterized in that, In S3, the literals appended to each entity node and relation node are feature-encoded, including feature encoding of text information in multimodal data: The pre-trained BERT model is used to convert the text information corresponding to entity nodes or relationship nodes into entity token sequences or relationship token sequences; Embed each token in the entity token sequence or relation token sequence to obtain the entity embedding vector sequence or relation embedding vector sequence. The calculation formula is as follows: ； in, Represents the first element in a sequence of entity tokens or relation tokens. Embedding of a token Indicates the first Each token corresponds to a word. Indicates the first The location of each token Indicates the first The segment identifier corresponding to each token. This refers to the wordPiece tokenizer. Represents the sine function. Represents the learning matrix; The entity embedding vector sequence or relation embedding vector sequence is linearly transformed into a query matrix, key matrix, and value matrix, and then passed through several layers of Transformer modules to obtain the entity deep text semantic vector or relation deep text semantic vector.

5. The method for learning the temporal knowledge graph representation of market entity dishonesty according to claim 1, characterized in that, In S3, the literals attached to each entity node and relation node are feature-encoded, including feature encoding of image information in multimodal data: The image information corresponding to entity nodes or relationship nodes is input into the Faster-R-CNN model to detect key visual regions, including enterprise logo areas, business license areas, penalty notice text areas, and areas of improper behavior. Each key visual region is processed by a ResNet encoder to obtain its corresponding local features; The local features are input into the gated recurrent unit model and aggregated to obtain the aggregated local semantic representation. The image information corresponding to entity nodes or relation nodes is input into the CLIP-ViT model, and global features are output. The global features are then mapped to a feature space that aggregates local features to obtain a global semantic representation. After concatenating the global semantic representation and the aggregated local semantic representation, and then performing layer normalization, the deep visual semantic vector of the entity or the deep visual semantic vector of the relationship is output.

6. The method for learning the temporal knowledge graph representation of market entity dishonesty according to claim 1, characterized in that, In S4, the semantic alignment of deep text semantic vectors and deep visual semantic vectors based on the cross-attention mechanism includes: Take any deep text semantic vector as the first query and any deep visual semantic vector as the first key and first value. Calculate the first attention score based on the first query, first key, and first value. Add the first attention score to the corresponding deep text semantic vector and then perform layer normalization to output the aligned entity deep text semantic vector or the aligned relation deep text semantic vector. Use any deep visual semantic vector as the second query and any deep text semantic vector as the second key and second value. Calculate the second attention score based on the second query, second key, and second value. Add the second attention score to the corresponding deep visual semantic vector and then perform layer normalization to output the aligned entity deep visual semantic vector or the aligned relation deep visual semantic vector.

7. The method for learning the temporal knowledge graph representation of market entity dishonesty according to claim 1, characterized in that, In S5, obtaining a complete entity node representation or a complete relation node representation includes: The time weights are mapped to the feature space of the unified entity node representation or the unified relation node representation by the multilayer perceptron, and then added to the corresponding node representation to obtain the complete entity node representation or the complete relation node representation.

Images