Time sequence knowledge graph representation learning method based on dishonesty information of market subject

Abstract

The invention relates to a time sequence knowledge graph representation learning method based on market subject dishonesty information. The method comprises the steps of obtaining market subject dishonesty data; preprocessing the dishonesty data of the market subject to obtain dishonesty text data and dishonesty image data; performing semantic analysis on the dishonesty text data to generate a dishonesty entity text vector; extracting the characteristics including the dishonesty information in the dishonesty image data to form a dishonesty entity visual vector; aggregating the untrusted entity text vector and the untrusted entity visual vector to obtain an untrusted entity vector; encoding the dishonesty text data and the dishonesty image data into a first sequence and a second sequence respectively, extracting time information as a time weight, and updating the first sequence and the second sequence converted based on the time weight respectively to obtain a first updated sequence and a second updated sequence; fusing the first update sequence and the second update sequence to obtain a dishonesty relationship vector; and generating a time sequence knowledge graph representation based on the dishonesty entity vector and the dishonesty relation vector.

Description

Technical Field

[0001] The present application relates to the technical field of temporal knowledge graph representation learning, and in particular to a temporal knowledge graph representation learning method based on market subject default information. Background Art

[0002] The dishonest information of market entities is the record and publicity of dishonest behavior of enterprises or individuals in commercial activities within a certain period of time. Representing the dishonest information of market entities as a time-series knowledge graph is a method to organize the dishonest information into a dynamic graph structure that changes over time. The dishonest information is encoded through entity recognition technology, and the names of relevant market entities are extracted to form dishonest entities. Relation extraction is used to identify the association between dishonest entities within a certain period of time. Finally, the entities and relationships in the time-series knowledge graph are converted into low-dimensional vector representations through global fusion, that is, the symbolic information is converted into numerical form. This representation learning method can capture the evolution process and time correlation of dishonest behavior of market entities, and provide a deeper understanding and more accurate judgment for the analysis and prediction of dishonest behavior.

[0003] The existing learning methods for temporal knowledge graph representation of dishonest information of market entities mainly have the following problems: 1. Poor reliability: In the untrustworthy information, there are many highly similar entity names, resulting in low reliability of the obtained matches. At the same time, the heterogeneity of data sources, uneven quality, and the ambiguity, synonyms and abbreviations of the text can lead to errors in entity matching.

[0004] 2. Single data source: Currently, the data source is mainly text data, which cannot fully capture the changes in various concepts covering different semantic directions.

[0005] 3. Lack of information interaction between relationships and entities; the contextual information of entities and relationships is not fully utilized, and entities and relationships are often processed separately, resulting in one-sidedness and limitations of information, making it difficult to fully understand the complex relationships between entities. At the same time, the establishment of relationships is based on static text data, without considering the dynamic evolution of entities and relationships, and lacking relevance information.

[0006] 4. There are structural differences in embeddings; different types of entities and relationships have irregularities and heterogeneity in embedding representations, making it difficult to unify them into a unified structural space. Summary of the invention

[0007] Based on this, it is necessary to provide a learning method for temporal knowledge graph representation based on the dishonesty information of market entities, which includes: S1: Acquire the market subject dishonesty data including images, texts and audios; pre-process the market subject dishonesty data to obtain dishonesty text data and dishonesty image data; S2: semantically analyze the untrustworthy text data to generate a untrustworthy entity text vector; extract features of the untrustworthy image data containing untrustworthy information to form a untrustworthy entity visual vector; aggregate the untrustworthy entity text vector and the untrustworthy entity visual vector to obtain a untrustworthy entity vector; S3: Encode the dishonest text data and the dishonest image data into a first sequence and a second sequence respectively, extract time information as a time weight, and update the first sequence and the second sequence converted based on the time weight respectively to obtain a first updated sequence and a second updated sequence; merge the first updated sequence and the second updated sequence to obtain a dishonest relationship vector; S4: Generate a temporal knowledge graph representation based on the default entity vector and the default relationship vector.

[0008] Beneficial effects: This method achieves effective representation and fusion of complex dishonesty information, provides feature extraction capabilities, and helps reduce the complexity of downstream tasks and improve processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0010] Figure 1 This is a flowchart of a learning method based on a temporal knowledge graph representation of market entity default information in an embodiment of the present application. DETAILED DESCRIPTION

[0011] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present application, so the present application is not limited by the specific embodiments disclosed below.

[0012] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0013] like Figure 1 As shown, this embodiment provides a learning method for temporal knowledge graph representation based on market subject dishonesty information, the method comprising: S1: Acquire the market subject's dishonesty data including images, texts and audios; pre-process the market subject's dishonesty data to obtain dishonesty text data and dishonesty image data.

[0014] Specifically, the preprocessing includes: Use audio recognition technology to convert the audio data in the market entity's dishonesty data into second text data; Combining the text data in the market entity dishonesty data with the second text data into the dishonesty text data; The image data in the market entity's dishonest data is used as the dishonest image data.

[0015] S2: semantically analyze the untrustworthy text data to generate a untrustworthy entity text vector; extract features containing untrustworthy information from the untrustworthy image data to form a untrustworthy entity visual vector; aggregate the untrustworthy entity text vector and the untrustworthy entity visual vector to obtain a untrustworthy entity vector.

[0016] Specifically, the semantic analysis of the dishonest text data includes: Step 1: By statistically learning several annotated corpora of dishonest information, the frequencies of multiple production rules are obtained, and then the probability of each production rule is calculated. The calculation formula is: ; in, Representing production rules probability; express count function; Represents a non-terminal symbol; Represents a combination of non-terminal symbols or a terminal symbol; Indicates is any right side of all production rules on the left side; Indicates is the set of all production rules on the left; Step 2: For each word in the dishonest text data, find all production rules that are on the right side of the production rule, and initialize the table, which is expressed as: ; in, Indicates that it contains i Characters to i +1 character table; Indicates all is the production rule on the right side; Representing production rules probability; Indicates i words; Step 3: For each substring length in the dishonest text data Fill the table from small to large, expressed as: ; ; ; in, Indicates that it contains i Characters to A table of characters; Indicates that from i Characters to k A substring of characters can be derived situation; Indicates that from k Characters to A substring of characters can be derived situation; Indicates that it contains i Characters to k A table of characters; Indicates that it contains k Characters to A table of characters; Indicates that from i Characters to All substrings of characters k A collection of characters; express The set of production rules for ; Step 4: Use the Lookback Table back [ i ][ j ] Record the generation path of the table each time it is filled, recursively traverse the backtracking table, and construct the parsing number; Step 5: Each untrustworthy entity name phrase extracted from the parsed data is used as input data to the trained Phrase2Vec model, and the corresponding untrustworthy entity text vector is output.

[0017] In this embodiment, the Phrase2Vec model is trained based on a large amount of text corpus (which includes dishonest text data and other relevant text data), and the training process is consistent with the conventional training process of the Phrase2Vec model.

[0018] Furthermore, the extracting of features containing dishonest information from the dishonest image data includes: Step 1: Take each image region in the dishonest image data as a node, take the spatial proximity of pixels between the image regions as a weighted edge, and construct a graph structure; Step 2: Use a first-order graph convolutional neural network to perform preliminary feature extraction on the graph structure to obtain preliminary features, which are expressed as: ; in, Indicates preliminary features; represents the activation function; D represents the degree matrix; A represents the adjacency matrix of the graph structure; X represents the feature representation of the node in the graph structure; Represents the weight matrix of a first-order graph convolutional neural network; Step 3: Take the default entity text vector as the semantic hint vector, introduce the attention weight through the attention mechanism, adjust the aggregation process of the first-order graph convolutional neural network, aggregate the preliminary features of all nodes, and obtain the aggregated features, so that it pays more attention to the features related to the semantic hint vector. The calculation formula is: ; ; in, represents the first aggregate feature; Indicates i The attention weight of each node; Indicates i Preliminary characteristics of nodes; represents the softmax function; represents the attention function; represents the semantic hint vector; Step 4: Use a second-order graph convolutional neural network to further aggregate the aggregated features based on the neighboring nodes of each node to obtain the default entity visual vector of each node. The calculation formula is: ; in, Indicates i The visual vector of the dishonest entity of each node; Indicates i The set of neighbor nodes of a node; Represents the weight matrix of a second-order graph convolutional neural network.

[0019] Furthermore, the aggregating the untrustworthy entity text vector and the untrustworthy entity visual vector includes: Step 1: Take the set of visual vectors of the dishonest entities as the source domain, the set of text vectors of the dishonest entities as the target domain, set the random walk length L, and use the source vector walk of length L generated by the flexible sampling strategy. x The elements are generated as follows: ; ; ; in, Indicates the source vector x The probability of generating an element; Indicates the source vector x elements; Indicates the source vector x- 1 element; Indicates i A text vector of a dishonest entity; Indicates j The untrustworthy text vector of nodes; represents the target domain; represents the source domain; express and The transition probability between Z represents the normalization constant; represents the walking guidance function; express and The weight between r represents the first hyperparameter; t represents the second hyperparameter; express and The distance between Step 2: Perform random walks in two directions and define the sequence obtained in one random walk as the transmission sequence; The first direction is to randomly select any vector in the source domain as the starting point, and access the next vector with its corresponding transition probability until the vector in the target domain is reached or the number of accessed vectors exceeds the threshold T, and then stop wandering; The second direction is to randomly select any vector in the target domain as the starting point, and access the next vector with its corresponding transition probability until the vector in the source domain is reached or the number of accessed vectors exceeds the threshold T, and then stop wandering; Step 3: Perform weighted summation on the untrustworthy entity visual vector and the untrustworthy entity text vector according to the transmission sequence to obtain the untrustworthy entity vector.

[0020] S3: Encode the dishonest text data and the dishonest image data into a first sequence and a second sequence respectively, extract time information as a time weight, and update the first sequence and the second sequence converted based on the time weight respectively to obtain a first updated sequence and a second updated sequence; fuse the first updated sequence and the second updated sequence to obtain a dishonest relationship vector.

[0021] Specifically, the process of obtaining the first update sequence and the second update sequence includes: Step 1: Use the pre-trained BERT model and the ViT model to encode the untrustworthy text data and the untrustworthy image data respectively, and obtain the first sequence and the second sequence respectively; and extract the time information in the untrustworthy text data and the untrustworthy image data respectively as the first time weight and the second time weight respectively; Step 2: Use the fully connected layer of the graph neural network, and based on the first time weight and the second time weight, perform dimensional transformation on the first sequence and the second sequence respectively to obtain the transformed first sequence and the transformed second sequence respectively. The calculation formula is: ; ; in, represents the first sequence after conversion, , Indicates the length of the first sequence after conversion, d Represents feature dimension; represents the first time weight; represents the first sequence; represents the first bias term; represents the second sequence after conversion, , Indicates the length of the second sequence after conversion; represents the second time weight; represents the second sequence; represents the second bias term; Step 3: Aggregate the first dishonest messages of the neighbor sequence of the first sequence after conversion to obtain the first dishonest relationship feature, which is calculated as follows: ; The first dishonest message is calculated based on the converted neighbor sequence of the first sequence and using a first message passing function. The calculation formula of the first message passing function is: ; in, Indicates i The first characteristic of dishonest relationship; Indicatesi The set of neighbor sequences of the first sequence after transformation; Indicates j The first dishonest message of a neighbor sequence; Indicates i The first sequence after the conversion j neighbor sequences; o represents the cross-modal relationship vector; Aggregate the second dishonest messages of the neighbor sequence of the second sequence after conversion to obtain the second dishonest relationship feature, which is calculated as follows: ; The second dishonest message is calculated based on the neighbor sequence of the converted second sequence and using a second message passing function, and the calculation formula of the second message passing function is: ; in, Indicates i The second characteristic of dishonest relationship; Indicates i A set of neighbor sequences of the converted second sequence; Indicates j The second dishonest message of the neighbor sequence; Indicates i The second sequence after the conversion j Neighbor sequence; Step 4: The converted first sequence is merged with the first default relationship feature to obtain the first updated sequence. The calculation formula is: ; in, Indicates i A first update sequence; represents the weight parameter; Indicates i The first sequence after conversion; The converted second sequence is combined with the second default relationship feature to obtain the second updated sequence, which is calculated as follows: ; in, Indicates i A second update sequence; Indicates i A second sequence after transformation; The first update sequence and the second update sequence both aggregate respective contextual breach of trust information.

[0022] Further, the fusing the first update sequence and the second update sequence includes: Step 1: Use the feedforward layer of the graph neural network to perform linear transformation on the first update sequence and the second update sequence to obtain text modality coding and image modality coding respectively. The calculation formula is: ; ; in, Indicates text modal encoding; Represents image modality encoding; ReLU activation function. Indicates i A first update sequence; Indicates i A second update sequence; A first weight matrix representing the text modality; A second weight matrix representing the text modality; A first weight matrix representing the image modality; a second weight matrix representing the image modality; Indicates the first deviation of the text modality; A second deviation representing the text modality; represents the first deviation of the image modality; represents the second deviation of the image modality; Step 2: Project the text modality code and the image modality code to a common space, and assign the first time weight and the second time weight extracted from the dishonest text data and the dishonest image data to each other, and connect the text modality code and the image modality code assigned with the time weight to obtain the modality fusion feature, and the calculation formula is: ; ; ; in, Represents modality fusion features; represents the text modality encoding given the second time weight; P represents the concatenation operation; represents the image modality encoding assigned to the first time weight; represents the first time weight; represents the second time weight; Step 3: In a set of fully connected layers, the modality fusion features are used as query, key, and value respectively, and the attention score is calculated as follows: ; ; ; ; in, represents the attention score; represents the softmax function; Q represents the query; K represents the key; V represents the value; T represents the transpose; Represents the dimension of the key; represents the query weight matrix; represents the key weight matrix; represents the value weight matrix; Step 4: Repeat step 3, traverse several groups of fully connected layers, and obtain the attention scores in each group of fully connected layers; Step 5: Apply the softmax function to normalize the attention scores, perform weighted summation on the normalized attention scores, and map the summation result to the default relationship vector through a fully connected layer.

[0023] Through the random walk strategy, the connection between different modal data in the target domain and the source domain is found to form a multimodal default entity vector; and the message passing function is used to aggregate the contextual default information separately to enhance the temporal characteristics; through multi-head attention extraction and integration, a multimodal default relationship vector is formed, which solves the problem of single data source.

[0024] S4: Generate a temporal knowledge graph representation based on the default entity vector and the default relationship vector.

[0025] Specifically, the step includes: S4.1: The default entity vector and the default relationship vector are reshaped into 2D matrix forms respectively, and the 2D transformed default entity vector and the 2D transformed default relationship vector are obtained respectively; the transformation process is expressed as: ; ; in, h Represents the default entity vector; g represents the default relationship vector; Represents the default entity vector after 2D transformation; represents the default relationship vector after 2D transformation; d Represents feature dimension; Represents the dimension of the default text vector after 2D transformation; Represents the dimension of the lost visual vector after 2D transformation; S4.2: The branched interactive convolutional network generates two parallel convolutional branches, each of which contains three different convolutional layers; the three convolutional layers contained in the first convolutional branch and the second convolutional branch are the same; In the first convolution branch, three different convolution layers are used to perform three convolutions on the 2D transformed untrustworthy entity vector, respectively, to obtain the first untrustworthy entity convolution result, the second untrustworthy entity convolution result, and the third untrustworthy entity convolution result, respectively; the convolution process is expressed as: ; ; ; in, Represents the convolution result of the first dishonest entity; represents the convolution result of the second dishonest entity; represents the convolution result of the third dishonest entity; Represents the first convolutional layer in the first convolutional branch; Represents the second convolutional layer in the first convolutional branch; Represents the third convolutional layer in the first convolutional branch; Represents the convolutional layer The convolution kernel of Represents the convolutional layer The convolution kernel of Represents the convolutional layer The convolution kernel of , Indicates the first branch i The convolution kernel of the convolution layer; Indicates The number of filters used in the convolutional layer; represents the first convolution branch The width of the convolution kernel of the convolution layer; represents the first The height of the convolution kernel of each convolutional layer; In the second convolution branch, three different convolution layers are used to perform three convolutions on the 2D transformed default relationship vector, respectively, to obtain the first default relationship convolution result, the second default relationship convolution result, and the third default relationship convolution result, respectively; the convolution process is expressed as: ; ; ; in, represents the convolution result of the first dishonest relationship; represents the convolution result of the second dishonest relationship; represents the convolution result of the third dishonest relationship; Represents the first convolutional layer in the second convolutional branch; Represents the second convolutional layer in the second convolutional branch; represents the third convolutional layer in the second convolutional branch; Represents the convolutional layer The convolution kernel of Represents the convolutional layer The convolution kernel of Represents the convolutional layer The convolution kernel of S4.3: The convolution result of the first dishonest entity is fused with the convolution result of the first dishonest relationship to obtain a first interaction result. The fusion formula is: ; in, represents the first interaction result; The convolution result of the second dishonest entity is fused with the convolution result of the second dishonest relationship to obtain the second interaction result. The fusion formula is: ; in, represents the second interaction result; The third dishonest entity convolution result and the third dishonest relationship convolution result are fused to obtain the third interaction result. The fusion formula is: ; in, represents the third interaction result; S4.4: Reshape the first interaction result, the second interaction result, and the third interaction result into 2D matrix forms respectively, and connect the reshaped first interaction result, the second interaction result, and the third interaction result to obtain a connection result; the calculation formula is: ; in, Indicates the connection result. , c represents the number of filters used in the convolutional layer; Indicates a connection operation; represents the first interaction result after reshaping; represents the second interaction result after reshaping; represents the third interaction result after reshaping; S4.5: Perform linear transformation and nonlinear mapping on the connection results to generate the temporal knowledge graph representation.

[0026] Further, step S4.5 includes: The connection result is expanded into a vector, and the connection result represented by the vector is input into the fully connected layer to obtain a mapping result; the expansion formula of the connection result is: ; in, Represents the concatenation result of the vector representation; Represents the reshape function; Indicates the connection result; Indicates the size of the first dimension after flattening; Represents the dimension of the convolution kernel in the convolution layer; Represents the total width of the convolution kernel in the first convolution branch; Represents the total height of the convolution kernel in the second convolution branch; The mapping process of the fully connected layer is expressed as: ; in, Indicates the mapping result; represents the first model parameter, ; represents the second model parameter, ; The mapping result is subjected to a ReLU activation function to obtain a nonlinearly represented mapping result; Use the reshape function to reshape the dimension of the mapping result of the nonlinear representation (reshape the dimension into ), and obtain the temporal knowledge graph representation.

[0027] Different processing methods are used to process image and text data, and the untrustworthy entity recognition and relationship extraction are performed in stages. At the same time, the pre-trained BERT model and ViT model are used to extract the time information. The fusion verification of entities and relationships is realized through the branched interactive convolutional network to form a temporal knowledge graph and embed it in a low-dimensional space, thereby achieving a compact and representative temporal knowledge graph representation for downstream tasks.

[0028] The learning method for temporal knowledge graph representation based on market subject dishonesty information provided in this embodiment has the following beneficial effects: This method uses a random walk dynamic matching strategy to fuse multi-source information through the first stage of untrustworthy entity recognition, which can effectively reduce the matching errors caused by the high similarity of entity names and the heterogeneity of data sources, and improve the reliability of entity matching. At the same time, the data of text modality and image modality are combined to comprehensively capture the multi-faceted concept changes covering different semantic directions, overcoming the problem of single data source. Through the second stage of relationship extraction, based on message passing function and graph neural network, the data context information is fully utilized to more comprehensively understand the complex relationship between entities and enhance the relevance information. Finally, the branch interaction convolutional network is used to realize the interaction between untrustworthy entities and relationships, effectively solving the irregularity and heterogeneity of different types of entities and relationships in embedding representation, and unifying them into a structural space. Through multimodal data processing, branch interaction convolution, random walk strategy and message passing function and other technologies, the effective representation and fusion of complex untrustworthy information is realized, providing feature extraction capabilities, which helps to reduce the complexity of downstream tasks and improve processing efficiency.

[0029] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0030] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.

Claims

1. A learning method for temporal knowledge graph representation based on market subject dishonesty information, characterized in that: include: S1: Obtaining dishonesty data of market entities including images, texts and audios; Preprocessing the market entity's dishonesty data to obtain dishonesty text data and dishonesty image data; S2: semantically analyze the untrustworthy text data to generate a untrustworthy entity text vector; extract features of the untrustworthy image data containing untrustworthy information to form a untrustworthy entity visual vector; aggregate the untrustworthy entity text vector and the untrustworthy entity visual vector to obtain a untrustworthy entity vector; S3: Encode the dishonest text data and the dishonest image data into a first sequence and a second sequence respectively, extract time information as a time weight, and update the first sequence and the second sequence converted based on the time weight respectively to obtain a first updated sequence and a second updated sequence; merge the first updated sequence and the second updated sequence to obtain a dishonest relationship vector; S4: Generate a temporal knowledge graph representation based on the default entity vector and the default relationship vector.

2. The learning method for temporal knowledge graph representation based on market subject dishonesty information according to claim 1 is characterized in that: The pre-processing comprises: Use audio recognition technology to convert the audio data in the market entity's dishonesty data into second text data; Combining the text data in the market entity dishonesty data with the second text data into the dishonesty text data; The image data in the market entity's dishonest data is used as the dishonest image data.

3. The learning method for temporal knowledge graph representation based on market subject dishonesty information according to claim 1 is characterized in that: The semantic analysis of the dishonest text data includes: Step 1: By statistically learning several annotated corpora of dishonest information, the frequencies of multiple production rules are obtained, and then the probability of each production rule is calculated. The calculation formula is: ； in, Representing production rules probability; express count function; Represents a non-terminal symbol; Represents a combination of non-terminal symbols or a terminal symbol; Indicates is any right side of all production rules on the left side; Indicates is the set of all production rules on the left; Step 2: For each word in the dishonest text data, find all production rules that are on the right side of the production rule, and initialize the table, which is expressed as: ； in, Indicates that it contains i Characters to i +1 character table; Indicates all is the production rule on the right side; Representing production rules probability; Indicates i words; Step 3: For each substring length in the dishonest text data Fill the table from small to large, expressed as: ； ； ； in, Indicates that it contains i Characters to A table of characters; Indicates that from i Characters to k A substring of characters can be derived situation; Indicates that from k Characters to A substring of characters can be derived situation; Indicates that it contains i Characters to k A table of characters; Indicates that it contains k Characters to A table of characters; Indicates that from i Characters to All substrings of characters k A collection of characters; express The set of production rules for ; Step 4: Use the Lookback Table back [ i ][ j ] Record the generation path of the table each time it is filled, recursively traverse the backtracking table, and construct the parsing number; Step 5: Each untrustworthy entity name phrase extracted from the parsed data is used as input data to the trained Phrase2Vec model, and the corresponding untrustworthy entity text vector is output.

4. The learning method for temporal knowledge graph representation based on market subject dishonesty information according to claim 1 is characterized in that: The extracting of the features containing the dishonest information in the dishonest image data comprises: Step 1: Take each image region in the dishonest image data as a node, take the spatial proximity of pixels between the image regions as a weighted edge, and construct a graph structure; Step 2: Use a first-order graph convolutional neural network to perform preliminary feature extraction on the graph structure to obtain preliminary features, which are expressed as: ； in, Indicates preliminary features; represents the activation function; D represents the degree matrix; A represents the adjacency matrix of the graph structure; X represents the feature representation of the node in the graph structure; Represents the weight matrix of a first-order graph convolutional neural network; Step 3: Using the default entity text vector as the semantic hint vector, the attention weight is introduced through the attention mechanism, the aggregation process of the first-order graph convolutional neural network is adjusted, the preliminary features of all nodes are aggregated, and the aggregated features are obtained. The calculation formula is: ； ； in, represents the first aggregate feature; Indicates i The attention weight of each node; Indicates i Preliminary characteristics of nodes; represents the softmax function; represents the attention function; represents the semantic hint vector; Step 4: Use a second-order graph convolutional neural network to further aggregate the aggregated features based on the neighboring nodes of each node to obtain the visual vector of the dishonest entity of each node. The calculation formula is: ； in, Indicates i The visual vector of the dishonest entity of each node; Indicates i The set of neighbor nodes of a node; Represents the weight matrix of a second-order graph convolutional neural network.

5. The learning method for temporal knowledge graph representation based on market subject dishonesty information according to claim 1 is characterized in that: The aggregating the untrustworthy entity text vector and the untrustworthy entity visual vector comprises: Step 1: Take the set of visual vectors of the dishonest entities as the source domain, the set of text vectors of the dishonest entities as the target domain, set the random walk length L, and use the source vector walk of length L generated by the flexible sampling strategy. x The elements are generated as follows: ； ； ； in, Indicates the source vector x The probability of generating an element; Indicates the source vector x elements; Indicates the source vector x- 1 element; Indicates i A text vector of a dishonest entity; Indicates j The untrustworthy text vector of nodes; represents the target domain; represents the source domain; express and The transition probability between Z represents the normalization constant; represents the walking guidance function; express and The weight between r represents the first hyperparameter; t represents the second hyperparameter; express and The distance between Step 2: Perform random walks in two directions and define the sequence obtained in one random walk as the transmission sequence; The first direction is to randomly select any vector in the source domain as the starting point, and access the next vector with its corresponding transition probability until the vector in the target domain is reached or the number of accessed vectors exceeds the threshold T, and then stop wandering; The second direction is to randomly select any vector in the target domain as the starting point, and access the next vector with its corresponding transition probability until the vector in the source domain is reached or the number of accessed vectors exceeds the threshold T, and then stop wandering; Step 3: Perform weighted summation on the untrustworthy entity visual vector and the untrustworthy entity text vector according to the transmission sequence to obtain the untrustworthy entity vector.

6. The learning method for temporal knowledge graph representation based on market subject dishonesty information according to claim 1 is characterized in that: The process of obtaining the first update sequence and the second update sequence includes: Step 1: Use the pre-trained BERT model and the ViT model to encode the untrustworthy text data and the untrustworthy image data respectively, and obtain the first sequence and the second sequence respectively; and extract the time information in the untrustworthy text data and the untrustworthy image data respectively as the first time weight and the second time weight respectively; Step 2: Use the fully connected layer of the graph neural network, and based on the first time weight and the second time weight, perform dimensional transformation on the first sequence and the second sequence respectively to obtain the transformed first sequence and the transformed second sequence respectively. The calculation formula is: ； ； in, represents the first sequence after conversion; represents the first time weight; represents the first sequence; represents the first bias term; represents the second sequence after conversion; represents the second time weight; represents the second sequence; represents the second bias term; Step 3: Aggregate the first dishonest messages of the neighbor sequence of the first sequence after conversion to obtain the first dishonest relationship feature, which is calculated as follows: ； ； in, Indicates i The first characteristic of dishonest relationship; Indicates i The set of neighbor sequences of the first sequence after transformation; Indicates j The first dishonest message of a neighbor sequence; Indicates i The first sequence after the conversion j neighbor sequences; o represents the cross-modal relationship vector; Aggregate the second dishonest messages of the neighbor sequence of the second sequence after conversion to obtain the second dishonest relationship feature, which is calculated as follows: ； ； in, Indicates i The second characteristic of dishonest relationship; Indicates i A set of neighbor sequences of the converted second sequence; Indicates j The second dishonest message of the neighbor sequence; Indicates i The second sequence after the conversion j Neighbor sequence; Step 4: The converted first sequence is merged with the first default relationship feature to obtain the first updated sequence. The calculation formula is: ； in, Indicates i A first update sequence; represents the weight parameter; Indicates i The first sequence after conversion; The converted second sequence is combined with the second default relationship feature to obtain the second updated sequence, which is calculated as follows: ； in, Indicates i A second update sequence; Indicates i A second sequence after conversion.

7. The learning method for temporal knowledge graph representation based on market subject dishonesty information according to claim 1 is characterized in that: The fusing the first update sequence and the second update sequence comprises: Step 1: Use the feedforward layer of the graph neural network to perform linear transformation on the first update sequence and the second update sequence to obtain text modality coding and image modality coding respectively. The calculation formula is: ； ； in, Indicates text modal encoding; Represents image modality encoding; ReLU activation function. Indicates i A first update sequence; Indicates i A second update sequence; A first weight matrix representing the text modality; A second weight matrix representing the text modality; A first weight matrix representing the image modality; a second weight matrix representing the image modality; Indicates the first deviation of the text modality; A second deviation representing the text modality; represents the first deviation of the image modality; represents the second deviation of the image modality; Step 2: Project the text modality code and the image modality code to a common space, and assign the first time weight and the second time weight extracted from the dishonest text data and the dishonest image data to each other, and connect the text modality code and the image modality code assigned with the time weight to obtain the modality fusion feature, and the calculation formula is: ； ； ； in, Represents modality fusion features; represents the text modality encoding given the second time weight; P represents the concatenation operation; represents the image modality encoding assigned to the first time weight; represents the first time weight; represents the second time weight; Step 3: In a set of fully connected layers, the modality fusion features are used as query, key, and value respectively, and the attention score is calculated as follows: ； ； ； ； in, represents the attention score; represents the softmax function; Q represents the query; K represents the key; V represents the value; T represents the transpose; Represents the dimension of the key; represents the query weight matrix; represents the key weight matrix; represents the value weight matrix; Step 4: Repeat step 3, traverse several groups of fully connected layers, and obtain the attention scores in each group of fully connected layers; Step 5: Apply the softmax function to normalize the attention scores, perform weighted summation on the normalized attention scores, and map the summation result to the default relationship vector through a fully connected layer.

8. The learning method for temporal knowledge graph representation based on market subject dishonesty information according to claim 1 is characterized in that S4 include: S4.1: reshape the default entity vector and the default relationship vector into 2D matrix forms respectively, and obtain the 2D transformed default entity vector and the 2D transformed default relationship vector respectively; S4.2: The branched interactive convolutional network generates two parallel convolutional branches, each of which contains three different convolutional layers; In the first convolution branch, three different convolution layers are used to perform three convolutions on the 2D transformed untrustworthy entity vector, respectively, to obtain a first untrustworthy entity convolution result, a second untrustworthy entity convolution result, and a third untrustworthy entity convolution result, respectively; In the second convolution branch, three different convolution layers are used to perform three convolutions on the 2D transformed default relationship vector, respectively, to obtain a first default relationship convolution result, a second default relationship convolution result, and a third default relationship convolution result, respectively; S4.3: Fusing the first dishonest entity convolution result with the first dishonest relationship convolution result to obtain a first interaction result; The second dishonest entity convolution result is merged with the second dishonest relationship convolution result to obtain a second interaction result; The third dishonest entity convolution result is merged with the third dishonest relationship convolution result to obtain a third interaction result; S4.4: reshape the first interaction result, the second interaction result, and the third interaction result into 2D matrix forms respectively, and connect the reshaped first interaction result, the second interaction result, and the third interaction result to obtain a connection result; S4.5: Perform linear transformation and nonlinear mapping on the connection results to generate the temporal knowledge graph representation.

9. The learning method for temporal knowledge graph representation based on market subject dishonesty information according to claim 8 is characterized in that: S4.5 includes: Expanding the connection result into a vector, and inputting the connection result represented by the vector into a fully connected layer to obtain a mapping result; The mapping result is subjected to a ReLU activation function to obtain a nonlinearly represented mapping result; The reshape function is used to reshape the dimension of the mapping result of the nonlinear representation to obtain the temporal knowledge graph representation.

10. The learning method for temporal knowledge graph representation based on market subject dishonesty information according to claim 9 is characterized in that: The expansion formula of the connection result is: ； in, Represents the concatenation result of the vector representation; Represents the reshape function; Indicates the connection result; Indicates the size of the first dimension after flattening; Represents the dimension of the convolution kernel in the convolution layer; Represents the total width of the convolution kernel in the first convolution branch; Represents the total height of the convolution kernel in the second convolution branch.

Images