Cross-modal abnormal semantic association analysis method for heterogeneous data

Abstract

The invention provides a cross-modal abnormal semantic association analysis method for heterogeneous data. The method comprises the following steps: S1, constructing by adopting an ontology modeling language to obtain an account entity class ontology, a transaction entity class ontology, a semantic entity class ontology and a time sequence entity class ontology, and establishing an association relationship to form a domain knowledge ontology library; s2, collecting multi-source heterogeneous data, wherein the multi-source heterogeneous data comprises at least two of text data, voice data and behavior data of an account or equipment; s3, performing feature extraction on the multi-source heterogeneous data to obtain a plurality of feature vectors; s4, on the basis of the incidence relation established in the domain knowledge ontology library, converting and splicing the feature vectors through the projection matrix to form a unified semantic representation vector; and S5, analyzing the semantic representation vector based on a pre-configured hierarchical analysis strategy to obtain a corresponding anomaly judgment result. The method has the beneficial effects that the accurate, efficient and explainable perception and judgment of the cross-modal abnormal behavior can be realized.

Description

Technical Field

[0001] This invention relates to the technical field of heterogeneous data analysis, and more specifically, to a method for cross-modal anomaly semantic association analysis of heterogeneous data. Background Technology

[0002] With the deep integration of the internet and big data technologies, data generated in fields such as finance, e-commerce, and social media exhibits significant characteristics of being multi-source and heterogeneous. This data typically includes multiple modalities such as text (e.g., chat logs, transaction notes), voice (e.g., customer service calls, voice messages), and user / device behavior data (e.g., operation logs, transaction records). Against this backdrop, accurately and efficiently identifying abnormal behavior (e.g., financial fraud, illegal operations, organized crime) from this heterogeneous data has become a core challenge in the field of information security and risk control.

[0003] Traditional anomaly detection methods are mostly designed for single-modality or structured data, such as rule-based transaction monitoring systems or statistical model-based text anomaly classification. However, these methods have fundamental limitations when dealing with complex cross-modal anomalies. First, the problem of modality fragmentation is prominent; data from different modalities are often analyzed independently, making it impossible to capture the deep semantic relationships between them. For example, a subtle threat in a voice call and subsequent abnormal transfer behavior are treated as isolated events in an independently analyzed system, resulting in a high false negative rate. Second, there is a semantic gap; there is a lack of a unified semantic understanding framework between heterogeneous data. Simple feature-level concatenation is insufficient to express the inherent logic between "emotional tension in the voice" and "sudden increase in transaction amount," leading to feature engineering failure and a high false positive rate. In addition, existing methods mostly rely on static rule bases, which are severely lacking in dynamic adaptability to constantly evolving new attack patterns, and have high maintenance costs.

[0004] To address this, while the industry has attempted to introduce technologies such as multimodal fusion and knowledge graphs, existing solutions still have significant shortcomings. For example, fusion methods based on simple feature concatenation cannot achieve deep semantic alignment; and knowledge graphs, which are only applicable to structured data, lack effective mapping mechanisms for unstructured text and speech. In summary, existing technologies have not yet provided a systematic solution that can truly break down the semantic barriers between heterogeneous modalities, achieve interpretable deep association analysis, and balance detection efficiency and accuracy.

[0005] Therefore, there is an urgent need in this field for a new cross-modal anomaly semantic association analysis method that can construct a unified semantic representation vector and place multi-source heterogeneous data under the same set of domain knowledge system for association analysis, thereby achieving accurate, efficient and interpretable perception and judgment of cross-modal anomaly behavior. Summary of the Invention

[0006] The technical problem to be solved by this invention is how to achieve accurate, efficient and interpretable perception and judgment of cross-modal anomalous behavior. In order to overcome the defects of the above-mentioned prior art (or related art), this invention provides a cross-modal anomalous semantic association analysis method for heterogeneous data. This invention provides a method for cross-modal anomaly semantic association analysis of heterogeneous data, comprising the following steps: Step S1: Using ontology modeling language, construct account entity ontology, transaction entity ontology, semantic entity ontology, and temporal entity ontology, and establish corresponding relationships to form a domain knowledge ontology library; Step S2: Collect multi-source heterogeneous data, wherein the multi-source heterogeneous data includes at least two of the following: text data, voice data, and account or device behavior data; Step S3: Extract features from the multi-source heterogeneous data to obtain multiple feature vectors; Step S4: Based on the association relationship established in the domain knowledge ontology, the feature vectors are transformed and concatenated to form a unified semantic representation vector through the projection matrix obtained by pre-training. Step S5: Analyze the semantic representation vector based on the pre-configured hierarchical analysis strategy to obtain the corresponding anomaly determination result.

[0007] Compared with existing technologies, the present invention provides a cross-modal anomaly semantic correlation analysis method for heterogeneous data, which has the following advantages: This invention, by constructing a domain knowledge ontology and a unified semantic representation vector, for the first time correlates and unifies fundamentally different data such as text, speech, and behavior at the semantic level. This enables the analysis of the inherent connections between different types of data, greatly improving the ability to identify hidden cross-modal associations. Furthermore, the entire process is guided by the domain knowledge ontology, forming a traceable and understandable analysis chain from ontology construction to feature projection and analysis decision-making. This makes the anomaly judgment results verifiable, greatly improving the credibility of the anomaly judgment results and their usability in fields such as financial risk control and security auditing. Moreover, by combining multi-source heterogeneous data with hierarchical analysis strategies, it can capture both rule-based, known, and explicit anomaly patterns, as well as model-based, unknown, and implicit anomaly patterns. This achieves multi-dimensional and three-dimensional detection of complex anomaly behavior, significantly reducing the false negative rate and enabling accurate, efficient, and interpretable perception and judgment of cross-modal anomaly behavior.

[0008] In one possible implementation, the attributes of the account entity class in step S1 include account ID, account type, and risk level; the attributes of the transaction entity class include transaction amount, transaction frequency, and associated account; the attributes of the semantic entity class ontology include keywords extracted from text and sentiment scores calculated from speech; and the attributes of the temporal entity class ontology include event timestamp and event sequence number.

[0009] Compared with existing technologies, the above technical solution can clearly define the key attributes of account entity ontology, transaction entity ontology, semantic entity ontology, and time-series entity ontology, so that cross-modal association is no longer a vague soft association, but a hard association based on specific attributes such as risk level, transaction frequency, sentiment score, and timestamp, which greatly reduces semantic ambiguity.

[0010] In one possible implementation, the sources of the text data in step S2 include application programming interfaces (APIs), web crawlers, and log files; the sources of the voice data include call recordings and voice conversation streams; and the sources of the behavioral data include user operation logs and transaction records.

[0011] Compared with existing technologies, the above technical solution can cover a variety of data sources, from structured interfaces to unstructured content, indicating that the method of the present invention can be flexibly deployed in a variety of practical business scenarios such as online risk control, customer service quality inspection, and internal audit, and has strong versatility.

[0012] In one possible implementation, during step S3, when extracting features from the text data, a pre-trained language model fine-tuned by the domain knowledge ontology is used to convert the text data into a high-dimensional semantic vector as the feature vector.

[0013] Compared with existing technologies, the above-mentioned technical solution can improve the representational ability of feature vectors by using a pre-trained language model finely tuned by a domain knowledge ontology, which enables the feature vectors obtained from text data conversion to not only contain general semantics but also domain-specific risk semantics.

[0014] In one possible implementation, in step S3, when extracting features from the speech data, the speech data is converted into speech text using an acoustic model and sentiment analysis is performed to obtain a sentiment score. Subsequently, the speech text and the sentiment score are vectorized to obtain the feature vector.

[0015] Compared with existing technologies, the above-mentioned technical solution can not only convert speech data to text, but also analyze emotions. It transforms both the "what was said" and "how it was said" information in the speech data into analyzable feature vectors, effectively capturing complex risk signals that combine content and emotion, such as "urgently requesting a transfer," thereby improving the representational ability of the feature vectors.

[0016] In one possible implementation, in step S3, when extracting features from the behavioral data, a time series graph is constructed using accounts or devices as nodes and the interactions between accounts or devices as edges, and the graph topology features between each node are calculated as the feature vector.

[0017] Compared with existing technologies, the above technical solution can transform isolated user behaviors into a global network relationship graph by constructing a time series graph and extracting topological features. This can intuitively discover group-related abnormal patterns such as central nodes controlling multiple accounts and abnormal transactions spreading in specific communities, thereby improving the representational ability of feature vectors.

[0018] In one possible implementation, in step S4, the projection matrix is ​​a dual-tower deep network, where one tower is used to process text modal features in each of the feature vectors, and the other tower is used to process behavioral modal features or speech modal features in each of the feature vectors. The alignment of the semantic representation vectors output by the two towers is optimized by an improved contrastive loss function.

[0019] Compared with existing technologies, the above technical solution can learn the deep abstract features in each feature vector through a dual-tower deep network, and then optimize the "narrowing" and "pushing" in a unified space through the improved contrastive loss function, thereby achieving accurate alignment of cross-modal semantics.

[0020] In one possible implementation, the hierarchical analysis strategy in step S5 is: The semantic representation vector is matched with a predefined abnormal pattern rule base by a rule engine to obtain a matching result. At the same time, a graph neural network is used to perform deep semantic association analysis on the heterogeneous graph constructed by the semantic representation vector to obtain the abnormal propagation path. Then, the abnormal determination result is obtained by combining the matching result and the abnormal propagation path.

[0021] Compared with existing technologies, the above technical solution can quickly and accurately target known threats through rule engines and deeply and covertly uncover unknown connections through graph neural networks. The combination of the two can ensure real-time performance without sacrificing the depth of analysis.

[0022] In one possible implementation, the graph neural network used in step S5 is a heterogeneous graph attention network. The node types of the heterogeneous graph attention network include the account entity ontology, the transaction entity ontology, the semantic entity ontology, and the temporal entity ontology. The heterogeneous graph attention network learns the importance weights of the associations between different types of nodes through a multi-layer attention mechanism to identify abnormal association paths as the abnormal propagation paths.

[0023] Compared with existing technologies, the above technical solution can utilize heterogeneous graph attention networks to not only discover abnormal propagation paths, but also clearly indicate "which account, through which transaction, and associated keyword" constitutes the main abnormal propagation path by weighting importance, providing clear clues for investigation.

[0024] In one possible implementation, in step S5, a weighted voting mechanism is used for the matching result and the anomaly propagation path, and the rule matching degree and model confidence degree are used as weighting factors to generate the anomaly determination result.

[0025] Compared with existing technologies, the above-mentioned technical solution can cleverly combine rule matching degree with model confidence degree through a weighted voting mechanism, avoiding the limitations of a single method and achieving a balance between high accuracy and low false positive rate. Attached Figure Description

[0026] Figure 1 This is a flowchart of the steps of the present invention. Detailed Implementation

[0027] First, those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention. Those skilled in the art can make adjustments as needed to adapt to specific application scenarios.

[0028] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0029] See Figure 1 This invention discloses a method for cross-modal anomaly semantic association analysis of heterogeneous data, comprising the following steps: Step S1: Using ontology modeling language, construct account entity ontology, transaction entity ontology, semantic entity ontology, and temporal entity ontology, and establish corresponding relationships to form a domain knowledge ontology library; Step S2: Collect multi-source heterogeneous data, and the multi-source heterogeneous data includes at least two of the following: text data, voice data, and account or device behavior data; Step S3: Extract features from the multi-source heterogeneous data to obtain multiple feature vectors; Step S4: Based on the relationships established in the domain knowledge ontology, the feature vectors are transformed and concatenated to form a unified semantic representation vector through the projection matrix obtained through pre-training. Step S5: Analyze the semantic representation vector based on the pre-configured hierarchical analysis strategy to obtain the corresponding anomaly determination result.

[0030] In this embodiment of the invention, step S1 is the cornerstone of breaking down the barriers of modal semantics. Its goal is to place scattered, multi-source heterogeneous data under a unified, machine-understandable semantic framework. In this step, the W3C standard OWL 2 DL language is used for ontology construction, and Protégé is used as the ontology editing tool. The OWL 2 DL language achieves a good balance between expressive power and computational integrity, making it suitable for complex knowledge reasoning.

[0031] In this embodiment of the invention, the core entity class individuals and attribute definitions are as follows: The attributes of the account entity class are defined as accountID: xsd:string (unique identifier for account ID), accountType: xsd:string (account type enumeration value: "individual", "corporate", "device account"), and riskLevel: xsd:float (risk level, range [0,1], initial value 0, dynamically updated), serving as the core anchor connecting all behavioral data; the attributes of the transaction entity class are defined as amount: xsd:decimal (transaction amount), frequency: xsd:integer (transaction frequency within a specific time window), fromAccount: owl:Thing (pointing to the account entity), and toAccount: owl:Thing (pointing to the account entity), used to characterize the flow of funds or value; the attributes of the semantic entity class are defined as keyword:xsd:string (keywords extracted from text, such as "margin", "emergency"), sentimentScore: xsd:float(sentiment score, range [-1,1], -1 for extremely negative, 1 for extremely positive), is used to carry abstract semantics extracted from unstructured data; the attributes of the time-series entity class are defined as timestamp: xsd:dateTime(the exact time the event occurred, i.e., the event timestamp, with millisecond precision) and sequenceID: xsd:integer(the order of the event in the sequence, i.e., the event sequence number), which are used to timestamp all behaviors and semantics, enabling them to have time-series analysis capabilities.

[0032] In this embodiment of the invention, object properties of the OWL 2 DL language are used to define the relationships between individuals, such as: initiatesTransaction (Domain: Account, Range: Transaction), containsKeyword (Domain: Transaction|Voice, Range: Semantic), hasTemporalContext (Domain: Account|Transaction|Semantic, Range: Temporal). At the same time, logical relationships between data properties are defined, such as through the SWRL rule: hasSentiment(?s, ?score) ^ greaterThan(?score, 0.7) -> hasUrgencyLevel(?t, "HIGH"), which means that if the sentiment score of a semantic entity is greater than 0.7, then the urgency level of its associated transaction entity is "high".

[0033] In this embodiment of the invention, step S2 is responsible for aggregating various types of data with high concurrency and low latency. The sources of text data include application programming interfaces (APIs), web crawlers, and log files. APIs obtain user comments and transaction notes in JSON / XML format by calling open APIs of financial institutions and social platforms. Web crawlers target forums and news websites, using the Scrapy framework to build distributed crawlers for incremental crawling. Log files are collected in real-time by the FileBeat log collector, which monitors and parses application logs and system operation logs. The sources of voice data include call recordings and voice conversation streams. Call recordings obtain WAV format audio streams from the call center system, while voice conversation streams receive Opus-encoded audio packets in real-time from Apps and IoT devices. The sources of behavioral data include user operation logs and transaction records. User operation logs are obtained by embedding points in the front-end / mobile SDK to record user clicks, browsing, login, and other events. Transaction records are directly obtained from the business database or data warehouse through Debezium's CDC real-time capture.

[0034] In this embodiment of the invention, step S3 aims to transform the raw data into standardized features for machine learning. When extracting features from text data, a pre-trained language model fine-tuned by a domain knowledge ontology is used to convert the text data into high-dimensional semantic vectors as feature vectors. When extracting features from speech data, the speech data is converted into speech text through an acoustic model and sentiment analysis is performed to obtain sentiment scores. Subsequently, the speech text and sentiment scores are vectorized to obtain feature vectors. When extracting features from behavioral data, a time series graph is constructed with accounts or devices as nodes and the interactions between accounts or devices as edges, and the graph topology features between each node are calculated as feature vectors.

[0035] In this embodiment of the invention, the feature extraction of text data includes clear word segmentation and vectorization. Clear word segmentation uses regular expressions to remove HTML tags and special characters, employs the Jieba word segmenter, and loads a custom domain risk dictionary (such as "pig butchering scam" or "money laundering") for enhanced word segmentation. Vectorization uses the RoBERTa-wwm-ext Chinese pre-trained model, using text descriptions and association rules generated by the constructed domain knowledge ontology as training data to further pre-train and supervised fine-tune the RoBERTa model, making it better understand the language of the financial risk control field. The 768-dimensional vector corresponding to the [CLS] label of the last layer of the RoBERTa model is taken as the feature vector of the text data.

[0036] In this embodiment of the invention, the feature extraction of speech data includes speech recognition processing, voiceprint sentiment analysis processing, and vector fusion processing. Speech recognition processing employs the Conformer model to transcribe the speech stream into text in real time. This Conformer model achieves higher recognition accuracy than traditional models in noisy environments. Voiceprint sentiment analysis processing uses the ECAPA-TDNN model to extract a 256-dimensional voiceprint feature vector for identity association. In parallel, prosodic features such as fundamental frequency, energy, and spectral centroid are extracted from the audio and input into a pre-trained LSTM sentiment classification network, outputting a continuous sentiment score. Vector fusion processing involves concatenating the vector obtained from the ASR-transcribed text with the sentiment score to form the feature vector of the speech data.

[0037] In this embodiment of the invention, the feature extraction of behavioral data includes time series graph construction and graph topology feature calculation. In the time series graph construction, each account or device is a node. If there are behaviors such as transactions, communication, or associated logins between nodes, an edge is established. The edge can be assigned weights, such as transaction amount and interaction frequency. The graph structure is dynamically maintained in memory using NetworkX. In the graph topology feature calculation, the degree centrality, betweenness centrality, and clustering coefficient of each node are calculated in real time. For transaction behaviors, the variance of transaction amount and the coefficient of variation of transaction time interval for each node are additionally calculated. These statistics are combined into a feature vector to characterize the behavioral pattern of the entity.

[0038] In this embodiment of the invention, step S4 is the core technology for achieving unified semantic understanding. The dual-tower deep network architecture includes a text tower, an action / speech tower, and a projection matrix. The text tower is based on a 3-layer MLP (Multilayer Perceptron), taking a 768-dimensional feature vector of text data as input. After two layers of nonlinear transformation, it outputs a 512-dimensional feature vector. The action / speech tower is also a 3-layer MLP, taking feature vectors of action or speech data as input and mapping them to another 512-dimensional vector space. The projection matrix is ​​essentially the trainable parameters of these two MLP networks. The improved contrastive loss function aims to make the same... The text data feature vector and behavioral data feature vector generated by an event (such as a fraud) are very close in the projected space, but far away from the features of different events. Specifically, the NT-Xent loss function is used and improved by introducing structured constraints for ontology association. During training and deployment, a large number of cross-modal samples are used to train the dual-tower deep network. After training, the two towers are used as fixed feature projectors. For new data, the 512-dimensional vectors after projection are obtained through the two towers respectively, and then concatenated to form a 1024-dimensional unified semantic representation vector.

[0039] In this embodiment of the invention, the hierarchical analysis strategy in step S5 involves matching the semantic representation vector with a predefined anomaly pattern rule base using a rule engine to obtain a matching result. Simultaneously, a graph neural network is used to perform deep semantic association analysis on the heterogeneous graph constructed from the semantic representation vector to obtain anomaly propagation paths. Then, the matching result and the anomaly propagation path are combined to obtain an anomaly determination result. The anomaly pattern rule base of the rule engine contains predefined rules based on a domain knowledge ontology. These rules include at least: single-modal threshold rules, cross-modal association rules, and temporal evolution rules. A single-modal threshold rule is, for example, IF new registered account AND transaction frequency > 5 within 24 hours THEN trigger alarm (A1). A cross-modal association rule is, for example, IF semantic entity.sentiment score < -0.8 AND transaction entity.amount > 10000 THEN trigger alarm (A2). A temporal evolution rule is, for example, IF account.risk level increases by 3 levels consecutively within 1 hour THEN trigger alarm (A3). Subsequently, the attribute values ​​in the unified semantic representation vector are matched with the anomaly pattern rule base, and a rule matching score of 0 to 1 is output as the matching result.

[0040] In this embodiment of the invention, the graph neural network used in step S5 is a heterogeneous graph attention network. The node types of the heterogeneous graph attention network include account entity ontology, transaction entity ontology, semantic entity ontology, and temporal entity ontology. The heterogeneous graph attention network learns the importance weights of the associations between different types of nodes through a multi-layer attention mechanism to identify abnormal association paths and abnormal propagation paths. In the construction of the heterogeneous graph attention network, a unified semantic representation vector is attached to the corresponding entity node. For example, account nodes have feature vectors of their behavioral data, and transaction nodes have vectorized attributes such as amount and frequency. In heterogeneous graph attention network analysis, node types include four categories: account, transaction, semantic, and temporal. The network structure adopts a 3-layer HGAT. In each layer, nodes aggregate their neighbor information through an attention mechanism. The calculation of attention weights considers the edge type (e.g., "initiated transaction" or "contains keywords") and node type. When identifying anomaly propagation paths, the HGAT model is trained to perform link prediction or anomaly node classification tasks. The model calculates an anomaly confidence score of 0 to 1 for edges and nodes in the heterogeneous graph attention network. By visualizing the path with the highest attention weight, the "anomaly propagation path" can be clearly located, for example: Account A - (high-frequency trading) -> Account B - (associated with the same risky keywords) -> semantic entity C. A weighted voting mechanism is used to calculate the final anomaly score, and the calculation formula is as follows:

[0041] in, Indicates the final abnormal score, This indicates adjustable hyperparameters. When pursuing high precision, the hyperparameters can be biased towards rules; when pursuing high recall, the hyperparameters can be biased towards the model. This represents the rule matching score. Indicates the confidence level of anomalies; Set a threshold θ; when S_final > θ, it is considered an anomaly.

[0042] In the description of this invention, the references to "one embodiment," "some embodiments," "in this embodiment," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0043] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for cross-modal anomaly semantic association analysis of heterogeneous data, characterized in that, Includes the following steps: Step S1: Using ontology modeling language, construct account entity ontology, transaction entity ontology, semantic entity ontology, and temporal entity ontology, and establish corresponding relationships to form a domain knowledge ontology library; Step S2: Collect multi-source heterogeneous data, wherein the multi-source heterogeneous data includes at least two of the following: text data, voice data, and account or device behavior data; Step S3: Extract features from the multi-source heterogeneous data to obtain multiple feature vectors; Step S4: Based on the association relationship established in the domain knowledge ontology, the feature vectors are transformed and concatenated to form a unified semantic representation vector through the projection matrix obtained by pre-training. Step S5: Analyze the semantic representation vector based on the pre-configured hierarchical analysis strategy to obtain the corresponding anomaly determination result.

2. The cross-modal anomaly semantic association analysis method according to claim 1, characterized in that, The attributes of the account entity class in step S1 include account ID, account type, and risk level; the attributes of the transaction entity class include transaction amount, transaction frequency, and associated account; the attributes of the semantic entity class ontology include keywords extracted from text and sentiment scores calculated from speech; and the attributes of the temporal entity class ontology include event timestamp and event sequence number.

3. The cross-modal anomaly semantic association analysis method according to claim 1, characterized in that, The sources of the text data in step S2 include application programming interfaces (APIs), web crawlers, and log files; the sources of the voice data include call recordings and voice conversation streams; and the sources of the behavioral data include user operation logs and transaction records.

4. The cross-modal anomaly semantic association analysis method according to claim 1, characterized in that, In step S3, when extracting features from the text data, a pre-trained language model fine-tuned by the domain knowledge ontology is used to convert the text data into a high-dimensional semantic vector as the feature vector.

5. The cross-modal anomaly semantic association analysis method according to claim 1, characterized in that, In step S3, when extracting features from the speech data, the speech data is converted into speech text through an acoustic model and sentiment analysis is performed to obtain a sentiment score. Then, the speech text and the sentiment score are vectorized to obtain the feature vector.

6. The cross-modal anomaly semantic association analysis method according to claim 1, characterized in that, In step S3, when extracting features from the behavioral data, a time series graph is constructed using accounts or devices as nodes and the interactions between accounts or devices as edges, and the graph topology features between each node are calculated as the feature vector.

7. The cross-modal anomaly semantic association analysis method according to claim 1, characterized in that, In step S4, the projection matrix is ​​a dual-tower deep network, where one tower is used to process the text modal features in each feature vector, and the other tower is used to process the behavioral modal features or speech modal features in each feature vector. The alignment of the semantic representation vectors output by the two towers is optimized by an improved contrastive loss function.

8. The cross-modal anomaly semantic association analysis method according to claim 1, characterized in that, The hierarchical analysis strategy in step S5 is as follows: The semantic representation vector is matched with a predefined abnormal pattern rule base by a rule engine to obtain a matching result. At the same time, a graph neural network is used to perform deep semantic association analysis on the heterogeneous graph constructed by the semantic representation vector to obtain the abnormal propagation path. Then, the abnormal determination result is obtained by combining the matching result and the abnormal propagation path.

9. The cross-modal anomaly semantic association analysis method according to claim 8, characterized in that, The graph neural network used in step S5 is a heterogeneous graph attention network. The node types of the heterogeneous graph attention network include the account entity ontology, the transaction entity ontology, the semantic entity ontology, and the temporal entity ontology. The heterogeneous graph attention network learns the importance weights of the association between different types of nodes through a multi-layer attention mechanism to identify abnormal association paths as the abnormal propagation paths.

10. The cross-modal anomaly semantic association analysis method according to claim 8, characterized in that, In step S5, a weighted voting mechanism is used for the matching result and the anomaly propagation path, and the rule matching degree and model confidence degree are used as weighting factors to generate the anomaly judgment result.

Images