A financial content risk control method and system based on a large model

Abstract

The present application provides a financial content risk control method and system based on a large model. First, multimodal semantic features are extracted from the target financial document to obtain a text semantic space and an image semantic space; the semantic association graph of the text modality is determined according to the relational semantics of each text semantic feature in the text semantic space in the context and the syntactic structure relationship of the text semantic features; the semantic structure relationship of the image semantic space is then converted into a semantic association graph of the image modality through the overall semantic features of the image and the syntactic features of the visual objects in the image; the semantic association graph of the text modality and the semantic association graph of the image modality are subjected to inter-modal fine-grained structural fusion to obtain a multimodal fusion graph; based on the multimodal fusion graph, the extracted multimodal features are subjected to abnormal classification to determine whether the target financial document has risks. The scheme of the present application can realize fine-grained multimodal feature fusion of financial documents, thereby improving the accuracy of risk control detection.

Description

Technical Field

[0001] The present application relates to the field of financial content risk control technology, and more specifically, to a financial content risk control method and system based on a large model. Background Art

[0002] Financial content risk control refers to the monitoring and analysis of content published and disseminated in the financial field (such as financial reports, news, social media information, etc.) to identify potential false information, misleading statements or illegal content, thereby preventing financial risks. Its core goal is to ensure the authenticity, accuracy and compliance of financial information through technical means and data analysis. Therefore, accurate false information detection in financial documents has become a key task.

[0003] In the existing technology, the existing risk control detection technology (such as false information detection) has many limitations when processing financial documents. Most traditional methods rely on a single modality of text or image for information analysis and cannot fully utilize the collaborative information of multiple modalities (such as text and image). This single-modality processing method is difficult to achieve refined detection in the face of complex financial scenarios. For example, in financial reports or news, text descriptions and image content often have close semantic associations. Analyzing text or images alone may miss important cross-modal information, resulting in inaccurate identification of false information. In addition, although traditional natural language processing technology can identify semantic anomalies to a certain extent, However, its ability to understand complex grammatical structures, implicit semantic relationships, etc. is limited. Especially in financial documents, a large number of terms, industry-specific languages, and implicit semantic relationships make traditional methods difficult to cope with. In addition, existing technologies usually process image information in financial documents in a rough manner, often only performing basic image detection or object recognition, ignoring the deep semantic fine-grained information within the image and its association with the text, resulting in a one-sided understanding of the combined content of images and text, further reducing the accuracy of risk control detection (such as false information detection). Therefore, how to achieve fine-grained multimodal feature fusion of financial documents to improve the accuracy of risk control detection has become a difficult problem faced by the industry. Summary of the invention

[0004] The present application provides a financial content risk control method and system based on a large model, which can realize fine-grained multimodal feature fusion of financial documents, thereby improving the accuracy of risk control detection.

[0005] In a first aspect, the present application provides a financial content risk control method based on a large model, comprising the following steps:

[0006] Collect target financial documents to be tested for risk control from financial content platforms;

[0007] Based on the pre-trained semantic big model, multimodal semantic features are extracted from the target financial document to obtain the text semantic space of the text modality and the image semantic space of the image modality.

[0008] Constructing a semantic association tree of text elements according to the relational semantics of each text semantic feature in the context in the text semantic space, and then determining a semantic association graph of text modality according to the semantic association tree and the syntactic structure relationship of the text semantic features in the text semantic space;

[0009] The semantic structural relationship between the image semantic features in the image semantic space is converted into a semantic association graph of the image modality through the overall semantic features of the image in the target financial document and the syntactic features of the visual objects in the image;

[0010] Performing inter-modal fine-grained structural fusion on the semantic association graph of the text modality and the semantic association graph of the image modality to obtain a multi-modal fusion graph;

[0011] Multimodal features are extracted from the multimodal fusion graph, and abnormal classification is performed on the extracted multimodal features. If the classification result is abnormal, it is determined that the target financial document has risks.

[0012] Preferably, multimodal semantic feature extraction is performed on the target financial document based on the pre-trained semantic big model to obtain the text semantic space of the text modality and the image semantic space of the image modality, specifically including:

[0013] Extract all text semantic features and image semantic features in the target financial document through the pre-trained semantic big model;

[0014] The set of all text semantic features is taken as the text semantic space of the text modality;

[0015] The set of all image semantic features is taken as the image semantic space of the image modality.

[0016] Preferably, constructing a semantic association tree of text elements according to the relationship semantics of each text semantic feature in the context in the text semantic space specifically includes:

[0017] Determine the relational semantics of each text semantic feature in the text semantic space in the context by using the context window sliding analysis technology;

[0018] Based on all the relational semantics, the text semantic features are hierarchically clustered to obtain multiple semantic association clusters;

[0019] According to all the semantic association clusters, each text semantic feature in the text semantic space is organized into a tree structure according to the semantic relationship, so as to obtain a semantic association tree of the text elements.

[0020] Preferably, determining the semantic association graph of the text modality by the semantic association tree and the syntactic structure relationship of the text semantic features in the text semantic space specifically includes:

[0021] Extract the syntactic structure of different segments in the text of the target financial document;

[0022] Determining the syntactic structure relationship of the text semantic features in the text semantic space through all syntactic structures, and generating a syntactic tree based on the syntactic structure relationship;

[0023] Structural semantic association is performed on the semantic association tree and the syntactic tree to generate a semantic association graph of the text modality.

[0024] Preferably, converting the semantic structural relationship between the image semantic features in the image semantic space into a semantic association graph of the image modality by using the overall semantic features of the image in the target financial document and the syntactic features of the visual objects in the image specifically includes:

[0025] Extract the overall semantic features of the images in the target financial document;

[0026] Extracting syntactic features of visual objects within images of target financial documents;

[0027] Based on the overall semantic features and the syntactic features, construct a semantic structure relationship model;

[0028] The semantic structure relationship model is used to convert the semantic structure relationship between each image semantic feature in the image semantic space into a semantic association graph of the image modality.

[0029] Preferably, extracting multimodal features from the multimodal fusion graph, using the extracted multimodal features to perform anomaly classification, and determining whether the target financial document is false information specifically includes:

[0030] Extract node features and edge features from multimodal fusion graphs;

[0031] The node features and the edge features are classified as abnormal. If the classification result is abnormal, it is determined that the target financial document has risks.

[0032] Preferably, the target financial document includes text data and image data.

[0033] In a second aspect, the present application provides a financial content risk control system based on a large model, including:

[0034] The collection module is used to collect target financial documents to be tested for risk control from the financial content platform;

[0035] A processing module, used for extracting multimodal semantic features of target financial documents based on a pre-trained semantic large model, and obtaining a text semantic space of a text modality and an image semantic space of an image modality;

[0036] The processing module is further used to construct a semantic association tree of text elements according to the relationship semantics of each text semantic feature in the context in the text semantic space, and then determine a semantic association graph of the text modality from the semantic association tree and the syntactic structure relationship of the text semantic features in the text semantic space;

[0037] The processing module is further used to convert the semantic structural relationship between the image semantic features in the image semantic space into a semantic association graph of the image modality through the overall semantic features of the image in the target financial document and the syntactic features of the visual objects in the image;

[0038] The processing module is further used to perform inter-modal fine-grained structural fusion on the semantic association graph of the text modality and the semantic association graph of the image modality to obtain a multi-modal fusion graph;

[0039] The execution module is used to extract multimodal features from the multimodal fusion graph, and perform abnormal classification on the extracted multimodal features. If the classification result is abnormal, it is determined that the target financial document has risks.

[0040] In a third aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory stores codes, and the processor is configured to obtain the codes and execute the above-mentioned large-model-based financial content risk control method.

[0041] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned financial content risk control method based on a large model.

[0042] The technical solution provided by the embodiments disclosed in this application has the following beneficial effects:

[0043] In an embodiment of the present application, target financial documents to be subject to risk control detection are collected from a financial content platform; multimodal semantic features are extracted from the target financial documents based on a pre-trained semantic macro model to obtain a text semantic space of a text modality and an image semantic space of an image modality; a semantic association tree of text elements is constructed according to the relational semantics of each text semantic feature in the text semantic space in a context, and then a semantic association graph of the text modality is determined from the semantic association tree and the syntactic structural relationship of the text semantic features in the text semantic space; the semantic structural relationship between each image semantic feature in the image semantic space is converted into a semantic association graph of the image modality through the overall semantic features of the image in the target financial document and the syntactic features of the visual objects in the image; fine-grained inter-modal structural fusion is performed on the semantic association graph of the text modality and the semantic association graph of the image modality to obtain a multimodal fusion graph; multimodal features are extracted from the multimodal fusion graph, and the extracted multimodal features are classified as abnormal. If the classification result is abnormal, it is determined that the target financial document is at risk.

[0044] It can be seen that the present application performs inter-modal fine-grained structural fusion on the semantic association graph of the text modality and the semantic association graph of the image modality to obtain a multimodal fusion graph, further extracts multimodal features from the multimodal fusion graph, performs abnormal classification on the extracted multimodal features, and determines whether the target financial document has risks; wherein, firstly, the semantic association graph of the text modality is determined by the semantic association tree of the text elements and the syntactic structure relationship of the text semantic features. By constructing the semantic association tree of the text elements, it is helpful to deeply understand the structure and logical relationship of the text content. This process can identify the key elements in the text and their interactions, so as to more accurately capture potential false information in a complex financial context. Further, by combining the semantic association tree with the syntactic structure relationship, a richer semantic association graph of the text modality is generated. The representation of the graph makes the relationship between each semantic feature in the text and other features clearer, improves the system's ability to understand text information, and thus helps to detect false information in the text and its possible propagation path; secondly, according to the overall semantic features of the image and the syntactic features of the visual objects in the image, the semantic structural relationship between each image semantic feature in the image semantic space is converted into the semantic relationship of the image modality. The semantic association graph transforms the semantic structural relationship of the image modality through the overall semantic features and the syntactic features of the visual objects, which can fully mine the deep semantic information inside the image. This process makes the association between the visual content and the text content in the image clearly presented, avoiding the rough analysis of the image content by traditional image processing methods. In addition, the semantic association graph of the image modality also helps to more comprehensively understand the synergy between images and texts in financial documents, further improving the accuracy of false information detection; then, by fusing the fine-grained structures of the text modality and the image modality to generate a multimodal fusion graph, different modes can be realized. The deep information integration between modalities enhances the system's overall understanding of financial documents. The fused graph not only contains the information of text and image, but also can capture the complex relationship between them, greatly improving the ability to identify false information; finally, based on the generated multimodal fusion graph, the extracted multimodal features are classified as abnormalities to determine whether the target financial document is risky. This can effectively identify potential abnormal information in financial documents, thereby completing risk control detection of false information in the target financial document; in summary, the present application scheme can realize fine-grained multimodal feature fusion of financial documents, thereby improving the accuracy of risk control detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is an exemplary flow chart of a financial content risk control method based on a large model according to some embodiments of the present application;

[0046] Figure 2 is a schematic diagram of a process of determining a semantic association tree according to some embodiments of the present application;

[0047] Figure 3 is a schematic diagram of the structure of a semantic association tree according to some embodiments of the present application;

[0048] Figure 4 is a schematic diagram of an exemplary software module of a financial content risk control system based on a large model according to some embodiments of the present application;

[0049] Figure 5 It is a structural diagram of a computer device for implementing a financial content risk control method based on a large model as shown in some embodiments of the present application. DETAILED DESCRIPTION

[0050] The core of this application is to extract multimodal semantic features of the target financial document to obtain text semantic space and image semantic space; determine the semantic association graph of the text modality according to the relational semantics of each text semantic feature in the text semantic space in the context and the syntactic structure relationship of the text semantic features; then convert the semantic structure relationship of the image semantic space into the semantic association graph of the image modality through the overall semantic features of the image and the syntactic features of the visual objects in the image; perform inter-modal fine-grained structural fusion of the semantic association graph of the text modality and the semantic association graph of the image modality to obtain a multimodal fusion graph; based on the multimodal fusion graph, perform abnormal classification on the extracted multimodal features to determine whether the target financial document has risks. The scheme of this application can realize fine-grained multimodal feature fusion of financial documents, thereby improving the accuracy of risk control detection.

[0051] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods. Figure 1 , which is an exemplary flow chart of a financial content risk control method based on a big model according to some embodiments of the present application. The financial content risk control method 100 based on a big model mainly includes the following steps:

[0052] In step 101, target financial documents to be tested for risk control are collected from a financial content platform.

[0053] It should be noted that the financial content platform in this application refers to an online platform that specializes in providing financial information, data and related services. These platforms usually include news, research reports, market data, analysis tools, etc. Their main function is to provide users with timely and accurate financial information to support investment decisions and investment management.

[0054] In specific implementation, the collection of target financial documents to be subject to risk control detection from the financial content platform can be achieved in the following manner, namely: the target financial documents to be subject to risk control detection can be obtained from the database of the financial content platform, wherein the target financial document is a multimodal data document, and the target financial document includes text data and image data. It should be noted that the present application can detect false information published in the financial content platform by performing false information detection on financial documents, thereby realizing risk control detection of financial content.

[0055] In step 102, multimodal semantic feature extraction is performed on the target financial document based on the pre-trained semantic big model to obtain a text semantic space of the text modality and an image semantic space of the image modality.

[0056] In some embodiments, multimodal semantic feature extraction is performed on the target financial document based on the pre-trained semantic big model to obtain the text semantic space of the text modality and the image semantic space of the image modality, which can be achieved by the following steps:

[0057] Extract all text semantic features and image semantic features in the target financial document through the pre-trained semantic big model;

[0058] The set of all text semantic features is taken as the text semantic space of the text modality;

[0059] The set of all image semantic features is taken as the image semantic space of the image modality.

[0060] It should be noted that the semantic big model in the present application is a machine learning model obtained by pre-training with a large amount of financial document data. The algorithm structure of the semantic big model in the present application adopts Transformer. Other algorithm structures can also be used in other embodiments, which are not limited here. It should also be noted that the principle of extracting all text semantic features in the target financial document through the pre-trained semantic big model is: using word embedding technology (such as Word2Vec) to convert each word in the text of the financial document into a high-dimensional vector, and then capturing the semantic relationship between words in the high-dimensional vector, thereby obtaining text semantic features; the principle of extracting all image semantic features in the target financial document through the pre-trained semantic big model is: performing convolution recognition on the image in the financial document through a convolutional neural network, and using the obtained recognition features as image semantic features.

[0061] In specific implementation, first, a pre-trained semantic big model (such as Generative Pre-trained Transformer, GPT) can be used to process the text in the target financial document, segment the text, and represent each word as a vector through the semantic big model, and all the obtained vectors are used as text semantic features; secondly, a convolutional neural network (CNN) is used to extract image features from the images in the financial document. These features will be represented in the form of vectors, and multiple semantic vectors will be generated by capturing the semantic information of the image, and all the generated semantic vectors will be used as image semantic features; then, all the extracted text semantic features are integrated to form a set, which can be represented by a high-dimensional vector space, and each vector corresponds to a text semantic feature, thereby forming a text semantic space of the text modality; finally, all the extracted image semantic features are integrated to form an image feature set, which is represented as a high-dimensional vector space in the same way as the text modality, to obtain the image semantic space of the image modality.

[0062] It should be noted that the text semantic features in this application refer to features that can express text content and semantic information. These features are usually used to understand and analyze the meaning, emotion and theme of the text; image semantic features refer to features that can express image content and semantic information. These features are usually used to understand and analyze the content, scene and object of the image.

[0063] In step 103, a semantic association tree of text elements is constructed according to the relational semantics of each text semantic feature in the context in the text semantic space, and then a semantic association graph of the text modality is determined by the semantic association tree and the syntactic structure relationship of the text semantic features in the text semantic space.

[0064] In some embodiments, reference Figure 2 As shown, this figure is a schematic diagram of the process of determining the semantic association tree in some embodiments of the present application. In this embodiment, the semantic association tree of text elements is constructed according to the relationship semantics of each text semantic feature in the context in the text semantic space, which can be implemented by the following steps:

[0065] In step 1031, the contextual semantics of each text semantic feature in the text semantic space is determined by using a context window sliding analysis technique;

[0066] In step 1032, hierarchical clustering is performed on the text semantic features based on all relational semantics to obtain multiple semantic association clusters;

[0067] In step 1033, the text semantic features in the text semantic space are organized into a tree structure according to semantic relationships based on all semantic association clusters, so as to obtain a semantic association tree of text elements.

[0068] It should be noted that the semantic association tree in the present application is a way of structurally representing the semantics of text elements. It is used to show the relationship and hierarchy between various semantic features in the text. In the form of a tree diagram, it can clearly show how semantic features (such as words, phrases or sentences) are related to each other and their positions in the semantic space; it should also be noted that the relational semantics in the present application refers to the semantic connection between words, phrases or sentences in the text. This relationship not only involves the direct connection between words, but also includes their changes in meaning and logical relationships in a specific context.

[0069] In the specific implementation, first, use the sliding window technology to set a fixed-size context window (such as 5 words) in the text, move the window step by step, perform semantic analysis on the words in each window, and calculate the similarity between the words in the window. Here, cosine similarity can be used to obtain the relationship matrix between words in the context window. The relationship matrix can be used to reflect the relationship semantics of each text semantic feature in the context; then, all the extracted relationship matrices are used as basic data, and a hierarchical clustering algorithm (such as agglomerative hierarchical clustering) is used to cluster all the relationship matrices, and features with high similarity are classified into one category to form multiple semantic association clusters; finally, based on the formed semantic association clusters, the text semantic features are organized into a tree structure according to the semantic relationship. The root node can be a theme or a central concept, and the child nodes are related semantic features, such as Figure 3 As shown, this figure is a structural diagram of the semantic association tree shown in some embodiments of the present application, with "finance" and "market" as root nodes, and the child nodes thereunder are set to "investors" and "confidence", thereby forming a semantic association tree.

[0070] It should be noted that the present application can extract and organize semantic features in text to form a structured semantic landscape through context window sliding analysis, hierarchical clustering and the construction of a semantic association tree. The semantic association tree not only helps to deeply understand the hierarchical semantics of the text, but also can effectively discover hidden semantic contradictions or unreasonable text organization in false information detection, thereby improving the detection accuracy of the sub-control detection system.

[0071] In some embodiments, determining the semantic association graph of the text modality from the semantic association tree and the syntactic structure relationship of the text semantic features in the text semantic space can be implemented by the following steps:

[0072] Extract the syntactic structure of different segments in the text of the target financial document;

[0073] Determining the syntactic structure relationship of the text semantic features in the text semantic space through all syntactic structures, and generating a syntactic tree based on the syntactic structure relationship;

[0074] Structural semantic association is performed on the semantic association tree and the syntactic tree to generate a semantic association graph of the text modality.

[0075] It should be noted that the syntactic tree in the present application is a way of structured representation of sentence structure and grammatical relations. It displays the hierarchical relationship and dependency relationship between the various components in the sentence through a tree structure, which can help understand the grammatical structure and semantic connotation of the sentence; it should also be noted that the semantic association graph of the text modality in the present application is a graphical data structure used to represent the various semantic features in the text and their mutual relationships. It displays the semantic connection between words and phrases in the text in the form of nodes and edges, which can help to more deeply understand the content and logical structure of the text.

[0076] In the specific implementation, first, a syntactic analysis tool (such as Stanford Parser) can be used to perform syntactic analysis on the target financial document to extract the syntactic structure of each paragraph. The syntactic structure includes phrase structure (words) and its dependencies, that is, identifying the subject, predicate, object and other components in the paragraph, and marking the dependencies between them. For example, in the sentence "Investors have increased confidence in the financial market", the syntactic analysis can identify "investors" as the subject, "increase" as the predicate, and "confidence" as the object, and mark the dependencies between them; then, a syntactic tree is constructed based on the extracted syntactic structure, in which each node represents a word or phrase, and the edge represents the syntactic relationship between words, and then the dependency relationship in the syntactic structure is used to determine the text. The syntactic structure relationship of each semantic feature in this semantic space. When constructing the syntactic tree, "investor" can be used as the root node, connected to the predicate "increase", and then connected to "confidence" as the object; finally, the semantic association tree and the syntactic tree are compared to identify the similarity between the two, check whether the semantic features match the syntactic structure, and generate a semantic association graph of the text modality on this basis, match the nodes in the semantic association tree with the corresponding nodes in the syntactic tree, connect them and generate edges to represent the semantic and syntactic relationship. For example, connect "investor" from the semantic association tree to the corresponding nodes in the syntactic tree to generate a new semantic association graph of the text modality.

[0077] It should be noted that the semantic association diagram of the text modality in this application can clearly display the relationship between various semantic features and syntactic structures in the text, and can provide rich semantic context features for subsequent risk control detection. This structured graphical representation can effectively support complex text analysis tasks, thereby improving the accuracy of information processing.

[0078] In step 104, the semantic structural relationship between the image semantic features in the image semantic space is converted into a semantic association graph of the image modality through the overall semantic features of the image in the target financial document and the syntactic features of the visual objects in the image.

[0079] In some embodiments, converting the semantic structural relationship between the image semantic features in the image semantic space into a semantic association graph of image modality by using the overall semantic features of the image in the target financial document and the syntactic features of the visual objects in the image can be achieved by using the following steps:

[0080] Extract the overall semantic features of the images in the target financial document;

[0081] Extracting syntactic features of visual objects within images of target financial documents;

[0082] Based on the overall semantic features and the syntactic features, construct a semantic structure relationship model;

[0083] The semantic structure relationship model is used to convert the semantic structure relationship between each image semantic feature in the image semantic space into a semantic association graph of the image modality.

[0084] It should be noted that the semantic association graph of the image modality in the present application is a graphical data structure used to represent the semantic relationship between different visual features in the image. It organizes the key visual objects, overall semantic features and the associations between them in the image into a graph structure of nodes and edges, which is used to describe the semantic associations between the elements in the image.

[0085] In the specific implementation, first, a pre-trained deep learning model (such as ResNet) can be used to process the images in the target financial document, and the overall semantic features of the image can be output through the last layer of the deep learning model (usually a fully connected layer). The overall semantic features are expressed in the form of a vector, which represents the semantic information of the image. For example, a picture containing a financial data chart is processed, and the output feature vector can reflect the trend, color, and form of the chart. Semantic information; secondly, an object detection model (such as YOLOV5) can be used to identify specific visual objects in the image, such as text boxes, chart elements, and graphics, and then the location information (bounding box coordinates) of each visual object, the category label and the relationship between them (such as "contains", "adjacent") are extracted, and the feature vector composed of the location information, category label and the relationship between them is used as the syntactic feature of the visual object. The syntactic feature in this application describes the visual object in the image. The core goal of the method is to reveal the layout, connection mode and semantic relationship between visual objects in the image by defining nodes and edges. Then, the visual objects extracted from the overall semantic features and syntactic features can be regarded as nodes of the graph. An adjacency matrix is ​​constructed according to the syntactic relationship between the visual objects. Each element in the matrix represents the strength of the relationship between the nodes. Then, a directed graph is created using a graph structure (such as NetworkX). The visual object nodes and the relationship edges between them are integrated into the graph, thereby completing the construction of the semantic structure relationship model. Finally, the constructed semantic structure relationship model can be used to identify the semantic relationship between the image semantic features in the image semantic space. Then, each image semantic feature is used as the feature node of the graph, and the semantic relationship between the image semantic features is used as the relationship edge of the graph. Then, all feature nodes and their corresponding relationship edges are integrated to form a semantic association graph of the complete image modality.

[0086] It should be noted that in this application, by extracting the overall semantic features of the image and the syntactic features of the visual object and combining them with the semantic structure relationship model, the semantic feature relationship in the image can be represented as a graphical structure. The semantic association graph of the image modality not only provides clear semantic structure support for image understanding, but also lays the foundation for subsequent multimodal fusion and risk control detection.

[0087] In step 105, the semantic association graph of the text modality and the semantic association graph of the image modality are subjected to inter-modal fine-grained structural fusion to obtain a multi-modal fusion graph.

[0088] In some embodiments, performing inter-modal fine-grained structural fusion on the semantic association graph of the text modality and the semantic association graph of the image modality to obtain a multi-modal fusion graph can be achieved by the following steps:

[0089] Extracting the structural granularity of the text in the semantic association graph of the text modality;

[0090] Extracting structural granularity of the image in a semantic association graph of the image modality;

[0091] Based on the structural fine-grainedness of the text and the structural fine-grainedness of the image, the semantic association graph of the text modality and the semantic association graph of the image modality are modally fused to obtain a multimodal fusion graph.

[0092] It should be noted that the structural granularity of the text in this application is a measure of the level of refinement of the analysis of the semantic structure information in the text, which is specifically manifested in parsing and organizing the language units such as words, phrases, sentences, etc. in the text according to their semantics, syntax, contextual dependencies and other relationships. It reveals the hierarchical structure and intrinsic semantic associations of the text by deeply analyzing the relationship between each semantic unit in the text; the structural granularity of the image in this application is a measure of the level of refinement of the analysis of the semantic structure information in the image.

[0093] In the specific implementation, first, the pre-trained convolutional neural network model can be used to perform convolution processing on the semantic association graph of the text modality, and then the fine-grained semantic structure in the semantic association graph can be identified through convolution, and then the fine-grained semantic structure can be used as the structural fine-grained text; then, the fine-grained semantic structure in the semantic association graph of the image modality can be extracted through the pre-trained Faster RCNN model, and the extracted result can be used as the structural fine-grained image; finally, the semantic association graph of the text and the semantic association graph of the image can be aligned through a cross-modal attention mechanism (such as multi-head self-attention), and then the semantic association graph of the text modality and the semantic association graph of the image modality can be mapped to the same feature space by two fully connected layers to bridge the difference in the representation features of different modalities, and then capture the correlation information between the two, and then connect each node in the semantic association graph of the text modality to each node on the semantic association graph of the image modality. Different from the connection method through addition or splicing, the method of establishing connecting edges between different graph nodes can help the model learn the correlation between objects contained in different modalities at a finer level, and then use the graph neural network (Graph The GNN (Graph Neural Network) fuses the fine-grained features of images and texts to reduce the differences in semantic features between different modalities, and then uses the output of the graph neural network as a multimodal fusion graph.

[0094] In step 106, multimodal features are extracted from the multimodal fusion graph, and abnormal classification is performed on the extracted multimodal features. If the classification result is abnormal, it is determined that the target financial document has risks.

[0095] In some embodiments, multimodal features are extracted from the multimodal fusion graph, and abnormal classification is performed on the extracted multimodal features. If the classification result is abnormal, determining that the target financial document has risks can be achieved by the following steps:

[0096] Extract node features and edge features from multimodal fusion graphs;

[0097] The node features and the edge features are classified as abnormal. If the classification result is abnormal, it is determined that the target financial document has risks.

[0098] In specific implementation, first, the feature encoder in the prior art can be used to read the node features and edge features within and between modalities from the multimodal fusion graph; then, the node features and edge features can be mapped to the correlation feature space, and then the correlation feature space can be converted into sentence-level correlation features and semantic-level correlation features between modalities through a convolutional layer, and then the sentence-level correlation features and semantic-level correlation features are used as inputs of a pre-trained classifier, the extracted multimodal features are classified by the classifier, and it is judged whether the target financial document is risky based on the classification results, that is, when the output result of the classifier is abnormal, the target financial document is judged to be risky.

[0099] It should be noted that the classifier in the present application may adopt a support vector machine, and other types of classifiers may also be adopted in other embodiments, which is not limited here.

[0100] On the other hand, in some embodiments, the present application provides a financial content risk control system based on a large model, referring to Figure 4 , which is a schematic diagram of exemplary software modules of a financial content risk control system based on a big model according to some embodiments of the present application. The financial content risk control system based on a big model 400 includes: a collection module 401, a processing module 402 and an execution module 403, which are described as follows:

[0101] Collection module 401, in this application, collection module 401 is mainly used to collect target financial documents to be tested for risk control from the financial content platform;

[0102] Processing module 402, in the present application, the processing module 402 is used to extract multimodal semantic features of the target financial document based on the pre-trained semantic large model, and obtain a text semantic space of the text modality and an image semantic space of the image modality;

[0103] The processing module 402 in the present application is also used to construct a semantic association tree of text elements according to the relationship semantics of each text semantic feature in the context in the text semantic space, and then determine the semantic association graph of the text modality from the semantic association tree and the syntactic structure relationship of the text semantic features in the text semantic space;

[0104] In the present application, the processing module 402 is also used to convert the semantic structural relationship between the image semantic features in the image semantic space into a semantic association graph of the image modality through the overall semantic features of the image in the target financial document and the syntactic features of the visual objects in the image;

[0105] In the present application, the processing module 402 is also used to perform inter-modal fine-grained structural fusion on the semantic association graph of the text modality and the semantic association graph of the image modality to obtain a multi-modal fusion graph;

[0106] Execution module 403, in the present application, the execution module 403 is mainly used to extract multimodal features from the multimodal fusion graph, and perform abnormal classification on the extracted multimodal features. If the classification result is abnormal, it is determined that the target financial document is at risk.

[0107] In addition, the present application also provides a computer device, which includes a memory and a processor, the memory stores code, and the processor is configured to obtain the code and execute the above-mentioned financial content risk control method based on a large model.

[0108] In some embodiments, reference Figure 5 , which is a schematic diagram of the structure of a computer device for implementing a financial content risk control method based on a large model according to some embodiments of the present application. The financial content risk control method based on a large model in the above embodiment can be Figure 5 The computer device 500 shown in the figure is implemented, and the computer device 500 includes at least one processor 501, a communication bus 502, a memory 503 and at least one communication interface 504.

[0109] The processor 501 may be a general-purpose central processing unit (CPU) or an application-specific integrated circuit (ASIC).

[0110] The communication bus 502 may be used to transmit information between the above-mentioned components.

[0111] The memory 503 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compressed optical disc, a laser disc, an optical disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 503 may exist independently and be connected to the processor 501 via the communication bus 502. The memory 503 may also be integrated with the processor 501.

[0112] The memory 503 is used to store the program code for executing the solution of the present application, and the execution is controlled by the processor 501. The processor 501 is used to execute the program code stored in the memory 503. The program code may include one or more software modules. The financial content risk control method based on the large model in the above embodiment can be implemented by the processor 501 and one or more software modules in the program code in the memory 503.

[0113] The communication interface 504 uses any transceiver or other device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0114] In a specific implementation, as an embodiment, a computer device may include multiple processors, each of which may be a single-CPU processor or a multi-CPU processor. The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0115] The above-mentioned computer device may be a general-purpose computer device or a special-purpose computer device. In a specific implementation, the computer device may be a desktop computer, a portable computer, a network server, a personal digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device or an embedded device. The embodiment of the present application does not limit the type of computer device.

[0116] In addition, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned financial content risk control method based on a large model.

[0117] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0118] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A financial content risk control method based on a large model, characterized in that: The steps include: Collect target financial documents to be tested for risk control from financial content platforms; Based on the pre-trained semantic big model, multimodal semantic features are extracted from the target financial document to obtain the text semantic space of the text modality and the image semantic space of the image modality. Constructing a semantic association tree of text elements according to the relational semantics of each text semantic feature in the context in the text semantic space, and then determining a semantic association graph of text modality according to the semantic association tree and the syntactic structure relationship of the text semantic features in the text semantic space; The semantic structural relationship between the image semantic features in the image semantic space is converted into a semantic association graph of the image modality through the overall semantic features of the image in the target financial document and the syntactic features of the visual objects in the image; Performing inter-modal fine-grained structural fusion on the semantic association graph of the text modality and the semantic association graph of the image modality to obtain a multi-modal fusion graph; Multimodal features are extracted from the multimodal fusion graph, and abnormal classification is performed on the extracted multimodal features. If the classification result is abnormal, it is determined that the target financial document has risks.

2. The method according to claim 1, characterized in that Based on the pre-trained semantic big model, multimodal semantic features are extracted from the target financial document to obtain the text semantic space of the text modality and the image semantic space of the image modality. Specifically, the following are included: Extract all text semantic features and image semantic features in the target financial document through the pre-trained semantic big model; The set of all text semantic features is taken as the text semantic space of the text modality; The set of all image semantic features is taken as the image semantic space of the image modality.

3. The method according to claim 1, characterized in that Constructing a semantic association tree of text elements according to the relationship semantics of each text semantic feature in the context in the text semantic space specifically includes: Determine the relational semantics of each text semantic feature in the text semantic space in the context by using the context window sliding analysis technology; Based on all the relational semantics, the text semantic features are hierarchically clustered to obtain multiple semantic association clusters; According to all the semantic association clusters, each text semantic feature in the text semantic space is organized into a tree structure according to the semantic relationship, so as to obtain a semantic association tree of the text elements.

4. The method according to claim 1, characterized in that Determining the semantic association graph of the text modality by the semantic association tree and the syntactic structure relationship of the text semantic features in the text semantic space specifically includes: Extract the syntactic structure of different segments in the text of the target financial document; Determining the syntactic structure relationship of the text semantic features in the text semantic space through all syntactic structures, and generating a syntactic tree based on the syntactic structure relationship; Structural semantic association is performed on the semantic association tree and the syntactic tree to generate a semantic association graph of the text modality.

5. The method according to claim 1, characterized in that The semantic structural relationship between the image semantic features in the image semantic space is converted into a semantic association graph of the image modality by using the overall semantic features of the image in the target financial document and the syntactic features of the visual objects in the image. Specifically, the semantic structural relationship between the image semantic features in the image semantic space is converted into a semantic association graph of the image modality. Specifically, the semantic structural relationship between the image semantic features in the image semantic space and the syntactic features of the visual objects in the image include: Extract the overall semantic features of the images in the target financial document; Extracting syntactic features of visual objects within images of target financial documents; Based on the overall semantic features and the syntactic features, construct a semantic structure relationship model; The semantic structure relationship model is used to convert the semantic structure relationship between each image semantic feature in the image semantic space into a semantic association graph of the image modality.

6. The method according to claim 1, characterized in that Extracting multimodal features from the multimodal fusion graph, and performing abnormal classification on the extracted multimodal features. If the classification result is abnormal, determining that the target financial document has risks specifically includes: Extract node features and edge features from multimodal fusion graphs; The node features and the edge features are classified as abnormal. If the classification result is abnormal, it is determined that the target financial document has risks.

7. The method according to claim 1, characterized in that The target financial document includes text data and image data.

8. A financial content risk control system based on a large model, characterized in that: include: The collection module is used to collect target financial documents to be tested for risk control from the financial content platform; A processing module, used for extracting multimodal semantic features of target financial documents based on a pre-trained semantic large model, and obtaining a text semantic space of a text modality and an image semantic space of an image modality; The processing module is further used to construct a semantic association tree of text elements according to the relationship semantics of each text semantic feature in the context in the text semantic space, and then determine a semantic association graph of the text modality from the semantic association tree and the syntactic structure relationship of the text semantic features in the text semantic space; The processing module is further used to convert the semantic structural relationship between the image semantic features in the image semantic space into a semantic association graph of the image modality through the overall semantic features of the image in the target financial document and the syntactic features of the visual objects in the image; The processing module is further used to perform inter-modal fine-grained structural fusion on the semantic association graph of the text modality and the semantic association graph of the image modality to obtain a multi-modal fusion graph; The execution module is used to extract multimodal features from the multimodal fusion graph, and perform abnormal classification on the extracted multimodal features. If the classification result is abnormal, it is determined that the target financial document has risks.

9. A computer device, comprising a memory and a processor, wherein the memory stores a code, characterized in that: The processor is configured to obtain the code and execute the financial content risk control method based on a large model as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the large-model-based financial content risk control method as described in any one of claims 1 to 7 is implemented.

Images