Text processing method and electronic device

Abstract

This application discloses a text processing method and electronic device, relating to the field of data processing technology. It includes defining a meta-path set for a target domain, where meta-paths describe compliance dependencies between domain entities and rule entities within the target domain. A heterogeneous graph attention network extracts features from the heterogeneous graph based on the meta-path set to obtain corresponding compliance knowledge vector representations. These compliance knowledge vector representations are then input into a pre-trained compliance text generation model to obtain target text that meets the rule requirements of the target domain. This method achieves the semantic range of entity feature aggregation constrained by meta-paths and the construction of semantically specific computational contexts, expanding the semantic receptive domain of entities and enhancing the expressive power of local structures, thereby improving the accuracy of model output. Therefore, it can solve the technical problem of inaccurate model output results in complex scenarios, achieving the technical effect of improving model output accuracy.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a text processing method and an electronic device. Background Technology

[0002] With the accelerated digital transformation of the financial industry, large language models are increasingly being used in financial scenarios such as intelligent customer service, risk assessment, and compliance review. However, the financial sector has strict compliance requirements and a complex regulatory system. Ordinary large language models are prone to generating outputs that violate compliance rules when generating financial-related content, which severely limits their practical application value in financial business.

[0003] Currently, two main technical approaches are used to address compliance issues in large models: one is to enhance the model's domain knowledge by incorporating financial corpora during pre-training; the other is to align the model's output behavior through instruction fine-tuning or reinforcement learning based on human feedback. However, while these methods improve model performance in financial scenarios to some extent, they still have significant limitations, especially when dealing with complex financial compliance knowledge. These methods struggle to accurately understand and apply fine-grained compliance rules, leading to inaccurate semantic understanding and consequently, inaccurate model output. Summary of the Invention

[0004] This application provides a text processing method and electronic device to at least solve the problem in related technologies where, when processing complex financial compliance knowledge, the model output results are inaccurate due to the difficulty in accurately understanding and applying fine-grained compliance rules.

[0005] This application provides a text processing method, including: Obtain the text to be processed in the target domain; Entity recognition and entity relation extraction are performed on the text to be processed to generate a heterogeneous graph corresponding to the text to be processed. The entity recognition includes domain entity recognition and rule entity recognition. The heterogeneous graph is used to extract features from the heterogeneous graph through the heterogeneous graph attention network of the target domain to obtain the compliance knowledge vector representation of the text to be processed. The heterogeneous graph attention network extracts features from the heterogeneous graph based on the meta-path set of the target domain. The meta-paths in the meta-path set are used to describe the compliance dependency relationship between domain entities and rule entities in the target domain. The compliance knowledge vector representation is input into a pre-trained compliance text generation model to obtain target text that matches the text to be processed, and the target text conforms to the rule requirements of the target domain.

[0006] This application also provides a text processing apparatus, including: The acquisition module is used to acquire the text to be processed in the target domain; The graph generation module is used to perform entity recognition and entity relation extraction on the text to be processed, and generate a heterogeneous graph corresponding to the text to be processed. The entity recognition includes domain entity recognition and rule entity recognition. The feature extraction module is used to extract features from the heterogeneous graph through the heterogeneous graph attention network of the target domain to obtain the compliance knowledge vector representation corresponding to the text to be processed. The heterogeneous graph attention network extracts features from the heterogeneous graph based on the meta-path set of the target domain. The meta-paths in the meta-path set are used to describe the compliance dependency relationship between domain entities and rule entities in the target domain. The text generation module is used to input the compliance knowledge vector representation into a pre-trained compliance text generation model to obtain target text that matches the text to be processed, and the target text conforms to the rule requirements of the target domain.

[0007] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described text processing methods.

[0008] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described text processing methods.

[0009] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described text processing methods.

[0010] By defining a meta-path set for the target domain, where meta-paths describe compliance dependencies between domain entities and rule entities, this application allows the heterogeneous graph attention network to extract features from the heterogeneous graph based on the meta-path set to obtain a corresponding compliance knowledge vector representation after generating the heterogeneous graph corresponding to the text to be processed. Inputting this compliance knowledge vector representation into a pre-trained compliance text generation model yields target text that meets the rule requirements of the target domain. By introducing the meta-path set to guide the heterogeneous graph attention network in feature extraction and encoding of the heterogeneous graph, and generating compliance knowledge vector representations, this application achieves the semantic scope of entity feature aggregation constrained by meta-paths and the construction of a semantically specific computational context. This expands the semantic receptive domain of entities and enhances the expressive power of local structures, thereby strengthening subtle differences between similar terms and improving semantic discriminability. Therefore, it can significantly improve the discriminability and accuracy of features of different entities, enhance the quality of the model's input features, and thus improve the accuracy of the model's output, resulting in compliant target text. Thus, it can solve the technical problem of inaccurate model output results in complex scenarios, achieving the technical effect of improving model output accuracy. Attached Figure Description

[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart illustrating a text processing method provided in an exemplary embodiment of this application; Figure 2 This illustration shows a schematic diagram of the nodes and relationships of a heterogeneous knowledge graph for financial compliance, as provided in an exemplary embodiment of this application. Figure 3 A flowchart illustrating a text processing method provided as another exemplary embodiment of this application; Figure 4 A flowchart illustrating the training process of a compliant text generation model provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a text processing device provided in an embodiment of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0014] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0015] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0016] This application provides a text processing method, which can be executed by a text processing device provided in this application. This device can be implemented in software and / or hardware and can be integrated into an electronic device. The method is described in detail below with reference to its execution flow.

[0017] Figure 1 A flowchart illustrating a text processing method provided in an exemplary embodiment of this application is shown below. Figure 1 As shown, the text processing method may include the following steps: Step 101: Obtain the text to be processed in the target domain.

[0018] The target areas can be sectors with compliance requirements, such as finance and healthcare.

[0019] In this embodiment, the text to be processed in the target domain can be text content input or uploaded by the user, such as financial consultation questions, text to be reviewed, etc.

[0020] Step 102: Perform entity recognition and entity relation extraction on the text to be processed to generate a heterogeneous graph corresponding to the text to be processed. Entity recognition includes domain entity recognition and rule entity recognition.

[0021] In this embodiment, for the acquired text to be processed, entity recognition and entity relationship extraction can be performed first, and a corresponding heterogeneous graph can be constructed. A node on the heterogeneous graph corresponds to an entity, which can be a domain entity or a rule entity.

[0022] Entity identification includes domain entity identification and rule entity identification. Domain entity identification identifies entities within a target domain, such as financial institutions and financial products in the financial field, and medical institutions, drug names, and disease names in the healthcare field. Rule entity identification identifies rule types and compliance risk labels within a target domain, such as financial regulations in the financial field, and medical clinical guidelines, medical insurance policies, and drug management regulations in the healthcare field. Entity relationships can include relationships between domain entities and relationships between domain entities and rule entities. For example, in the financial field, a financial institution launching a financial product constitutes an entity relationship between the financial institution and the financial product, and the requirement for financial institutions to comply with financial regulations constitutes an entity relationship between the financial institution and the regulations. In the healthcare field, entity relationships may include, for example, clinical contraindications for drug use arising from a disease.

[0023] In one optional embodiment of this application, a named entity recognition model can be used to perform target domain entity recognition and rule entity recognition on the text to be processed to obtain multiple entities, including domain entities and rule entities. Then, an entity relationship extraction model based on a dual-channel encoder is used to extract the entity relationship from the multiple entities to obtain a set of triples corresponding to the multiple entities. The triples in the triple set include two entities and the relationship type between the two entities. Then, a heterogeneous graph is generated based on the triple set.

[0024] Among them, the named entity recognition model is a pre-trained model in natural language processing technology. This model is used to identify domain entities and rule-based entities within the target domain. The model operates on input sequences... The above calculation of the conditional probability distribution outputs a set of entities including the organization name, product type (domain entity), and compliance risk label (rule entity) of the target domain. The named entity recognition model is based on a bidirectional Transformer encoder and obtains context-sensitive semantic vectors through a self-attention mechanism. Its optimization objective still follows the language modeling loss function as follows: .

[0025] in, θ The parameter space is inherited from the pre-trained model, ensuring consistency between domain semantic mapping and general language features; T The length of the input sequence; This indicates that the model predicts based on previous inputs. x t The probability of.

[0026] After entity recognition is completed, the entity relation extraction process continues. A relation discrimination model based on a dual-channel encoder is used to extract entity relations from the identified entities, identifying semantic relationships between entities and generating a set of triples. ,in For entities and The types of compliance relationships between them include regulatory constraints, risk associations, and compliance dependencies. The extracted triples are organized in the form of a knowledge graph, forming a heterogeneous graph structure. , where the node set ν An edge set consists of entities of different categories in the target domain. ɛ It reflects multiple types of semantic dependencies between entities. This heterogeneous graph has highly complex connection features, and the semantic roles of different node types in the graph differ significantly.

[0027] For example, taking the financial sector as the target field, Figure 2 This illustration shows a schematic diagram of nodes and relationships in a heterogeneous knowledge graph for financial compliance, as described in an exemplary embodiment of this application. Figure 2 As shown, this heterogeneous diagram illustrates the relationship between financial institutions and financial products, the relationship between financial institutions and regulatory provisions, and the relationship between financial products and regulatory provisions.

[0028] Step 103: The heterogeneous graph is used to extract features from the heterogeneous graph through the heterogeneous graph attention network of the target domain to obtain the compliance knowledge vector representation of the text to be processed. The heterogeneous graph attention network extracts features from the heterogeneous graph based on the meta-path set of the target domain. The meta-paths in the meta-path set are used to describe the compliance dependency relationship between domain entities and rule entities in the target domain.

[0029] The heterogeneous graph attention network for the target domain is trained using a heterogeneous graph constructed from a dataset in the target domain. Guided by predefined meta-paths in the target domain, this network utilizes meta-path attention mechanisms and residual connections to achieve information fusion and structured encoding on the input heterogeneous graph, outputting an encoded compliance knowledge vector representation. This addresses the problem of dilution of subtle semantic differences in node representations caused by graph neural networks aggregating highly heterogeneous compliance knowledge graphs, thereby improving semantic discriminability.

[0030] In the specific implementation of the meta-path-guided heterogeneous graph attention network, the first step is to define a set of meta-paths covering the semantic dependencies between multiple types of entities in the target domain (e.g., financial institutions, financial products, regulations, and risk events in the financial field). Each meta-path type characterizes a specific compliant semantic pattern, used to constrain different semantic connection patterns. Each meta-path... This describes the sequence of compliance dependencies between entities (e.g., financial institutions and products, regulations) to guide the aggregation of multi-hop neighborhoods in heterogeneous graphs. For example, a financial institution's involvement with financial products is subject to regulatory provisions or risk events triggering regulatory clauses. For instance, a meta-path in the financial field might be: Financial Institution → Financial Product → Regulatory Provisions; a meta-path in the healthcare field might be: Medical Institution → Drug → Disease → Clinical Guidelines. Heterogeneous graph attention networks utilize path traversal algorithms within heterogeneous knowledge graphs. The process involves searching for node sequences that conform to the meta-path, instantiating the meta-path, and thus constructing a multi-hop, domain-compliant context within the graph structure. This expands the semantic receptive domain of nodes and enhances their local structural expressiveness. The entire mechanism minimizes the comprehensive loss function during training via gradient descent. L = The value of λ is used to achieve the fusion learning of language knowledge and compliance graph knowledge, thereby ensuring that the output compliance knowledge vector maintains stability in semantic consistency and structural relevance, providing a solid domain feature foundation for subsequent steps. Here, λ is the compliance constraint coefficient. Modeling a loss function for language, To ensure compliance with loss alignment requirements, the specific formula will be described in subsequent embodiments.

[0031] In this embodiment, the obtained heterogeneous graph is input into the heterogeneous graph attention network of the target domain for feature extraction. The heterogeneous graph attention network is processed under the guidance of the meta-path set and outputs the obtained compliance knowledge vector representation.

[0032] Step 104: Input the compliance knowledge vector representation into the pre-trained compliance text generation model to obtain the target text that matches the text to be processed. The target text conforms to the rule requirements of the target domain.

[0033] The compliant text generation model is obtained by fine-tuning a pre-trained language model using training samples from the target domain. The specific training process will be explained in detail in subsequent embodiments and will not be elaborated here. The pre-trained language model is a neural network model trained on a general corpus. For example, the pre-trained language model can be a deep learning model based on the Transformer architecture, obtained through training on massive amounts of internet text.

[0034] The idea behind pre-training is to first train a set of model parameters on a single task, then use these parameters to initialize the network model parameters, and finally use the initialized network model to train on other tasks, resulting in models adapted for those tasks. By pre-training on large-scale corpora, neural language representation models can learn powerful language representation capabilities, extracting rich syntactic and semantic information from text. Pre-trained language models can provide tokens containing rich semantic information and sentence-level features for downstream tasks. Fine-tuning can also be performed directly on the pre-trained model for downstream tasks, quickly and easily obtaining downstream-specific models. The neural network algorithm structure used to train the pre-trained language model can be a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), a Long Short-Term Memory (LSTM), or a model built with attention networks, such as Transformer or BERT; this application does not impose any limitations on this approach. Attention networks are network models trained using attention mechanisms. These models assign different weights to each part of the input sequence, thereby extracting more important feature information and resulting in more accurate output. Fine-tuning refers to performing small-scale training on a pre-trained language model for a specific task objective (downstream task) and task data (downstream data). This allows for minor adjustments to the parameters of the pre-trained language model, ultimately resulting in a model adapted to the specific task and data, such as the compliant text generation model in this application.

[0035] In this embodiment, the obtained compliance knowledge vector representation is input into a pre-trained compliance text generation model for processing. The compliance text generation model generates text based on the input compliance knowledge vector representation and outputs target text that meets the rule requirements of the target domain.

[0036] The text processing method of this application defines a meta-path set for the target domain. The meta-paths in the meta-path set are used to describe the compliance dependencies between domain entities and rule entities in the target domain. Therefore, after generating a heterogeneous graph corresponding to the text to be processed, the heterogeneous graph attention network extracts features from the heterogeneous graph based on the meta-path set of the target domain to obtain the corresponding compliance knowledge vector representation. Inputting the compliance knowledge vector representation into a pre-trained compliance text generation model yields target text that meets the rule requirements of the target domain. By introducing the meta-path set of the target domain to guide the heterogeneous graph attention network in extracting and encoding features from the heterogeneous graph and generating compliance knowledge vector representation, the method achieves the semantic range of entity feature aggregation constrained by meta-paths and the construction of semantically specific computational contexts. This expands the semantic receptive domain of entities and enhances the expressive power of local structures, thereby strengthening the subtle differences between similar terms and improving semantic discriminability. Therefore, it can significantly improve the discriminability and accuracy of features of different entities, enhance the quality of input features of the model, and thus improve the accuracy of model output, resulting in compliant target text. Therefore, it can solve the technical problem of inaccurate model output results in complex scenarios and achieve the technical effect of improving the accuracy of model output.

[0037] In one alternative embodiment of this application, such as Figure 3 As shown, based on the foregoing embodiments, step 103 may include the following sub-steps: Step 201: Through the attention layer of the heterogeneous graph attention network, determine the attention weight of the neighbor nodes associated with the entity in the heterogeneous graph. The attention layer determines the neighbor nodes associated with the entity based on the meta-path set of the target domain.

[0038] In this embodiment of the application, the heterogeneous graph attention network includes an attention layer, a residual connection layer and a graph convolutional layer. First, the attention layer determines the neighboring nodes associated with entities in the input heterogeneous graph based on the metapath set of the target domain, and then determines the attention weight of the neighboring nodes of the entity for that entity.

[0039] In one optional embodiment of this application, when the attention layer determines the attention weights of neighboring nodes associated with an entity in a heterogeneous graph, for each node in the heterogeneous graph, the attention layer can take that node as the starting node, determine the multi-hop node sequence to which the node belongs according to the meta-path set of the target domain, and take all nodes in the multi-hop node sequence (excluding the starting node) as the neighboring nodes of the starting node, and then determine the attention weights of each neighboring node to the starting node based on the attention mechanism. Here, the multi-hop node sequence matches a meta-path in the meta-path set, that is, the multi-hop node sequence is consistent with or partially consistent with that meta-path.

[0040] In one optional embodiment of this application, when the attention layer determines the attention weights of neighboring nodes associated with an entity in a heterogeneous graph, for each node (entity) in the heterogeneous graph, the attention layer can traverse the metapaths in the metapath set to find a multi-hop node sequence that matches the metapath in the heterogeneous graph, with the starting node of the multi-hop node sequence being the entity; then, the ending node of the multi-hop node sequence is determined as the neighboring node associated with the entity; next, based on the entity and the aforementioned multi-hop node sequence corresponding to the entity, the heterogeneous subgraph corresponding to the entity is determined from the heterogeneous graph; finally, based on the heterogeneous subgraph, the attention weights of the neighboring nodes associated with the entity are determined. When determining the attention weights, the attention weights of each neighboring node for the entity are determined on the heterogeneous subgraph based on the attention mechanism.

[0041] In this embodiment, metapaths guide the definition and filtering of neighbors. At the compute node... i When representing the graph, first, based on each predefined meta-path (e.g., financial institution → financial product → regulatory terms), a path traversal algorithm is used to find the nodes in the graph. i The starting point is any sequence of multi-hop nodes that conforms to the metapath pattern; the endpoint nodes of these sequences constitute the nodes. i Meta-path neighbors under a specific meta-path semantics. Different meta-paths will produce different subsets of semantic neighbors. It is understandable that nodes... i The node in the matched meta-path could be the first node or a node in the middle of the meta-path. Next, semantically specific attention weights are calculated based on the meta-path neighbors. For each meta-path, the attention weight formula is used to calculate the attention weights at each node. i Weights are calculated on the subgraph formed by all the corresponding multi-hop node sequences. This ensures that the determined neighbor nodes and the calculated attention weights are specific to the semantic context of the metapath. The metapath deeply participates in the generation of attention weights by constraining the semantic scope of aggregation and constructing a semantically specific computational context.

[0042] In the attention mechanism, to achieve information aggregation among heterogeneous nodes under the meta-path, the feature vector of each node... Query vectors are generated using linear projection. Key vector AND value vector ,in For any node, the projection matrix is ​​independent. i Its set of neighboring nodes is denoted as Attention mechanism for each neighbor Assign weights ,node i with neighboring nodes j The attention weights between individuals are determined by a similarity function, and the formula for calculating the attention weights is as follows: .

[0043] In the above formula, and They are nodes i with neighboring nodes j The input feature vector is W, which is a trainable linear transformation matrix; a is a learnable attention weight vector used to measure the structural importance between nodes. It is a non-linear activation function, with the symbol... This represents a vector concatenation operation. The formula calculates the importance of each neighbor under meta-path instantiation, and uses exponential normalization to ensure that the sum of the weights of all neighbors is 1, thus achieving semantically weighted aggregation.

[0044] In this embodiment, for each node on the heterogeneous graph, the metapaths in the metapath set are traversed to find the multi-hop node sequence that conforms to the metapath in the heterogeneous graph, thereby determining the neighbor nodes of each node. Then, the attention weights of the neighbor nodes associated with the entity are determined on the heterogeneous subgraph determined from the heterogeneous graph based on the entity and the multi-hop node sequence. The weights are calculated in parallel in different projection subspaces and averaged through a multi-head attention mechanism. The attention layer captures the node relevance in multiple semantic dimensions, which helps to improve the expressive power and stability of the nodes.

[0045] Step 202: Through the residual connection layer of the heterogeneous graph attention network, feature aggregation is performed on the entity and neighbor nodes based on the attention weights of the neighbor nodes to obtain the aggregated features corresponding to the entity.

[0046] In this embodiment, after the attention layer obtains the attention weights of each entity's neighboring nodes for that entity, the residual connection layer of the heterogeneous graph attention network performs a residual connection operation to prevent the gradient vanishing problem during deep propagation. The residual connection layer aggregates features of the entity and its neighboring nodes based on their attention weights, obtaining the aggregated features corresponding to the entity. The aggregated features of the entity output by the residual connection layer can be expressed by the following formula: .

[0047] in, For nodes i The original feature vector, For nodes after heterogeneous aggregation i The output feature representation, i.e., node i The aggregation characteristics. In the summation term... Ensure communication with neighboring nodes j The semantic interactions are weighted according to importance, the residual structure preserves the node’s own characteristics to prevent it from being overwhelmed by neighbor information, and improves training stability during the training phase.

[0048] Step 203: The aggregated features of entities are nonlinearly mapped through the graph convolutional layer of the heterogeneous graph attention network to obtain the compliance knowledge vector representation corresponding to the text to be processed.

[0049] In this embodiment, after the residual connection layer outputs the aggregated features of each entity and its associated neighbor nodes, the graph convolutional layer performs nonlinear mapping on the aggregated features of each entity. The features of all entities are mapped and nonlinearly combined by a multilayer perceptron in the graph convolutional layer to form a globally consistent compliance knowledge vector representation. It can be understood that the compliance knowledge vector representation can be represented in the form of a matrix.

[0050] In one optional embodiment of this application, the process of the graph convolutional layer performing nonlinear mapping on the aggregated features of entities and outputting a compliance knowledge vector representation includes: using the aggregated features of entities as input data for the first layer, determining the output features of entities at the m-th layer layer by layer, where 1≤m≤M, M is the total number of layers in the graph convolutional layer, and using the output features of the previous layer as input for the current layer; determining the compliance feature representation of entities based on the aggregated features of entities and the output features of entities at the M-th layer; and generating and outputting the compliance knowledge vector representation corresponding to the text to be processed based on the compliance feature representation of entities.

[0051] In other words, in this embodiment, the graph convolutional layer has a total of M layers. The aggregated features of entities output from the residual connection layer serve as the input data for the first layer of the graph convolutional layer. The first layer processes the input aggregated features and outputs the entity's output features at that layer. The output features of the first layer serve as the input data for the second layer, and so on. The entity's output features at the second layer serve as the input data for the third layer, and so on. The entity's output features at the Mth layer are then concatenated with the entity's aggregated features to obtain the corresponding compliance feature representation of the entity. It is understood that the aggregated features of each entity undergo the above processing to obtain the corresponding compliance feature representation.

[0052] In one optional embodiment of this application, when determining the compliance feature representation of an entity based on its aggregated features and its output features at layer M, the aggregated features of the entity can be added to its output features at layer M to obtain the compliance feature representation of the entity. That is, in the forward propagation of the deep graph convolutional network (M layers in total), the initial input features of the network (i.e., the aggregated features of the entity) are added to the output features at layer M to obtain the compliance feature representation of the entity. By bypassing intermediate layers and directly connecting to the output features of the Mth layer, the problem of gradient vanishing / exploding in deep networks can be solved, ensuring the stability of the optimization process.

[0053] In summary, during the processing stage of the graph convolutional layer, each node... i The output feature of the m-th layer is represented as ,nodei The input features of the first layer are the aforementioned aggregated features. After nonlinear mapping by the multilayer perceptron, the calculation formula for the updated node features of the next layer is as follows: .

[0054] in, Let m be the weight matrix of the m-th layer. For bias vectors, ReLU is a non-linear activation function. A multilayer perceptron consists of two fully connected layers and one ReLU activation layer. The first layer performs linear projection to compress the feature dimension, and the second layer performs non-linear expansion to capture higher-order semantic features.

[0055] Finally, through the interlayer residual mechanism To achieve stable propagation of gradient flow, the nodes are obtained. i compliance characteristics ,in For nodes i The output feature vector of the Mth layer, For nodes i The aggregated features retain the original semantics of the nodes during forward propagation and avoid gradient vanishing during backpropagation, ensuring the continuity of the feature optimization process. The compliance feature representations of all nodes constitute a global compliance knowledge embedding matrix (i.e., compliance knowledge vector representation), where each dimension contains a structured representation jointly defined by domain semantic constraints and risk logic.

[0056] The text processing method of this application embodiment determines the attention weights of neighboring nodes associated with an entity in a heterogeneous graph through the attention layer of a heterogeneous graph attention network. The attention layer determines the neighboring nodes associated with the entity based on the meta-path set of the target domain. Then, through the residual connection layer of the heterogeneous graph attention network, feature aggregation is performed on the entity and neighboring nodes based on the attention weights of the neighboring nodes to obtain the aggregated features corresponding to the entity. Finally, the graph convolutional layer of the heterogeneous graph attention network performs nonlinear mapping on the aggregated features of the entity to obtain the compliance knowledge vector representation corresponding to the text to be processed. Thus, the meta-path structure is used to express multiple categories of... The entity relationship model, the multi-hop neighborhood structure generated by the meta-path instantiation, is attention-weighted and summed in the aggregation stage to form aggregated features for feature fusion. This ensures that semantic interactions with neighboring nodes are weighted according to importance, achieving weighted fusion of local information in the high-dimensional feature space. It accurately encodes regulatory constraints and risk dependencies at the semantic level, improving the accuracy of feature extraction. Furthermore, by performing nonlinear mapping on features through graph convolutional layers, it can form a globally consistent compliance representation, ensuring that subsequent models can improve their understanding of domain compliance rules and generalization ability based on rich graph structure knowledge, providing domain feature support for accurate compliance text generation.

[0057] In one alternative embodiment of this application, such as Figure 4 As shown, the compliance text generation model in the aforementioned embodiment is trained through the following steps: Step 301: Obtain training samples in the target domain. The training samples include sample text and the corresponding sample compliance knowledge vector representation.

[0058] The target areas can be the financial sector, the healthcare sector, etc.

[0059] In the financial sector, when acquiring sample text, one can first obtain data such as financial regulations, compliance cases, and risk warnings. Data preprocessing is then used to remove noise and redundant information, outputting standardized financial compliance knowledge data. This financial compliance knowledge data is then used as sample text. Entity recognition and entity relationship extraction are performed on the financial compliance knowledge data to construct a sample heterogeneous graph. This heterogeneous graph is then input into the aforementioned heterogeneous graph attention network for feature extraction, yielding a sample compliance knowledge vector representation. The sample text and its corresponding sample compliance knowledge vector representation constitute a training sample.

[0060] In the healthcare field, when acquiring sample text, one can first obtain data such as clinical guidelines, medical insurance policies, drug management regulations, and medical error cases. Then, through preprocessing operations such as standardizing medical terminology (e.g., International Classification of Diseases (ICD) codes and generic drug names), and removing noise and redundant information, standardized medical compliance knowledge data is output. This obtained medical compliance knowledge data is then used as sample text, and the corresponding sample compliance knowledge vector representation is determined. The sample text and its corresponding sample compliance knowledge vector representation constitute a training sample.

[0061] In one optional embodiment of this application, when acquiring training samples in the target domain, compliance knowledge text in the target domain can be acquired first. This acquired compliance knowledge text is standardized domain compliance knowledge data that has undergone data cleaning, removal of sensitive information, and quality screening. Next, data augmentation is performed on the compliance knowledge text to obtain augmented text. For example, common data augmentation techniques such as back-translation and synonym replacement can be used. Based on the compliance knowledge text and the augmented text, sample text is constructed. Then, entity recognition and entity relation extraction are performed on the sample text to generate a sample heterogeneous graph corresponding to the sample text. Features are extracted from the sample heterogeneous graph using a heterogeneous graph attention network in the target domain to obtain the sample compliance knowledge vector representation corresponding to the sample text. Finally, training samples are constructed based on the sample text and the sample compliance knowledge vector representation.

[0062] It should be noted that, in this embodiment, the specific process of obtaining the sample heterogeneous graph and the sample compliance knowledge vector representation can be referred to the specific implementation method of obtaining the heterogeneous graph and compliance knowledge vector representation in the aforementioned embodiments. The implementation principle is similar and will not be repeated here.

[0063] Taking the financial sector as an example, raw financial compliance knowledge data can be obtained from public databases of financial regulatory agencies and industry compliance knowledge bases. This data includes financial regulations, compliance case documents, and risk warning records. Data preprocessing is performed on the raw data, including text cleaning using natural language processing tools to remove Hypertext Markup Language (HTML) tags, special symbols, and irrelevant numbers; applying text standardization techniques to unify date formats and institution names; and detecting and deleting redundant and duplicate content using rule engines and statistical methods. Data preprocessing also includes data augmentation steps, generating synthetic samples through back-translation, synonym replacement, and entity replacement to increase data diversity. Back-translation involves translating text into other languages ​​and then back into the original language to generate new samples. Synonym replacement and entity replacement can create rare compliance scenario data, ensuring that the generated dataset covers various types of financial risk such as credit risk, market risk, and operational risk. After quality checks, sensitive information is removed and the data is encrypted and stored, outputting standardized financial compliance knowledge data. This data is organized in plain text format and annotated with data source and version information for subsequent knowledge extraction and structured coding. The financial compliance knowledge dataset covers multiple financial sub-sectors such as banking, securities, and insurance, ensuring the comprehensiveness of compliance knowledge.

[0064] In this embodiment of the application, compliance knowledge text is augmented using data augmentation technology to obtain augmented text. The augmented text and the obtained compliance knowledge text together constitute sample text, and the corresponding sample compliance knowledge vector representation is determined for constructing training samples. This improves the diversity and coverage of training samples and supports compliance learning of the model in rare scenarios.

[0065] Step 302: Input the sample text and sample compliance knowledge vector representation into the pre-trained language model, and obtain the prediction results and hidden state vectors output by the pre-trained language model.

[0066] In this embodiment, the sample text and sample compliance knowledge vector representation are input into the pre-trained language model. The multi-head attention layer of the pre-trained language model integrates compliance information in the query key-value calculation, and outputs the prediction result and hidden state vector after processing. The hidden state vector refers to the hidden state output by the last Transformer layer before the output layer of the pre-trained language model, and the prediction result is the output of the output layer of the pre-trained language model, including the predicted values ​​from the input sample text. xt The probability of.

[0067] Step 303: Determine the loss value of the pre-trained language model based on the sample compliance knowledge vector representation, hidden state vector, and prediction results.

[0068] In this embodiment, after obtaining the prediction result and the hidden state vector, the current loss value of the pre-trained language model can be calculated by combining the sample compliance knowledge vector representation in the training samples.

[0069] In one optional embodiment of this application, the language modeling loss of the pre-trained language model can be determined based on the prediction results. The language modeling loss can be calculated based on the cross-entropy loss function and is used to characterize the difference between the model's predictions of the pre-real sample text. The formula for calculating the language modeling loss is described in the aforementioned language modeling loss function. L lm , which will not be elaborated here; and, based on the sample compliance knowledge vector representation and the hidden state vector, the compliance alignment loss of the pre-trained language model is determined. The compliance alignment loss can be calculated based on the cosine similarity function and is used to characterize the semantic distance between the model output vector and the sample compliance knowledge vector representation. The calculation formula of the compliance alignment loss is shown in the following formula (1); then, based on the language modeling loss and the compliance alignment loss, the loss value of the pre-trained language model is determined.

[0070] (1) In the above formula (1), This represents the hidden state vector output by the model. This represents the sample compliance knowledge vector.

[0071] As an example, when determining the loss value of a pre-trained language model, the average of the language modeling loss and the compliance alignment loss can be used as the final loss value.

[0072] As an example, the loss value of a pre-trained language model can be determined by weighted summation of the language modeling loss and the compliance alignment loss. , This is the final determined loss value. Therefore, in this embodiment, the first weight corresponding to the language modeling loss (i.e., ...) can be obtained first. The second weight corresponding to the compliance alignment loss (i.e.) The sum of the first and second weights is 1. Then, the language modeling loss and compliance alignment loss are weighted and summed based on the first and second weights to obtain the loss value of the pre-trained language model.

[0073] The first and second weights can be preset or dynamically adjusted during training based on a dynamic weight adjustment mechanism.

[0074] As an example, the first and second weights can both be set to 0.5.

[0075] As an example, the first and second weights are adjusted based on a dynamic weight adjustment mechanism. This mechanism calculates the second weight based on the number of violations during the training process. The calculation formula for the second weight is shown in formula (2) below. Specifically, multiple predicted texts output by the pre-trained model can be obtained. Then, the number of violating texts that violate compliance rules among the multiple predicted texts is determined. Violation detection of the predicted texts can be implemented based on rule engines and semantic matching technology to ensure the accuracy of the violation text statistics. Then, the second weight is determined based on the number of violation texts. Finally, the first weight is determined based on the second weight. The first weight = 1 - the second weight.

[0076] (2) In the above formula (2), This indicates the number of violating texts identified at the current moment. α These are preset values; the specific values ​​can be determined through cross-validation, for example... α The value is 0.05.

[0077] In this embodiment, by acquiring multiple predicted texts output by the pre-trained model and determining the number of non-compliant texts among them, a second weight is determined based on the number of non-compliant texts, and a first weight is determined based on the second weight. This achieves a dynamic weight adjustment mechanism that can dynamically adjust the weights of language modeling loss and compliance alignment loss according to the number of non-compliant texts output by the model during training. This ensures that language patterns are learned first in the early stages of model training, compliance constraints are strengthened in the later stages of training, and content generation that violates domain rules is dynamically penalized during the fine-tuning of the pre-trained language model. This precisely balances the model's language fluency and compliance, improves the accuracy and robustness of security alignment, and ensures that the content generated by the trained model maintains both language fluency and strict compliance with domain regulatory norms.

[0078] In practical applications, the second and first weights can be dynamically adjusted according to a preset update cycle. For example, the weights can be iteratively updated every 100 outputs. During training, the model output is monitored, and when generated content violates domain rules, the number of violations is counted. This number can be output to the training log, and the violation count can be retrieved from the training log when weights need to be updated. The weight coefficients are used to balance the language modeling loss and compliance alignment loss. During training, the initial value of the second weight can be set to 0.1, and it is dynamically adjusted based on the number of violations. It should be noted that the count of violations is accumulated as training progresses. After each adjustment of the weight coefficients, the count does not reset from zero; instead, it continues to accumulate based on the original number of violations to ensure the accuracy of weight adjustments.

[0079] Step 304: Adjust the model parameters of the pre-trained language model with the training objective of minimizing the loss value of the pre-trained language model and iterate the training until the loss value of the pre-trained language model is less than the preset value or the preset number of training rounds is reached. The training is then completed, and a compliant text generation model is obtained.

[0080] In this embodiment, after determining the loss value of the pre-trained model, if the loss value is not less than the preset value, that is, the model has not yet converged, the model parameters of the pre-trained language model are adjusted with the goal of minimizing the loss value of the pre-trained language model and iterative training is performed until the obtained loss value is less than the preset value, or although the loss value is not less than the preset value, the number of iterations has reached the preset number of training rounds. If either of these two conditions is met, the training is considered complete, and the model at this time is determined as the trained compliant text generation model, which generates compliant content in the target domain.

[0081] When adjusting model parameters, a mini-batch gradient descent algorithm can be used to update the parameters, combined with a learning rate scheduling strategy to gradually reduce the learning rate. The learning rate scheduling strategy uses exponential decay, and the learning rate formula is as follows: ,in, The initial learning rate, , As the attenuation factor, ; t This represents the number of epochs during training; it is an integer counter that monotonically increases as training progresses. The learning rate is updated after each training epoch, adjusted exponentially to ensure a smooth decrease in the learning rate, avoiding training oscillations and ensuring the model converges to the optimal solution. Gradient clipping is used to prevent gradient explosion, and the gradient norm threshold is set to 1.0. The Adam optimizer is used for training.

[0082] The text processing method of this application embodiment inputs sample text and corresponding sample compliance knowledge vector representation into a pre-trained language model, obtains the prediction result and hidden state vector output by the pre-trained language model, determines the loss value of the pre-trained language model based on the sample compliance knowledge vector representation, hidden state vector, and prediction result, and adjusts the model parameters of the pre-trained language model with the loss value of the pre-trained language model as the training objective and iteratively trains until a compliant text generation model is obtained. Thus, by introducing sample compliance knowledge vector representation as an additional input and using it for loss value calculation during the fine-tuning process of the pre-trained language model, the model's ability to perceive domain compliance rules can be enhanced, and a compliant text generation model that can output securely aligned compliant text can be trained, thereby improving the security and compliance of the large model in the target domain and promoting the deep application of artificial intelligence technology in the target domain.

[0083] During training, the accuracy can be calculated periodically using a validation set. When the accuracy falls below a preset threshold, an early stopping mechanism is triggered to prevent overfitting. Therefore, in one optional embodiment of this application, the pre-trained language model can be validated using a validation dataset according to a preset validation cycle during training to obtain the prediction accuracy of the pre-trained language model. When the prediction accuracy gradually decreases and falls below the accuracy threshold, an early stopping mechanism is triggered, and the current pre-trained language model is used as the compliant text generation model.

[0084] The validation cycle can be set according to actual needs. For example, it can be set to validate once after each round of training, i.e., the validation cycle is one round. The prediction accuracy can be determined by the ratio of the number of accurate outputs in the text content of the current model (i.e., the model obtained after the most recent round of training) to the total number of outputs in the validation dataset. The accuracy threshold can be set according to actual needs. By comparing the prediction accuracy of multiple validations, if the prediction accuracy is found to gradually decrease and fall below the accuracy threshold, the early stopping mechanism is triggered, training is stopped, and the latest model is determined as the compliant text generation model.

[0085] It is understandable that the type of accuracy assessment may differ across different sectors. For example, in the financial sector, the prediction accuracy determined through the validation set could be the financial compliance accuracy; in the healthcare sector, the prediction accuracy determined through the validation set could be based on medical compliance rate, diagnosis and treatment accuracy, and allergy risk detection rate as core indicators.

[0086] In this embodiment, the pre-trained language model is validated using a validation dataset during the training process according to a preset validation cycle. The prediction accuracy of the pre-trained language model is obtained. When the prediction accuracy gradually decreases and falls below the accuracy threshold, an early stopping mechanism is triggered, and the current pre-trained language model is used as a compliant text generation model. Thus, the training is stopped when the model prediction accuracy continues to decline, which can prevent overfitting. Furthermore, the learning rate scheduling and the early stopping mechanism work together during the training process to optimize training efficiency and model performance.

[0087] In one optional implementation of this application, before deployment, the performance of the obtained compliance text generation model can be evaluated using a test dataset derived from real business scenarios. For example, in the financial field, a test dataset containing financial Q&A and compliance scenarios can be used to evaluate model performance. Model performance metrics may include, but are not limited to, compliance rate, risk detection accuracy, and response time. Based on the evaluation results, parameter pruning and quantization techniques are applied to optimize the model size and inference speed to ensure that it meets the performance and resource constraints of the production environment. When deploying the compliance text generation model, a real-time monitoring module is integrated to continuously record all target texts generated by the model, detect potential compliance risks in the target text that violate the regulatory and business rules of the target domain, and generate corresponding risk tags. Business personnel verify the target text based on the risk tags. If the risk is confirmed to be real, a correction instruction is triggered, and the corrected compliance text is recorded. If the risk is confirmed to be a false detection, a compliance confirmation instruction is triggered, and a compliance label is recorded. The risk tags, corrected text, and compliance labels are aggregated into a compliance dataset. Based on this compliance dataset, the model's operating parameters are adjusted to correct compliance deviations and output a deployable large model that meets the domain's compliance requirements. This model ensures security and compliance in practical applications. The real-time monitoring mechanism ensures the feasibility and stability of the model in practical applications, providing reliable technical support for the compliant application of large language models in the target domain.

[0088] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0089] Embodiments of this application also provide a text processing device, which can be implemented in software and / or hardware and can be integrated into an electronic device.

[0090] Figure 5 This is a schematic diagram of the structure of a text processing device provided in an embodiment of this application, as shown below. Figure 5 As shown, the text processing device 50 includes: an acquisition module 510, an image generation module 520, a feature extraction module 530, and a text generation module 540.

[0091] Among them, the acquisition module 510 is used to acquire the text to be processed in the target domain; The graph generation module 520 is used to perform entity recognition and entity relation extraction on the text to be processed, and generate a heterogeneous graph corresponding to the text to be processed. Entity recognition includes domain entity recognition and rule entity recognition. The feature extraction module 530 is used to extract features from the heterogeneous graph through the heterogeneous graph attention network of the target domain to obtain the compliance knowledge vector representation of the text to be processed. The heterogeneous graph attention network extracts features from the heterogeneous graph based on the meta-path set of the target domain. The meta-paths in the meta-path set are used to describe the compliance dependency relationship between domain entities and rule entities in the target domain. The text generation module 540 is used to input the compliance knowledge vector representation into a pre-trained compliance text generation model to obtain target text that matches the text to be processed. The target text conforms to the rule requirements of the target domain.

[0092] Optionally, the graph generation module 520 is also used for: The named entity recognition model is used to perform target domain entity recognition and rule entity recognition on the text to be processed, resulting in multiple entities, including domain entities and rule entities. By using a relation discrimination model based on a dual-channel encoder, entity relations are extracted from multiple entities to obtain a set of triples corresponding to multiple entities. Each triple in the set of triples includes two entities and the relationship type between the two entities. Generate heterogeneous graphs based on sets of triples.

[0093] Optionally, the feature extraction module 530 includes: The weight determination unit is used to determine the attention weights of neighboring nodes associated with entities in a heterogeneous graph through the attention layer of the heterogeneous graph attention network. The attention layer determines the neighboring nodes associated with entities based on the meta-path set of the target domain. The feature aggregation unit is used to aggregate features of entities and neighboring nodes based on the attention weights of neighboring nodes through the residual connection layer of the heterogeneous graph attention network, so as to obtain the aggregated features corresponding to the entity. The mapping unit is used to perform nonlinear mapping on the aggregated features of entities through the graph convolutional layer of the heterogeneous graph attention network to obtain the compliance knowledge vector representation corresponding to the text to be processed.

[0094] Optionally, the process by which the attention layer determines the attention weights of neighboring nodes associated with an entity in a heterogeneous graph includes: traversing the metapaths in the metapath set, finding a multi-hop node sequence that matches the metapath in the heterogeneous graph, with the starting node of the multi-hop node sequence being the entity; determining the ending node of the multi-hop node sequence as the neighboring node associated with the entity; determining the heterogeneous subgraph corresponding to the entity from the heterogeneous graph based on the entity and the multi-hop node sequence; and determining the attention weights of neighboring nodes associated with the entity based on the heterogeneous subgraph.

[0095] Optionally, the process of the graph convolutional layer performing nonlinear mapping on the aggregated features of entities and outputting a compliance knowledge vector representation includes: using the aggregated features of entities as input data for the first layer, determining the output features of entities at the m-th layer layer by layer, where 1≤m≤M, M is the total number of layers in the graph convolutional layer, and using the output features of the previous layer as input to the current layer; determining the compliance feature representation of entities based on the aggregated features of entities and the output features of entities at the M-th layer; and generating and outputting the compliance knowledge vector representation corresponding to the text to be processed based on the compliance feature representation of entities.

[0096] Optionally, the aggregated features of the entity are added to the output features of the entity at the Mth layer to obtain the compliance feature representation of the entity.

[0097] Optionally, the text processing device 50 further includes: The sample acquisition module is used to acquire training samples in the target domain. The training samples include sample text and the sample compliance knowledge vector representation corresponding to the sample text. The input module is used to input sample text and sample compliance knowledge vector representation into the pre-trained language model, and obtain the prediction results and hidden state vectors output by the pre-trained language model. The loss determination module is used to determine the loss value of the pre-trained language model based on the sample compliance knowledge vector representation, hidden state vector, and prediction results. The model training module is used to adjust the model parameters of the pre-trained language model with the training objective of minimizing the loss value of the pre-trained language model and iteratively train it until the loss value of the pre-trained language model is less than the preset value or the preset number of training rounds is reached. Once training is complete, a compliant text generation model is obtained.

[0098] Optionally, the loss determination module includes: The first determining unit is used to determine the language modeling loss of the pre-trained language model based on the prediction results; The second determining unit is used to determine the compliance alignment loss of the pre-trained language model based on the sample compliance knowledge vector representation and the hidden state vector. The third determining unit is used to determine the loss value of the pre-trained language model based on the language modeling loss and compliance alignment loss.

[0099] Further optionally, the third determining unit is also used for: Obtain the first weight corresponding to the language modeling loss and the second weight corresponding to the compliance alignment loss, and the sum of the first weight and the second weight is 1; The loss value of the pre-trained language model is obtained by weighting and summing the language modeling loss and compliance alignment loss based on the first and second weights.

[0100] Optionally, the third determining unit is also used for: Obtain multiple predicted texts output by the pre-trained model; Determine the number of non-compliant texts that violate compliance rules among multiple predicted texts; The second weight is determined based on the number of violating texts; The first weight is determined based on the second weight.

[0101] Optionally, the sample acquisition module is also used for: Obtain compliance knowledge texts in the target domain; Data augmentation is performed on compliance knowledge texts to obtain augmented text; Sample text is constructed based on compliance knowledge text and enhanced text; Entity recognition and entity relation extraction are performed on the sample text to generate a sample heterogeneity graph corresponding to the sample text; Feature extraction of sample heterogeneous graphs is performed by using a heterogeneous graph attention network in the target domain to obtain the sample compliance knowledge vector representation corresponding to the sample text; Training samples are constructed based on sample text and sample compliance knowledge vector representations.

[0102] Optionally, the text processing device 50 further includes: a verification module, used for: During the training process, the pre-trained language model is validated using a validation dataset according to a preset validation cycle to obtain the prediction accuracy of the pre-trained language model. An early stopping mechanism is triggered when the prediction accuracy gradually decreases and falls below the accuracy threshold, and the current pre-trained language model is used as the compliant text generation model.

[0103] For a description of the features in the embodiments corresponding to the text processing device, please refer to the relevant descriptions in the embodiments corresponding to the text processing method, which will not be repeated here.

[0104] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described text processing method embodiments.

[0105] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described text processing method embodiments when it is run.

[0106] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0107] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described text processing method embodiments.

[0108] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described text processing method embodiments.

[0109] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be executed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), application-specific standard parts (ASSP), a system-on-chip (SoC), a complex programmable logic device (CPLD), a microcontroller unit (MCU), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination thereof. The aforementioned computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0110] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0111] The foregoing has provided a detailed description of a text processing method, electronic device, storage medium, and program product provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to aid in understanding the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A text processing method, characterized in that, include: Obtain the text to be processed in the target domain; Entity recognition and entity relation extraction are performed on the text to be processed to generate a heterogeneous graph corresponding to the text to be processed. The entity recognition includes domain entity recognition and rule entity recognition. The heterogeneous graph is used to extract features from the heterogeneous graph through the heterogeneous graph attention network of the target domain to obtain the compliance knowledge vector representation of the text to be processed. The heterogeneous graph attention network extracts features from the heterogeneous graph based on the meta-path set of the target domain. The meta-paths in the meta-path set are used to describe the compliance dependency relationship between domain entities and rule entities in the target domain. The compliance knowledge vector representation is input into a pre-trained compliance text generation model to obtain target text that matches the text to be processed, and the target text conforms to the rule requirements of the target domain.

2. The text processing method according to claim 1, characterized in that, The step of performing entity recognition and entity relation extraction on the text to be processed to generate a heterogeneous graph corresponding to the text to be processed includes: The text to be processed is subjected to domain entity recognition and rule entity recognition in the target domain by a named entity recognition model to obtain multiple entities, including domain entities and rule entities. By extracting entity relationships from the multiple entities using a relationship discrimination model based on a dual-channel encoder, a set of triples corresponding to the multiple entities is obtained. Each triple in the set of triples includes two entities and the relationship type between the two entities. Based on the set of triples, a heterogeneous graph is generated.

3. The text processing method according to claim 1, characterized in that, The step of extracting features from the heterogeneous graph using a heterogeneous graph attention network in the target domain to obtain a compliance knowledge vector representation corresponding to the text to be processed includes: The attention layer of the heterogeneous graph attention network determines the attention weights of the neighboring nodes associated with an entity in the heterogeneous graph. The attention layer determines the neighboring nodes associated with the entity based on the meta-path set of the target domain. The entity and its neighboring nodes are aggregated based on the attention weights of the neighboring nodes through the residual connection layer of the heterogeneous graph attention network to obtain the aggregated features corresponding to the entity. The aggregated features of the entity are nonlinearly mapped through the graph convolutional layer of the heterogeneous graph attention network to obtain the compliance knowledge vector representation corresponding to the text to be processed.

4. The text processing method according to claim 3, characterized in that, The process by which the attention layer determines the attention weights of neighboring nodes associated with an entity in the heterogeneous graph includes: Traverse the metapaths in the metapath set, and find the multi-hop node sequence that matches the metapath from the heterogeneous graph, wherein the starting node of the multi-hop node sequence is the entity; The endpoint node of the multi-hop node sequence is determined as the neighbor node associated with the entity; Based on the entity and the multi-hop node sequence, determine the heterogeneous subgraph corresponding to the entity from the heterogeneous graph; Based on the heterogeneous subgraph, the attention weights of the neighboring nodes associated with the entity are determined.

5. The text processing method according to claim 3, characterized in that, The process by which the graph convolutional layer performs a non-linear mapping on the aggregated features of the entity and outputs the compliance knowledge vector representation includes: The aggregated features of the entity are used as the input data of the first layer, and the output features of the entity at the m-th layer are determined layer by layer, where 1≤m≤M, M is the total number of layers of the graph convolutional layer, and the output features of the previous layer are used as the input of the current layer. Based on the aggregation characteristics of the entity and the output characteristics of the entity at the Mth layer, the compliance characteristic representation of the entity is determined; Based on the compliance feature representation of the entity, a compliance knowledge vector representation corresponding to the text to be processed is generated and output.

6. The text processing method according to claim 5, characterized in that, The determination of the compliance feature representation of the entity based on the entity's aggregation features and the entity's output features at layer M includes: The aggregated features of the entity are added to the output features of the entity at the Mth layer to obtain the compliance feature representation of the entity.

7. The text processing method according to any one of claims 1-6, characterized in that, The compliant text generation model is trained in the following manner: Obtain training samples from the target domain, the training samples including sample text and sample compliance knowledge vector representations corresponding to the sample text; The sample text and the sample compliance knowledge vector representation are input into the pre-trained language model, and the prediction results and hidden state vectors output by the pre-trained language model are obtained. Based on the sample compliance knowledge vector representation, the hidden state vector, and the prediction result, the loss value of the pre-trained language model is determined; The model parameters of the pre-trained language model are adjusted with the training objective of minimizing the loss value of the pre-trained language model, and training is iterated until the loss value of the pre-trained language model is less than a preset value or the preset number of training rounds is reached. The training is then completed, and the compliant text generation model is obtained.

8. The text processing method according to claim 7, characterized in that, The step of determining the loss value of the pre-trained language model based on the sample compliance knowledge vector representation, the hidden state vector, and the prediction result includes: The language modeling loss of the pre-trained language model is determined based on the prediction results; Based on the sample compliance knowledge vector representation and the hidden state vector, the compliance alignment loss of the pre-trained language model is determined; The loss value of the pre-trained language model is determined based on the language modeling loss and the compliance alignment loss.

9. The text processing method according to claim 8, characterized in that, Determining the loss value of the pre-trained language model based on the language modeling loss and the compliance alignment loss includes: Obtain the first weight corresponding to the language modeling loss and the second weight corresponding to the compliance alignment loss, wherein the sum of the first weight and the second weight is 1; The language modeling loss and the compliance alignment loss are weighted and summed based on the first weight and the second weight to obtain the loss value of the pre-trained language model.

10. The text processing method according to claim 9, characterized in that, The step of obtaining the first weight corresponding to the language modeling loss and the second weight corresponding to the compliance alignment loss includes: Obtain multiple predicted texts output by the pre-trained model; Determine the number of non-compliant texts that violate compliance rules among the multiple predicted texts; The second weight is determined based on the number of the violating texts; The first weight is determined based on the second weight.

11. The text processing method according to claim 7, characterized in that, The step of obtaining training samples in the target domain includes: Obtain compliance knowledge text in the target domain; The compliance knowledge text is augmented to obtain augmented text; The sample text is constructed based on the aforementioned compliance knowledge text and the aforementioned enhanced text; Entity recognition and entity relation extraction are performed on the sample text to generate a sample heterogeneity graph corresponding to the sample text; Feature extraction is performed on the sample heterogeneous graph using a heterogeneous graph attention network in the target domain to obtain the sample compliance knowledge vector representation corresponding to the sample text. The training samples are constructed based on the sample text and the sample compliance knowledge vector representation.

12. The text processing method according to claim 7, characterized in that, The method further includes: During the training process, the pre-trained language model is validated using a validation dataset according to a preset validation period to obtain the prediction accuracy of the pre-trained language model. An early stopping mechanism is triggered when the prediction accuracy gradually decreases and falls below the accuracy threshold, and the current pre-trained language model is used as the compliant text generation model.

13. The method according to claim 1, characterized in that, The method further includes: The performance of the compliant text generation model was evaluated using a test dataset derived from real-world business scenarios. Based on the performance evaluation results, the model size and inference speed of the compliant text generation model are optimized. The step of inputting the compliance knowledge vector representation into a pre-trained compliance text generation model includes: The compliance knowledge vector representation is input into the pre-trained and optimized compliance text generation model.

14. The method according to claim 1, characterized in that, The method further includes: Record the target text generated by the compliant text generation model; Detect potential compliance risks in the target text that violate the rules of the target domain, and generate risk markers; The system receives instructions triggered by relevant personnel after verifying the target text based on the risk marker: if the risk is confirmed to be real, a correction instruction is triggered and the corrected compliant text is recorded; if the risk is confirmed to be a false detection, a compliance confirmation instruction is triggered and the compliance label is recorded. The compliance dataset is determined based on the risk markers, the revised compliance text, and the compliance labels. The parameters of the compliance text generation model are updated based on the compliance dataset to obtain the updated compliance text generation model.

15. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the text processing method as described in any one of claims 1 to 14.

Images