Children long-sequence medical data processing method and device based on attention distribution optimization and program product

Abstract

The invention relates to the field of intelligent medical treatment, in particular to a child long-sequence medical data processing method and device based on attention distribution optimization and a program product. Comprising the steps of obtaining child long-sequence text data; the child long-sequence text is input to a neural network, the neural network comprises N stacked attention layers, N is a natural number larger than 1, the stacked attention layers are cross-stacked standard attention and sparse attention, and N is a natural number larger than 1; performing long-sequence text context feature extraction on the child long-sequence text through standard attention and sparse attention which are crossed and stacked to obtain processed long-sequence feature data; wherein when the sparse attention is subjected to attention distribution calculation, one is added to a denominator of a probability distribution conversion function, and the total attention of the sparse attention of the current layer is obtained to be smaller than one. The application has good clinical value.

Description

Technical Field

[0001] This application relates to the field of intelligent healthcare, specifically to a method, device, program product, and computer-readable storage medium for processing long-sequence medical data of children based on attention distribution optimization. Background Technology

[0002] The significance of researching long windows lies in improving the model's ability to process long-sequence data. Long windows mean the model can consider more contextual information, which is crucial for understanding and generating high-quality output. This is important for many fields, including Natural Language Processing (NLP), speech recognition, and time series analysis. Particularly in the field of pediatric medicine, when performing NLP on textual data such as disease-related medical knowledge, medical guidelines, and medical case studies, long windows are needed for contextual understanding of the text sequence. However, long window processing techniques still have significant limitations, including: 1. Pre-filling time: Pre-filling long input sequences requires significantly more computation time and GPU memory than short input sequences; 2. Concurrency limitations: The large key-value cache of long-context models consumes a large amount of GPU high-bandwidth memory (HBM), limiting the number of concurrent users that can be served simultaneously; 3. Decoding latency: Repeatedly reading the key-value cache from the HBM during the decoding phase significantly increases latency; 4. Context switching latency: When the key-value cache overflows into the CPU's DDR memory, context switching introduces significant latency. Summary of the Invention

[0003] Therefore, research on long-window models faces challenges such as high deployment costs and low computational efficiency, especially compared to short-context models. Current research focuses on reducing the deployment cost of long-context models, enabling them to handle longer sequences, while maintaining costs comparable to short-context models, to achieve widespread deployment of long-window models in practical applications. To address the above problems, this invention provides a method for processing long-sequence medical data in children based on attention distribution optimization, specifically including: Obtain long-sequence text data of children; The long sequence text of the child is input into a neural network, which includes N stacked attention layers, where N is a natural number greater than 1. The stacked attention layers are cross-stacked standard attention and sparse attention. The long sequence text of the child is processed by extracting contextual features of the long sequence text through cross-stacked standard attention and sparse attention to obtain the processed long sequence feature data. In the sparse attention, when calculating the attention distribution, the denominator of the probability distribution transformation function is increased by one to obtain that the total attention of the current layer of sparse attention is less than one.

[0004] Optionally, the neural network further includes a data embedding layer. After the long sequence text of the child is passed through the data embedding layer, a token vector of the long sequence is obtained. The token vector of the long sequence is then input to N stacked attention layers to extract the contextual features of the long sequence text and obtain the processed long sequence feature data. The data embedding layer is used to dynamically encode the token vector of the long sequence text by embedding the long sequence text input length and rotation position vector.

[0005] Optionally, the rotation position vector embedding is obtained by extrapolating or interpolating the position vector embedding to obtain a long sequence of token vectors.

[0006] Optionally, the rotation position vector embedding process is as follows: the long sequence text data of the child is input to the data embedding layer and converted into a token vector; the length of the long sequence is calculated; the rotation position vector is embedded based on the length of the long sequence to generate a dynamic position code; and the token vector and the dynamic position code are fused to obtain the token vector of the long sequence.

[0007] Optionally, the long sequence of token vectors is input to the attention layer, where the relative distance is first calculated through dynamic position encoding, and an attention score matrix is ​​obtained by calculating the attention score based on the relative distance, the query vector, and the key vector. The output of the current attention layer is then calculated using the attention score matrix.

[0008] Optionally, the attention layer further includes generating a pediatric semantic relevance bias matrix, obtaining any L positions in the attention score matrix and the corresponding word vectors, where L is a natural number greater than 1, querying the entities of the word vectors in the knowledge graph, calculating the relevance between the entities to obtain a relevance bias value matrix, fusing the relevance bias value matrix with the attention score matrix to obtain an attention weight matrix, and obtaining the output of the current attention layer based on the attention weight matrix.

[0009] Optionally, the word vector is a word vector or character vector in a long sequence of text for children.

[0010] Optionally, the query further includes entity determination, querying the mapping data of the word vector in the knowledge graph, determining whether the mapping data is an entity, calculating the relevance bias value between entities when the determination result is an entity, and assigning the relevance bias value to zero when the determination result is not an entity, thereby obtaining a relevance bias value matrix.

[0011] Optionally, when the attention layer is standard attention, the standard attention is multi-head attention. The multi-head attention is grouped, and each group of attention heads is guided to perform a preset medical type hierarchical perception to obtain a hierarchical perception attention weight matrix. The output of the current attention layer is obtained through the hierarchical perception attention weight matrix.

[0012] Optionally, the fixed guidance obtains the entity corresponding to the position through the attention score matrix, determines the preset type of the entity, and obtains the attention weight matrix for layered perception by performing layered perception calculation based on the preset type.

[0013] Optionally, the hierarchical perception calculation is based on a preset type to calculate the corresponding type relevance bias value; otherwise, the bias is assigned to zero to obtain a second relevance bias value matrix. The second relevance bias value matrix is ​​then fused with the attention score matrix to obtain the hierarchical perception attention weight matrix. The output of the current attention layer is obtained based on the hierarchical perception attention weight matrix.

[0014] Optionally, the preset type includes any one or more of the following: symptom-diagnosis, description-signs, examination-symptoms, diagnosis-treatment method.

[0015] Optionally, the output of the current attention layer is obtained by first calculating a second relevance bias matrix through hierarchical perception, then querying entities at corresponding positions based on the attention score matrix, and calculating the relevance between the entities to obtain a third relevance bias value matrix. The third relevance bias value matrix is ​​then fused with the attention score matrix to obtain a third attention weight matrix. The output of the current attention layer is obtained based on the third attention weight matrix. Optionally, the output of the current attention layer is obtained by first querying the entity at the corresponding position through the attention score matrix, calculating the correlation between the entities to obtain the correlation bias matrix, then calculating the fourth correlation bias matrix through hierarchical perception, fusing the fourth correlation bias matrix with the attention score matrix to obtain the fourth attention weight matrix, and obtaining the output of the current attention layer based on the fourth attention weight matrix.

[0016] Optionally, the output of the current attention layer is obtained by performing an element-wise AND operation between the relevance bias matrix and the second relevance bias matrix to obtain the fifth relevance bias matrix, fusing the fifth relevance bias matrix with the attention score matrix to obtain the fifth attention weight matrix, and obtaining the output of the current attention layer based on the fifth attention weight matrix.

[0017] Optionally, the standard attention includes one or more of the following: multi-head attention, self-attention, and cross attention.

[0018] Optionally, the sparse attention includes one or more of the following: sliding window attention, dilation attention, adaptive sparse attention, block sparse attention, and grouped query attention.

[0019] The purpose of this invention is to provide a computer program product that includes a computer program or instructions, which are executed by a processor to implement the above-described method for processing long-sequence medical data of children based on attention distribution optimization.

[0020] The purpose of this invention is to provide a computer device comprising a memory, a processor, and a computer program or instructions stored in the memory, wherein the computer program or instructions are executed by the processor to implement the above-described method for processing long-sequence medical data of children based on attention distribution optimization.

[0021] The purpose of this invention is to provide a computer-readable storage medium having a computer program or instructions stored thereon, which is executed by a processor to implement the above-described method for processing long-sequence medical data of children based on attention distribution optimization.

[0022] Advantages of this invention: 1. Long sequence data processing is performed by combining sparse attention and standard attention. Specifically, long sequence data processing is performed by sliding window attention and multi-head attention. After the long sequence data information is obtained by sliding the window, it is fed into multi-head attention for further processing. In this way, the overall computational consumption is linearly related to the input length. After the length exceeds a certain limit, this method can significantly reduce the GPU memory usage without reducing the ability to understand the context of long texts.

[0023] 2. The combined sparse attention and standard attention are positionally encoded by rotating position encoding and calculating the relative distance between vectors. Extrapolation dynamic inference is performed based on long sequences, and the attention mask is adjusted by interpolation. This provides dynamic position encoding for combined attention, which enhances the model's ability to model long sequence dependencies while ensuring resolution.

[0024] 3. The standard Transformer's self-attention mechanism treats all word pairs equally when calculating inter-word associations, ignoring the specific semantic constraints in pediatric consultations. Therefore, this invention introduces a pediatric semantic relevance bias matrix and hierarchical attention head division of labor into the Transformer, transforming entity relationship distances in the knowledge graph into attention priors, guiding the model to focus on medical associations. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A schematic flowchart of a method for processing long-sequence medical data of children based on attention distribution optimization provided in an embodiment of the present invention; Figure 2 A schematic diagram of a system for processing long-sequence medical data of children based on attention distribution optimization, provided in an embodiment of the present invention; Figure 3 A schematic diagram of a device for processing long-sequence medical data of children based on attention distribution optimization, provided in an embodiment of the present invention; Figure 4 A comparison of Attention Masks for global attention mechanisms and local attention mechanisms provided in embodiments of the present invention; Figure 5 Extrapolation and interpolation based on rotational position encoding provided in this embodiment of the invention; Figure 6 Comparison of the accuracy of the large model Longeval test provided in the embodiments of the present invention; Figure 7 Comparison of perplexity of large models provided for embodiments of the present invention; Figure 8 The RoPE provided in this embodiment of the invention demonstrates the ability of the attention score level to focus on similar tokens under different bases. Figure 9 This is a fitting graph of the relationship between the base in RoPE and the context windows it can support, provided for embodiments of the present invention.

[0027] Figure 10 This is a performance comparison chart of the model under the small base case provided in the embodiments of the present invention. The three columns from left to right are the three evaluation metrics: perplexity, rangeval, and needle-in-a-haystack. From top to bottom, they are the evaluation results of the small base case in the fine-tuning and pre-training scenarios. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0029] In some of the processes described in the specification, claims, and accompanying drawings of this invention, multiple operations appearing in a specific order are included. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The operation numbers, such as S101, S102, etc., are merely used to distinguish different operations and do not represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0030] Figure 1 A schematic diagram of the method for processing long-sequence medical data of children based on attention distribution optimization provided in this embodiment of the invention is shown, specifically including: S1: Obtain long-sequence text data for children; In one embodiment, the long sequence text for children includes one or more of the following: medical guidelines, children's disease cases, examination data, and medical Q&A data.

[0031] In one specific embodiment, the collection of children's question-and-answer text data primarily comprises two data sources: first, anonymized authentic doctor-patient dialogue records obtained from partner pediatric hospitals or online health consultation platforms, covering complete question-and-answer pairs from parents describing symptoms to doctors providing diagnostic advice; second, relevant question-and-answer texts collected from publicly available medical question-and-answer communities and pediatric health science articles. All of this text data undergoes rigorous privacy anonymization processing to remove personally identifiable information, ensuring the compliance and security of data use. Children's disease case records and examination data are obtained from partner pediatric hospitals, and medical guidelines are based on currently published medical guidelines for the diagnosis of pediatric diseases.

[0032] Data cleaning and standardization are performed on the collected data from various children's fields, including but not limited to data entity identification and labeling, data relationship extraction, data structuring, and data filtering.

[0033] In one specific embodiment, an external knowledge base is constructed and collected to obtain or construct a structured pediatric medical knowledge graph. This graph should contain a rich set of entities such as pediatric diseases, symptoms, drugs, and examinations, as well as semantic relationships between entities (such as "disease-accompanying symptoms" and "drug-treatment of disease"). This knowledge graph can be derived from the extraction and organization of pediatric-related parts of public medical knowledge bases (such as UMLS), or it can be self-built based on authoritative medical textbooks and clinical guidelines.

[0034] Construct or collect several key dictionaries and terminology databases: First, a "Dictionary of Vague Expressions in Pediatrics," which specifically collects colloquial and vague words commonly used by children and parents to describe discomfort (such as "feeling unwell," "stomach upset," and "wheezing"); second, a "Dictionary of Standard Medical Terminology," used to standardize the expression of medical concepts.

[0035] In addition, a lightweight named entity recognition model for the pediatric domain can be obtained by fine-tuning a general biomedical BERT model (such as BioBERT) on labeled pediatric medical text. Its function is to accurately identify pediatric medical entities defined by the aforementioned labeling categories from the text.

[0036] The collected original long-sequence texts of children were subjected to refined annotation. The annotation work mainly included two aspects: First, medical entity annotation of the text. Professionals with medical backgrounds identified and annotated the pediatric medical entities involved in the text according to predefined annotation specifications. The annotation categories need to be closely integrated with subsequent steps, and mainly include: Symptom description (e.g., "cough" or "abdominal pain"), disease name (e.g., "acute bronchitis" or "allergic rhinitis"), body part (e.g., "throat" or "abdomen"), examination items (e.g., "complete blood count" or "lung auscultation"), treatment method (e.g., "nebulizer therapy" or "oral rehydration salts"), etc. Second, for some high-quality question-and-answer data, we will label the intent and logical relationships, such as the disease category to which the question belongs and the logical order of key suggestions in the answer. This labeled data will be used for subsequent loss function construction and model evaluation.

[0037] In one embodiment, long sequence text of children is preprocessed to obtain enhanced text, which is then fed into a neural network.

[0038] In one embodiment, the process of enhancing text involves: acquiring long-sequence text data of children; Named entity recognition and fuzzy expression extraction were performed on the long sequence text data of children to obtain medical terminology entities and fuzzy words; Calculate the semantic similarity and knowledge graph correlation between the fuzzy words and medical term entities, and match each fuzzy word with a medical term entity to obtain the matching result; In the matching results, medical term entities are retrieved from the external pediatric knowledge graph to construct a local subgraph by searching all relevant entities and relationships within the skip neighborhood. In the local subgraph, connection paths are randomly generated and the paths are converted into a set of knowledge statements. The knowledge statement set is fused with children's long-sequence text data to obtain enhanced text. The skip neighborhood is obtained by using different numbers of skips in the entity neighborhood of medical terms.

[0039] Optionally, the fusion of the knowledge statement set with the children's long sequence text data is carried out through a dynamic gating mechanism. The dynamic gating mechanism calculates the gating weight of each knowledge statement in relation to the context of the children's long sequence text data, and only merges the relevant knowledge with higher weights to obtain the enhanced text.

[0040] In one specific embodiment, the present invention employs a data preprocessing method that combines an external pediatric medical knowledge graph with context-aware semantic diffusion. Through semantic enhancement and knowledge injection, it generates enhanced text sequences rich in structured semantic information. The specific steps are as follows: 1) Context-aware medical terminology extraction and fuzzy word alignment; Given a long sequence of text data from children, a lightweight pediatric domain named entity recognition model is first used to identify standard medical terminology entities from the long sequence of text. Simultaneously, a fuzzy word dictionary is used to match fuzzy expressions in the sequence of text. Furthermore, by calculating semantic similarity and knowledge graph relevance, each fuzzy word is aligned to the most relevant standard medical terminology entity, as shown below:

[0041] In the formula, Indicates the relationship with the first The first sequence of text The standard medical terms aligned with the fuzzy words are the standard medical term entities determined after alignment. They are most relevant to the fuzzy words described by children in terms of semantics and medical knowledge, and represent the professional and precise transformation results of the original fuzzy and imprecise descriptions. This represents the sample index, used to identify the first... A sequence of text; Indicates a fuzzy vocabulary index; Indicates from the first Text in a sequence The fuzzy word instances identified are specifically determined by a pre-built "pediatric fuzzy expression dictionary". This dictionary collects non-professional, colloquial words that children often use to describe symptoms, such as "feeling uncomfortable", "fussy", "stomach rumbling", and "feeling a little unwell", and uses them as a pattern library. Represents an external pediatric medical knowledge graph The set consisting of all the entities included; Represents an external pediatric medical knowledge graph; The entity representing standard medical terminology is a set. Candidate elements, such as "fever", "acute upper respiratory tract infection", "abdominal pain", etc. Indicates in set Find the entity that maximizes the objective function value. The operation; This represents a pre-trained word vector model function, which maps input words or entities into dense semantic vectors of fixed dimensions. For example, a pre-trained word vector model can be a word embedding layer of a Transformer model pre-trained on a large general corpus, such as Word2Vec, GloVe, or a Transformer model, to provide a general semantic vector representation of words. This represents the cosine similarity calculation function, used to measure the directional similarity between two word vectors in the semantic space, with a range of [value range missing]. ; Indicates instances of fuzzy words With entity In knowledge graph The strength of the association between two entities is specifically measured by the reciprocal of the number of edges traversed by the shortest path between them; the closer the distance, the stronger the association. The hyperparameter represents the weights that balance semantic similarity and graph relevance, controlling the proportion of each factor's contribution to the final decision. Its value ranges from [value range missing]. Example of a value: .

[0042] In practical implementation, lightweight pediatric domain-specific named entity recognition models can be obtained by fine-tuning general medical named entity recognition models (such as BERT-based models) on publicly available pediatric medical texts (such as pediatric textbook abstracts, clinical guidelines, and de-identified electronic medical record data), or by directly fine-tuning models pre-trained on biomedical corpora and available for download (such as BioBERT and ClinicalBERT).

[0043] In practical implementation, an external pediatric medical knowledge graph is used. It can be derived from publicly constructed pediatric disease knowledge graphs (such as those constructed by extracting pediatric-related parts from the medical knowledge base UMLS), or it can be self-built using medical ontology (such as SNOMEDCT, ICD-10) and combined with pediatric clinical practice knowledge.

[0044] 2) Subgraph retrieval and semantic diffusion based on knowledge graphs; For each sequence of text corresponding to an entity set, these entities are retrieved from an external pediatric medical knowledge graph. All relevant entities and relationships within the skip neighborhood constitute a local subgraph. A random walk algorithm is then used to generate multiple paths on this local subgraph, and these paths are converted into natural language descriptions, resulting in a set of knowledge statements related to the original sequence text, represented as:

[0045] In the formula, Indicates the first A set of knowledge statements for a sequence of texts, containing A natural language description is a collection of natural language sentences that converts the structured "entity-relationship-entity" relationships in a knowledge graph into a text form that the model can understand, representing the structured and semi-structured background medical knowledge related to the current sequence of text. Indicates the first The first sequence text related to the first Each knowledge statement text is generated by knowledge graph path transformation and encodes structured semantic relationships extracted from the knowledge graph; The number of knowledge statements is controlled by the random walk parameter, with examples of possible values. ; This represents the number of hops in a knowledge graph retrieval, defining the scope of the neighborhood retrieval. Examples of possible values ​​are provided. .

[0046] In one embodiment, assuming the original sequence text is: "The child has a cough with phlegm and is a little wheezing," then the named entity recognition model identifies "cough," "phlegm," and "wheezing," and uses these entities as a starting point in the knowledge graph. Search for it in the middle All relevant entities within the jump (such as "bronchitis", "asthma", "lung auscultation") and their relationships (such as "possible accompaniment", "is a symptom", "needs examination") are analyzed, and then a random walk is performed on the subgraph to generate paths such as "cough -> possible accompaniment -> expectoration -> is a symptom -> bronchitis". Finally, the paths are converted into natural language descriptions. For example, "A cough may be accompanied by phlegm, which is one of the symptoms of bronchitis."

[0047] 3) Dynamic gating fusion generates enhanced sequence text; To avoid introducing irrelevant information through simple concatenation, a dynamic gating mechanism is employed to fuse the original sequence text with a set of relevant knowledge statements. This mechanism calculates a gating weight for each knowledge statement based on its context with the original sequence text, fusing only the knowledge statements with higher weights to generate the semantically enhanced sequence text, as follows:

[0048] In the formula, Indicates the first A semantically enhanced sequence text is a sequence text that has been semantically enhanced. It retains the context and intent of the original long sequence text for children, while integrating external medical knowledge to make its expression more precise and its information richer. Indicates the first The original children's sequence texts usually contain a lot of colloquial, vague, emotional, and non-professional expressions; Indicates text concatenation operators; Indicates index from arrive All knowledge statements Perform a weighted splicing operation; This represents a knowledge statement index, with values ​​ranging from... arrive ; Indicates the first The first sequence text related to the first A knowledge statement text; Indicates the first The first sequence of texts A knowledge statement text The dynamic gating weights, with a value range of This characterizes the semantic relevance of the knowledge statement to the current sequence of text, and is calculated as follows: ; This represents the Sigmoid activation function, which maps real numbers to... interval; This represents the parameter vector of the dynamic gating network, which consists of trainable parameters with dimension 1. ; express Transpose of; The weight matrix of the dynamic gating network is a trainable parameter with dimension . ; This represents a function that uses a pre-trained language model to obtain a [CLS] tag vector representation of the input text. This vector is used to characterize the overall semantics of the input text. The pre-trained language model can be a standard pre-trained Transformer encoder model such as BERT, RoBERTa, or ALBERT. This represents a vector concatenation operation; This represents the hyperbolic tangent activation function, which maps input values ​​to... interval; This represents the dimension of the hidden layer in the pre-trained language model, with examples of possible values. .

[0049] It should be noted that "Related Knowledge:" is a fixed text prefix used to identify in the concatenated enhanced text that subsequent content is a statement of related knowledge imported from an external knowledge base.

[0050] Furthermore, the sequence text is standardized using only medical terminology to obtain enhanced text; In one implementation, medical terminology standardization involves replacing non-standard expressions, abbreviations, or synonyms that may exist in the sequence text with unified, standard medical terms by consulting a medical standard terminology dictionary. For example, "runny nose" is standardized to "nasal discharge," and "high fever" is standardized to "high fever." Specifically, definition Indicates the first The semantically enhanced text is the enhanced text after medical terminology standardization. The final output is the enhanced long-sequence text dataset for children. ,in, The total number of samples.

[0051] S2: The long sequence text of the child is input into the neural network, which includes N stacked attention layers, where N is a natural number greater than 1. The stacked attention layers are cross-stacked standard attention and sparse attention. The long sequence text of the child is processed by extracting contextual features of the long sequence text through cross-stacked standard attention and sparse attention to obtain the processed long sequence feature data. Among them, when the sparse attention is used to calculate the attention distribution, the denominator of the probability distribution transformation function is increased by one to obtain that the total attention of the current layer of sparse attention is less than one.

[0052] In one embodiment, the standard attention includes one or more of the following: multi-head attention, self-attention, and cross attention; the sparse attention includes one or more of the following: sliding window attention, dilated attention, adaptive sparse attention, block sparse attention, and grouped query attention. In one embodiment, the neural network further includes a data embedding layer. After the long sequence text of the child is passed through the data embedding layer, a token vector of the long sequence is obtained. The token vector of the long sequence is then input to N stacked attention layers to extract the contextual features of the long sequence text and obtain the processed long sequence feature data. The data embedding layer is used to dynamically encode the token vector of the long sequence text by embedding the long sequence text input length and rotation position vector.

[0053] In one embodiment, the rotation position vector embedding is obtained by extrapolating or interpolating the position vector embedding to obtain a long sequence of token vectors; Optionally, the rotation position vector embedding process is as follows: the long sequence text data of the child is input to the data embedding layer and converted into a token vector; the length of the long sequence is calculated; the rotation position vector is embedded based on the length of the long sequence to generate a dynamic position code; and the token vector and the dynamic position code are fused to obtain the token vector of the long sequence.

[0054] In one embodiment, the long sequence of token vectors is input to the attention layer. In the attention layer, the relative distance is first calculated through dynamic position encoding. Based on the relative distance, the query vector, and the key vector, an attention score matrix is ​​calculated. The output of the current attention layer is then calculated using the attention score matrix.

[0055] In one embodiment, the attention layer further includes generating a pediatric semantic relevance bias matrix, obtaining any L positions in the attention score matrix and the corresponding word vectors, where L is a natural number greater than 1, querying the entities of the word vectors in the knowledge graph, calculating the relevance between the entities to obtain a relevance bias value matrix, fusing the relevance bias value matrix with the attention score matrix to obtain an attention weight matrix, and obtaining the output of the current attention layer based on the attention weight matrix.

[0056] Optionally, the word vector is a word vector or character vector in a long sequence of text for children.

[0057] In one embodiment, the query further includes entity determination, querying the mapping data of the word vector in the knowledge graph, determining whether the mapping data is an entity, calculating the relevance bias value between entities when the determination result is an entity, and assigning the relevance bias value to zero when the determination result is not an entity, thereby obtaining a relevance bias value matrix.

[0058] In one embodiment, when the attention layer is standard attention, the standard attention is multi-head attention. The multi-head attention is grouped, and each group of attention heads is guided to perform a preset medical type hierarchical perception to obtain a hierarchical perception attention weight matrix. The output of the current attention layer is obtained through the hierarchical perception attention weight matrix.

[0059] In one embodiment, the fixed guidance obtains the entity corresponding to the position through the attention score matrix, determines the preset type of the entity, and obtains the attention weight matrix for layered perception by performing layered perception calculation based on the preset type.

[0060] In one embodiment, the hierarchical perception calculation is based on a preset type to calculate the corresponding type relevance bias value; otherwise, the bias value is assigned to zero to obtain a second relevance bias value matrix. The second relevance bias value matrix is ​​then fused with the attention score matrix to obtain the hierarchical perception attention weight matrix. The output of the current attention layer is obtained based on the hierarchical perception attention weight matrix.

[0061] In one embodiment, the preset type includes any one or more of the following: symptom-diagnosis, description-signs, examination-symptoms, diagnosis-treatment method.

[0062] In one embodiment, the output of the current attention layer is obtained by first calculating a second relevance bias matrix through hierarchical perception, then querying the entity at the corresponding position based on the attention score matrix, and calculating the relevance between the entities to obtain a third relevance bias value matrix. The third relevance bias value matrix is ​​then fused with the attention score matrix to obtain a third attention weight matrix, and the output of the current attention layer is obtained based on the third attention weight matrix.

[0063] In another embodiment, the output of the current attention layer is obtained by first querying the entity at the corresponding position through the attention score matrix, calculating the correlation between the entities to obtain the correlation bias matrix, then calculating the fourth correlation bias matrix through hierarchical perception, fusing the fourth correlation bias matrix with the attention score matrix to obtain the fourth attention weight matrix, and obtaining the output of the current attention layer based on the fourth attention weight matrix.

[0064] In another embodiment, the output of the current attention layer is obtained by performing an element-wise AND operation between the relevance bias matrix and the second relevance bias matrix to obtain a fifth relevance bias matrix, fusing the fifth relevance bias matrix with the attention score matrix to obtain a fifth attention weight matrix, and obtaining the output of the current attention layer based on the fifth attention weight matrix.

[0065] In one specific embodiment, the self-attention mechanism of the standard Transformer treats all word pairs equally when calculating the association between word pairs, ignoring specific semantic constraints in pediatric consultation, such as the stronger association between symptom description and disease diagnosis word pairs, and the strong prior correlation between entities with direct relationships in the knowledge graph during attention calculation.

[0066] Conventional multi-head attention lacks the utilization of such prior domain knowledge, making it difficult to quickly learn these important connections in fine-tuning scenarios with limited data.

[0067] This invention improves upon standard scaled dot product attention by employing a learnable pediatric semantic relevance bias matrix. This matrix is ​​dynamically generated from the relational distances between medical entities corresponding to lexical terms in the knowledge graph, thereby guiding the model to focus on more medically relevant lexical pairs. Simultaneously, it adopts a hierarchical attention head division of labor, allowing different attention heads to focus on different levels of medical semantic associations. The specific steps are as follows: 1) Calculation of basic attention score; For the input sequence representation of each Transformer layer, the query matrix and key matrix are first obtained through a relative distance linear transformation, and then the scaling dot product attention score is calculated, as follows:

[0068] In the formula, This represents the basic attention score matrix, with dimensions of . The first of the basic attention score matrix Line number Column elements Indicates position Position Basic attention weights; Indicates the first The query matrix of the layer is calculated as follows: ; Indicates the first The key matrix of the layer is calculated as follows: ; express Transpose of; This represents the dimension of the key vector, used to scale the dot product result and prevent excessively large values ​​from causing gradient instability. A specific value is preferred. ; Indicates the number of heads of attention; Indicates the first Each sample was processed before The output of the layer Transformer is the first layer. Each sample was processed before The output sequence after processing by the layer Transformer encoder is used as the first... The input to the layer attention mechanism has a dimension of ; This represents the layer index of the Transformer encoder; Indicates the first The weight matrix used to generate the query matrix in the layer is a trainable parameter with dimension 1. ; Indicates the first The weight matrix used to generate the key matrix in the layer is a trainable parameter with dimension 1. .

[0069] At the same time, define Indicates the first The value matrix of the layer is calculated as follows: ; Indicates the first The weight matrix used to generate the value matrix in the layer is a trainable parameter with dimension 1. ; The dimension of the value vector is preferably set to... .

[0070] 2) Dynamic generation of the pediatric semantic relevance bias matrix; For any two positions in the attention score matrix, query the entity in the knowledge graph corresponding to their respective words. If both words correspond to entities, calculate the semantic relevance bias based on their relational distance in the knowledge graph; otherwise, the semantic relevance bias is zero, as shown below:

[0071] In the formula, Indicates the first The layer attention mechanism applies to the location Position Pediatric semantic relevance bias value; Represents the position in the sequence lexical units in knowledge graphs The corresponding entity in the knowledge graph is determined based on the input lexical sequence currently being processed by the l-th layer Transformer. The corresponding entity in the middle; Represents the position in the sequence lexical units in knowledge graphs The corresponding entity in the middle; Representing entities and In knowledge graph The relational distance in the equation is defined as the number of edges traversed by the shortest path between two entities. Indicates the first The layer dependency scalar parameter is a trainable parameter used to control the overall dependency of the layer on the semantic bias. This represents the temperature hyperparameter, used to adjust the decay rate of the influence of the relational distance on the bias value. A larger value results in faster decay. Example values ​​are provided. ; This represents the natural exponential function.

[0072] It should be noted that during the construction process, a mapping relationship is established between lexical units and knowledge graph entities. For each lexical unit in the input lexical unit sequence currently being processed by the l-th layer Transformer, the knowledge graph is queried. If the word element Able to match knowledge graphs If a standard medical entity in the word "fever" matches the entity "fever", then... That is, the entity, if the word element If no entity can be matched, such as punctuation marks, the common verb "is", or non-medical terms not included in the knowledge graph, then... Considered empty or not part of the entity set That is, the semantic relevance bias value for pediatric cases corresponding to "other situations". The value is 0. Based on this, and During the preprocessing and embedding layer construction stages, external knowledge graphs are queried. , which are the words in the input sequence and word elements When calculating attention bias, the location is checked in relation to the corresponding medical entity. and location Have all the lexical terms been successfully associated with entities? If so, then based on the entity... and entity Calculate the bias value for relational distances in a knowledge graph.

[0073] Furthermore, define Indicates the first The pediatric semantic relevance bias matrix of the layer has dimensions of Pediatric semantic relevance bias matrix The Line number Column elements are .

[0074] 3) Bias fusion and attention output calculation; The dynamically generated pediatric semantic relevance bias matrix is ​​element-wise added to the basic attention score matrix, and then normalized using the Softmax function to obtain the attention weight matrix. This weight matrix is ​​then used to perform a weighted summation of the value matrix to obtain the output of this layer's attention mechanism, represented as:

[0075] In the formula, Indicates the first The output of the layer with a pediatric semantic bias attention mechanism incorporates a medical knowledge-driven prior relevance bias into the standard attention mechanism, representing the aggregation result of contextual information guided by domain knowledge. When guiding the model to calculate attention, it tends to focus on word pairs that are strongly related in medical knowledge, thereby accelerating the model's learning in data-scarce scenarios and improving the medical logic of its reasoning. Indicates the first The attention weight matrix of the layer, the final attention weight matrix after fusing pediatric semantic bias, is calculated as follows: Attention weight matrix The element in the t-th row and s-th column is Characterizing position Position Normalized attention weights; This represents the function that performs Softmax normalization row by row, ensuring that the sum of the attention weights of each position to all positions is equal. .

[0076] 4) Implementation of attention head division of labor in hierarchical perception; In multi-head attention mechanisms, different medical semantic constraints are applied to different attention heads to achieve functional division of labor. Specifically, focus on the number of heads The attention heads are divided into several groups, each responsible for focusing on a specific type of association between medical entities, such as "symptom-diagnosis" associations or "description-sign" associations. For attention heads focusing on "symptom-diagnosis" associations, attention is only given when the position... and location Corresponding entity and The pediatric semantic relevance bias value is calculated only when the symptoms belong to the "symptom" category and the disease category, respectively. Otherwise, the bias is zero. For attention heads that focus on the "description-sign" association, the bias is zero only if the position... and location Corresponding entity and The pediatric semantic relevance bias value is calculated only when the terms belong to the categories of "patient descriptive words" and "clinical signs," respectively. The bias calculation rules for other attention heads remain unchanged; In practice, entity category information is derived from a knowledge graph. The predefined category labels of the entities in the model guide the different attention heads to focus on different levels of medical logical relationships through this division of labor. Finally, the outputs of all attention heads are processed by standard Transformer layer components such as concatenation and feedforward networks to obtain the final output of this layer. Specifically, firstly, the output calculated for each attention head... The outputs are concatenated along the last dimension. The concatenated result is then passed through a learnable linear projection layer to unify the dimensions. Residual connections and layer normalization are then applied, and the normalized result is input into a standard feedforward neural network. Finally, residual connections and layer normalization are applied again, resulting in the final representation. Layer output ; in, Indicates the first The sample was processed by the first... The output representation sequence is processed by a layer-by-layer perceptual Transformer encoder, which includes a multi-head attention and feedforward network with pediatric semantic bias.

[0077] In one embodiment, the loss calculation of the neural network is as follows: construct an ideal attention prior matrix, and calculate the alignment loss by comparing the average attention distribution of each group of attention heads in the coding layer with the ideal attention prior matrix; Optionally, the loss calculation also includes differential loss calculation, which calculates the difference loss in attention distribution between different groups, and obtains the attention consistency loss through the difference loss and the alignment loss.

[0078] In one specific embodiment, in the hierarchical perception multi-head attention mechanism, different attention heads are designed to focus on specific types of medical entity associations. The ideal model should exhibit intra-group consistency and inter-group specificity in attention patterns; that is, the attention distribution of attention heads within the same group towards strongly associated medical entities should be similar, while the attention focus of attention heads in different groups should be distinct. To achieve this goal: This invention first utilizes the entity alignment results from the preprocessing stage and the entity category information from the knowledge graph to construct an ideal attention prior matrix for each sample in a batch. This matrix identifies the word pairs that the model should give high attention to. Then, it calculates the average attention distribution of each group of attention heads in a specific layer and encourages them to move closer to the ideal prior through alignment loss, while simultaneously encouraging the separation of attention distributions between different groups through differentiation loss. Finally, it calculates the attention consistency loss, expressed as:

[0079] In the formula, It represents the loss of attentional consistency, used to measure and guide the degree to which the model's internal attention mechanism learns and follows prior knowledge in pediatric medicine; This indicates the training batch size, which is the number of samples contained in the training batch. This indicates the total number of attention head groups, for example, if grouped by "symptom-diagnosis", "description-signs", and "other". ; Indicates the first The weight coefficients of each attention head group are trainable parameters used to balance the importance of different groups in the total loss function. This represents the Kullback-Leibler divergence, used to measure the difference between two probability distributions; Indicates temperature coefficient The row-wise Softmax normalization function, temperature coefficient Used to control the smoothness of the output distribution; a value less than 1 makes the distribution sharper, and a value greater than 1 makes the distribution smoother. Indicates the first The sample at the th The average attention matrix within a group of attention heads is obtained by averaging the attention weight matrices of all attention heads within that group. Indicates the first The sample at the th The average attention matrix within each attention head group; Indicates the first The ideal attention prior matrix for each sample has dimensions of . It is constructed from the fuzzy word alignment results in the preprocessing stage and the entity category information of the knowledge graph, and is used to identify the word pairs that the model should pay high attention to; Indicates the first Attention mask matrices for attention heads, with dimensions of . Based on the predefined medical entity association types for this grouping, irrelevant associations are filtered out when calculating the alignment loss, for two positions in the sequence. and location Query its corresponding entity and entity In a knowledge graph, if the category labels are... The category combination conforms to the first The type of group focus (e.g., "symptom-diagnosis"), then The corresponding Set it to 1 otherwise to 0, and then when calculating the Kullback-Leibler divergence, it is compared with the ideal attention prior matrix. Element-wise multiplication is used to filter out irrelevant associations; This represents the element-wise multiplication operator; The weight hyperparameter represents the differential loss, which controls the contribution ratio of the differential loss term to the total loss function. An example value is 0.1. Index representing the attention head group; Indicates difference from Attention Head Grouping Index Let represent the Jensen-Shannon divergence, a symmetric smoothed version of the KL divergence, used to measure the similarity between two distributions, with a range of . ; Indicates the first The cross-batch average attention matrix of the attention head group, within a training batch, for the _th ... Calculate the number of samples. The average attention matrix of the group Then, the average attention matrix of all samples in the batch is averaged over the batch dimension to obtain the first... The average attention matrix across batches for each attention head group ; This represents the scaling factor, used to adjust the sensitivity of the differential loss to the Jensen-Shannon divergence value; an example value is 1.0.

[0080] In the actual implementation, for any two positions in the sequence and location If their lexical units and word elements All correspond to entities in the knowledge graph, and (a) in fuzzy word alignment, Alignment from fuzzy words to (or vice versa); or (b) in knowledge graphs In the middle, their relational distance If the value is less than a preset threshold (e.g., 1 or 2), then within the ideal attention prior matrix... The corresponding Set it to 1, otherwise set it to 0, thus identifying word pairs that "should be given high attention".

[0081] In one embodiment, the method further includes inter-layer information redundancy calculation. The N stacked attention layers are judged to be similar by calculating inter-layer similarity. When the similarity is greater than a preset threshold, they are judged to be similar layers. Similar layers share parameters or are connected to non-similar layers by inter-layer skipping. Otherwise, feature data processing is performed based on the stacking order, and inter-layer information redundancy calculation is repeated until the last layer to obtain the processed long sequence feature data.

[0082] In one embodiment, the neural network further includes an expert system, with an expert system module connected in series after each attention layer as a feature layer. Long sequence text from children undergoes feature processing through N sets of feature layers to obtain processed long sequence feature data. The expert system includes a token, an expert module, and a first module. Each expert module contains S experts, where S is a natural number greater than 1. The expert module and the first module are parallel. One expert in the expert module is fixed as the fixed expert. After receiving data from the attention layer, the expert system inputs the data to both the expert module and the first module. The first module performs feature processing to obtain a first output. The expert module obtains fixed outputs through the fixed expert and dynamically selects one of the S-1 experts using the token to obtain a dynamic output. The fixed output and the dynamic output are fused to obtain a first fused output. The first fused output is then fused with the first output to obtain the output of the current expert system, which is input to the next attention layer. The number of experts in different layers of the expert system is not equal.

[0083] In one specific embodiment, the fundamental reason for proposing a hybrid model is that existing attention mechanisms, whether global attention or KV-saving mechanisms like GQA and MHA, cannot reduce prefill time, i.e., the time it takes for the model to return the first token response to the user's input. To reduce this time, this invention will explore hybridizing standard global attention with other models.

[0084] Currently, most language models are based on the Transformer architecture, whose most prominent feature is the attention mechanism. This mechanism's resource consumption is proportional to the square of the input length; as the model's input length increases, the required space and time grow exponentially. While methods such as recomputation and FlashAttention can reduce the space consumption of attention computation, these conventional optimization techniques become increasingly ineffective once the window length exceeds a certain threshold.

[0085] A major current trend in optimizing long windows is sparse attention, which approximates fully connected attention by using sliding windows or fixed-size attention. This makes the overall computational cost linearly related to the input length, and significantly reduces memory usage beyond a certain length. Traditional methods like LongFormer and BigBird, and more recent approaches like StreamingLLM and LM-Infinite, all utilize this approach. However, since most sparse attention methods are merely approximations of standard attention, some performance degradation is inevitable. In scenarios requiring refined long text understanding, this approach often loses crucial local information. Therefore, while sparse attention improves processing efficiency, it can reduce the processing capability for long texts in many scenarios, such as... Figure 4 As shown, with global attention, each token can pay attention to every previous token, while with local attention, it can only pay attention to a few recent tokens.

[0086] Secondly, another major direction in the current industry for handling long windows is to increase the processing length of the model only during the model deployment and inference phase, without changing the context window length during the training phase (Train-Short, Test-Long). The core idea of ​​this approach is that most current language models are based on relative positional encoding, and during inference, interpolation or extrapolation is performed using relative positional encoding to allow the model to handle lengths not encountered during training. Systems using this approach have achieved some success in handling long windows. However, this method approximates longer windows at the positional encoding layer, adapting to longer windows by modifying the positional encoding of shorter windows. This often results in situations where longer windows improve, but shorter windows within the previously trained length deteriorate. Furthermore, this method can cause inconsistencies between training and inference, leading to significant changes in the dynamic performance of Attention. In practice, while metrics such as PPL may improve, the actual ability to understand the context of long texts remains unchanged. Figure 5 As shown, extrapolation and interpolation based on rotational position encoding are used. This invention employs standard Attention and sparse Attention to ensure the model's performance in understanding long texts, while modifying the position encoding to adapt to contexts of varying lengths, extending the window length to over 1000K. The accuracy and perplexity of using hybrid attention and individual attention on different windows are compared, such as... Figure 6 , Figure 7 As shown.

[0087] The strategy of combining sliding window attention and multi-head attention aims to address the efficiency and effectiveness issues in processing long sequences while maintaining the model's ability to capture complex contextual dependencies. In practice, these two attention mechanisms can be applied to different layers of the model or different parts of the same layer.

[0088] This hybrid strategy not only improves the model's efficiency in processing long sequences but also ensures that the model possesses rich representational capabilities, enabling it to handle a wide range of task requirements, achieving an ideal balance from basic syntactic analysis to advanced semantic understanding. Meanwhile, the superior extrapolation capability exhibited by sliding window attention (if the training length exceeds the empirical receptive field) opens a door to future infinitely long window models.

[0089] In addition, the sensor head mechanism was explored in depth, which posits that LLM can utilize a two-layer attention mechanism for context learning. Through in-depth exploration of its underlying mechanisms, it was discovered that the replication function required for the first layer of attention can be accomplished solely through local attention. This finding reveals that a hybrid model employing sliding window attention and global attention may not lead to performance degradation compared to a model using only global attention. Furthermore, to better utilize the hybrid model, this invention optimizes the Softmax function in the attention mechanism. Traditional attention mechanisms force the sum of attention for each token to be 1; however, when applying window attention, situations inevitably arise where no token needs attention. Therefore, this invention optimizes Softmax by adding 1 to the denominator to ensure that the total attention can be less than 1. Preliminary experimental explorations show that using the hybrid model not only saves on KV cache and reduces prefill time but also maintains long window capabilities and even surpasses the global attention model in terms of perplexity extrapolation.

[0090] In one specific embodiment, in-depth analysis and research of the Transformer attention mechanism and improvements to positional encoding techniques will optimize the model's extrapolation capability from two aspects: optimizing the attention distribution and positional encoding extrapolation (interpolation). This invention found that as sequences lengthen, on the one hand, the model exhibits an explosion in the Hidden_States of the corresponding tokens in the network due to Out-of-Depth (OOD); on the other hand, the variance of the Attention Score continuously decreases with sequence length. By proposing a dynamic positional encoding extrapolation scheme for RoPE and ALiBi, different degrees of Attention-Mask dynamic interpolation are performed on ALiBi_Masks at different resolutions, enhancing the model's ability to model long sequence dependencies while maintaining resolution.

[0091] Positional embeddings are a core component of current large-scale language models. Rotated Positional Embeddings (RoPEs), a technique that encodes positional information using rotation matrices, has become the de facto choice for positional embeddings in many LLMs (such as LLaMA sequences). RoPEs have been further used to extend long context capabilities, largely based on adjusting the only hyperparameter of RoPE, `base`, to mitigate the out-of-distribution (OOD) problem in positional embeddings. However, this invention finds that LLMs based on object-oriented theory may achieve only superficial long context capabilities. The role of RoPEs in LLMs is re-examined, and a new long-term decay property is proposed, deriving a lower bound of the RoPE base with respect to the context length. This work reveals the relationship between context length and the key RoPE parameter `base` from both theoretical and empirical perspectives, providing insights for future long-window model training.

[0092] Contribution 1: The long-range attenuation properties of RoPE were re-examined, such as Figure 8 As shown, the two graphs on the left depict how the attention score changes with increasing distance. The two graphs on the right depict how the model's ability to focus on similar tokens changes with increasing distance.

[0093] Contribution 2: like Figure 9 As shown in the two figures on the right, this capability should be greater than 0, which leads to the conclusion that the key parameter base of RoPE has an absolute lower bound given a long context length.

[0094] Contribution Three: This reveals that when the base is below the aforementioned lower bound, it exhibits a "long window capability on the surface," such as... Figure 10 As shown, it can be observed that when the base is small, the model exhibits the ability to speak normally (low perplexity), but has completely lost the ability to retrieve information from a distance.

[0095] By revealing the limitations of current RoPE-based location coding, Baichuan Intelligence will conduct research on more reasonable location coding in the future. Exploring more reasonable location coding generally requires meeting three conditions: first, long-range attenuation, meaning that attention should be paid to nearby information more than to distant information; second, non-negativity, meaning that attention should be paid to similar tokens more than to random tokens; and third, smoothness, meaning that the perception of location information should be smoother, reducing oscillations.

[0096] The present invention also discloses a computer program product or system, including a computer program that, when executed by a processor, implements the above-described steps of the method for processing long-sequence medical data of children based on attention distribution optimization.

[0097] Figure 2 A schematic diagram of a system for processing long-sequence medical data of children based on attention distribution optimization provided in this embodiment of the invention specifically includes: Acquisition Unit: Acquire long sequence text data of children; Processing Unit: The long sequence text of the child is input into a neural network. The neural network includes N stacked attention layers, where N is a natural number greater than 1. The stacked attention layers are cross-stacked standard attention and sparse attention. The long sequence text of the child is processed by extracting contextual features of the long sequence text through cross-stacked standard attention and sparse attention to obtain processed long sequence feature data. Among them, when the sparse attention is used to calculate the attention distribution, the denominator of the probability distribution transformation function is increased by one to obtain that the total attention of the current layer of sparse attention is less than one.

[0098] Figure 3 An embodiment of the present invention provides a schematic diagram of a computer device, specifically including: A memory and a processor; the memory is used to store program instructions; the processor is used to invoke the program instructions when any of the above-described processing of long-sequence pediatric medical data based on attention distribution optimization is executed.

[0099] The present invention also discloses a computer-readable storage medium storing a computer program that, when executed by a processor, performs any of the above-described processing of long-sequence medical data of children based on attention distribution optimization.

[0100] The verification results of this verification embodiment show that assigning inherent weights to indications can improve the performance of this method compared to the default settings. Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separated; the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of this embodiment. Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. This program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0101] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0102] The computer device provided by the present invention has been described in detail above. For those skilled in the art, there will be changes in the specific implementation and application scope based on the ideas of the embodiments of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for processing long-sequence medical data of children based on attention distribution optimization, characterized in that, include: Obtain long-sequence text data of children; The long sequence text of the child is input into a neural network, which includes N stacked attention layers, where N is a natural number greater than 1. The stacked attention layers are cross-stacked standard attention and sparse attention. The long sequence text of the child is processed by extracting contextual features of the long sequence text through cross-stacked standard attention and sparse attention to obtain the processed long sequence feature data. In the sparse attention, when calculating the attention distribution, the denominator of the probability distribution transformation function is increased by one to obtain that the total attention of the current layer of sparse attention is less than one.

2. The method for processing long-sequence medical data of children based on attention distribution optimization according to claim 1, characterized in that, The neural network also includes a data embedding layer. After the long sequence text of the child is passed through the data embedding layer, a token vector of the long sequence is obtained. The token vector of the long sequence is input to N stacked attention layers to extract the contextual features of the long sequence text to obtain the processed long sequence feature data. The data embedding layer is a dynamic position vector encoding of the long sequence text of the child through the embedding of the long sequence text input length and rotation position vector to obtain the token vector of the long sequence.

3. The method for processing long-sequence medical data of children based on attention distribution optimization according to claim 2, characterized in that, The rotation position vector embedding is obtained by extrapolating or interpolating the position vector embedding to obtain a long sequence of token vectors; Optionally, the rotation position vector embedding process is as follows: the long sequence text data of the child is input to the data embedding layer and converted into a token vector; the length of the long sequence is calculated; the rotation position vector is embedded based on the length of the long sequence to generate a dynamic position code; and the token vector and the dynamic position code are fused to obtain the token vector of the long sequence.

4. The method for processing long-sequence medical data of children based on attention distribution optimization according to claim 2, characterized in that, The long sequence of token vectors is input to the attention layer. In the attention layer, the relative distance is first calculated through dynamic position encoding. Based on the relative distance, the query vector, and the key vector, the attention score is calculated to obtain the attention score matrix. The output of the current attention layer is calculated through the attention score matrix. Optionally, the attention layer further includes generating a pediatric semantic relevance bias matrix, obtaining any L positions in the attention score matrix and the corresponding word vectors, where L is a natural number greater than 1, querying the entities of the word vectors in the knowledge graph, calculating the relevance between the entities to obtain a relevance bias value matrix, fusing the relevance bias value matrix with the attention score matrix to obtain an attention weight matrix, and obtaining the output of the current attention layer based on the attention weight matrix; Optionally, the word vector is a word vector or character vector in a long sequence of text for children; Optionally, the query further includes entity determination, querying the mapping data of the word vector in the knowledge graph, determining whether the mapping data is an entity, calculating the relevance bias value between entities when the determination result is an entity, and assigning the relevance bias value to zero when the determination result is not an entity, thereby obtaining a relevance bias value matrix.

5. The method for processing long-sequence medical data of children based on attention distribution optimization according to any one of claims 1-4, characterized in that, When the attention layer is standard attention, the standard attention is multi-head attention. The multi-head attention is grouped, and each group of attention heads is guided to perform a preset medical type hierarchical perception to obtain a hierarchical perception attention weight matrix. The output of the current attention layer is obtained through the hierarchical perception attention weight matrix. Optionally, the fixed guidance obtains the entity corresponding to the position through the attention score matrix, determines the preset type of the entity, and obtains the attention weight matrix for layered perception by performing layered perception calculation based on the preset type. Optionally, the hierarchical perception calculation is based on a preset type to calculate the corresponding type relevance bias value, otherwise the bias is assigned to zero to obtain a second relevance bias value matrix. The second relevance bias value matrix is ​​fused with the attention score matrix to obtain the hierarchical perception attention weight matrix, and the output of the current attention layer is obtained based on the hierarchical perception attention weight matrix. Optionally, the preset type includes any one or more of the following: symptom-diagnosis, description-signs, examination-symptoms, diagnosis-treatment method.

6. The method for processing long-sequence medical data of children based on attention distribution optimization according to claim 5, characterized in that, The output of the current attention layer is obtained by first calculating the second relevance bias matrix through hierarchical perception, then querying the entity at the corresponding position based on the attention score matrix, and calculating the relevance between the entities to obtain the third relevance bias value matrix. The third relevance bias value matrix is ​​then fused with the attention score matrix to obtain the third attention weight matrix, and the output of the current attention layer is obtained based on the third attention weight matrix. Optionally, the output of the current attention layer is obtained by first querying the entity at the corresponding position through the attention score matrix, calculating the correlation between the entities to obtain the correlation bias matrix, then calculating the fourth correlation bias matrix through hierarchical perception, fusing the fourth correlation bias matrix with the attention score matrix to obtain the fourth attention weight matrix, and obtaining the output of the current attention layer based on the fourth attention weight matrix. Optionally, the output of the current attention layer is obtained by performing an element-wise AND operation between the relevance bias matrix and the second relevance bias matrix to obtain the fifth relevance bias matrix, fusing the fifth relevance bias matrix with the attention score matrix to obtain the fifth attention weight matrix, and obtaining the output of the current attention layer based on the fifth attention weight matrix.

7. The method for processing long-sequence medical data of children based on attention distribution optimization according to claim 1, characterized in that, The standard attention includes one or more of the following: multi-head attention, self-attention, and cross attention; Optionally, the sparse attention includes one or more of the following: sliding window attention, dilation attention, adaptive sparse attention, block sparse attention, and grouped query attention.

8. A computer program product comprising a computer program or instructions, characterized in that, The computer program or instructions are executed by a processor to implement the method for processing long-sequence medical data of children based on attention distribution optimization as described in any one of claims 1-7.

9. A computer device comprising a memory, a processor, and a computer program or instructions stored in the memory, characterized in that, The computer program or instructions are executed by a processor to implement the method for processing long-sequence medical data of children based on attention distribution optimization as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, The computer program or instructions are executed by a processor to implement the method for processing long-sequence medical data of children based on attention distribution optimization as described in any one of claims 1-7.

Images