Human-machine cooperation Transform model method and framework for online inquiry
Through the human-computer collaboration Transformer model for online consultation, combined with the Transformer coding layer and the doctor enhancement layer, the problem of insufficient human-computer collaboration in the existing technology is solved, and high-accuracy disease diagnosis and ineffective symptoms are achieved.
Patent Information
- Application Number
- CN202410987484.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-12
AI Technical Summary
The existing human-machine collaborative disease diagnosis methods only require cooperation between humans and artificial intelligence on individual issues, and fail to effectively utilize the additional knowledge of human feedback, resulting in excessive ineffective symptoms generated by machines and lack of global human-machine collaboration processes.
Using the human-computer collaborative Transformer model method for online consultation, the target user's symptoms and attribute information is obtained through the inquiry layer, and the Transformer encoding layer and the doctor-enhanced Transformer layer are used for processing, output disease information and confidence, and combine with multiple doctors to enhance their self-enhanced Attention level and decision-making level to improve diagnosis results and reduce ineffective symptoms.
It improves the accuracy and efficiency of diagnosis, reduces the generation of ineffective symptoms, and uses doctors’ prior knowledge to enhance the symptom inquiry ability of artificial intelligence, achieving high-accuracy diagnosis.
Smart Images

Figure CN118942675A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of human-computer collaboration, and in particular to a human-computer collaboration Transformer model method and framework for online medical consultation. Background Art
[0002] At present, the consultation modes of online disease diagnosis are mainly divided into two categories: manual diagnosis and machine-based diagnosis. The manual diagnosis mode is that real doctors provide high-quality medical services to patients, but the consultation fees are relatively high. In contrast, the machine-based diagnosis mode utilizes intelligent systems based on artificial intelligence models and has made significant progress in recent years. These machine-based diagnosis systems have performed well in improving the efficiency of medical services and reducing costs, but the accuracy is still not reliable enough.
[0003] At present, the human-computer computing (HMC) conceptual framework is proposed for online medical consultation. The HMC framework has shown the ability to solve complex tasks in the medical field. For example, the human-computer collaboration protocol improves the performance of knee lesion detection, surpassing the capabilities of a single subject; the collaboration between humans and large language models shows promise in improving the differential diagnosis of brain magnetic resonance imaging; the human-computer collaborative navigation system achieves higher precision and recall when navigating tumor images. Studies have shown that human-computer collaboration has broad application prospects in high-risk medical decision-making; for example, Fogliato et al. studied the design details of human-computer collaboration in radiological diagnosis and found that when the AI inference results were displayed immediately, the radiologist's diagnosis was more consistent with the AI suggestion; Calisto et al. constructed a clinician-AI workflow for breast cancer image classification. The results showed that the clinician's diagnostic accuracy was significantly improved, and the clinical field is accepting human-computer systems for breast cancer diagnosis.
[0004] Online disease diagnosis has also begun to adopt human-machine collaboration; for example, Chen et al. proposed a human-machine collaborative diagnosis system to ensure the accuracy of diagnosis. In the inquiry stage, AI uses a deep Q-network-based method to inquire about symptoms; in the diagnosis stage, doctors check AI's preliminary diagnosis and provide a final diagnosis; Zhao et al. proposed a human-machine diagnosis team framework, in which doctors and AI jointly conduct symptom inquiries through a reinforcement learning-based allocation strategy; this framework minimizes the workload of doctors while ensuring high diagnostic accuracy; however, most frameworks only involve cooperation between humans and AI on individual issues, such as classification problems, where AI provides advice to humans during the decision-making process; in this case, AI fails to fully utilize the additional knowledge of human feedback to enhance itself, and in addition, there is a lack of human-machine collaboration rules throughout the entire complex workflow. Summary of the invention
[0005] The embodiment of the present invention provides a human-computer collaborative Transformer model method and framework for online medical consultation, so as to at least solve the technical problems that the existing human-computer collaborative disease diagnosis methods only require cooperation between humans and artificial intelligence on individual issues, and the human-computer collaborative framework does not effectively utilize the additional knowledge in human feedback, and lacks the establishment of a human-computer collaborative process, resulting in a large number of invalid symptoms generated by the machine.
[0006] According to one aspect of an embodiment of the present invention, a human-computer collaborative Transformer model method for online medical consultation is provided. The method may include: when the query layer of the human-computer collaborative disease diagnosis framework based on Transformer performs the Kth round of query, the displayed symptoms, displayed symptom attributes, additional symptoms and additional symptom attributes of the target user are obtained, wherein K is greater than or equal to the target number of times; the displayed symptoms, displayed symptom attributes, additional symptoms and additional symptom attributes are added to obtain the target information; the target information is processed through the Transformer encoding layer, and the disease information corresponding to the target information and the predicted confidence of the disease information are output; when the predicted confidence of the disease information is greater than the preset threshold of the disease, the disease information corresponding to the target information is determined as the diagnosis result of the target user, and when the predicted confidence of the disease information is less than or equal to the preset threshold of the disease, the target information is processed through the input embedding layer in the doctor-enhanced Transformer encoding layer to obtain the initial context representation; the initial context representation is processed through the doctor-enhanced Transformer layer in the doctor-enhanced Transformer encoding layer. The doctor-enhanced Transformer layer includes a multi-head doctor-enhanced self-attention layer, a first residual regularization layer, a feedforward layer, and a second residual regularization layer, and the multi-head doctor-enhanced self-attention layer includes processing the initial context representation and the doctor matrix; the final context representation is linearly transformed by the decision layer in the doctor-enhanced Transformer encoding layer, and the first new symptom and the confidence value of the first new symptom are output; when the confidence value of the first new symptom is less than the preset threshold of the new symptom and the symptoms output by the previous two rounds of the doctor-enhanced Transformer encoding layer are invalid, the second new symptom corresponding to the target information is obtained, the second new symptom interacts with the target user, and the attribute information corresponding to the second new symptom is obtained; when the confidence value of the first new symptom is greater than or equal to the preset threshold of the new symptom, or the symptoms output by any round of the previous two rounds of the doctor-enhanced Transformer encoding layer are valid, the first new symptom interacts with the target user to obtain the attribute information corresponding to the first new symptom.
[0007] Optionally, the display symptom, the display symptom attribute, the additional symptom and the additional symptom attribute are added together to obtain the expression of the target information: X0 = Embedding (Sex ,S add )+Embedding(A ex ,A add )(1), where X0 is the target information, S ddd For additional symptoms, S ex To show symptoms, A ex To display the symptom attributes, A add It is an additional symptom attribute.
[0008] Optionally, the target information is processed through a Transformer encoding layer to output disease information corresponding to the target information and a prediction confidence of the disease information, including: after the target sequence of the Nth layer is output by the decision layer in the Transformer encoding layer, the target sequence is aggregated into a single representation through an average pooling layer, wherein the expression of the single representation is: (2), where l represents the length of the target sequence, x i represents the i-th symptom vector in the target sequence, AvgPooling(x) represents the average pooling operation on x; the single representation is mapped to the output space required for the disease classification task through the fully connected layer, and the disease information corresponding to the target information and the predicted confidence of the disease information are output.
[0009] Optionally, the initial context representation is processed by the doctor-enhanced Transformer layer in the doctor-enhanced Transformer encoding layer to obtain a final context representation, including: processing the initial context representation by the multi-head doctor-enhanced self-attention layer in the doctor-enhanced Transformer layer to obtain first output information; processing the first output information through a feedforward layer to obtain a final context representation, wherein each multi-head doctor-enhanced self-attention layer is connected to a first residual regularization layer, and each feedforward layer is connected to a second residual regularization layer.
[0010] Optionally, the context information is processed by a multi-head doctor enhanced self-attention layer in the doctor enhanced Transformer layer to obtain first output information, including: the multi-head doctor enhanced self-attention layer applies three linear transformations W i q ,W i k , and W i v Transform the context information to obtain the query Q for each head i , key K i , and value V i ; Based on the query Q of each head i and key K i ,, get the global attention score; based on the query Q of each headi , key K i and the doctor matrix, the enhanced doctor's attention score is obtained; based on the global attention score, the enhanced doctor's attention score, and the value V i And hyper parameters, get the value of each head of the multi-head doctor enhanced self-attention layer; connect the value of each head to get the first output information.
[0011] Optionally, based on the query Q for each head i and key K i , and the expression of the global attention score is: in, is the global attention score, d h is the dimension of each head, T is K i The transpose of .
[0012] Optionally, based on the query Q for each head i , key K i And the doctor matrix, the expression of the enhanced doctor's attention score is: in, To enhance the doctor’s attention score, M doc For the doctor matrix.
[0013] Optionally, based on the global attention score, the enhanced doctor's attention score, the value V i And hyperparameters, the expression of the value of each head of the multi-head doctor enhanced self-attention layer is obtained as: Among them, head i represents the value of each head, and γ∈[0,1] represents a score used to adjust the enhanced doctor's attention score. Hyperparameters of weights.
[0014] Optionally, the calculation process of the doctor matrix is: obtain the target data set, and build a symptom graph based on the target data set; query the node features of the doctor's symptoms from the symptom graph, scale the node features with the initial context representation, and obtain the dot product diagonal vector; write the dot product diagonal vector into the diagonal matrix to obtain a diagonal matrix; build a sparse matrix; add the diagonal matrix and the sparse matrix to obtain the doctor matrix.
[0015] According to one aspect of an embodiment of the present invention, a human-machine collaborative Transformer model framework for online medical consultation is provided. The framework may include: an inquiry layer and a diagnosis layer, wherein the inquiry layer includes an activation module and a machine module, and the diagnosis layer includes a diagnosis module and a diagnosis result. The inquiry layer and the diagnosis layer interact with each other, the activation module is connected to the machine module, and the diagnosis module is connected to the diagnosis result;
[0016] The diagnosis layer is used to obtain the displayed symptoms, displayed symptom attributes, additional symptoms and additional symptom attributes of the target user when the inquiry layer makes the Kth round of inquiry, add the displayed symptoms, displayed symptom attributes, additional symptoms and additional symptom attributes to obtain the target information, process the target information, and output the disease information corresponding to the target information and the predicted confidence of the disease information; when the predicted confidence of the disease information is greater than the preset threshold of the disease, the disease information corresponding to the target information is determined as the diagnosis result of the target user, and when the predicted confidence of the disease information is less than or equal to the preset threshold of the disease, the inquiry layer starts working, wherein K is greater than or equal to the target number of times;
[0017] The query layer is used to process the target information through the input embedding layer in the doctor-enhanced Transformer encoding layer in the machine module to obtain an initial context representation, process the initial context representation through the doctor-enhanced Transformer layer in the doctor-enhanced Transformer encoding layer to obtain a final context representation, perform a linear transformation on the final context representation through the decision layer in the doctor-enhanced Transformer encoding layer, output the first new symptom and the confidence value of the first new symptom, and the activation module starts working. When the confidence value of the first new symptom is less than the preset threshold of the new symptom and the previous two rounds of doctor-enhanced Transformer When all symptoms output by the encoding layer are invalid, a second new symptom corresponding to the target information is obtained, and the second new symptom interacts with the target user to obtain attribute information corresponding to the second new symptom. When the confidence value of the first new symptom is greater than or equal to the preset threshold of the new symptom, or the symptoms output by any of the previous two rounds of doctor-enhanced Transformer encoding layers are valid, the first new symptom interacts with the target user to obtain attribute information corresponding to the first new symptom. The doctor-enhanced Transformer layer includes a multi-head doctor-enhanced self-attention layer, a first residual regularization layer, a feedforward layer, and a second residual regularization layer. The multi-head doctor-enhanced self-attention layer includes processing the initial context representation and the doctor matrix.
[0018] Beneficial effects of the present invention:
[0019] The present invention proposes a human-machine collaborative Transformer model method for online consultation. When the query layer of the human-machine collaborative disease diagnosis framework based on Transformer performs the Kth round of query, the displayed symptoms, displayed symptom attributes, additional symptoms and additional symptom attributes of the target user are obtained; the displayed symptoms, displayed symptom attributes, additional symptoms and additional symptom attributes are added to obtain the target information; the target information is processed through the Transformer encoding layer, and the disease information corresponding to the target information and the prediction confidence of the disease information are output; when the prediction confidence of the disease information is greater than the preset threshold of the disease, the disease information corresponding to the target information is determined as the diagnosis result of the target user; when the prediction confidence of the disease information is less than or equal to the preset threshold of the disease, the prediction confidence of the disease information is enhanced by the doctor. The input embedding layer in the nsformer encoding layer processes the target information to obtain an initial context representation; the initial context representation is processed by the doctor-enhanced Transformer layer in the doctor-enhanced Transformer encoding layer to obtain a final context representation, wherein K is greater than or equal to the target number of times, the doctor-enhanced Transformer layer includes a multi-head doctor-enhanced self-attention layer, a first residual regularization layer, a feedforward layer, and a second residual regularization layer, and the multi-head doctor-enhanced self-attention layer includes processing the initial context representation and the doctor matrix; the final context representation is linearly transformed by the decision layer in the doctor-enhanced Transformer encoding layer to output the first new symptom and the confidence value of the first new symptom; when the confidence value of the first new symptom is less than the preset threshold of the new symptom and the symptoms output by the previous two rounds of doctor-enhanced Transformer encoding layers are invalid, the second new symptom corresponding to the target information is obtained. The second new symptom interacts with the target user to obtain the attribute information corresponding to the second new symptom. When the confidence value of the first new symptom is greater than or equal to the preset threshold of the new symptom, or the symptoms output by any round of the Transformer encoding layer enhanced by the previous two rounds of doctors are valid, the first new symptom interacts with the target user to obtain the attribute information corresponding to the first new symptom. This solves the technical problems that the existing human-computer collaborative disease diagnosis methods only require humans and artificial intelligence to cooperate on individual issues, and the human-computer collaborative framework does not effectively utilize the additional knowledge in human feedback, and lacks the establishment of a human-computer collaborative process, which will generate a large number of invalid symptoms generated by the machine. It provides a doctor-enhanced attention mechanism, uses the doctor's prior knowledge in consultation to enhance the ability of artificial intelligence in symptom inquiry, and allows doctors to handle complex cases that machines cannot solve, thereby reducing the generation of invalid symptoms and achieving a high-accuracy diagnosis within a limited number of inquiry rounds. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:
[0021] Figure 1 is a flow chart of a human-machine collaborative Transformer model method for online medical consultation according to an embodiment of the present invention;
[0022] Figure 2 is a schematic diagram of a machine module of a doctor-enhanced Transformer according to an embodiment of the present invention;
[0023] Figure 3 is a schematic diagram of a process of forming a doctor matrix according to an embodiment of the present invention;
[0024] Figure 4 is a schematic diagram of a human-machine collaborative Transformer model framework for online medical consultation according to an embodiment of the present invention;
[0025] Figure 5 is a schematic diagram of a human-computer collaborative disease diagnosis framework algorithm flow according to an embodiment of the present invention;
[0026] Figure 6 is a schematic diagram for explaining a software and hardware experimental environment according to an embodiment of the present invention;
[0027] Figure 7 is a schematic diagram of statistics of an experimental data set according to an embodiment of the present invention;
[0028] Figure 8 is a schematic diagram of an experimental evaluation comparison according to an embodiment of the present invention;
[0029] Fig. 9 is a schematic diagram of a case study of a human-machine collaborative disease diagnosis method according to an embodiment of the present invention. DETAILED DESCRIPTION
[0030] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only embodiments of a part of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0032] Example 1
[0033] According to an embodiment of the present invention, a human-computer collaborative Transformer model method for online medical consultation is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system containing at least one set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0034] Figure 1 is a flowchart of a human-machine collaborative Transformer model method for online medical consultation according to an embodiment of the present invention. Figure 1 As shown, the method may include the following steps:
[0035] Step S101, when the query layer of the Transformer-based human-machine collaborative disease diagnosis framework performs the Kth round of query, the displayed symptoms, displayed symptom attributes, additional symptoms and additional symptom attributes of the target user are obtained, where K is greater than or equal to the target number of times.
[0036] In the technical solution provided in the above step S101 of the present invention, when the query layer of the human-machine collaborative disease diagnosis framework based on Transformer performs the Kth round of query, the human doctor and the artificial intelligence model ask the target user for additional symptoms S according to the self-reported symptoms (explicit symptoms) of the target user before the Kth round. add , additional symptoms are expressed as in represents the M symptoms asked by human doctors, Represents the N symptoms asked by the artificial intelligence model. The displayed symptom attribute can be the target user's answer to the human doctor's question about the displayed symptoms, yes, no, or unknown. The additional symptom attribute can be the target user's answer to the artificial intelligence model's question about additional symptoms, yes, no, or unknown, where the target number is 2.
[0037] Step S102: sum the displayed symptoms, displayed symptom attributes, additional symptoms and additional symptom attributes to obtain target information.
[0038] In the technical solution provided in the above step S102 of the present invention, the displayed symptoms, displayed symptom attributes, additional symptoms and additional symptom attributes are added together to obtain the target information.
[0039] Step S103, processing the target information through the Transformer coding layer, and outputting the disease information corresponding to the target information and the prediction confidence of the disease information.
[0040] In the technical solution provided in the above step S103 of the present invention, the target information is processed by the diagnostic layer Transformer coding layer to output the disease information corresponding to the target information and the prediction confidence of the disease information.
[0041] Step S104, when the prediction confidence of the disease information is greater than the preset threshold of the disease, the disease information corresponding to the target information is determined as the diagnosis result of the target user. When the prediction confidence of the disease information is less than or equal to the preset threshold of the disease, the target information is processed by the input embedding layer in the doctor-enhanced Transformer encoding layer to obtain the initial context representation.
[0042] In the technical solution provided in the above step S104 of the present invention, if the prediction confidence of the disease information is greater than the preset threshold of the disease, the disease information corresponding to the target information is determined as the diagnosis result of the target user, and there is no need to inquire through the inquiry layer. When the prediction confidence of the disease information is less than or equal to the preset threshold of the disease, the inquiry layer works, and the target information is processed by the input embedding layer in the doctor-enhanced Transformer encoding layer to obtain the initial context representation.
[0043] Step S105, the initial context representation is processed by the doctor-enhanced Transformer layer in the doctor-enhanced Transformer encoding layer to obtain the final context representation, wherein the doctor-enhanced Transformer layer includes a multi-head doctor-enhanced self-attention layer, a first residual regularization layer, a feedforward layer, and a second residual regularization layer, and the multi-head doctor-enhanced self-attention layer includes processing the initial context representation and the doctor matrix.
[0044] In the technical solution provided in the above step S105 of the present invention, the initial context representation is processed by the doctor-enhanced Transformer layer in the doctor-enhanced Transformer encoding layer to obtain the final context representation.
[0045] Step S106, linearly transform the final context representation through the decision layer in the doctor-enhanced Transformer encoding layer, and output the first new symptom and the confidence value of the first new symptom.
[0046] In the technical solution provided in the above step S106 of the present invention, the decision layer performs a linear change on the final context representation, and outputs the first new symptom corresponding to the target information and the confidence value of the first new symptom.
[0047] Step S107, when the confidence value of the first new symptom is less than the preset threshold of the new symptom and the symptoms output by the Transformer coding layer of the previous two rounds of doctor enhancement are invalid, the second new symptom corresponding to the target information is obtained, the second new symptom interacts with the target user, and the attribute information corresponding to the second new symptom is obtained. When the confidence value of the first new symptom is greater than or equal to the preset threshold of the new symptom, or the symptoms output by any one of the previous two rounds of doctor enhancement Transformer coding layers are valid, the first new symptom interacts with the target user to obtain the attribute information corresponding to the first new symptom.
[0048] In the technical solution provided in the above step S107 of the present invention, when the confidence value of the first new symptom is less than the preset threshold of the new symptom and the symptoms output by the two rounds of doctor-enhanced Transformer coding layers k-1 and k-2 are invalid, the doctor participates in the consultation. During the doctor's consultation, the second new symptom corresponding to the target information is obtained, and the second new symptom interacts with the target user to obtain attribute information corresponding to the second new symptom. When the confidence value of the first new symptom is greater than or equal to the preset threshold of the new symptom, or the symptoms output by any round of the two rounds of doctor-enhanced Transformer coding layers k-1 and k-2 are valid, the doctor does not participate in the consultation, the inquiry layer continues to work, the first new symptom interacts with the target user, and the attribute information corresponding to the first new symptom is obtained. The attribute information is valid or invalid. Valid means that the target user answers yes or no, and invalid means that the target user answers unknown.
[0049] The above method of this embodiment is further introduced below.
[0050] As an optional embodiment, in step S102, the displayed symptoms, displayed symptom attributes, additional symptoms, and additional symptom attributes are added together to obtain the expression of the target information: X0 = Embedding (S ex ,S add )+Embedding(A ex ,A add )(1), where X0 is the target information, S add For additional symptoms, S ex To show symptoms, A ex To display the symptom attributes, Aadd It is an additional symptom attribute.
[0051] In this embodiment, the symptoms and attributes of the additional symptoms and explicit symptoms are added together to obtain target information.
[0052] As an optional implementation method, step S103, processing the target information through the Transformer coding layer, outputting the disease information corresponding to the target information and the prediction confidence of the disease information, including: after the target sequence of the Nth layer output by the decision layer in the Transformer coding layer, aggregating the target sequence into a single representation through an average pooling layer, wherein the expression of the single representation is: Where l represents the length of the target sequence, x i represents the i-th symptom vector in the target sequence, AvgPooling(x) represents the average pooling operation on x; the single representation is mapped to the output space required for the disease classification task through the fully connected layer, and the disease information corresponding to the target information and the predicted confidence of the disease information are output.
[0053] In this embodiment, the target information is processed by two layers of encoders in the Transformer encoding layer in the diagnosis layer to obtain the target sequence of the Nth layer, and the target sequence is aggregated into a single representation through the average pooling layer. The expression is: Where l represents the length of the target sequence, x i represents the i-th symptom vector in the target sequence, AvgPooling(x) represents the average pooling operation on x; the single representation is mapped to the output space required for the disease classification task through the fully connected layer, and the disease information corresponding to the target information and the predicted confidence of the disease information are output.
[0054] As an optional implementation method, step S105, the initial context representation is processed by the doctor-enhanced Transformer layer in the doctor-enhanced Transformer encoding layer to obtain the final context representation, including: processing the initial context representation by the multi-head doctor-enhanced self-attention layer in the doctor-enhanced Transformer layer to obtain first output information; processing the first output information through a feedforward layer to obtain the final context representation, wherein each multi-head doctor-enhanced self-attention layer is connected to a first residual regularization layer, and each feedforward layer is connected to a second residual regularization layer.
[0055] In this embodiment, according to an embodiment of the present invention, a machine module of a doctor-enhanced Transformer is provided, Figure 2 is a schematic diagram of a machine module of a doctor-enhanced Transformer according to an embodiment of the present invention, such as Figure 2As shown, the multi-head doctor enhanced self-attention layer processes the context information and obtains the first output information, such as Figure 2 The yellow multi-head doctor enhanced self-attention layer processes the first output information through the feedforward layer to obtain the final context representation, where each multi-head doctor enhanced self-attention layer is connected to a first residual regularization layer, each feedforward layer is connected to a second residual regularization layer, and the doctor-enhanced Transformer layer includes N layers.
[0056] As an optional implementation method, the context information is processed by the multi-head doctor enhanced self-attention layer in the doctor enhanced Transformer layer to obtain the first output information, including: the multi-head doctor enhanced self-attention layer applies three linear transformations W i q ,W i k , and W i v Transform the context information to obtain the query Q for each head i , key K i , and value V i ; Based on the query Q of each head i and key K i ,, get the global attention score; based on the query Q of each head i , key K i and the doctor matrix, the enhanced doctor's attention score is obtained; based on the global attention score, the enhanced doctor's attention score, and the value V i And hyper parameters, get the value of each head of the multi-head doctor enhanced self-attention layer; connect the value of each head to get the first output information.
[0057] In this embodiment, the multi-head doctor enhanced self-attention layer applies three linear transformations W i q ,W i k , and W i v Transform the context information to obtain the query Q for each head i , key K i , and value V i ; Based on the query Q of each head i and key K i , calculate and get the global attention score; according to the query Q of each head i , key K i Calculate with the doctor matrix to get the enhanced doctor's attention score; according to the global attention score, the enhanced doctor's attention score, and the value V iand hyperparameters to obtain the value of each head of the multi-head doctor enhanced self-attention layer; connect the values of each head to obtain the first output information;
[0058] As an optional implementation mode, according to the query Q of each head i and key K i , calculate, and get the expression of global attention score: in, is the global attention score, d h is the dimension of each head, T is K i The transpose of .
[0059] In this embodiment, the key K i Transpose, Q i and After dot product multiplication, it is divided by the square root of the dimension of each head and then softmax calculation is performed to obtain the global attention score.
[0060] As an optional implementation method, based on the query Q of each head i , key K i And the doctor matrix, the expression of the enhanced doctor's attention score is: in, To enhance the doctor’s attention score, M doc For the doctor matrix.
[0061] In this embodiment, the key K i Transpose, Q i and After dot product multiplication, it is divided by the square root of the dimension of each head and added to the doctor matrix, and then softmax calculation is performed to obtain the enhanced doctor's attention score.
[0062] As an optional embodiment, based on the global attention score, the enhanced doctor's attention score, the value V i And hyperparameters, the expression of the value of each head of the multi-head doctor enhanced self-attention layer is obtained as: (5), where head i represents the value of each head, and γ∈[0,1] represents a score used to adjust the enhanced doctor's attention score. Hyperparameters of weights.
[0063] In this embodiment, the value of each head of the multi-head doctor enhanced self-attention layer is obtained through formula (5).
[0064] As an optional implementation method, the calculation process of the doctor matrix is: obtain the target data set, and build a symptom graph based on the target data set; query the node features of the doctor's symptoms from the symptom graph, scale the node features and the initial context representation to obtain the dot product diagonal vector; write the dot product diagonal vector into the diagonal matrix to obtain a diagonal matrix; build a sparse matrix; add the diagonal matrix and the sparse matrix to obtain the doctor matrix.
[0065] In this embodiment, according to an embodiment of the present invention, a process for forming a doctor matrix is provided. Figure 3 is a schematic diagram of a process of forming a doctor matrix according to an embodiment of the present invention, such as Figure 3 As shown, by learning the doctor matrix M doc Incorporating the doctor’s prior knowledge into the Transformer enables the machine module to generate symptoms that are more relevant to the doctor. The motivation for doing so is to enhance the attention score of the doctor-related symptoms so that the attention head can focus on more valuable features. Figure 3 In the example, "Symptom 3" is the symptom the doctor asked about. The doctor matrix is filled with green or orange non-zero elements from the sparse matrix and the diagonal matrix.
[0066] The formation process of the doctor matrix is as follows:
[0067] In order to learn this matrix M doc First, a unique symptom graph is constructed for each target dataset and pre-trained by neighbor masking. Then, the doctor’s symptom s is queried from the symptom graph. H The node feature g∈R d , perform a scaled dot product with the initial context representation X, and transform the resulting diagonal vector V dia ∈R l Fill in the diagonal matrix M dia ∈R l×l , as shown in formulas (6) and (7):
[0068]
[0069] Among them, W g ∈R d×d is the trainable parameter matrix, X T is the initial context representation, when i=j, for When i≠j, In addition, since the self-attention operation needs to normalize the row vector through the softmax function, in order to enhance the weight of the doctor's symptoms, a sparse matrix M is also constructed. spa ∈R 1×1 , as shown in formula (8):
[0070]
[0071] Among them, s j represents the jth symptom in the symptom input sequence. By combining and Get the doctor matrix To enhance the symptoms related to doctors, as shown in formula (9):
[0072]
[0073] For each directional symptom graph corresponding to the target dataset, each node corresponds to a symptom in the target dataset, and the edge represents the relationship between the symptoms. The graph attention network is used to aggregate the relationship between symptoms and extract the structural feature g of each node i. i , as shown in formula (10):
[0074]
[0075] Where W1 and W2 are trainable parameters. j is the expression of symptom i, N i represents the neighbors of symptom i, σ is the sigmoid function. The attention coefficient α is calculated by the attention mechanism applying the LeakyReLU activation function ij , as shown in formula (11):
[0076]
[0077] According to an embodiment of the present invention, a human-machine collaborative Transformer model framework for online medical consultation is provided. Figure 4 is a schematic diagram of a human-machine collaborative Transformer model framework for online medical consultation according to an embodiment of the present invention. Figure 4 As shown, a human-machine collaborative Transformer model framework for online medical consultation includes an inquiry layer and a diagnosis layer, wherein the inquiry layer includes an activation module and a machine module, and the diagnosis layer includes a diagnosis module and a diagnosis result. The inquiry layer and the diagnosis layer interact with each other, the activation module is connected to the machine module, and the diagnosis module is connected to the diagnosis result.
[0078] The diagnosis layer is used to obtain the displayed symptoms, displayed symptom attributes, additional symptoms and additional symptom attributes of the target user when the inquiry layer makes the Kth round of inquiry, add the displayed symptoms, displayed symptom attributes, additional symptoms and additional symptom attributes to obtain the target information, process the target information, and output the disease information corresponding to the target information and the predicted confidence of the disease information; when the predicted confidence of the disease information is greater than the preset threshold of the disease, the disease information corresponding to the target information is determined as the diagnosis result of the target user, and when the predicted confidence of the disease information is less than or equal to the preset threshold of the disease, the inquiry layer starts working.
[0079] The query layer is used to process the target information through the input embedding layer in the doctor-enhanced Transformer encoding layer in the machine module to obtain an initial context representation, process the initial context representation through the doctor-enhanced Transformer layer in the doctor-enhanced Transformer encoding layer to obtain a final context representation, perform a linear transformation on the final context representation through the decision layer in the doctor-enhanced Transformer encoding layer, output the first new symptom and the confidence value of the first new symptom, and the activation module starts working. When the confidence value of the first new symptom is less than the preset threshold of the new symptom and the previous two rounds of doctor-enhanced Transformer When all symptoms output by the encoding layer are invalid, a second new symptom corresponding to the target information is obtained, and the second new symptom interacts with the target user to obtain attribute information corresponding to the second new symptom. When the confidence value of the first new symptom is greater than or equal to the preset threshold of the new symptom, or the symptoms output by any of the previous two rounds of doctor-enhanced Transformer encoding layers are valid, the first new symptom interacts with the target user to obtain attribute information corresponding to the first new symptom. The doctor-enhanced Transformer layer includes a multi-head doctor-enhanced self-attention layer, a first residual regularization layer, a feedforward layer, and a second residual regularization layer. The multi-head doctor-enhanced self-attention layer includes processing the initial context representation and the doctor matrix.
[0080] exist Figure 4The present invention illustrates a Transformer-based human-machine collaborative disease diagnosis framework, in which the patient initiates an online consultation (left side), 3 and then interacts with the inquiry layer and the diagnosis layer (central body), and finally the system provides a diagnosis result (right side). The dotted arrow (termination) indicates that the conversation will be terminated if the known symptom information is sufficient. The activation module is implemented by cascading two rule units. The machine module is implemented by a doctor-enhanced Transformer model, in which the symptom graph is learned through a graph attention network. The doctor module is implemented by label-based truth, while the diagnosis module is implemented by a Transformer model, in which the early stopping criterion is implemented by a rule unit. Specifically, the machine module consists of a four-layer encoder, the diagnosis module consists of a two-layer encoder, and the symptom graph uses a one-layer graph network. The experimental settings for different target datasets are different. For the MZ-4 target dataset and the Dxy target dataset, λ d , max , δ s , δ d and γ are set to 10, 15, 0.92, 0.97, and 0.9, respectively. For the MZ-10 target dataset, λ d , max , δ s , δ d and γ are set to 15, 15, 0.996, 0.97, and 0.9, respectively. For the MDD target dataset, λ d , max , δ s , δ d and γ are set to 10, 10, 0.86, 0.97, and 0.7, respectively.
[0081] According to an embodiment of the present invention, a human-computer collaborative disease diagnosis framework algorithm flow is provided. Figure 5 is a schematic diagram of a human-computer collaborative disease diagnosis framework algorithm flow according to an embodiment of the present invention, such as Figure 5 As shown in Figure 1, the method includes 13 steps. The specific steps are as follows: First, a graph structure containing symptoms and their relationships is created based on each data set. The symptom graph is pre-trained using the neighbor masking technique to capture the relationships and features between symptoms. Then, the parameters θ of the machine module are pre-trained using the self-supervised learning method. m , and pre-train the parameters θ of the diagnostic module using the original sample data d Next, we enter a double loop structure: In the outer loop (repeated N1 times), the machine module takes symptom query actions based on the current input state. Then, according to the actions of the machine module, the environment generates corresponding reward feedback R. Based on the reward feedback, by minimizing the loss function To update the parameters θ of the machine module mIn the inner loop (repeated N2 times), the machine module obtains more symptom information through further inquiries and fine-tunes the parameters θ of the diagnosis module by optimizing the cross entropy loss function. d Finally, after the outer loop ends, the pre-trained diagnostic module parameters θ d Reassign the values to the current diagnosis module. The purpose of this process is to improve the accuracy of symptom inquiry and disease diagnosis by continuously training and fine-tuning the machine module and the diagnosis module.
[0082] According to an embodiment of the present invention, a schematic diagram of a software and hardware experimental environment of an embodiment of the present invention is provided. Figure 6 is a schematic diagram of a software and hardware experimental environment according to an embodiment of the present invention, such as Figure 6 As shown in the figure, the hardware environment is based on the Linux operating system of Ubuntu 18.04.6, and the NVIDIA GeForce RTX 3090 graphics card with 24.0GiB video memory is used to accelerate the calculation. The software is developed based on the GPU version of the deep learning framework Pytorch using Python 3.8.
[0083] According to an embodiment of the present invention, a schematic diagram of statistics of an experimental data set of an embodiment of the present invention is provided. Figure 7 is a schematic diagram of statistics of an experimental data set according to an embodiment of the present invention, such as Figure 7 As shown in the figure, details of the four datasets used in the experiment, including the number of symptoms, the number of diseases, and the number of samples. The MZ-4 dataset is the first manually annotated dataset for evaluating the inquiry dialogue strategy, constructed by Wei et al. It is derived from the pediatric records of the Baidu Health platform and contains four pediatric diseases: upper respiratory tract infection, pediatric bronchitis, pediatric diarrhea, and pediatric indigestion. The Dxy dataset is a medical dialogue dataset collected by Xu et al. from the Chinese online medical community. It contains five diseases: allergic rhinitis, pediatric hand, foot and mouth disease, pneumonia, pediatric diarrhea, and upper respiratory tract infection. The MZ-10 dataset is a corpus for automatic medical inquiry systems created by Chen et al. It contains multi-level annotations and is suitable for a variety of different tasks. This paper studies the diagnostic dialogue strategy task, so the present invention uses the structured symptom features in the dataset to train the agent. The MDD dataset comes from the ICL R 2021Workshop. The dataset contains 2,374 user goals (samples) and 118 symptoms, covering 12 types of diseases. All conversations in MDD come from real patients in offline hospitals in the real world, and are therefore closer to real clinical diagnosis scenarios.
[0084] According to an embodiment of the present invention, a schematic diagram of experimental evaluation comparison of an embodiment of the present invention is provided. Figure 8 is a schematic diagram of an experimental evaluation comparison according to an embodiment of the present invention, such as Figure 8 As shown, blue indicates the symptoms asked by the doctor; red indicates the real disease. The figure shows a success story from the MDD dataset, demonstrating the effectiveness of artificial intelligence under extremely poor performance under human-machine collaboration. Before the conversation began, the diagnosis module made an incorrect preliminary diagnosis based only on the dominant symptoms, with a low prediction confidence score (67.4%), indicating that more symptom inquiries were needed. Initially, the machine module asked the patient about two invalid symptoms and therefore did not contribute to the diagnosis of the disease. Subsequently, the activation module called the doctor to ask about a valid symptom (chest tightness and shortness of breath) in the third round, which directly led to the success of the preliminary diagnosis (asthma). More importantly, in the fourth round, the machine module asked about valid symptoms and increased the confidence in the correct disease (99.0%), which shows that the doctor's enhanced attention mechanism can improve the performance of artificial intelligence.
[0085] According to an embodiment of the present invention, a schematic diagram of a case study of a human-machine collaborative disease diagnosis method according to an embodiment of the present invention is provided. Fig. 9 is a schematic diagram of a case study of a human-machine collaborative disease diagnosis method according to an embodiment of the present invention, such as Fig. 9 As shown, Fig. 9 Includes: disease labels, overt symptoms and covert symptoms.
[0086] In an embodiment of the present invention, when the query layer of the human-computer collaborative disease diagnosis framework based on Transformer performs the Kth round of query, the displayed symptoms, displayed symptom attributes, additional symptoms and additional symptom attributes of the target user are obtained; the displayed symptoms, displayed symptom attributes, additional symptoms and additional symptom attributes are added to obtain the target information; the target information is processed by the Transformer encoding layer, and the disease information corresponding to the target information and the predicted confidence of the disease information are output; when the predicted confidence of the disease information is greater than the preset threshold of the disease, the disease information corresponding to the target information is determined as the diagnosis result of the target user, and when the predicted confidence of the disease information is less than or equal to the preset threshold of the disease, the target information is processed by the input embedding layer in the doctor-enhanced Transformer encoding layer to obtain the initial context representation; the initial context representation is processed by the doctor-enhanced Transformer layer in the doctor-enhanced Transformer encoding layer to obtain the final context representation, wherein the doctor-enhanced Transformer layer includes a multi-head doctor-enhanced self-attention layer, a first residual regularization layer, a feedforward layer, and a second residual regularization layer, and the multi-head doctor-enhanced self-attention layer includes processing the initial context representation and the doctor matrix; the doctor-enhanced Tr The decision layer in the ansformer encoding layer performs a linear transformation on the final context representation, and outputs the first new symptom and the confidence value of the first new symptom; when the confidence value of the first new symptom is less than the preset threshold of the new symptom and the symptoms output by the previous two rounds of the doctor-enhanced Transformer encoding layer are invalid, the second new symptom corresponding to the target information is obtained, and the second new symptom interacts with the target user to obtain the attribute information corresponding to the second new symptom. When the confidence value of the first new symptom is greater than or equal to the preset threshold of the new symptom, or the symptoms output by any round of the previous two rounds of the doctor-enhanced Transformer encoding layer are valid, the first new symptom is interacted with the target user. The user interacts with the user to obtain the attribute information corresponding to the first new symptom, which solves the technical problem that the existing human-machine collaborative disease diagnosis method only requires humans and artificial intelligence to cooperate on individual issues, and the human-machine collaborative framework does not effectively utilize the additional knowledge in human feedback, and lacks the establishment of a human-machine collaborative process, which will generate a large number of invalid symptoms generated by the machine. It provides a mechanism for doctors to enhance their attention, uses the doctors' prior knowledge in consultation to enhance the ability of artificial intelligence in symptom inquiry, and allows doctors to handle complex cases that machines cannot solve, thereby reducing the generation of invalid symptoms and achieving a high-accuracy diagnosis within a limited number of inquiry rounds.
[0087] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0088] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0089] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0090] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0091] In addition, each functional unit in each embodiment of the present invention may be integrated into a first processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0092] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A human-machine collaborative Transformer model method for online medical consultation, characterized in that: include: When the query layer of the Transformer-based human-machine collaborative disease diagnosis framework performs the Kth round of query, the displayed symptoms, displayed symptom attributes, additional symptoms, and additional symptom attributes of the target user are obtained, where K is greater than or equal to the target number of times; Adding the displayed symptoms, displayed symptom attributes, additional symptoms, and additional symptom attributes to obtain target information; The target information is processed through the Transformer encoding layer, and the disease information corresponding to the target information and the prediction confidence of the disease information are output; When the prediction confidence of the disease information is greater than the preset threshold of the disease, the disease information corresponding to the target information is determined as the diagnosis result of the target user. When the prediction confidence of the disease information is less than or equal to the preset threshold of the disease, the target information is processed by the input embedding layer in the doctor-enhanced Transformer encoding layer to obtain the initial context representation; The initial context representation is processed by the doctor-enhanced Transformer layer in the doctor-enhanced Transformer encoding layer to obtain the final context representation, wherein the doctor-enhanced Transformer layer includes a multi-head doctor-enhanced self-attention layer, a first residual regularization layer, a feedforward layer, and a second residual regularization layer, and the multi-head doctor-enhanced self-attention layer includes processing the initial context representation and the doctor matrix; The final context representation is linearly transformed through the decision layer in the doctor-enhanced Transformer encoding layer to output the first new symptom and the confidence value of the first new symptom; When the confidence value of the first new symptom is less than the preset threshold of the new symptom and the symptoms output by the previous two rounds of doctor-enhanced Transformer coding layers are invalid, the second new symptom corresponding to the target information is obtained, and the second new symptom interacts with the target user to obtain the attribute information corresponding to the second new symptom. When the confidence value of the first new symptom is greater than or equal to the preset threshold of the new symptom, or the symptoms output by any of the previous two rounds of doctor-enhanced Transformer coding layers are valid, the first new symptom interacts with the target user to obtain the attribute information corresponding to the first new symptom.
2. The method according to claim 1, characterized in that The expression for the target information is obtained by adding the displayed symptoms, displayed symptom attributes, additional symptoms, and additional symptom attributes: X0=Embedding(S ex ,S add )+Embedding(A ex ,A add ) (1) Among them, X0 is the target information, S add For additional symptoms, S ex To show symptoms, A ex To display the symptom attributes, A add It is an additional symptom attribute.
3. The method according to claim 1, characterized in that The target information is processed through the Transformer encoding layer, and the disease information corresponding to the target information and the prediction confidence of the disease information are output, including: After the target sequence of the Nth layer is output by the decision layer in the Transformer encoding layer, the target sequence is aggregated into a single representation through the average pooling layer, where the expression of the single representation is: Where l represents the length of the target sequence, x i represents the i-th symptom vector in the target sequence, and AvgPooling(x) represents the average pooling operation on x; The single representation is mapped to the output space required for the disease classification task through a fully connected layer, and the disease information corresponding to the target information and the predicted confidence of the disease information are output.
4. The method according to claim 1, characterized in that Transforme enhanced by doctors r Doctor-enhanced Transforme in the encoding layer r The layer processes the initial context representation to obtain the final context representation, including: Transforme enhanced by doctors r The multi-head doctor enhanced self-attention layer in the layer processes the initial context representation to obtain the first output information; The first output information is processed through a feedforward layer to obtain the final context representation, wherein each multi-head doctor enhanced self-attention layer is connected to a first residual regularization layer, and each feedforward layer is connected to a second residual regularization layer.
5. The method according to claim 4, characterized in that The context information is processed by the multi-head doctor-enhanced self-attention layer in the doctor-enhanced Transformer layer to obtain the first output information, including: The multi-headed doctor enhanced self-attention layer applies three linear transformations W i q , W i k , and W i v Transform the context information to obtain the query Q for each head i , key K i , and the value V i ; Based on each head query Q i and key K i ,,get the global attention score; Based on each head query Q i , key K i and the doctor matrix, to obtain the enhanced doctor's attention score; Based on the global attention score, the doctor's attention score is enhanced, and the value V i and hyperparameters to obtain the value of each head of the multi-head doctor enhanced self-attention layer; Connect the values of each header to obtain the first output information.
6. The method according to claim 5, characterized in that Based on each head query Q i and key K i , and the expression of the global attention score is: in, is the global attention score, d h is the dimension of each head, T is K i The transpose of .
7. The method according to claim 5, characterized in that Based on each head query Q i , key K i And the doctor matrix, the expression of the enhanced doctor's attention score is: in, To enhance the doctor’s attention score, M doc For the doctor matrix.
8. The method according to claim 5, characterized in that Based on the global attention score, the doctor's attention score is enhanced, and the value V i And hyperparameters, the expression of the value of each head of the multi-head doctor enhanced self-attention layer is obtained as: Among them, head i represents the value of each head, and γ∈[0, 1] represents a score used to adjust the enhanced doctor's attention score. Hyperparameters of weights.
9. The method according to claim 5, characterized in that The calculation process of the doctor matrix is: Obtain a target data set, and build a symptom graph based on the target data set; Query the node features of the doctor's symptoms from the symptom graph, perform a scaled dot product between the node features and the initial context representation, and obtain a dot product diagonal vector; Write the dot product diagonal vector into the diagonal matrix to get the diagonal matrix; Construct sparse matrix; The diagonal matrix is added to the sparse matrix to obtain the doctor matrix.
10. A human-machine collaborative Transformer model framework for online medical consultation, characterized in that: include: An inquiry layer and a diagnosis layer, wherein the inquiry layer includes an activation module and a machine module, and the diagnosis layer includes a diagnosis module and a diagnosis result. The inquiry layer and the diagnosis layer interact with each other, the activation module is connected to the machine module, and the diagnosis module is connected to the diagnosis result; The diagnosis layer is used to obtain the displayed symptoms, displayed symptom attributes, additional symptoms and additional symptom attributes of the target user when the inquiry layer makes the Kth round of inquiry, add the displayed symptoms, displayed symptom attributes, additional symptoms and additional symptom attributes to obtain the target information, process the target information, and output the disease information corresponding to the target information and the predicted confidence of the disease information; when the predicted confidence of the disease information is greater than the preset threshold of the disease, the disease information corresponding to the target information is determined as the diagnosis result of the target user, and when the predicted confidence of the disease information is less than or equal to the preset threshold of the disease, the inquiry layer starts working, wherein K is greater than or equal to the target number of times; The query layer is used to process the target information through the input embedding layer in the doctor-enhanced Transformer encoding layer in the machine module to obtain an initial context representation, process the initial context representation through the doctor-enhanced Transformer layer in the doctor-enhanced Transformer encoding layer to obtain a final context representation, perform a linear transformation on the final context representation through the decision layer in the doctor-enhanced Transformer encoding layer, output the first new symptom and the confidence value of the first new symptom, and the activation module starts working. When the confidence value of the first new symptom is less than the preset threshold of the new symptom and the previous two rounds of doctor-enhanced Transformer When all symptoms output by the encoding layer are invalid, a second new symptom corresponding to the target information is obtained, and the second new symptom interacts with the target user to obtain attribute information corresponding to the second new symptom. When the confidence value of the first new symptom is greater than or equal to the preset threshold of the new symptom, or the symptoms output by any of the previous two rounds of doctor-enhanced Transformer encoding layers are valid, the first new symptom interacts with the target user to obtain attribute information corresponding to the first new symptom. The doctor-enhanced Transformer layer includes a multi-head doctor-enhanced self-attention layer, a first residual regularization layer, a feedforward layer, and a second residual regularization layer. The multi-head doctor-enhanced self-attention layer includes processing the initial context representation and the doctor matrix.
