Virtual patient dynamic dialogue generation method and system based on reinforcement learning

Abstract

The application relates to the fields of artificial intelligence and medical simulation, and particularly discloses a virtual patient dynamic dialogue generation method and system based on reinforcement learning, which comprises the following steps: acquiring an initial continuous state of a virtual patient; fusing the initial continuous state and external interaction information to generate an action signal containing a natural language output signal and an implicit intervention signal; constructing a dynamic evolution process, taking the current continuous state and the implicit intervention signal as driving input, and generating an updated continuous state through nonlinear transformation and time integration; comparing the updated continuous state with a reference trajectory to generate a time sequence consistency cumulative measure; and adjusting strategy parameters according to the measure, so that the implicit intervention signal generated in subsequent rounds drives the updated continuous state to approach the reference trajectory; the application realizes deep coupling of disease continuous evolution and dialogue behavior, and effectively improves the time sequence authenticity and interaction consistency of virtual patient simulation.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and medical simulation technology, specifically to a method and system for generating dynamic dialogues of virtual patients based on reinforcement learning. Background Technology

[0002] Currently, virtual patient technology is widely used in medical education and clinical training. Its core objective is to simulate the symptoms and responses of real patients through human-computer interaction, providing a training environment for medical students and young physicians. Existing virtual patient implementations mainly fall into two categories: one is a rule-driven dialogue system that constructs the patient's response path through predefined medical scripts and branching logic; the other is based on machine learning, particularly using large language models trained on static datasets, and constructing an end-to-end dialogue generation model through supervised learning fine-tuning, fitting patient responses using a large amount of historical consultation records.

[0003] Existing virtual patient methods cannot accurately simulate the strong temporal coupling between symptom evolution and physiological parameter changes during disease progression in the continuous time domain, resulting in state jumps and temporal disorder during virtual patient interaction. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for generating dynamic dialogues of virtual patients based on reinforcement learning, so as to solve the problems mentioned above.

[0005] The objective of this invention can be achieved through the following technical solutions: A method for generating dynamic dialogues for virtual patients based on reinforcement learning includes the following steps: S1, obtain the initial continuous state of the virtual patient. The initial continuous state is a time-continuous implicit vector. The implicit vector is used to characterize the distribution of the virtual patient's physiological parameters in the continuous time domain. S2, integrates the initial continuous state with the real-time acquired external interaction information to generate the action signal for the current round. The action signal includes natural language output signal and implicit intervention signal. S3, construct a dynamic evolution process to describe the evolution of a continuous state over time, take the current continuous state and the implicit intervention signal as the driving input of the dynamic evolution process, and perform nonlinear transformation and time integration on the current continuous state through the dynamic evolution process to generate an updated continuous state; S4. The updated continuous state is compared with the preset reference trajectory, and a time consistency cumulative metric is generated based on the comparison result. The time consistency cumulative metric is used to quantify the degree of fit between the updated continuous state and the reference trajectory along the time axis. S5, adjust the strategy parameters for generating action signals based on the cumulative temporal consistency metric, so that the updated continuous state output by the implicit intervention signals generated in subsequent rounds drives the dynamic evolution process to approach the reference trajectory.

[0006] As a further aspect of the present invention: S1 specifically includes: Acquire preset static attribute data and dynamic disease course data of virtual patients, and fuse the preset static attribute data and dynamic disease course data into initial fusion features; The initial fused features are mapped to a continuous temporal space to generate an initial temporal distribution; A continuity constraint is applied to the initial time-domain distribution to make it differentiable on the time axis. The initial time-domain distribution after the continuity constraint is used as the initial continuous state.

[0007] As a further aspect of the present invention: S2 specifically includes: The initial continuous state is input into the first mapping path to obtain the state feature representation, and the external interaction information is input into the second mapping path to obtain the interaction feature representation. The state feature representation and the interaction feature representation are nonlinearly coupled to generate a fused control basis vector. The fusion control basis vectors are input in parallel to the first generation path and the second generation path. The first generation path outputs the natural language output signal, and the second generation path outputs the implicit intervention signal.

[0008] As a further aspect of the present invention: the generation of the fusion control basis vector specifically includes: The state feature representation and the interaction feature representation are respectively subjected to multidimensional expansion processing to obtain expanded state features and expanded interaction features; Calculate the tensor product of the extended state features and the extended interaction features in the joint embedding space to obtain the higher-order coupling tensor; The higher-order coupled tensors are subjected to dimensionality reduction and aggregation, and the principal components of the higher-order coupled tensors are extracted as the fusion control basis vectors.

[0009] As a further aspect of the present invention: S3 specifically includes: The current continuous state is fused with the implicit intervention signal to obtain the fused driving state; The fusion-driven state is decomposed into multiple components with different time scales; Perform integration operations with corresponding step sizes on the components at each time scale to obtain each integral component; Each integral component is reorganized and corrected, and the reorganized and corrected result is used as the updated continuous state.

[0010] As a further aspect of the present invention: the fusion of the current continuous state with the implicit intervention signal to obtain the fused driving state specifically includes: The current continuous state and the implicit intervention signal are subjected to nonlinear extension processing to obtain extended state features and extended intervention features. The extended state features and extended intervention features are orthogonally projected into the joint embedding space to obtain the joint projection representation. An energy constraint is applied to the joint projection representation so that the modulus of the joint projection representation falls within a preset range, and the energy-constrained joint projection representation is used as the fusion driving state.

[0011] As a further aspect of the present invention: S4 specifically includes: Local morphological features of the updated continuous state are extracted along the time axis, and local morphological features of the reference trajectory at the corresponding time are also extracted. Calculate the morphological similarity between the local morphological features of the updated continuous state and the local morphological features of the reference trajectory to obtain a temporal similarity sequence; A decay accumulation operation is performed on the temporal similarity sequence along the time axis, and the result of the decay accumulation operation is used as the temporal consistency accumulation measure.

[0012] As a further aspect of the present invention: the local morphological features specifically include: Centered on the current moment, extract a local segment of a preset length from the updated continuous state to obtain a local state segment; Perform multi-order difference operations on the local state segments to obtain first-order difference sequences and second-order difference sequences, and combine the first-order difference sequences and second-order difference sequences into a local morphological vector. The local morphological vector is normalized so that the magnitudes of each component of the local morphological vector are under a unified dimension. The normalized local morphological vector is then used as the local morphological feature of the updated continuous state. When extracting the local morphological features of the reference trajectory at the corresponding time, the same steps as those used for extracting the local morphological features of the updated continuous state are employed.

[0013] As a further aspect of the present invention: S5 specifically includes: Convert the cumulative metric for time-series consistency into a baseline for parameter tuning; Obtain the implicit intervention signal sequence generated in the historical rounds, perform time-series correlation calculation between the parameter adjustment baseline and the implicit intervention signal sequence to obtain the parameter update direction; An adaptive step size constraint is applied to the parameter update direction to keep the magnitude of the parameter update direction within a preset range, and the parameter update direction after the adaptive step size constraint is used as the parameter adjustment amount. The parameter adjustment amount is added to the current strategy parameters to obtain the updated strategy parameters.

[0014] A reinforcement learning-based virtual patient dynamic dialogue generation system includes: The continuous state initialization module obtains the initial continuous state of the virtual patient. The initial continuous state is a time-continuous implicit vector, which is used to characterize the distribution of the virtual patient's physiological parameters in the continuous time domain. The action signal generation module fuses the initial continuous state with the external interaction information acquired in real time to generate the action signal for the current round. The action signal includes natural language output signal and implicit intervention signal. The state evolution update module constructs a dynamic evolution process to describe the evolution of a continuous state over time. It takes the current continuous state and the implicit intervention signal as the driving input of the dynamic evolution process. Through the dynamic evolution process, it performs nonlinear transformation and time integration on the current continuous state to generate the updated continuous state. The temporal consistency measurement module compares the updated continuous state with the preset reference trajectory and generates a temporal consistency cumulative measurement based on the comparison result. The temporal consistency cumulative measurement is used to quantify the degree of fit between the updated continuous state and the reference trajectory along the time axis. The strategy parameter adjustment module adjusts the strategy parameters of the generated action signals based on the cumulative temporal consistency metric, so that the updated continuous state output by the implicit intervention signals generated in subsequent rounds drives the dynamic evolution process to approach the reference trajectory.

[0015] The beneficial effects of this invention are: (1) This invention constructs the physiological state of a virtual patient as a time-continuous implicit vector and uses the dynamic evolution process to decompose and integrate it at multiple time scales, thereby realizing the continuous and progressive simulation of pathological evolution. This effectively avoids the temporal disorder and mechanical feeling caused by discrete state jumps, and makes the disease evolution process strictly follow the temporal progression of the clinical course, thus improving the simulation realism of virtual patients in scenarios with strong temporal dependence such as critical illness.

[0016] (2) This invention uses the cumulative measurement of temporal consistency as a fine-grained reward signal to adjust the strategy parameters in a closed loop, so that the generation of implicit intervention signals and the evolution of physiological state are synergistically optimized. Without relying on artificially designed sparse rewards, it can guide the continuous state output by the dynamic evolution process to approach the preset reference trajectory, thereby reducing the interaction cost and realizing the intrinsic consistency between the dialogue content and the patient's physiological state. Attached Figure Description

[0017] The invention will now be further described with reference to the accompanying drawings.

[0018] Figure 1This is a flowchart of the method of the present invention; Figure 2 This is a system block diagram of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Please see Figure 1 As shown, this invention is a method for generating dynamic dialogue for virtual patients based on reinforcement learning, comprising the following steps: S1, obtain the initial continuous state of the virtual patient. The initial continuous state is a time-continuous implicit vector. The implicit vector is used to characterize the distribution of the virtual patient's physiological parameters in the continuous time domain. S2, integrates the initial continuous state with the real-time acquired external interaction information to generate the action signal for the current round. The action signal includes natural language output signal and implicit intervention signal. S3, construct a dynamic evolution process to describe the evolution of a continuous state over time, take the current continuous state and the implicit intervention signal as the driving input of the dynamic evolution process, and perform nonlinear transformation and time integration on the current continuous state through the dynamic evolution process to generate an updated continuous state; S4. The updated continuous state is compared with the preset reference trajectory, and a time consistency cumulative metric is generated based on the comparison result. The time consistency cumulative metric is used to quantify the degree of fit between the updated continuous state and the reference trajectory along the time axis. S5, adjust the strategy parameters for generating action signals based on the cumulative temporal consistency metric, so that the updated continuous state output by the implicit intervention signals generated in subsequent rounds drives the dynamic evolution process to approach the reference trajectory.

[0021] In S1, the initial continuous state of the virtual patient is obtained. The initial continuous state is a time-continuous implicit vector. The implicit vector is used to characterize the distribution of the virtual patient's physiological parameters in the continuous time domain, specifically including: First, preset static attribute data and dynamic disease course data of the virtual patient are acquired and fused into an initial fusion feature. The preset static attribute data originates from a pre-entered patient basic information database stored locally. This database includes the virtual patient's age, gender, height, weight, and preset basic disease tags. The dynamic disease course data originates from a pre-constructed disease course parameter table, set according to clinical treatment guidelines. This table includes an initial symptom type identifier, an initial symptom severity value, a disease course start timestamp, and a preset disease stage identifier. During the data acquisition phase, the values ​​of each field from the static attribute data and the dynamic disease course data are read through a data interface. The values ​​from the static attribute data are combined to form a first vector, and the values ​​from the dynamic disease course data are combined to form a second vector. The first and second vectors are then concatenated to obtain the initial fusion feature, which is a multi-dimensional numerical vector.

[0022] Secondly, the initial fusion features are mapped to a continuous temporal space to generate an initial temporal distribution. This mapping process is implemented through a nonlinear transformation unit consisting of three cascaded nonlinear transformation layers. Each layer performs the following operation: each element of the input vector is input to a preset nonlinear activation function, which is a hyperbolic tangent function. The input of the hyperbolic tangent function is the current element value, and the output is the element value after nonlinear transformation. The first layer uses the initial fusion features as input and outputs a first intermediate vector; the second layer uses the first intermediate vector as input and outputs a second intermediate vector; the third layer uses the second intermediate vector as input and outputs a third intermediate vector. Each nonlinear transformation layer has a corresponding weight coefficient matrix and a bias vector. The dimension of the weight coefficient matrix is ​​preset according to the dimension of the input vector, and the dimension of the bias vector is consistent with the dimension of the output vector. After completing the three nonlinear transformations, the third intermediate vector is used as the initial temporal distribution, which is a set of numerical sequences at discrete time points. Each time point corresponds to a numerical vector, and each element in the numerical vector represents the initial distribution value of different physiological parameters at the corresponding time point.

[0023] Finally, a continuity constraint is applied to the initial time-domain distribution to make it differentiable on the time axis. The initial time-domain distribution after the continuity constraint is taken as the initial continuous state. This continuity constraint is achieved through cubic spline interpolation. The specific process is as follows: extract each discrete time point and its corresponding numerical vector from the initial time-domain distribution. Use the time point as the independent variable and each element in the numerical vector as the dependent variable. Construct a cubic spline interpolation curve for each element in the numerical vector. The construction process of each cubic spline interpolation curve is as follows: divide the interval between two adjacent discrete time points into an interpolation segment. Construct a cubic polynomial function on each interpolation segment. This cubic polynomial function satisfies that the function value at both ends of the interpolation segment is equal to the element value at the corresponding time point in the initial time-domain distribution, and that the first derivative and second derivative are continuous at both ends of the interpolation segment. By solving the coefficients of the cubic polynomial function of each interpolation segment, the spline function expression describing the continuous change of the element with time is obtained. For each element in the numerical vector, the above cubic spline interpolation operation is performed to obtain a set of spline function expressions. This set of spline function expressions together constitutes a time-continuous implicit vector. This implicit vector has a clear numerical definition and a first derivative at any point in time on the time axis. This implicit vector is used as the initial continuous state of the virtual patient.

[0024] In S2, the initial continuous state is fused with real-time acquired external interaction information to generate the action signal for the current round. The action signal includes natural language output signals and implicit intervention signals, specifically including: First, the initial continuous state is input into the first mapping path to obtain a state feature representation. Simultaneously, external interaction information is input into the second mapping path to obtain an interaction feature representation. The first mapping path consists of three cascaded linear transformation layers. Each linear transformation layer performs the following operation: multiplying the input vector by a pre-defined first weight matrix and adding it to a pre-defined first bias vector to obtain an output vector. The first layer takes the value vector of the initial continuous state at a preset reference time as input and outputs a first state intermediate vector with a dimension of 128. The second layer takes the first state intermediate vector as input and outputs a second state intermediate vector with a dimension of 64. The third layer takes the second state intermediate vector as input and outputs a state feature representation with a dimension of 32. External interaction information refers to the user input text received in the current round, which is converted into an interaction feature representation in the following way: First, the user input text is segmented to obtain a segmentation sequence, and each segment is mapped to a corresponding word vector. The word vector has a dimension of 256, resulting in a word vector sequence. Then, a weighted average operation is performed on the word vector sequence, that is, the arithmetic mean of all word vectors is calculated to obtain a 256-dimensional interaction feature representation.

[0025] Secondly, a nonlinear coupling operation is performed on the state feature representation and the interaction feature representation to generate a fused control basis vector. This nonlinear coupling operation is implemented through the following steps: First, the state feature representation and the interaction feature representation are subjected to multidimensional expansion processing to obtain expanded state features and expanded interaction features. The specific method of multidimensional expansion processing is as follows: For the 32-dimensional vector of the state feature representation, each element is multiplied by a preset first expansion coefficient set, which contains three different expansion coefficient values: 0.5, 1.0, and 2.0, thereby expanding each element into three derived values ​​and expanding the 32-dimensional vector into a 96-dimensional expanded state feature; For the 256-dimensional vector of the interaction feature representation, each element is multiplied by a preset second expansion coefficient set, which contains two different expansion coefficient values: 0.8 and 1.2, thereby expanding each element into two derived values ​​and expanding the 256-dimensional vector into a 512-dimensional expanded interaction feature. The second step involves calculating the tensor product of the extended state features and extended interaction features in the joint embedding space to obtain a higher-order coupling tensor. The tensor product is calculated by multiplying each element of the extended state features by each element of the extended interaction features, resulting in a 96-row, 512-column two-dimensional matrix. This two-dimensional matrix is ​​the higher-order coupling tensor, containing 49,152 elements. The third step involves dimensionality reduction and aggregation of the higher-order coupling tensor, extracting its principal components as fusion control basis vectors. The specific method of dimensionality reduction and aggregation is as follows: the two-dimensional matrix of the high-order coupling tensor is concatenated along the row direction to convert it into a 49152-dimensional one-dimensional vector. Principal component analysis is then performed on this one-dimensional vector. The process of principal component analysis is as follows: 1000 samples of high-order coupling tensor are collected in advance as training samples. The covariance matrix of these 1000 samples is calculated. The eigenvalues ​​and eigenvectors of the covariance matrix are solved. The top 10 eigenvectors with the largest eigenvalues ​​are selected to form a projection matrix. The current 49152-dimensional one-dimensional vector is multiplied by the projection matrix to obtain a 10-dimensional fusion control basis vector.

[0026] Finally, the fusion control basis vector is input in parallel to the first and second generation paths. The first generation path outputs a natural language output signal, and the second generation path outputs an implicit intervention signal. The first generation path consists of an intent classification unit and a text filling unit: the intent classification unit inputs the 10-dimensional fusion control basis vector into a linear classification layer. This layer contains a pre-defined classification weight matrix and a classification bias vector. The classification weight matrix has a dimension of 10 rows and 20 columns, and the classification bias vector has a dimension of 20. By multiplying the fusion control basis vector by the classification weight matrix and then adding it to the classification bias vector, a 20-dimensional intent score vector is obtained. The intent label corresponding to the dimension with the highest score in the intent score vector is selected as the output intent for the current round. This intent label is selected from a pre-defined intent label set containing 20 different dialogue intents. The text filling unit retrieves the corresponding text template from a pre-defined text template library based on the output intent label. The first five elements of the fusion control basis vector are filled into the placeholders of the text template according to a pre-defined mapping relationship, generating a complete natural language output signal. The second generation path consists of a linear mapping unit: the linear mapping unit contains a pre-defined mapping weight matrix with a dimension of 10 rows and 8 columns. The 10-dimensional fusion control basis vector is multiplied by the mapping weight matrix to obtain an 8-dimensional vector, which is then output as an implicit intervention signal.

[0027] In S3, a dynamic evolution process is constructed to describe the evolution of a continuous state over time. The current continuous state and the implicit intervention signal are used as the driving inputs of the dynamic evolution process. The current continuous state is nonlinearly transformed and integrated over time through the dynamic evolution process to generate an updated continuous state. Specifically, this includes: First, the current continuous state is fused with the implicit intervention signal to obtain the fused driving state. The current continuous state refers to the distribution of the virtual patient's physiological parameters in the continuous time domain at the start of the current round, existing in the form of a time-continuous function. The implicit intervention signal refers to the 8-dimensional vector output by the second generation path. The fusion process is specifically implemented through the following steps: The first step involves performing nonlinear expansion processing on the current continuous state and the implicit intervention signal to obtain expanded state features and expanded intervention features, respectively. The specific method of nonlinear expansion processing is as follows: For the current continuous state, its value vector at the current time is obtained. This value vector has a dimension of 32. Each element of this 32-dimensional vector is input to a preset first nonlinear expansion function. This first nonlinear expansion function adopts an exponential function form, that is, for an input value x, the output value is e raised to the power of x. Through this operation, the 32-dimensional vector is expanded into 32-dimensional expanded state features, and each element value is mapped to a non-negative value. For the implicit intervention signal, an 8-dimensional vector, each element is input to a preset second nonlinear expansion function. This second nonlinear expansion function adopts a hyperbolic tangent function form, that is, for an input value y, the output value is the value of the hyperbolic tangent function at y. The value range of this function is between -1 and +1. Through this operation, the 8-dimensional vector is expanded into 8-dimensional expanded intervention features.

[0028] The second step involves performing orthogonal projection operations on the extended state features and extended intervention features in the joint embedding space to obtain a joint projected representation. The specific method for orthogonal projection is as follows: A projection matrix with dimensions of 40 rows and 40 columns is pre-defined. This projection matrix is ​​constructed as follows: 500 sets of 40-dimensional vectors, formed by concatenating extended state features and extended intervention features, are collected as training samples. The covariance matrix of these training samples is calculated, and eigenvalue decomposition is performed on the covariance matrix. The top 40 eigenvectors with the largest eigenvalues ​​are selected to form an orthogonal basis matrix, which is the projection matrix. The extended state features and extended intervention features are concatenated to obtain a 40-dimensional concatenated vector. This concatenated vector is then multiplied by the projection matrix to obtain a 40-dimensional joint projected representation.

[0029] The third step involves applying energy constraints to the joint projection representation, ensuring its magnitude falls within a preset range. The energy-constrained joint projection representation is then used as the fusion driving state. The energy constraint is implemented by calculating the magnitude of the joint projection representation, which is obtained by calculating the square root of the sum of squares of all elements in the 40-dimensional vector. The preset lower limit is 0.5, and the preset upper limit is 2.0. If the magnitude is less than 0.5, each element in the joint projection representation is multiplied by a scaling factor (0.5 divided by the magnitude) to adjust the magnitude to 0.5. If the magnitude is greater than 2.0, each element in the joint projection representation is multiplied by a scaling factor (2.0 divided by the magnitude) to adjust the magnitude to 2.0. If the magnitude is between 0.5 and 2.0, the joint projection representation remains unchanged. The 40-dimensional vector after the energy constraint process is then used as the fusion driving state.

[0030] Secondly, the fusion driving state is decomposed into multiple components with different time scales. Specifically, three different time scale parameters are pre-defined: a short-term parameter of 0.1 seconds, a medium-term parameter of 1.0 second, and a long-term parameter of 10.0 seconds. This 40-dimensional vector of the fusion driving state is then input to three scale filters. Each scale filter uses a first-order low-pass filter, and its output is related to the input using the following mathematical formula: ;in, Represents component vectors The differential, The derivative of time, This indicates that the fusion driving state is used as the input vector. Indicates the first The component vectors corresponding to each time scale This indicates the corresponding time scale parameter, with the short time scale corresponding to... The value is 0.1 seconds, corresponding to the medium time scale. The value is 1.0 second, corresponding to a long-term scale. The value is 10.0 seconds. By solving this first-order differential equation, three component vectors with different time scales are obtained, each with a dimension of 40.

[0031] Then, integration operations with corresponding step sizes are performed on the components at each time scale to obtain each integral component. The specific integration method is as follows: for short-time scale components, the Euler integration method is used with an integration step size of 0.05 seconds, and numerical integration is performed on the component within the time interval to obtain the short-time integral component; for medium-time scale components, the Euler integration method is used with an integration step size of 0.5 seconds, and numerical integration is performed on the component within the time interval to obtain the medium-time integral component; for long-time scale components, the Euler integration method is used with an integration step size of 5.0 seconds, and numerical integration is performed on the component within the time interval to obtain the long-time integral component. Each integral component has 40 dimensions.

[0032] Finally, the integral components are recombined and corrected, and the recombined and corrected results are used as the updated continuous state. The specific method of recombination is as follows: the short-time integral component, the medium-time integral component, and the long-time integral component are weighted and summed, with preset weight coefficients of 0.2, 0.3, and 0.5, respectively. Each integral component is multiplied by its corresponding weight coefficient and then summed to obtain a 40-dimensional recombined vector. The specific correction process is as follows: The reconstructed vector is input into a nonlinear correction unit, which consists of three cascaded layers: The first layer performs a linear transformation, multiplying the 40-dimensional reconstructed vector by a 40x40 correction weight matrix, and then adding it to a 40-dimensional correction bias vector to obtain the first intermediate correction vector; the second layer performs a normalization operation, subtracting the mean of each element in the first intermediate correction vector from the vector's mean, and then dividing by the vector's standard deviation to obtain the second intermediate correction vector; the third layer performs a limiting operation, restricting each element in the second intermediate correction vector to the range of -3 to +3. If the element value is less than -3, it is set to -3; if the element value is greater than +3, it is set to +3; otherwise, it remains unchanged. The 40-dimensional vector obtained after the third limiting operation is used as the updated continuous state, which is used to characterize the distribution of physiological parameters of the virtual patient at the next time point.

[0033] In S4, the updated continuous state is compared with a preset reference trajectory, and a temporal consistency cumulative metric is generated based on the comparison result. The temporal consistency cumulative metric is used to quantify the degree of fit between the updated continuous state and the reference trajectory along the time axis, specifically including: First, local morphological features of the updated continuous state are extracted along the time axis, and local morphological features of the reference trajectory at the corresponding time points are also extracted. The extraction process specifically involves: taking the current time point as the center, extracting local segments of a preset length from the updated continuous state to obtain local state segments. The preset length is set to 11 time points, i.e., selecting the current time point, the 5 time points before the current time point, and the 5 time points after the current time point, for a total of 11 time points constituting the local state segments. Each time point corresponds to a 40-dimensional vector. Multi-order difference operations are performed on the local state segments to obtain first-order difference sequences and second-order difference sequences. The first-order difference sequence is calculated by subtracting the 40-dimensional vectors of two adjacent time points, i.e., subtracting the vector of the previous time point from the vector of the later time point, resulting in 10 first-order difference vectors, each with a dimension of 40. These 10 first-order difference vectors are arranged in chronological order to form a first-order difference sequence. The second-order difference sequence is calculated as follows: Subtraction is performed on any two adjacent first-order difference vectors in the first-order difference sequence, i.e., the first-order difference vector at the later time point is subtracted from the first-order difference vector at the previous time point, resulting in nine second-order difference vectors, each with a dimension of 40. These nine second-order difference vectors are arranged in chronological order to form the second-order difference sequence. The first-order and second-order difference sequences are then combined to form a local morphological vector. Specifically, the ten vectors in the first-order difference sequence are expanded sequentially to obtain a 400-dimensional vector, and the nine vectors in the second-order difference sequence are expanded sequentially to obtain a 360-dimensional vector. These two vectors are then concatenated to obtain a 760-dimensional local morphological vector. The local morphological vector is normalized to ensure that the magnitudes of its components are on a uniform scale. This normalization is achieved by calculating the square root of each component in the 760-dimensional local morphological vector (i.e., the square root of the sum of squares of all components), obtaining the modulus, and then dividing each component by this modulus to obtain the normalized local morphological vector. This normalized vector is then used as the local morphological feature of the updated continuous state. When extracting the local morphological features of the reference trajectory at corresponding time points, the same steps are used as when extracting the local morphological features of the updated continuous state. Specifically, 11 time-point segments of the reference trajectory are extracted centered on the current time point, and the first-order and second-order difference sequences are calculated, combined to form a 760-dimensional local morphological vector, and then normalized to obtain the local morphological features of the reference trajectory.

[0034] Secondly, the morphological similarity between the local morphological features of the updated continuous state and the local morphological features of the reference trajectory is calculated to obtain a temporal similarity sequence. The morphological similarity is calculated as follows: the local morphological feature vector of the updated continuous state and the local morphological feature vector of the reference trajectory are subjected to a dot product operation, i.e., the sum of the products of the corresponding positional components of the two vectors is calculated to obtain the dot product value; simultaneously, the magnitude of each vector is calculated, i.e., the square root of the sum of the squares of each vector component is calculated; the dot product value is divided by the product of the two magnitudes to obtain the cosine similarity value, which is between -1 and +1, with the value closer to 1 indicating greater similarity between the two local morphological features. For each time point, the above calculation is performed to obtain the corresponding morphological similarity value. The morphological similarity values ​​corresponding to all time points are arranged in chronological order to form a temporal similarity sequence.

[0035] Finally, a decay accumulation operation is performed on the temporal similarity sequence along the time axis, and the result of the decay accumulation operation is used as the temporal consistency accumulation metric. The specific method of the decay accumulation operation is as follows: a decay coefficient of 0.95 is preset, and each value in the temporal similarity sequence is traversed in chronological order from earliest to latest. The initial accumulation value is set to 0. For the t-th value in the sequence, it is multiplied by the decay coefficient raised to the power of t minus 1 and then added to the accumulated value. That is, similarity values ​​from earlier times are given higher accumulation weights, and similarity values ​​from later times are given lower accumulation weights. After traversing the entire temporal similarity sequence, the final accumulated value is used as the temporal consistency accumulation metric. This metric is used to quantify the degree of fit between the updated continuous state and the reference trajectory along the entire time axis.

[0036] In S5, the strategy parameters for generating action signals are adjusted based on the cumulative temporal consistency metric, so that the implicit intervention signals generated in subsequent rounds drive the updated continuous states output by the dynamic evolution process to approach the reference trajectory. Specifically, this includes: First, the cumulative timing consistency metric is converted into a parameter adjustment baseline. The cumulative timing consistency metric is a value output from step four, ranging from 0 to positive infinity. The conversion is implemented using a nonlinear mapping unit that employs a piecewise linear function: when the cumulative timing consistency metric is less than 0.5, the parameter adjustment baseline is set to -0.1; when the cumulative timing consistency metric is between 0.5 and 1.0, the parameter adjustment baseline is set to the cumulative timing consistency metric minus 0.5 multiplied by 0.2, i.e., linearly mapped to the range of -0.1 to 0.1; when the cumulative timing consistency metric is between 1.0 and 2.0, the parameter adjustment baseline is set to the cumulative timing consistency metric minus 1.0 multiplied by 0.2, i.e., linearly mapped to the range of 0 to 0.2; when the cumulative timing consistency metric is greater than 2.0, the parameter adjustment baseline is set to 0.2. Through this piecewise linear mapping, a parameter adjustment baseline between -0.1 and 0.2 is obtained.

[0037] Secondly, the implicit intervention signal sequence generated in historical rounds is obtained. A time-series correlation operation is performed between the parameter adjustment baseline and the implicit intervention signal sequence to obtain the parameter update direction. The implicit intervention signal sequence generated in historical rounds refers to the 8-dimensional implicit intervention signals output by the second generation path in the most recent 10 rounds. These signals are arranged in chronological order to form a 10x8 matrix. The specific method of the time-series correlation operation is as follows: the average value of each column in the matrix is ​​calculated to obtain an 8-dimensional average intervention vector; the parameter adjustment baseline is multiplied by each component of the average intervention vector to obtain an 8-dimensional product vector; this product vector is element-wise added to the sub-parameters related to the generation of implicit intervention signals in the current strategy parameters to obtain an 8-dimensional candidate direction vector; the sign of this candidate direction vector is determined, setting the positive components to 1, the negative components to -1, and the zero components to 0, to obtain an 8-dimensional sign vector, which is used as the parameter update direction.

[0038] Then, an adaptive step size constraint is applied to the parameter update direction to keep its magnitude within a preset range. The parameter update direction after the adaptive step size constraint is used as the parameter adjustment amount. The specific method of the adaptive step size constraint is as follows: the preset upper limit of the step size is 0.01, and the preset lower limit is 0.001. First, the historical average gradient of the current policy parameter is calculated, that is, the average absolute value of each component of the parameter update direction in the last 100 rounds is calculated to obtain an 8-dimensional average gradient magnitude vector. For each component in the parameter update direction, it is multiplied by a step size coefficient. This step size coefficient is obtained by multiplying the preset baseline step size of 0.005 by the ratio of the average gradient magnitude of the corresponding dimension of the component to the average gradient magnitude of all dimensions, so that the step size coefficient is adaptively adjusted between 0.001 and 0.01. Each component in the parameter update direction is multiplied by its corresponding step size coefficient to obtain an 8-dimensional adjusted vector, which is used as the parameter adjustment amount.

[0039] Finally, the parameter adjustments are superimposed onto the current policy parameters to obtain the updated policy parameters. The current policy parameters refer to the weight matrix and bias vector of the intent classification unit in the first generation path, and the weight matrix of the mapping unit in the second generation path. The superposition method is as follows: the first five components of the parameter adjustments are superimposed onto the corresponding rows of the intent classification unit's weight matrix; the sixth component is superimposed onto the corresponding position of the intent classification unit's bias vector; and the seventh and eighth components are superimposed onto the corresponding columns of the mapping unit's weight matrix in the second generation path. The resulting weight matrix and bias vector are used as the updated policy parameters for generating action signals in subsequent rounds.

[0040] Please see Figure 2 As shown, the virtual patient dynamic dialogue generation system based on reinforcement learning includes: The continuous state initialization module obtains the initial continuous state of the virtual patient. The initial continuous state is a time-continuous implicit vector, which is used to characterize the distribution of the virtual patient's physiological parameters in the continuous time domain. The action signal generation module fuses the initial continuous state with the external interaction information acquired in real time to generate the action signal for the current round. The action signal includes natural language output signal and implicit intervention signal. The state evolution update module constructs a dynamic evolution process to describe the evolution of a continuous state over time. It takes the current continuous state and the implicit intervention signal as the driving input of the dynamic evolution process. Through the dynamic evolution process, it performs nonlinear transformation and time integration on the current continuous state to generate the updated continuous state. The temporal consistency measurement module compares the updated continuous state with the preset reference trajectory and generates a temporal consistency cumulative measurement based on the comparison result. The temporal consistency cumulative measurement is used to quantify the degree of fit between the updated continuous state and the reference trajectory along the time axis. The strategy parameter adjustment module adjusts the strategy parameters of the generated action signals based on the cumulative temporal consistency metric, so that the updated continuous state output by the implicit intervention signals generated in subsequent rounds drives the dynamic evolution process to approach the reference trajectory.

[0041] The working principle of this invention is as follows: First, preset static attribute data and dynamic disease course data of a virtual patient are acquired and fused into initial fusion features. These are then mapped to a continuous time domain space through nonlinear transformation, and continuous constraints are applied through cubic spline interpolation to generate a time-continuous implicit vector as the initial continuous state. Next, the initial continuous state and external interaction information are extracted as state feature representation and interaction feature representation through linear transformation and word vector weighted averaging, respectively. These are then subjected to multidimensional expansion, tensor product operation, and principal component analysis to generate a fusion control basis vector. This vector is then input in parallel to the intent classification and text filling path to generate a natural language output signal, and to the linear mapping path to generate an implicit intervention signal. Finally, a dynamic evolution process is constructed, fusing the current continuous state and implicit intervention signal into a fusion driving state through nonlinear expansion, orthogonal projection, and energy constraints. This state is then decomposed into... The components at three different time scales—short-term, medium-term, and long-term—are weighted and recombined after numerical integration at corresponding step sizes. Following linear transformation, normalization, and amplitude limiting correction, an updated continuous state is generated. Subsequently, local segments are extracted along the time axis, and local morphological vectors are extracted and normalized using first-order and second-order difference operations. The cosine similarity to the local morphological features of the preset reference trajectory is calculated, resulting in a temporal similarity sequence. This sequence is then attenuated and accumulated using a decay coefficient to generate a temporal consistency accumulation metric. Finally, this metric is converted into a parameter adjustment benchmark through piecewise linear mapping. A temporal correlation operation is performed with the historical implicit intervention signal sequence to obtain the parameter update direction. An adaptive step-size constraint is then applied to generate the parameter adjustment amount, which is superimposed on the current strategy parameters to complete the update. This ensures that the updated continuous state output by the implicit intervention signals generated in subsequent rounds, driven by the dynamic evolution process, approximates the reference trajectory.

[0042] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.

Claims

1. A method for generating dynamic dialogues for virtual patients based on reinforcement learning, characterized in that, Includes the following steps: S1, obtain the initial continuous state of the virtual patient. The initial continuous state is a time-continuous implicit vector. The implicit vector is used to characterize the distribution of the virtual patient's physiological parameters in the continuous time domain. S2, integrates the initial continuous state with the real-time acquired external interaction information to generate the action signal for the current round. The action signal includes natural language output signal and implicit intervention signal. S3, construct a dynamic evolution process to describe the evolution of a continuous state over time, take the current continuous state and the implicit intervention signal as the driving input of the dynamic evolution process, and perform nonlinear transformation and time integration on the current continuous state through the dynamic evolution process to generate an updated continuous state; S4. The updated continuous state is compared with the preset reference trajectory, and a time consistency cumulative metric is generated based on the comparison result. The time consistency cumulative metric is used to quantify the degree of fit between the updated continuous state and the reference trajectory along the time axis. S5, adjust the strategy parameters for generating action signals based on the cumulative temporal consistency metric, so that the updated continuous state output by the implicit intervention signals generated in subsequent rounds drives the dynamic evolution process to approach the reference trajectory.

2. The method for generating dynamic dialogue for virtual patients based on reinforcement learning according to claim 1, characterized in that, S1 specifically includes: Acquire preset static attribute data and dynamic disease course data of virtual patients, and fuse the preset static attribute data and dynamic disease course data into initial fusion features; The initial fused features are mapped to a continuous temporal space to generate an initial temporal distribution; A continuity constraint is applied to the initial time-domain distribution to make it differentiable on the time axis. The initial time-domain distribution after the continuity constraint is used as the initial continuous state.

3. The method for generating dynamic dialogue for virtual patients based on reinforcement learning according to claim 1, characterized in that, S2 specifically includes: The initial continuous state is input into the first mapping path to obtain the state feature representation, and the external interaction information is input into the second mapping path to obtain the interaction feature representation. The state feature representation and the interaction feature representation are nonlinearly coupled to generate a fused control basis vector. The fusion control basis vectors are input in parallel to the first generation path and the second generation path. The first generation path outputs the natural language output signal, and the second generation path outputs the implicit intervention signal.

4. The method for generating dynamic dialogue for virtual patients based on reinforcement learning according to claim 3, characterized in that, The generation of the fusion control basis vectors specifically includes: The state feature representation and the interaction feature representation are respectively subjected to multidimensional expansion processing to obtain expanded state features and expanded interaction features; Calculate the tensor product of the extended state features and the extended interaction features in the joint embedding space to obtain the higher-order coupling tensor; The higher-order coupled tensors are subjected to dimensionality reduction and aggregation, and the principal components of the higher-order coupled tensors are extracted as the fusion control basis vectors.

5. The method for generating dynamic dialogue for virtual patients based on reinforcement learning according to claim 1, characterized in that, S3 specifically includes: The current continuous state is fused with the implicit intervention signal to obtain the fused driving state; The fusion-driven state is decomposed into multiple components with different time scales; Perform integration operations with corresponding step sizes on the components at each time scale to obtain each integral component; Each integral component is reorganized and corrected, and the reorganized and corrected result is used as the updated continuous state.

6. The method for generating dynamic dialogue for virtual patients based on reinforcement learning according to claim 5, characterized in that, The process of fusing the current continuous state with the implicit intervention signal to obtain the fused driving state specifically includes: The current continuous state and the implicit intervention signal are subjected to nonlinear extension processing to obtain extended state features and extended intervention features. The extended state features and extended intervention features are orthogonally projected into the joint embedding space to obtain the joint projection representation. An energy constraint is applied to the joint projection representation so that the modulus of the joint projection representation falls within a preset range, and the energy-constrained joint projection representation is used as the fusion driving state.

7. The method for generating dynamic dialogue for virtual patients based on reinforcement learning according to claim 1, characterized in that, S4 specifically includes: Local morphological features of the updated continuous state are extracted along the time axis, and local morphological features of the reference trajectory at the corresponding time are also extracted. Calculate the morphological similarity between the local morphological features of the updated continuous state and the local morphological features of the reference trajectory to obtain a temporal similarity sequence; A decay accumulation operation is performed on the temporal similarity sequence along the time axis, and the result of the decay accumulation operation is used as the temporal consistency accumulation measure.

8. The method for generating dynamic dialogue for virtual patients based on reinforcement learning according to claim 7, characterized in that, The local morphological features specifically include: Centered on the current moment, extract a local segment of a preset length from the updated continuous state to obtain a local state segment; Perform multi-order difference operations on the local state segments to obtain first-order difference sequences and second-order difference sequences, and combine the first-order difference sequences and second-order difference sequences into a local morphological vector. The local morphological vector is normalized so that the magnitudes of each component of the local morphological vector are under a unified dimension. The normalized local morphological vector is then used as the local morphological feature of the updated continuous state. When extracting the local morphological features of the reference trajectory at the corresponding time, the same steps as those used for extracting the local morphological features of the updated continuous state are employed.

9. The method for generating dynamic dialogue for virtual patients based on reinforcement learning according to claim 1, characterized in that, S5 specifically includes: Convert the cumulative metric for time-series consistency into a baseline for parameter tuning; Obtain the implicit intervention signal sequence generated in the historical rounds, perform time-series correlation calculation between the parameter adjustment baseline and the implicit intervention signal sequence to obtain the parameter update direction; An adaptive step size constraint is applied to the parameter update direction to keep the magnitude of the parameter update direction within a preset range, and the parameter update direction after the adaptive step size constraint is used as the parameter adjustment amount. The parameter adjustment amount is added to the current strategy parameters to obtain the updated strategy parameters.

10. A virtual patient dynamic dialogue generation system based on reinforcement learning, characterized in that, A method for generating dynamic dialogue for virtual patients based on reinforcement learning as described in any one of claims 1-9, comprising: The continuous state initialization module obtains the initial continuous state of the virtual patient. The initial continuous state is a time-continuous implicit vector, which is used to characterize the distribution of the virtual patient's physiological parameters in the continuous time domain. The action signal generation module fuses the initial continuous state with the external interaction information acquired in real time to generate the action signal for the current round. The action signal includes natural language output signal and implicit intervention signal. The state evolution update module constructs a dynamic evolution process to describe the evolution of a continuous state over time. It takes the current continuous state and the implicit intervention signal as the driving input of the dynamic evolution process. Through the dynamic evolution process, it performs nonlinear transformation and time integration on the current continuous state to generate the updated continuous state. The temporal consistency measurement module compares the updated continuous state with the preset reference trajectory and generates a temporal consistency cumulative measurement based on the comparison result. The temporal consistency cumulative measurement is used to quantify the degree of fit between the updated continuous state and the reference trajectory along the time axis. The strategy parameter adjustment module adjusts the strategy parameters of the generated action signals based on the cumulative temporal consistency metric, so that the updated continuous state output by the implicit intervention signals generated in subsequent rounds drives the dynamic evolution process to approach the reference trajectory.

Images