A method for constructing a dataset for multimodal identity recognition

Abstract

The present invention relates to the technical field of data processing, and discloses a data set construction method for multimodal identity recognition, comprising the following steps: collecting multimodal data; generating plane data based on the multimodal data; inputting the plane data into a classification model, and the classification model outputs results representing classifications of all patients; inputting the plane data and the patient classification data into an evaluation model, and the evaluation model outputs results representing rehabilitation evaluations of all patients; inputting the plane data, the patient classification data and the rehabilitation evaluation data into a rehabilitation model, and the rehabilitation model outputs results representing rehabilitation plans of all patients; the present invention obtains the classification data of the patients by analyzing the multimodal data of different patients, thereby evaluating the rehabilitation effects of the patients, improving the effectiveness of the rehabilitation evaluation, and providing rehabilitation plans according to the rehabilitation data and evaluation data of each patient. Compared with only evaluating the rehabilitation effects of the patients, the method can provide a rehabilitation plan with good rehabilitation effects, low difficulty and low risk.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more specifically, to a method for constructing a data set for multimodal identity recognition. Background Art

[0002] The demands on rehabilitation medicine are increasing day by day. Objectively and accurately evaluating the rehabilitation effects of patients is crucial to formulating scientific rehabilitation treatment plans. After constructing a multimodal data set, deep learning technology is used to integrate multi-source heterogeneous data from different patients, including the patient's movement data, physiological data, psychological data, and environmental data, so as to judge the rehabilitation effects of different patients.

[0003] However, there are great individual differences among patients, and the evaluation of rehabilitation effects varies greatly among patients of different types, with different severities, and whether complications occur. At the same time, personalized rehabilitation treatment plans should be generated based on the patient's rehabilitation assessment and individual characteristics to help patients recover their physical functions faster and better. Summary of the invention

[0004] The purpose of the present invention is to provide a method for constructing a data set for multimodal identity recognition in order to solve the above-mentioned problem.

[0005] The present invention provides a method for constructing a data set for multimodal identity recognition, comprising the following steps:

[0006] Step 100, collecting multimodal data, which is collected once at each time sampling point, including doctor information, nurse information, and patient information;

[0007] The time sampling point refers to the moment when data is collected at a predetermined time interval;

[0008] Step 200, generating plane data based on multimodal data, including facets for storing data, a serial data stream indexed by a facet to a data object, the plane data further comprising a relationship list for storing association relationships between data objects indexed by the facets, the element value of the i-th row and j-th column of the relationship list representing a quantized value of the association relationship between the data objects indexed by the i-th and j-th facets;

[0009] The association relationships include high and low, corresponding to two quantitative values;

[0010] Data subjects include doctors, nurses, and patients;

[0011] Data objects with high association relationships are defined as: patients and doctors who provide diagnosis and treatment services, patients and nurses who provide nursing services, patients in the same ward, doctors in the same department, and nurses in the same department;

[0012] A serial data stream No. 1 of a data object includes n No. 1 data units ordered in time, and the t-th No. 1 data unit represents multimodal data collected at the t-th moment of the data object;

[0013] Step 300, inputting the plane data into a classification model, the classification model comprising a first backbone network, the first backbone network comprising a first encoder, a second encoder and a first FC layer, the classification model outputting a result representing the classification of all patients;

[0014] Step 400, inputting the plane data and the patient classification data into an evaluation model, the evaluation model comprising a second backbone network, the second backbone network comprising a third encoder, a fourth encoder, a fifth encoder, a first feature fusion layer and a second FC layer, the evaluation model outputs results representing rehabilitation evaluations of all patients;

[0015] Step 500, input the plane data, patient classification data and rehabilitation assessment data into the rehabilitation model, the rehabilitation model includes a third backbone network, the third backbone network includes a sixth encoder, a seventh encoder, an eighth encoder, a ninth encoder, a second feature fusion layer and a third FC layer, and the rehabilitation model output represents the results of all patient rehabilitation plans.

[0016] Furthermore, in the classification model, a serial data stream No. 1 is input into a first encoder, the first encoder outputs a first identification feature to a second encoder, the second encoder also inputs a relationship list, the second encoder outputs a second identification feature to a first FC layer, and the first FC layer outputs a result representing the classification of all patients.

[0017] Furthermore, the loss function calculation method of the classification model is:

[0018] ;

[0019] in Represents the classification model loss value, represents the total number of classification samples, represents the total number of patient types, The default value is 10, including patients with stroke, spinal cord injury, bone and joint disease, nerve injury, muscle disease, children with developmental disorders, geriatric syndrome, stable cardiopulmonary disease, postoperative recovery, and pain. It is The patient type symbol value of the classification sample, if the The true patient type of the classification sample is The value is 1 for each patient type, otherwise it is 0. is the first The classification samples belong to The probability of each patient type, Indicates the total number of severity levels, The default value is 3, including mild, moderate and severe. It is The severity symbol value of the classification sample, if the The true severity of the classified samples is The severity level is 1, otherwise it is 0. is the first The classification samples belong to The probability of severity, It is The true complication label of a classification sample has a value of 0 or 1, 1 means that the classification sample has a complication, and 0 means that the classification sample has no complication. is the first The probability of complications occurring in the classified samples is Represents the logarithmic function with the natural constant e as the base.

[0020] Furthermore, in the evaluation model, the first serial data stream is input into the third encoder, the third encoder outputs the third identification feature to the first feature fusion layer, the second serial data stream is input into the fourth encoder, the fourth encoder outputs the fourth identification feature to the first feature fusion layer, the first feature fusion layer outputs the first combined feature to the fifth encoder, the fifth encoder also inputs the relationship list, the fifth encoder outputs the fifth identification feature to the second FC layer, and the second FC layer outputs the results of the rehabilitation evaluation of all patients.

[0021] Furthermore, the patient classification data is encoded into a second serial data stream; the second serial data stream includes B second data units sorted by time, and the zth second data unit represents the patient classification output by the classification model at the zth moment.

[0022] Furthermore, the loss function calculation method of the evaluation model is:

[0023] ;

[0024] in Represents the loss value of the evaluation model, represents the total number of evaluation samples, indicates the total number of evaluation types, The default value is 3, including physical function assessment, cognitive ability assessment, and psychological status assessment. Indicates The evaluation sample The true evaluation score value of each evaluation type, Represents the output of the evaluation model The evaluation sample The assessment score value for each assessment type.

[0025] Furthermore, in the rehabilitation model, the serial data stream No. 1 is input into the sixth encoder, the sixth encoder outputs the sixth identification feature to the second feature fusion layer, the serial data stream No. 3 is input into the seventh encoder, the seventh encoder outputs the seventh identification feature to the second feature fusion layer, the serial data stream No. 2 is input into the eighth encoder, the eighth encoder outputs the eighth identification feature to the second feature fusion layer, the second feature fusion layer outputs the second combined feature to the ninth encoder, the ninth encoder also inputs a relationship list, the ninth encoder outputs the ninth identification feature to the third FC layer, and the third FC layer outputs the results representing the rehabilitation plans for all patients.

[0026] Furthermore, the rehabilitation assessment data is encoded into a third serial data stream; the third serial data stream includes A third data units sorted in time, and the yth third data unit represents the rehabilitation assessment data output by the assessment model at the yth moment.

[0027] Furthermore, the training steps of the rehabilitation model are as follows:

[0028] Step 401, copy the rehabilitation model to generate an Actor network and a TargetActor network, use a feedforward neural network as a Critic network, copy the Critic network to obtain a TargetCritic network, and randomly initialize the parameters of the Actor network, the TargetActor network, the Critic network, and the TargetCritic network;

[0029] Step 402, input Hybrid data consisting of multimodal data collected, patient classification data, and rehabilitation assessment data Go to the Actor network to get the Rehabilitation program implemented ;

[0030] No. Rehabilitation program implemented Add random noise to get Noise-adding rehabilitation program ;

[0031] The first Noise-adding rehabilitation program Input into the environment and get immediate rewards from feedback and Mixed data collected ; Experience Deposit into experience pool;

[0032] Step 403, randomly obtain N experiences from the experience pool;

[0033] enter Go to the TargetActor network to get the Rehabilitation program implemented ;

[0034] Step 404, calculate the first loss value and the second loss value , gradient descent method updates the Actor network and Critic network;

[0035] ;

[0036] ;

[0037] ;

[0038] in Represents input and The output of the Critic network is Indicates that and Input the output of the Critic network, Indicates execution The comprehensive Q value when express and The output of the TargetCritic network;

[0039] is the discount factor, which is a value between 0 and 1, used to balance the weights of immediate rewards and future rewards. The default value is 0.6;

[0040] ;

[0041] in, represents the total number of patients, indicates the total number of evaluation types, The default value is 3, including physical function assessment, cognitive ability assessment, and psychological status assessment. and Respectively represent Patient No. Second and After the first rehabilitation program The evaluation score value of each evaluation type, and Respectively, all patients The maximum and minimum values ​​of the evaluation score for each evaluation type. , , Respectively represent the first, second, and third weight coefficients of instant rewards, and Rehabilitation program The difficulty and risk value of and The calculation method is:

[0042] ;

[0043] in, represents the total number of all possible strategies, Indicates The difficulty score of the strategy, is the indicator function, when the rehabilitation program Included in The value is 1 when the strategy is adopted, otherwise it is 0;

[0044] ;

[0045] in, Indicates The risk score of the strategy, Indicates a rehabilitation plan Included in The maximum value of the risk score of the strategy;

[0046] pass Update the Actor network by Update the Critic network;

[0047] Step 405, every fixed number of network updates, update the TargetActor network so that its parameters are the same as those of the current Actor network, and update the TargetCritic network so that its parameters are the same as those of the current Critic network;

[0048] Step 406, when the Actor network and the Critic network converge, the step is terminated, or when the number of cycles of the loop steps 403-405 reaches a set value, the Actor network replaces the rehabilitation model and the step is terminated.

[0049] The present invention provides a computer storage medium for storing computer-readable instructions, which, when read, can execute the aforementioned method for constructing a data set for multimodal identity recognition.

[0050] The beneficial effects of the present invention are as follows: the present invention obtains classification data of patients by analyzing multimodal data of different patients, thereby evaluating the rehabilitation effect of patients, improving the effectiveness of rehabilitation evaluation, and providing rehabilitation plans based on the rehabilitation data and evaluation data of each patient. Compared with only evaluating the rehabilitation effect of patients, this method can provide a rehabilitation plan with good rehabilitation effect, low difficulty and risk. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a flow chart of a method for constructing a data set for multimodal identity recognition of the present invention. DETAILED DESCRIPTION

[0052] The subject matter described herein will now be discussed with reference to example embodiments. It should be understood that the discussion of these embodiments is only to enable those skilled in the art to better understand and implement the subject matter described herein, and the functions and arrangements of the elements discussed may be changed without departing from the scope of protection of the contents of this specification. Each example may omit, replace or add various processes or components as needed. In addition, the features described relative to some examples may also be combined in other examples.

[0053] At least one embodiment of the present invention discloses a method for constructing a data set for multimodal identity recognition, such as Figure 1 As shown, the following steps are included:

[0054] Step 100, collecting multimodal data, which is collected once at each time sampling point, including doctor information, nurse information, and patient information;

[0055] Doctor information includes the doctor’s basic information (doctor ID, name, gender, age, professional title), professional field, contact information, and department to which the doctor belongs;

[0056] Nurse information includes basic information (nurse ID, name, gender, age, job title), professional field, contact information, and department to which they belong;

[0057] Patient information includes basic patient information (patient ID, name, age, gender, medical history, ward), exercise data, physiological data, psychological data, environmental data, and rehabilitation goals;

[0058] The time sampling point refers to the moment when data is collected at a predetermined time interval;

[0059] In one embodiment of the present invention, the default value of the predetermined time interval is 12 hours.

[0060] In one embodiment of the present invention, multimodal data is collected to construct a data set, including collecting doctor information, nurse information, and patient information, and preprocessing and annotating the collected data; the annotated content includes the patient type, severity, complications, physical function assessment, cognitive ability assessment, and psychological state assessment of the patient;

[0061] The dataset is randomly divided into training set, validation set and test set, and the ratio of training set, validation set and test set is 6:2:2.

[0062] Step 200, generating plane data based on multimodal data, including facets for storing data, a serial data stream indexed by a facet to a data object, the plane data further comprising a relationship list for storing association relationships between data objects indexed by the facets, the element value of the i-th row and j-th column of the relationship list representing a quantized value of the association relationship between the data objects indexed by the i-th and j-th facets;

[0063] The association relationship includes high and low, corresponding to two quantitative values ​​respectively.

[0064] Data subjects include doctors, nurses, and patients;

[0065] Data objects with high association relationships are defined as: patients and doctors who provide diagnosis and treatment services, patients and nurses who provide nursing services, patients in the same ward, doctors in the same department, and nurses in the same department;

[0066] A serial data stream No. 1 of a data object includes n No. 1 data units ordered in time, and the t-th No. 1 data unit represents multimodal data collected at the t-th moment of the data object;

[0067] In one embodiment of the present invention, the doctor information, nurse information, and patient information are input into the classification model, the evaluation model, and the rehabilitation model after feature engineering;

[0068] Patient information includes video and text modalities, while doctor and nurse information includes text modalities. For data of various modalities, corresponding feature engineering methods are used to encode and generate features, and then they are spliced.

[0069] The information of video modality is used as a feature engineering method through CNN algorithm;

[0070] The text modality information is used as a feature engineering method through the Word Embedding algorithm.

[0071] Step 300, input the plane data into the classification model, the classification model includes a first backbone network, the first backbone network includes a first encoder, a second encoder and a first FC (fully connected layer) layer, wherein a serial data stream No. 1 is input into the first encoder, the first encoder outputs a first identification feature to the second encoder, the second encoder also inputs a relationship list, the second encoder outputs a second identification feature to the first FC layer, and the first FC layer outputs a result representing the classification of all patients.

[0072] In one embodiment of the present invention, the formula of the first encoder is as follows:

[0073]

[0074] in Represents the plane data The first bin corresponds to the first serial data stream The first identifying feature, Indicates The first bin corresponds to the first serial data stream The first identifying feature, Indicates The first serial data stream of the bin Number one data unit, and are the first and second serial recognition weight matrices of the first encoder, is the serial identification bias parameter of the first encoder, yes Activation function.

[0075] In one embodiment of the present invention, the formula of the second encoder is as follows:

[0076]

[0077] Indicates Layer The second facet feature of the facets, Indicates Layer The second facet feature of the facets, and Respectively represent and and face yuan A collection of facets with high associations. represents the cardinality of a set, Indicates The plane recognition weight matrix of the layer, , Indicates the total number of layers, hour , Indicates The facet index of the object The first identifying feature, represents the total number of data items No. 1 in serial data stream No. 1, hour Equal to the surface element The second identification feature, is the sigmoid function;

[0078] In one embodiment of the present invention, the formula of the first FC layer is as follows:

[0079]

[0080] in represents a patient classification vector, the first The component value represents the The patient type vector, severity vector, and complication vector of each patient. The first patient type vector of the patient The component represents the The patient belongs to The probability of the patient type, The severity vector of the patient The component represents the The patient belongs to The probability of severity, The vector representing whether the patient has complications The probability of a patient developing complications. A probability greater than 0.5 indicates that a complication has occurred, otherwise it indicates that no complication has occurred. Represents the plane data The second identification feature of the facet, Indicates the total number of layers, The set of cells representing all plane data, Indicates full connection.

[0081] In one embodiment of the present invention, the loss function calculation method of the classification model is:

[0082]

[0083] in Represents the classification model loss value, represents the total number of classification samples, represents the total number of patient types, The default value is 10, including patients with stroke, spinal cord injury, bone and joint disease, nerve injury, muscle disease, children with developmental disorders, geriatric syndrome, stable cardiopulmonary disease, postoperative recovery, and pain. It is The patient type symbol value of the classification sample, if the The true patient type of the classification sample is The value is 1 for each patient type, otherwise it is 0. is the first The classification samples belong to The probability of each patient type, Indicates the total number of severity levels, The default value is 3, including mild, moderate and severe. It is The severity symbol value of the classification sample, if the The true severity of the classified samples is The severity level is 1, otherwise it is 0. is the first The classification samples belong to The probability of severity, It is The true complication label of a classification sample has a value of 0 or 1, 1 means that the classification sample has a complication, and 0 means that the classification sample has no complication. is the first The probability of complications occurring in the classified samples is Represents the logarithmic function with the natural constant e as the base.

[0084] Step 400, input the plane data into the evaluation model, the evaluation model includes a second backbone network, the second backbone network includes a third encoder, a fourth encoder, a fifth encoder, a first feature fusion layer and a second FC layer, wherein the first serial data stream is input into the third encoder, the third encoder outputs the third recognition feature to the first feature fusion layer, the second serial data stream is input into the fourth encoder, the fourth encoder outputs the fourth recognition feature to the first feature fusion layer, the first feature fusion layer outputs the first combination feature to the fifth encoder, the fifth encoder also inputs a relationship list, the fifth encoder outputs the fifth recognition feature to the second FC layer, and the second FC layer outputs the results of the rehabilitation evaluation of all patients.

[0085] In one embodiment of the present invention, the patient classification data is encoded into a second serial data stream; the second serial data stream includes B second data units sorted by time, and the zth second data unit represents the patient classification output by the classification model at the zth moment.

[0086] In one embodiment of the present invention, the formula of the third encoder is as follows:

[0087]

[0088] in Represents the plane data The first bin corresponds to the first serial data stream A third identifying feature, Indicates The first bin corresponds to the first serial data stream A third identifying feature, Indicates The first serial data stream of the bin Number one data unit, and are the first and second serial recognition weight matrices of the third encoder, is the serial identification bias parameter of the third encoder, yes Activation function.

[0089] In one embodiment of the present invention, the formula of the fourth encoder is as follows:

[0090]

[0091] in Indicates the first The fourth identifying feature, Indicates the first The fourth identifying feature, Indicates the first The second data unit, and are the first and second serial recognition weight matrices of the fourth encoder, is the serial identification bias parameter of the fourth encoder, yes Activation function.

[0092] In one embodiment of the present invention, the formula of the first feature fusion layer is as follows:

[0093]

[0094] in, Represents the plane data The first combination feature corresponding to the face element, Represents the plane data The first bin corresponds to the first serial data stream A third identifying feature, represents the total number of data items No. 1 in serial data stream No. 1, Indicates the first The fourth identifying feature, represents the total number of data items No. 2 of the No. 2 serial data stream, Represents the feature fusion function (splicing function).

[0095] In one embodiment of the present invention, the formula of the fifth encoder is as follows:

[0096]

[0097]

[0098] in Indicates Layer The fifth aggregation of facets is represented by and Respectively represent Layer and Layer The fifth facet feature of the facets is Representation and Facets A collection of facets with high association relationships; Represents the universal quantifier symbol; Indicates Layer The fifth facet feature of the facets is , Indicates the total number of layers, hour , ,in and Respectively represent and A first combined feature of an object indexed by facets; hour Equal to the surface element The fifth identifying feature, Indicates Aggregation functions of layers, such as average or max pooling functions, Indicates the fifth encoder The first plane of the layer identifies the weight matrix, σ represents the Sigmoid function, Represents the concatenation function.

[0099] In one embodiment of the present invention, the formula of the second FC layer is as follows:

[0100]

[0101] in Represents a rehabilitation assessment vector. The dth component value of the rehabilitation assessment vector represents the rehabilitation assessment of the dth patient. The rehabilitation assessment includes the assessment scores of physical function, cognitive ability, and psychological state. Represents the plane data The fifth identification feature of the facets, Indicates the total number of layers, The set of cells representing all plane data, Indicates full connection.

[0102] In one embodiment of the present invention, the loss function calculation method of the evaluation model is:

[0103]

[0104] in Represents the loss value of the evaluation model, represents the total number of evaluation samples, indicates the total number of evaluation types, The default value is 3, including physical function assessment, cognitive ability assessment, and psychological status assessment. Indicates The evaluation sample The true evaluation score value of each evaluation type, Represents the output of the evaluation model. The evaluation sample The assessment score value for each assessment type.

[0105] Step 500, input the plane data into the rehabilitation model, the rehabilitation model includes a third backbone network, the third backbone network includes a sixth encoder, a seventh encoder, an eighth encoder, a ninth encoder, a second feature fusion layer and a third FC layer, wherein the first serial data stream is input into the sixth encoder, the sixth encoder outputs the sixth recognition feature to the second feature fusion layer, the third serial data stream is input into the seventh encoder, the seventh encoder outputs the seventh recognition feature to the second feature fusion layer, the second serial data stream is input into the eighth encoder, the eighth encoder outputs the eighth recognition feature to the second feature fusion layer, the second feature fusion layer outputs the second combined feature to the ninth encoder, the ninth encoder also inputs a relationship list, the ninth encoder outputs the ninth recognition feature to the third FC layer, and the third FC layer outputs the results representing the rehabilitation plans for all patients.

[0106] In one embodiment of the present invention, the rehabilitation assessment data is encoded into a third serial data stream; the third serial data stream includes A third data units sorted by time, and the yth third data unit represents the rehabilitation assessment data output by the assessment model at the yth moment.

[0107] In one embodiment of the present invention, the formula of the sixth encoder is as follows:

[0108]

[0109]

[0110]

[0111] in Represents the plane data The first bin corresponds to the first serial data stream The sixth identifying feature, Indicates The first bin corresponds to the first serial data stream The sixth identifying feature, and Respectively represent the sixth encoder The first and second intermediate features of the facet, Indicates The first serial data stream of the bin Number one data unit, and are the first and second serial recognition weight matrices of the sixth encoder, is the first serial identification bias parameter of the sixth encoder, is the hyperbolic tangent function, represents the dot product, Represents the Sigmoid function.

[0112] In one embodiment of the present invention, the formula of the seventh encoder is as follows:

[0113]

[0114] in Indicates the third serial data stream The seventh identifying feature, Indicates the third serial data stream The seventh identifying feature, Indicates the third serial data stream The third data unit, and are the first and second serial recognition weight matrices of the seventh encoder, is the serial identification bias parameter of the seventh encoder, yes Activation function.

[0115] In one embodiment of the present invention, the formula of the eighth encoder is as follows:

[0116]

[0117] in Indicates the first The eighth identifying feature, Indicates the first The eighth identifying feature, Indicates the first The second data unit, and are the first and second serial recognition weight matrices of the eighth encoder, is the serial identification bias parameter of the eighth encoder, yes Activation function.

[0118] In one embodiment of the present invention, the formula of the second feature fusion layer is as follows:

[0119]

[0120] in, Represents the plane data The second combination feature corresponding to the face element, Represents the plane data The first bin corresponds to the first serial data stream The sixth identifying feature, represents the total number of data items No. 1 in serial data stream No. 1, Indicates the third serial data stream The fourth identifying feature, represents the total number of data items No. 3 of serial data stream No. 3, Indicates the first The eighth identifying feature, represents the total number of data items No. 2 of the No. 2 serial data stream, Represents the feature fusion function (splicing function).

[0121] In one embodiment of the present invention, the formula of the ninth encoder is as follows:

[0122]

[0123] in Indicates Layer The ninth facet feature of the facets is Representation and Facets A collection of facets with high associations. Indicates the ninth encoder The first plane identification weight matrix of the layer, It is Layer Opposite element The attention weight is calculated by the following formula:

[0124]

[0125] in is a learnable attention vector, represents the vector concatenation operation, Indicates the ninth encoder The second plane identification weight matrix of the layer, represents transpose;

[0126] and Respectively represent Layer The facet and The fifth facet feature of the facets is , Indicates the total number of layers, hour , , hour Equal to the surface element The ninth identification feature of and Respectively represent and The second combined feature of the object indexed by the facets.

[0127] In one embodiment of the present invention, the formula of the third FC layer is as follows:

[0128]

[0129] in Represents a rehabilitation program vector. The Gth component value of the rehabilitation program vector represents the probability value of the Gth strategy. The strategy with the largest probability value is selected as the output. The strategy group contains all executable strategies. A strategy includes rehabilitation training, rehabilitation education, psychological intervention, and assistive device measures. Represents the plane data The ninth identification feature of the facets, Indicates the total number of layers, The set of cells representing all plane data, Indicates full connection.

[0130] In one embodiment of the present invention, the training steps of the rehabilitation model are as follows:

[0131] Step 401, copy the rehabilitation model to generate an Actor network and a TargetActor network, use a feedforward neural network as a Critic network, copy the Critic network to obtain a TargetCritic network, and randomly initialize the parameters of the Actor network, the TargetActor network, the Critic network, and the TargetCritic network;

[0132] Step 402, input Hybrid data consisting of multimodal data collected, patient classification data, and rehabilitation assessment data Go to the Actor network to get the Rehabilitation program implemented ;

[0133] No. Rehabilitation program implemented Add random noise to get Noise-adding rehabilitation program ;

[0134] The first Noise-adding rehabilitation program Input into the environment and get immediate rewards from feedback and Mixed data collected ; Experience Deposit into experience pool;

[0135] Step 403, randomly obtain N experiences from the experience pool;

[0136] enter Go to the TargetActor network to get the Rehabilitation program implemented ;

[0137] Step 404, calculate the first loss value and the second loss value , gradient descent method updates the Actor network and Critic network;

[0138]

[0139]

[0140]

[0141] in Represents input and The output of the Critic network is Indicates that and Input the output of the Critic network, Indicates execution The comprehensive Q value when express and The output of the TargetCritic network;

[0142] is the discount factor, which is a value between 0 and 1, used to balance the weights of immediate rewards and future rewards. The default value is 0.6;

[0143]

[0144] in, represents the total number of patients, indicates the total number of evaluation types, The default value is 3, including physical function assessment, cognitive ability assessment, and psychological status assessment. and Respectively represent Patient No. Second and After the first rehabilitation program The evaluation score value of each evaluation type, and Respectively, all patients The maximum and minimum values ​​of the evaluation score for each evaluation type. , , Respectively represent the first, second, and third weight coefficients of instant rewards, and Rehabilitation program Difficulty and risk values;

[0145] pass Update the Actor network by Update the Critic network;

[0146] Step 405, every fixed number of network updates, update the TargetActor network so that its parameters are the same as those of the current Actor network, and update the TargetCritic network so that its parameters are the same as those of the current Critic network;

[0147] Step 406, when the Actor network and the Critic network converge, the step is terminated, or when the number of cycles of the loop steps 403-405 reaches a set value, the Actor network replaces the rehabilitation model and the step is terminated.

[0148] In one embodiment of the present invention, and The calculation method is:

[0149]

[0150] in, represents the total number of all possible strategies, Indicates The difficulty score of the strategy, is the indicator function, when the rehabilitation program Included in The value is 1 when the strategy is adopted, otherwise it is 0;

[0151]

[0152] in, Indicates The risk score of the strategy, Indicates a rehabilitation plan Included in The maximum risk score of the strategy.

[0153] In one embodiment of the present invention, the classification model, the evaluation model and the rehabilitation model are trained using the training set, the model is verified and the parameters are adjusted using the validation set during the training process, and the model performance is evaluated on the test set;

[0154] The training goal of the classification model is to minimize the loss function of the classification model , the training goal of the evaluation model is to minimize the loss function of the evaluation model , the training goal of the rehabilitation model is to maximize the immediate reward ;

[0155] The classification model and evaluation model use the Adam optimizer, with the initial learning rate set to 0.001, and the learning rate is multiplied by 0.5 for decay in each training cycle;

[0156] The training method of the rehabilitation model is performed according to steps 401-406;

[0157] When the classification accuracy of the classification model on the validation set and the mean absolute error of the evaluation model on the validation set do not improve after reaching the set training threshold during the training cycle, the training is terminated.

[0158] Evaluate the classification accuracy of the classification model on the test set, the mean absolute error of the evaluation model, and the average instant reward of the rehabilitation model , as the final performance indicator of the model.

[0159] In at least one embodiment of the present invention, a computer storage medium is provided for storing computer-readable instructions, which, when read, can execute the aforementioned method for constructing a data set for multimodal identity recognition.

[0160] The above describes an embodiment of the present invention, but this embodiment is not limited to the above-mentioned specific implementation mode. The above-mentioned specific implementation mode is merely illustrative and not restrictive. Under the guidance of this embodiment, ordinary technicians in this field can also make more forms of equivalent embodiments, all of which are within the protection of this embodiment.

Claims

1. A method for constructing a dataset for multimodal identity recognition, characterized in that: The following steps are involved: Step 100, collecting multimodal data, which is collected once at each time sampling point, including doctor information, nurse information, and patient information; The time sampling point refers to the moment when data is collected at a predetermined time interval; Step 200, generating plane data based on multimodal data, including facets for storing data, a serial data stream indexed by a facet to a data object, the plane data further comprising a relationship list for storing association relationships between data objects indexed by the facets, the element value of the i-th row and j-th column of the relationship list representing a quantized value of the association relationship between the data objects indexed by the i-th and j-th facets; The association relationships include high and low, corresponding to two quantitative values; Data subjects include doctors, nurses, and patients; Data objects with high association relationships are defined as: patients and doctors who provide diagnosis and treatment services, patients and nurses who provide nursing services, patients in the same ward, doctors in the same department, and nurses in the same department; A serial data stream No. 1 of a data object includes n No. 1 data units ordered in time, and the t-th No. 1 data unit represents multimodal data collected at the t-th moment of the data object; Step 300, inputting the plane data into a classification model, the classification model comprising a first backbone network, the first backbone network comprising a first encoder, a second encoder and a first FC layer, the classification model outputting a result representing the classification of all patients; Step 400, inputting the plane data and the patient classification data into an evaluation model, the evaluation model comprising a second backbone network, the second backbone network comprising a third encoder, a fourth encoder, a fifth encoder, a first feature fusion layer and a second FC layer, the evaluation model outputs results representing rehabilitation evaluations of all patients; Step 500, input the plane data, patient classification data and rehabilitation assessment data into the rehabilitation model, the rehabilitation model includes a third backbone network, the third backbone network includes a sixth encoder, a seventh encoder, an eighth encoder, a ninth encoder, a second feature fusion layer and a third FC layer, and the rehabilitation model output represents the results of all patient rehabilitation plans.

2. The method for constructing a data set for multimodal identity recognition according to claim 1, characterized in that: In the classification model, a serial data stream No. 1 is input into the first encoder, the first encoder outputs a first identification feature to the second encoder, the second encoder also inputs a relationship list, the second encoder outputs a second identification feature to the first FC layer, and the first FC layer outputs a result representing the classification of all patients.

3. The method for constructing a data set for multimodal identity recognition according to claim 2, characterized in that: The loss function calculation method of the classification model is: ； in Represents the classification model loss value, represents the total number of classification samples, represents the total number of patient types, The default value is 10, including patients with stroke, spinal cord injury, bone and joint disease, nerve injury, muscle disease, children with developmental disorders, geriatric syndrome, stable cardiopulmonary disease, postoperative recovery, and pain. It is The patient type symbol value of the classification sample. If The true patient type of the classification sample is The value is 1 for each patient type, otherwise it is 0. is the first The classification samples belong to The probability of each patient type, Indicates the total number of severity levels, The default value is 3, including mild, moderate and severe. It is The severity symbol value of the classification sample, if the The true severity of the classified samples is The value is 1 for each severity level, otherwise it is 0. is the first The classification samples belong to The probability of severity, It is The true complication label of a classification sample has a value of 0 or 1, 1 means that the classification sample has a complication, and 0 means that the classification sample has no complication. is the first The probability of complications occurring in the classified samples is Represents the logarithmic function with the natural constant e as the base.

4. The method for constructing a data set for multimodal identity recognition according to claim 1, characterized in that: In the evaluation model, serial data stream No. 1 is input into the third encoder, the third encoder outputs the third identification feature to the first feature fusion layer, serial data stream No. 2 is input into the fourth encoder, the fourth encoder outputs the fourth identification feature to the first feature fusion layer, the first feature fusion layer outputs the first combination feature to the fifth encoder, the fifth encoder also inputs the relationship list, the fifth encoder outputs the fifth identification feature to the second FC layer, and the second FC layer outputs the results of the rehabilitation evaluation of all patients.

5. The method for constructing a data set for multimodal identity recognition according to claim 4, characterized in that: The patient classification data is encoded into a second serial data stream; the second serial data stream includes B second data units sorted by time, and the zth second data unit represents the patient classification output by the classification model at the zth moment.

6. The method for constructing a data set for multimodal identity recognition according to claim 4, characterized in that: The loss function calculation method for the evaluation model is: ； in Represents the loss value of the evaluation model, represents the total number of evaluation samples, indicates the total number of evaluation types, The default value is 3, including physical function assessment, cognitive ability assessment, and psychological status assessment. Indicates The evaluation sample The true evaluation score value of each evaluation type, Represents the output of the evaluation model. The evaluation sample The assessment score value for each assessment type.

7. The method for constructing a data set for multimodal identity recognition according to claim 1, characterized in that: In the rehabilitation model, the first serial data stream is input into the sixth encoder, the sixth encoder outputs the sixth identification feature to the second feature fusion layer, the third serial data stream is input into the seventh encoder, the seventh encoder outputs the seventh identification feature to the second feature fusion layer, the second serial data stream is input into the eighth encoder, the eighth encoder outputs the eighth identification feature to the second feature fusion layer, the second feature fusion layer outputs the second combined feature to the ninth encoder, the ninth encoder also inputs a relationship list, the ninth encoder outputs the ninth identification feature to the third FC layer, and the third FC layer outputs the results representing the rehabilitation plans for all patients.

8. The method for constructing a data set for multimodal identity recognition according to claim 7, characterized in that: The rehabilitation assessment data is encoded into a No. 3 serial data stream; the No. 3 serial data stream includes A No. 3 data units sorted by time, and the yth No. 3 data unit represents the rehabilitation assessment data output by the assessment model at the yth moment.

9. The method for constructing a data set for multimodal identity recognition according to claim 7, characterized in that: The training steps of the rehabilitation model are as follows: Step 401, copy the rehabilitation model to generate an Actor network and a TargetActor network, use a feedforward neural network as a Critic network, copy the Critic network to obtain a TargetCritic network, and randomly initialize the parameters of the Actor network, the TargetActor network, the Critic network, and the TargetCritic network; Step 402, input Hybrid data consisting of multimodal data collected, patient classification data, and rehabilitation assessment data Go to the Actor network to get the Rehabilitation program implemented ; No. Rehabilitation program implemented Add random noise to get Noise-adding rehabilitation program ; The first Noise-adding rehabilitation program Input into the environment and get immediate rewards from feedback and Mixed data collected ; Experience Deposit into experience pool; Step 403, randomly obtain N experiences from the experience pool; enter Go to the TargetActor network to get the Rehabilitation program implemented ; Step 404, calculate the first loss value and the second loss value , gradient descent method updates the Actor network and Critic network; ； ； ； in Represents input and The output of the Critic network is Indicates that and Input the output of the Critic network, Indicates execution The comprehensive Q value when express and The output of the TargetCritic network; is the discount factor, which is a value between 0 and 1, used to balance the weights of immediate rewards and future rewards. The default value is 0.6; ； in, represents the total number of patients, indicates the total number of evaluation types, The default value is 3, including physical function assessment, cognitive ability assessment, and psychological status assessment. and Respectively represent Patient No. Second and After the first rehabilitation program The evaluation score value of each evaluation type, and Respectively, all patients The maximum and minimum values ​​of the evaluation score for each evaluation type. , , Respectively represent the first, second, and third weight coefficients of instant rewards, and Rehabilitation program The difficulty and risk value of and The calculation method is: ； in, represents the total number of all possible strategies, Indicates The difficulty score of the strategy, is the indicator function, when the rehabilitation program Included in The value is 1 when the strategy is adopted, otherwise it is 0; ； in, Indicates The risk score of the strategy, Indicates a rehabilitation plan Included in The maximum value of the risk score of the strategy; pass Update the Actor network by Update the Critic network; Step 405, every fixed number of network updates, update the TargetActor network so that its parameters are the same as those of the current Actor network, and update the TargetCritic network so that its parameters are the same as those of the current Critic network; Step 406, when the Actor network and the Critic network converge, the step is terminated, or when the number of cycles of the loop steps 403-405 reaches a set value, the Actor network replaces the rehabilitation model and the step is terminated.

10. A computer storage medium, characterized in that: It is used to store computer-readable instructions, which, when read, can execute a method for constructing a data set for multimodal identity recognition as described in any one of claims 1-9.

Images