A pediatric nursing quality assessment method based on multimodal data interaction

Abstract

This invention relates to the field of medical and nursing technology, and in particular to a method for assessing the quality of pediatric nursing care based on multimodal data interaction. The method includes multimodal data acquisition, data preprocessing, nursing behavior feature analysis, infant status assessment, and nursing quality scoring and feedback. This approach collects multi-source heterogeneous data during the nursing process and preprocesses and extracts features from image, audio, and text multimodal data using ResNet50, GloVe, and Mel-frequency cepstral coefficients respectively, improving the completeness and efficiency of feature fusion. This approach employs a multi-granularity feature interaction fusion method to extract and fuse key features from multiple dimensions during the nursing process, comprehensively acquiring nursing behavior features and infant status features, thereby significantly improving the accuracy and intelligence of nursing quality assessment and enabling real-time monitoring and scientific feedback support for the nursing process.

Description

Technical Field

[0001] This invention relates to the field of medical and nursing technology, and in particular to a method for assessing the quality of pediatric care based on multimodal data interaction. Background Technology

[0002] With the development of artificial intelligence, the Internet of Things, and big data analytics, the field of pediatric care is gradually evolving towards intelligence and digitalization. Traditional nursing quality assessment mainly relies on manual observation and recording, which suffers from problems such as strong subjectivity, low efficiency, and delayed feedback, making it difficult to meet the needs of modern high-quality nursing services. Nursing processes generate various types of data, which come from different sources, have diverse formats, and are not synchronized in time, making unified analysis and utilization difficult. Nursing staff may perform non-standard operations when executing nursing procedures, but the lack of effective monitoring and timely correction mechanisms affects nursing quality and infant safety. Summary of the Invention

[0003] To address the aforementioned issues and overcome the shortcomings of existing technologies, this invention provides a method for assessing the quality of pediatric care based on multimodal data interaction. Addressing the problem that various types of data are generated during the nursing process, with different sources, formats, and time synchronizations, making unified analysis and utilization difficult, this solution collects multi-source heterogeneous data during the nursing process. It preprocesses and extracts features from image, audio, and text multimodal data using ResNet50, GloVe, and Mel-frequency cepstral coefficients respectively, achieving time synchronization and structured representation, thereby improving the completeness of feature fusion and analytical efficiency. Furthermore, addressing the issue of potential non-standard operations by nursing staff during nursing procedures, lack of effective monitoring and timely correction mechanisms, which affect nursing quality and infant safety, this solution employs a multi-granularity feature interaction fusion method. This method extracts and fuses key features from multiple dimensions during the nursing process, comprehensively acquiring nursing behavioral characteristics and infant state characteristics, thereby significantly improving the accuracy and intelligence level of nursing quality assessment and enabling real-time monitoring and scientific feedback support for the nursing process.

[0004] The technical solution adopted by this invention is as follows: This invention provides a method for assessing the quality of pediatric care based on multimodal data interaction, the method comprising the following steps:

[0005] Step S1: Multimodal data acquisition, acquiring multi-source data during the nursing process, including video image data, audio data, physiological data collected by wearable devices, nursing log text data, and environmental sensor data;

[0006] Step S2: Data preprocessing, cleaning, synchronizing and normalizing multi-source data, extracting features from multi-source data, constructing multimodal feature vectors, and classifying multimodal feature vectors into nursing feature vectors and infant feature vectors;

[0007] Step S3: Nursing behavior feature analysis. The nursing feature vector in the multimodal feature vector is fused using a multi-granularity feature interaction fusion method, and the fused features are classified to obtain nursing behavior features.

[0008] Step S4: Infant status assessment. The infant feature vector is analyzed using a multi-granularity feature interaction fusion method. The current status of the infant is judged by combining physiological indicators, facial expressions, and crying features to obtain the infant status features.

[0009] Step S5: Nursing quality scoring and feedback. Establish a nursing quality assessment model, conduct a comprehensive analysis of nursing behavior characteristics and infant condition characteristics, output a comprehensive nursing quality score, and generate a visual report.

[0010] Furthermore, in step S2, feature extraction is performed on the multi-source data to construct a multimodal feature vector, specifically including the following steps:

[0011] Step S21: Image feature extraction. The video image data during the nursing process is processed into image data with uniform width, height and number of channels. The ResNet50 model is used to extract feature information from the image data, and a fully connected layer is used to obtain the final image features.

[0012] Step S22: Text feature extraction. The pre-trained GloVe model is used to convert the words in the nursing log text data in the nursing process into d-dimensional word vectors and unify the text length to K words. Text features are obtained through a fully connected layer.

[0013] Step S23: Audio feature extraction. Mel frequency cepstral coefficients are used to extract features from the audio data during the nursing process to obtain audio features.

[0014] Step S24: Feature mapping. Image features, text features, and audio features are used as the original features. An interaction control matrix is ​​constructed using the original features, and the original features are mapped to the same space to obtain mapped features. The mapping features from different modalities are aligned and fused using the interaction control matrix to construct a multimodal feature vector. The formula used is as follows:

[0015] ;

[0016] ;

[0017] ;

[0018] In the formula, It is the sigmoid activation function. and For trainable weights, and For trainable bias, Indicates original features, For interactive control matrix, For mapping features, Represents a multimodal feature vector;

[0019] Furthermore, in step S3, a multi-granularity feature interaction fusion method is used to perform feature interaction fusion on the nursing feature vector in the multimodal feature vector, including the following steps:

[0020] Step S31: Interaction feature fusion, constructing a feature interaction module, capturing interaction features at different granularities, and obtaining multi-granularity interaction features through the concat operation;

[0021] The construction of the feature interaction module, which captures interaction features at different granularities, specifically includes the following steps:

[0022] Step S311: Feature similarity calculation. Calculate the similarity scores between text, audio, and image features in the nursing feature vector. The formula used is as follows:

[0023] ;

[0024] In the formula, This represents the similarity score from text to image. Represents the image feature vector. To represent the initial features of the text, similarly, calculate the similarity between text and audio, audio and text, audio and image, image and text, and image and audio.

[0025] Step S312: Calculate the alternative output features. Calculate the features in the nursing feature vector that can be used for alternative outputs to obtain the alternative output features. The formula used is as follows:

[0026] ;

[0027] In the formula, Indicates alternative output features, The weight matrix is ​​the linear transformation matrix. Representing image features, text features, and audio features. For bias vectors, It is the hyperbolic tangent activation function;

[0028] Step S313: Calculate the attention weighting vector using the following formula:

[0029] ;

[0030] In the formula, Weighted vectors for text attention. Similarly, using the attention weighting function, we calculate the attention weighting vectors for image and audio features to obtain the image attention weighting vector and the audio attention weighting vector.

[0031] Step S314: Interact the text attention weighted vector, image attention weighted vector, and audio attention weighted vector at different granularities, and fuse the interaction features at different granularities through the concat operation to obtain multi-granularity interaction features;

[0032] Step S32: Reduce redundant information and optimize redundant information in multi-granularity interaction features. The formula used is as follows:

[0033] ;

[0034] In the formula, Indicates interactive fusion features, This represents the Sigmoid activation function. Indicates original features, and Let represent the trainable weight matrix and bias vector, respectively. Represents multi-granularity interaction features;

[0035] Step S33: Feature fusion. The multi-granularity interaction features obtained are integrated using a fully connected layer to obtain the interaction fusion features. The formula used is as follows:

[0036] ;

[0037] In the formula, As an interactive fusion feature, For multi-granularity interaction features, and These represent the trainable weight matrix and bias vector, respectively.

[0038] Step S34: Dynamic filtering. The interactive fusion features are dynamically filtered and classified through the gating unit to retain the feature information most relevant to the nursing quality score, thereby obtaining nursing behavior features.

[0039] The beneficial effects achieved by the present invention using the above solution are as follows:

[0040] (1) In response to the problem that various types of data are generated during the nursing process, and that these data have different sources, diverse formats, and asynchronous time, making it difficult to analyze and utilize them in a unified manner, this solution collects multi-source heterogeneous data during the nursing process, and preprocesses and extracts features from multimodal data of images, audio, and text using ResNet50 model, Glove model, and Mel frequency cepstral coefficients, respectively, and maps the extracted features to the same space to achieve time synchronization and structured representation, thereby improving the integrity of feature fusion and analysis efficiency;

[0041] (2) In response to the problem that nursing staff may perform non-standard operations and lack effective monitoring and timely correction mechanisms when performing nursing procedures, which affects nursing quality and infant safety, this solution adopts a multi-granularity feature interaction fusion method to extract and fuse key features in the nursing process from multiple dimensions, comprehensively obtain nursing behavior features and infant status features, thereby significantly improving the accuracy and intelligence level of nursing quality assessment, and realizing real-time monitoring and scientific feedback support for the nursing process. Attached Figure Description

[0042] Figure 1 This is a flowchart illustrating a method for assessing the quality of pediatric care based on multimodal data interaction proposed in this invention.

[0043] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. Detailed Implementation

[0044] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0045] Example 1, see Figure 1 This invention provides a method for assessing the quality of pediatric care based on multimodal data interaction, the method comprising the following steps:

[0046] Step S1: Multimodal data acquisition, acquiring multi-source data during the nursing process, including video image data, audio data, physiological data collected by wearable devices, nursing log text data, and environmental sensor data;

[0047] Step S2: Data preprocessing, cleaning, synchronizing and normalizing multi-source data, extracting features from multi-source data, constructing multimodal feature vectors, and classifying multimodal feature vectors into nursing feature vectors and infant feature vectors;

[0048] Step S3: Nursing behavior feature analysis. The nursing feature vector in the multimodal feature vector is fused using a multi-granularity feature interaction fusion method, and the fused features are classified to obtain nursing behavior features.

[0049] Step S4: Infant status assessment. The infant feature vector is analyzed using a multi-granularity feature interaction fusion method. The current status of the infant is judged by combining physiological indicators, facial expressions, and crying features to obtain the infant status features.

[0050] Step S5: Nursing quality scoring and feedback. Establish a nursing quality assessment model, conduct a comprehensive analysis of nursing behavior characteristics and infant condition characteristics, output a comprehensive nursing quality score, and generate a visual report.

[0051] Example 2, based on the above example, involves feature extraction from multi-source data and the construction of a multimodal feature vector in step S2, specifically including the following steps:

[0052] Step S21: Image feature extraction. The video image data during the nursing process is processed into image data with uniform width, height and number of channels. The ResNet50 model is used to extract feature information from the image data, and a fully connected layer is used to obtain the final image features.

[0053] Step S22: Text feature extraction. The pre-trained GloVe model is used to convert the words in the nursing log text data in the nursing process into d-dimensional word vectors and unify the text length to K words. Text features are obtained through a fully connected layer.

[0054] Step S23: Audio feature extraction. Mel frequency cepstral coefficients are used to extract features from the audio data during the nursing process to obtain audio features.

[0055] Step S24: Feature mapping. Image features, text features, and audio features are used as the original features. An interaction control matrix is ​​constructed using the original features, and the original features are mapped to the same space to obtain mapped features. The mapping features from different modalities are aligned and fused using the interaction control matrix to construct a multimodal feature vector. The formula used is as follows:

[0056] ;

[0057] ;

[0058] ;

[0059] In the formula, It is the sigmoid activation function. and For trainable weights, and For trainable bias, Indicates original features, For interactive control matrix, For the mapped features, This represents a multimodal feature vector.

[0060] By performing the aforementioned operations, this solution addresses the problem of various types of data generated during the nursing process. These data come from different sources, have diverse formats, and are not synchronized in time, making them difficult to analyze and utilize uniformly. This solution collects multi-source heterogeneous data from the nursing process and preprocesses and extracts features from multimodal data of images, audio, and text using ResNet50, GloVe, and Mel-frequency cepstral coefficients, respectively. The extracted features are then mapped to the same space to achieve time synchronization and structured representation, thereby improving the completeness of feature fusion and the efficiency of analysis.

[0061] Example 3, based on the above examples, includes the following steps in step S22 for text feature extraction:

[0062] Step S221: Word vector representation. Use a pre-trained GloVe model to represent all words as d-dimensional word vectors, forming a word vector dictionary;

[0063] Step S222: Text length normalization and embedding. Set the text length to K words. If the length exceeds K words, truncate it. If the length is less than K words, pad it with zeros. Use an embedding layer to represent each input text as a matrix.

[0064] Step S223: Multi-channel feature extraction, using BiLSTM and self-attention mechanism to obtain rich text features, the formula used is as follows:

[0065] ;

[0066] ;

[0067] In the formula, The text is in matrix form. For indexing, , and For trainable parameters, Indicates will The result obtained after inputting into a bidirectional LSTM This represents a bidirectional LSTM model. For the output of the self-attention mechanism, This is a self-attention mechanism;

[0068] Step S24: Multi-head attention mechanism, sum all text features and obtain global information features through multi-head attention mechanism, and obtain the final text features through a fully connected layer.

[0069] Example 4, based on the above examples, in step S3, uses a multi-granularity feature interaction fusion method to perform feature interaction fusion on the nursing feature vector in the multimodal feature vector, including the following steps:

[0070] Step S31: Interaction feature fusion, constructing a feature interaction module, capturing interaction features at different granularities, and obtaining multi-granularity interaction features through the concat operation;

[0071] Step S32: Reduce redundant information and optimize redundant information in multi-granularity interaction features. The formula used is as follows:

[0072] ;

[0073] In the formula, Indicates interactive fusion features, This represents the Sigmoid activation function. Indicates original features, and Let represent the trainable weight matrix and bias vector, respectively. Represents multi-granularity interaction features;

[0074] Step S33: Feature fusion. The multi-granularity interaction features obtained are integrated using a fully connected layer to obtain the interaction fusion features. The formula used is as follows:

[0075] ;

[0076] In the formula, As an interactive fusion feature, For multi-granularity interaction features, and These represent the trainable weight matrix and bias vector, respectively.

[0077] Step S34: Dynamic filtering. The interactive fusion features are dynamically filtered and classified through the gating unit to retain the feature information most relevant to the nursing quality score, thereby obtaining nursing behavior features.

[0078] Example 5, based on the above examples, describes the construction of a feature interaction module to capture interaction features at different granularities, specifically including the following steps:

[0079] Step S311: Feature similarity calculation. Calculate the similarity scores between text, audio, and image features in the nursing feature vector. The formula used is as follows:

[0080] ;

[0081] In the formula, This represents the similarity score from text to image. Represents the image feature vector. To represent the initial features of the text, similarly, calculate the similarity between text and audio, audio and text, audio and image, image and text, and image and audio.

[0082] Step S312: Calculate the alternative output features. Calculate the features in the nursing feature vector that can be used for alternative outputs to obtain the alternative output features. The formula used is as follows:

[0083] ;

[0084] In the formula, Indicates alternative output features, The weight matrix is ​​the linear transformation matrix. Representing image features, text features, and audio features. For bias vectors, It is the hyperbolic tangent activation function;

[0085] Step S313: Calculate the attention weighting vector using the following formula:

[0086] ;

[0087] In the formula, Weighted vectors for text attention. Similarly, using the attention weighting function, we calculate the attention weighting vectors for image and audio features to obtain the image attention weighting vector and the audio attention weighting vector.

[0088] Step S314: Interact the text attention weighted vector, image attention weighted vector, and audio attention weighted vector at different granularities, and fuse the interaction features at different granularities through the concat operation to obtain multi-granularity interaction features.

[0089] Example 6: Based on the above examples, in step S4, the infant state assessment process uses the same processing method as the care feature vector to process the infant feature vector, and extracts the infant state features through classification operations; identifying whether the infant is in a state of pleasure, crying, anger, drowsiness, etc.

[0090] The nursing behavior characteristics obtained in step S3 include actions, operation sequence and attitudes during processes such as feeding, diaper changing, soothing, and bathing; the infant state characteristics include infant facial features, crying type features, limb movement features and physiological state features.

[0091] Example 7: Based on the above examples, in step S5, the nursing quality scoring and feedback analyzes the correlation between nursing behavior characteristics and infant state characteristics, and uses context modeling to construct an interpretable nursing quality assessment model in conjunction with the standard specifications of the nursing process. Based on the correlation between characteristics, the compliance, timeliness, and gentleness of nursing behaviors are scored respectively, and the analytic hierarchy process is used to construct a judgment matrix, calculate the weights of different characteristics, and finally calculate the comprehensive nursing quality score by weighted average.

[0092] By performing the aforementioned operations, this solution addresses the issue of non-standard operations by nursing staff during nursing procedures, the lack of effective monitoring and timely correction mechanisms, which affect nursing quality and infant safety. This solution employs a multi-granularity feature interaction fusion method to extract and fuse key features in the nursing process from multiple dimensions, comprehensively acquiring nursing behavior features and infant state features. This significantly improves the accuracy and intelligence level of nursing quality assessment, enabling real-time monitoring and scientific feedback support for the nursing process.

[0093] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0094] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

[0095] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1. A method for assessing the quality of pediatric care based on multimodal data interaction, characterized in that: The method includes the following steps: Step S1: Multimodal data acquisition, acquiring multi-source data during the nursing process, including video image data, audio data, physiological data collected by wearable devices, nursing log text data, and environmental sensor data; Step S2: Data preprocessing, cleaning, synchronizing and normalizing multi-source data, extracting features from multi-source data, constructing multimodal feature vectors, and classifying multimodal feature vectors into nursing feature vectors and infant feature vectors; Step S3: Nursing behavior feature analysis. The nursing feature vector in the multimodal feature vector is fused using a multi-granularity feature interaction fusion method, and the fused features are classified to obtain nursing behavior features. Step S4: Infant status assessment. The infant feature vector is analyzed using a multi-granularity feature interaction fusion method. The current status of the infant is judged by combining physiological indicators, facial expressions, and crying features to obtain the infant status features. Step S5: Nursing quality scoring and feedback. Establish a nursing quality assessment model, comprehensively analyze nursing behavior characteristics and infant condition characteristics, output a comprehensive nursing quality score, and generate a visual report. In step S2, feature extraction is performed on the multi-source data to construct a multimodal feature vector, which specifically includes the following steps: Step S21: Image feature extraction. The video image data during the nursing process is processed into image data with uniform width, height and number of channels. The ResNet50 model is used to extract feature information from the image data, and a fully connected layer is used to obtain the final image features. Step S22: Text feature extraction. The pre-trained GloVe model is used to convert the words in the nursing log text data in the nursing process into d-dimensional word vectors and unify the text length to K words. Text features are obtained through a fully connected layer. Step S23: Audio feature extraction. Mel frequency cepstral coefficients are used to extract features from the audio data during the nursing process to obtain audio features. Step S24: Feature mapping. Image features, text features, and audio features are used as original features. An interaction control matrix is ​​constructed using the original features, and the original features are mapped to the same space to obtain mapped features. The mapping features in different modalities are aligned and fused using the interaction control matrix to construct a multimodal feature vector. In step S3, a multi-granularity feature interaction fusion method is used to perform feature interaction fusion on the nursing feature vector in the multimodal feature vector, including the following steps: Step S31: Interaction feature fusion, constructing a feature interaction module, capturing interaction features at different granularities, and obtaining multi-granularity interaction features through the concat operation; Step S32: Reduce redundant information and optimize redundant information in multi-granularity interaction features. The formula used is as follows: ； In the formula, Indicates interactive fusion features, This represents the Sigmoid activation function. Indicates original features, and Let represent the trainable weight matrix and bias vector, respectively. Represents multi-granularity interaction features; Step S33: Feature fusion. The multi-granularity interaction features obtained are integrated using a fully connected layer to obtain the interaction fusion features. The formula used is as follows: ； In the formula, As an interactive fusion feature, For multi-granularity interaction features, and These represent the trainable weight matrix and bias vector, respectively. Step S34: Dynamic filtering. The interactive fusion features are dynamically filtered through the gating unit to retain the feature information most relevant to the nursing quality score, thereby obtaining nursing behavior features.

Images