Pediatric nursing quality evaluation method based on multi-modal data interaction

Abstract

The invention relates to the technical field of medical care, in particular to a pediatric care quality evaluation method based on multi-modal data interaction. The method comprises the steps of multi-modal data acquisition, data preprocessing, nursing behavior feature analysis, infant state evaluation, and nursing quality scoring and feedback. According to the scheme, the multi-source heterogeneous data in the nursing process is collected, the ResNet50 model, the Glove model and the Mel-frequency cepstral coefficient are used for preprocessing and feature extraction of the multi-modal data of the image, the audio and the text, and the integrity and the analysis efficiency of feature fusion are improved; according to the scheme, a multi-granularity feature interactive fusion method is adopted, key features in the nursing process are extracted and fused from multiple dimensions, and the nursing behavior features and the infant state features are comprehensively obtained, so that the accuracy and the intelligent level of nursing quality evaluation are remarkably improved, and real-time monitoring and scientific feedback support of the nursing process are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical care, and particularly to a method for evaluating the quality of pediatric care based on multi-modal data interaction. Background Art

[0002] With the development of artificial intelligence, Internet of Things, and big data analysis technologies, the field of child care has gradually evolved towards intelligence and digitization. Traditional nursing quality assessment mainly relies on manual observation and recording, which has problems such as strong subjectivity, low efficiency, and lagging feedback, and it is difficult to meet the needs of modern high-quality nursing services; various types of data are generated during the nursing process, these data have different sources, diverse formats, and are out of sync in time, making it difficult to uniformly analyze and utilize; when nursing staff execute the nursing process, there may be non-standard operations, but there is a lack of effective monitoring and timely correction mechanisms, which affect the quality of nursing and the safety of infants. Summary of the Invention

[0003] In view of the above situation, to overcome the defects of the prior art, the present invention provides a method for evaluating the quality of pediatric care based on multi-modal data interaction. For the problem that various types of data are generated during the nursing process, these data have different sources, diverse formats, and are out of sync in time, making it difficult to uniformly analyze and utilize, this solution collects multi-source heterogeneous data during the nursing process, and respectively preprocesses and extracts features from multi-modal data of images, audio, and text using the ResNet50 model, Glove model, and Mel-frequency cepstral coefficients to achieve time synchronization and structured representation, thereby improving the integrity of feature fusion and analysis efficiency; for the problem that when nursing staff execute the nursing process, there may be non-standard operations, and there is a lack of effective monitoring and timely correction mechanisms, which affect the quality of nursing and the safety of infants, this solution adopts a multi-granularity feature interaction and fusion method to extract and fuse key features in the nursing process from multiple dimensions, comprehensively obtain nursing behavior features and infant state features, thereby significantly improving the accuracy and intelligence level of nursing quality assessment, and realizing real-time monitoring and scientific feedback support for the nursing process.

[0004] The technical solution adopted by the present invention is as follows: A method for evaluating the quality of pediatric care based on multi-modal data interaction provided by the present invention, the method includes the following steps:

[0005] Step S1: Multi-modal data collection, obtaining multi-source data during the nursing process, including video image data, audio data, physiological data collected by wearable devices, nursing log text data, and environmental sensor data;

[0006] Step S2: Data preprocessing, cleaning, synchronizing, and normalizing the multi-source data, and extracting features from the multi-source data to construct a multi-modal feature vector, and classifying the multi-modal feature vector into a nursing feature vector and an infant feature vector;

[0007] Step S3: Analysis of nursing behavior characteristics. Use the multi-granularity feature interaction and fusion method to perform feature interaction and fusion on the nursing feature vectors in the multi-modal feature vectors, and classify the fused features to obtain nursing behavior characteristics;

[0008] Step S4: Infant state assessment. Use the multi-granularity feature interaction and fusion method to analyze the infant feature vectors, comprehensively judge the current state of the infant based on physiological indicators, facial expressions, and cry characteristics, and obtain infant state characteristics;

[0009] Step S5: Nursing quality scoring and feedback. Establish a nursing quality assessment model, comprehensively analyze the nursing behavior characteristics and infant state characteristics, output the comprehensive nursing quality score, and generate a visual report.

[0010] Furthermore, in step S2, perform feature extraction on the multi-source data to construct a multi-modal feature vector, which specifically includes the following steps:

[0011] Step S21: Image feature extraction. Process the video image data during the nursing process into image data with a unified width, height, and number of channels. Use the ResNet50 model to extract feature information from the image data, and use a fully connected layer to obtain the final image features;

[0012] Step S22: Text feature extraction. Use the pre-trained Glove model to convert the words in the nursing log text data during the nursing process into d-dimensional word vectors and unify the text length to K words, and obtain text features through a fully connected layer;

[0013] Step S23: Audio feature extraction. Use the Mel-frequency cepstral coefficients to extract features from the audio data during the nursing process to obtain audio features;

[0014] Step S24: Feature mapping. Take the image features, text features, and audio features as the original features, use the original features to construct an interaction control matrix, and map the original features into the same space to obtain the mapped features. Use the mutual control matrix to align and fuse the mapped features in different modalities to construct a multi-modal feature vector. The formula used is as follows: ; ; ;

[0015] In the formula, is the sigmoid activation function, and are trainable weights, and are trainable biases, Represents the original feature, is the interaction control matrix, is the mapping feature, represents the multi-modal feature vector;

[0016] Furthermore, in step S3, a multi-granularity feature interaction fusion method is used to perform feature interaction fusion on the care feature vector in the multi-modal feature vector, including the following steps:

[0017] Step S31: Interaction feature fusion, constructing a feature interaction module to capture interaction features at different granularities, and obtaining multi-granularity interaction features through a concat operation;

[0018] The construction of the feature interaction module to capture interaction features at different granularities specifically includes the following steps:

[0019] Step S311: Feature similarity calculation, calculating the similarity scores between text, audio, and image features in the care feature vector. The formula used is as follows: ;

[0020] In the formula, represents the similarity score from text to image, represents the image feature vector, represents the preliminary feature of the text. Similarly, calculate the similarities from text to audio, audio to text, audio to image, image to text, and image to audio;

[0021] Step S312: Calculate the alternative output feature, calculating the features in the care feature vector that can be alternatively output to obtain the alternative output feature. The formula used is as follows: ;

[0022] In the formula, represents the alternative output feature, is the weight matrix of the linear transformation, represents the image feature, text feature, and audio feature, is the bias vector, is the hyperbolic tangent activation function;

[0023] Step S313: Calculate the attention weighted vector. The formula used is as follows: ;

[0024] In the formula, is the text attention weighted vector, is the attention weighted function. Similarly, calculate the attention weighted vectors of image and audio features to obtain the image attention weighted vector and the audio attention weighted vector;

[0025] Step S314: Interact the text attention weighted vector, the image attention weighted vector, and the audio attention weighted vector at different granularities, and fuse the interaction features at different granularities through the concat operation to obtain multi-granularity interaction features;

[0026] Step S32: Reduce redundant information and optimize the redundant information in the multi-granularity interaction features. The formula used is as follows: ;

[0027] In the formula, represents the interaction fusion feature, represents the Sigmoid activation function, represents the original feature, and respectively represent the trainable weight matrix and bias vector, represents the multi-granularity interaction feature;

[0028] Step S33: Fuse features. Use a fully connected layer to integrate the obtained multi-granularity interaction features to obtain the interaction fusion feature. The formula used is as follows: ;

[0029] In the formula, is the interaction fusion feature, is the multi-granularity interaction feature, and respectively represent the trainable weight matrix and bias vector;

[0030] Step S34: Dynamically screen. Dynamically screen and classify the interaction fusion feature through a gated unit, and retain the feature information most relevant to the nursing quality score to obtain the nursing behavior feature.

[0031] The beneficial effects achieved by the present invention using the above solution are as follows:

[0032] (1) Aiming at the problem that multiple types of data are generated during the nursing process, these data are from different sources, have diverse formats, and are out of sync in time, making it difficult to uniformly analyze and utilize. This solution collects multi-source heterogeneous data during the nursing process, and respectively preprocesses and extracts features from multi-modal data of images, audio, and text using the ResNet50 model, the Glove model, and the Mel-frequency cepstral coefficients, and maps the extracted features into the same space to achieve time synchronization and structured representation, thereby improving the integrity of feature fusion and the analysis efficiency;

[0033] (2)Regarding the problem that nursing staff may have non-standard operations during the implementation of the nursing process, lacking effective monitoring and timely correction mechanisms, which affect the nursing quality and infant safety, this solution adopts a multi-granularity feature interaction and fusion method to extract and fuse key features in the nursing process from multiple dimensions, comprehensively obtain nursing behavior features and infant state features, thereby significantly improving the accuracy and intelligence level of nursing quality assessment, and realizing real-time monitoring and scientific feedback support for the nursing process. Brief Description of the Drawings

[0034] Figure 1 It is a schematic flowchart of a method for evaluating the quality of pediatric nursing based on multi-modal data interaction proposed by the present invention;

[0035] The drawings are used to provide a further understanding of the present invention, and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the present invention, and do not constitute a limitation to the present invention. Detailed Embodiments

[0036] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work belong to the scope of protection of the present invention.

[0037] Embodiment 1, refer to Figure 1 , a method for evaluating the quality of pediatric nursing based on multi-modal data interaction provided by the present invention, the method includes the following steps:

[0038] Step S1: Multi-modal data collection, obtaining multi-source data in the nursing process, including video image data, audio data, physiological data collected by wearable devices, nursing log text data, and environmental sensor data;

[0039] Step S2: Data preprocessing, cleaning, synchronizing, and normalizing the multi-source data, extracting features from the multi-source data, constructing a multi-modal feature vector, and classifying the multi-modal feature vector into a nursing feature vector and an infant feature vector;

[0040] Step S3: Nursing behavior feature analysis, using the multi-granularity feature interaction and fusion method to perform feature interaction and fusion on the nursing feature vector in the multi-modal feature vector, and classifying the fused features to obtain nursing behavior features;

[0041] Step S4: Infant state assessment, using the multi-granularity feature interaction and fusion method to analyze the infant feature vector, comprehensively judging the current state of the infant based on physiological indicators, facial expressions, and cry features, and obtaining infant state features;

[0042] Step S5: Nursing quality scoring and feedback. Establish a nursing quality assessment model, comprehensively analyze nursing behavior characteristics and infant status characteristics, output a comprehensive nursing quality score, and generate a visualization report.

[0043] Example 2. Based on the above example, in step S2, perform feature extraction on multi-source data to construct a multi-modal feature vector, which specifically includes the following steps:

[0044] Step S21: Image feature extraction. Process the video image data during the nursing process into image data with a unified width, height, and number of channels. Use the ResNet50 model to extract feature information from the image data, and use a fully connected layer to obtain the final image features;

[0045] Step S22: Text feature extraction. Use the pre-trained Glove model to convert the words in the nursing log text data during the nursing process into d-dimensional word vectors and unify the text length to K words, and obtain text features through a fully connected layer;

[0046] Step S23: Audio feature extraction. Use Mel-frequency cepstral coefficients to extract features from the audio data during the nursing process to obtain audio features;

[0047] Step S24: Feature mapping. Take the image features, text features, and audio features as the original features, use the original features to construct an interaction control matrix, and map the original features into the same space to obtain the mapped features. Use the mutual control matrix to align and fuse the mapped features in different modalities to construct a multi-modal feature vector. The formula used is as follows: ; ; ;

[0048] In the formula, is the sigmoid activation function, and are trainable weights, and are trainable biases, represents the original feature, is the interaction control matrix, is the mapped feature, represents the multi-modal feature vector.

[0049] By performing the above operations, in view of the problem that various types of data are generated during the nursing process, these data are from different sources, have diverse formats, and are out of sync in time, making it difficult to unify analysis and utilization. This solution collects multi-source heterogeneous data during the nursing process, and respectively uses the ResNet50 model, the Glove model, and the Mel-frequency cepstral coefficients to preprocess and extract features from multi-modal data of images, audio, and text, and maps the extracted features to the same space to achieve time synchronization and structured representation, thereby improving the integrity of feature fusion and analysis efficiency.

[0050] Embodiment 3. Based on the above embodiment, in step S22, text feature extraction specifically includes the following steps:

[0051] Step S221: Word vector representation. Use the pre-trained Glove model to represent all words as d-dimensional word vectors to form a word vector dictionary;

[0052] Step S222: Text length normalization and embedding. Set the text length to K words, truncate if it exceeds, and pad with zeros if it is insufficient, and use the embedding layer to represent each input text in matrix form;

[0053] Step S223: Multi-channel feature extraction. Adopt BiLSTM and self-attention mechanism to obtain rich text features. The used formula is as follows: ; ;

[0054] In the formula, is the matrix form of the text, is the index, , and are trainable parameters, represents the result obtained by inputting into the bidirectional LSTM, represents the bidirectional LSTM model, is the output of the self-attention mechanism, is the self-attention mechanism;

[0055] Step S24: Multi-head attention mechanism. Add all the text features and obtain global information features through the multi-head attention mechanism, and obtain the final text features through a fully connected layer.

[0056] Embodiment 4. Based on the above embodiment, in step S3, use the multi-granularity feature interaction fusion method to perform feature interaction fusion on the nursing feature vectors in the multi-modal feature vectors, including the following steps:

[0057] Step S31: Interaction feature fusion, constructing a feature interaction module to capture interaction features at different granularities, and obtain multi-granularity interaction features through concat operation;

[0058] Step S32: Reduce redundant information and optimize the redundant information in the multi-granularity interaction features. The formula used is as follows: ;

[0059] Where, represents interactive fusion features, represents the Sigmoid activation function, represents the original features, and Represent the trainable weight matrix and bias vector respectively, Represents multi-granularity interaction features;

[0060] Step S33: Fusion features: Use a fully connected layer to integrate the obtained multi-granularity interaction features to obtain interactive fusion features. The formula used is as follows: ;

[0061] Where, For interactive fusion features, is a multi-granularity interaction feature, and Represent the trainable weight matrix and bias vector respectively;

[0062] Step S34: Dynamic screening: Dynamically screen and classify the interactive fusion features through the gating unit, retain the feature information most relevant to the nursing quality score, and obtain the nursing behavior characteristics.

[0063] Example 5: Based on the above example, this example constructs a feature interaction module to capture interaction features at different granularities, specifically including the following steps:

[0064] Step S311: Feature similarity calculation, calculating the similarity scores between the text, audio, and image features in the nursing feature vector. The formula used is as follows: ;

[0065] Where, represents the similarity score from text to image, represents the image feature vector, Represent the preliminary features of the text. Similarly, calculate the similarity between text to audio, audio to text, audio to image, image to text, and image to audio;

[0066] Step S312: Calculate the alternative output features. Calculate the features in the nursing feature vector that can be used for alternative output to obtain the alternative output features. The formula used is as follows: ;

[0067] In the formula, represents the alternative output features, is the weight matrix of the linear transformation, represents the image features, text features, and audio features, is the bias vector, is the hyperbolic tangent activation function;

[0068] Step S313: Calculate the attention weighted vector. The formula used is as follows: ;

[0069] In the formula, is the text attention weighted vector, is the attention weighted function. Similarly, calculate the attention weighted vectors of the image and audio features to obtain the image attention weighted vector and the audio attention weighted vector;

[0070] Step S314: Interact the text attention weighted vector, the image attention weighted vector, and the audio attention weighted vector at different granularities, and fuse the interaction features at different granularities through the concat operation to obtain the multi-granularity interaction features.

[0071] Example Six: This example is based on the above example. In step S4, the baby state evaluation process processes the baby feature vector using the same processing method as the nursing feature vector, and extracts the baby state features through a classification operation; identify whether the baby is in a happy, crying, angry, sleepy, etc. state

[0072] The nursing behavior features obtained in step S3 include actions, operation sequences, and attitudes during processes such as feeding, diaper changing, soothing, and bathing; the baby state features include baby facial features, cry type features, limb movement features, and physiological state features.

[0073] Example Seven: This example is based on the above example. In step S5, for the nursing quality scoring and feedback, perform a correlation analysis on the nursing behavior features and the baby state features, and use the context modeling method to construct an interpretable nursing quality assessment model in combination with the standard specifications of the nursing process. According to the correlation relationship between the features, score the compliance, timeliness, gentleness, etc. dimensions of the nursing behavior respectively, and use the analytic hierarchy process to construct a judgment matrix to calculate the weights of different features, and finally calculate the comprehensive nursing quality score through weighted calculation.

[0074] By performing the above operations, aiming at the problems that there may be irregular operations, lack of effective monitoring and timely correction mechanisms when nursing staff execute the nursing process, which affect the nursing quality and the safety of infants, this solution adopts a multi-granularity feature interaction and fusion method to extract and fuse key features in the nursing process from multiple dimensions, comprehensively obtain nursing behavior features and infant state features, thereby significantly improving the accuracy and intelligent level of nursing quality assessment, and realizing real-time monitoring and scientific feedback support for the nursing process.

[0075] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device.

[0076] Although the embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

[0077] The above describes the present invention and its implementation manners. This description is not restrictive. What is shown in the drawings is only one of the implementation manners of the present invention, and the actual structure is not limited thereto. Generally speaking, if those of ordinary skill in the art are inspired by it and design similar structural manners and embodiments without creative efforts without departing from the purpose of the present invention, they shall fall within the protection scope of the present invention.

Claims

1. A method for evaluating the quality of pediatric care based on multi-modal data interaction, characterized in that: The method includes the following steps: Step S1: Multimodal data collection, obtaining multi-source data during the nursing process, including video image data, audio data, physiological data collected by wearable devices, nursing log text data, and environmental sensor data; Step S2: Data preprocessing, cleaning, synchronizing, and normalizing the multi-source data, extracting features from the multi-source data, constructing a multimodal feature vector, and classifying the multimodal feature vector into a nursing feature vector and an infant feature vector; Step S3: Nursing behavior feature analysis, using a multi-granularity feature interaction fusion method to perform feature interaction fusion on the nursing feature vector in the multimodal feature vector, and classifying the fused features to obtain nursing behavior features; Step S4: Infant state assessment, using a multi-granularity feature interaction fusion method to analyze the infant feature vector, comprehensively judging the current state of the infant based on physiological indicators, facial expressions, and cry features, and obtaining infant state features; Step S5: Nursing quality scoring and feedback, establishing a nursing quality assessment model, comprehensively analyzing the nursing behavior features and infant state features, outputting a comprehensive nursing quality score, and generating a visual report.

2. The method for evaluating the quality of pediatric care based on multi-modal data interaction according to claim 1, wherein: In step S2, features are extracted from the multi-source data to construct a multimodal feature vector, which specifically includes the following steps: Step S21: Image feature extraction, processing the video image data during the nursing process into image data with a unified width, height, and number of channels, using the ResNet50 model to extract feature information from the image data, and obtaining the final image features using a fully connected layer; Step S22: Text feature extraction, using the pre-trained Glove model to convert the words in the nursing log text data during the nursing process into d-dimensional word vectors and unifying the text length to K words, and obtaining text features through a fully connected layer; Step S23: Audio feature extraction, using Mel-frequency cepstral coefficients to extract features from the audio data during the nursing process to obtain audio features; Step S24: Feature mapping, taking the image features, text features, and audio features as original features, constructing an interaction control matrix using the original features, mapping the original features into the same space to obtain mapped features, and aligning and fusing the mapped features in different modalities using the mutual control matrix to construct a multimodal feature vector.

3. The pediatric nursing quality assessment method based on multi-modal data interaction according to claim 1, characterized in that: In step S3, a multi-granularity feature interaction fusion method is used to perform feature interaction fusion on the nursing feature vector in the multimodal feature vector, including the following steps: Step S31: Interaction feature fusion, constructing a feature interaction module to capture interaction features at different granularities, and obtaining multi-granularity interaction features through a concat operation; Step S32: Reducing redundant information, optimizing the redundant information in the multi-granularity interaction features, and the formula used is as follows: ； In the formula, represents the interactive fusion feature, represents the Sigmoid activation function, represents the original feature, and respectively represent the trainable weight matrix and bias vector, represents the multi-granularity interactive feature; Step S33: Fusing features, using a fully connected layer to integrate the obtained multi-granularity interaction features to obtain interaction fusion features, and the formula used is as follows: ； In the formula, is the interactive fusion feature, is the multi-granularity interaction feature, and respectively represent the trainable weight matrix and the bias vector; Step S34: Dynamic screening, dynamically screening the interaction fusion features through a gating unit, retaining the feature information most relevant to the nursing quality score, and obtaining nursing behavior features.

Images