Nursing quality management and control system based on multi-modal data

Abstract

The invention provides a nursing quality management and control system based on multi-modal data, relates to the field of intelligent nursing, and adopts a data processing technology based on artificial intelligence to perform comprehensive analysis on a set described by acquired nursing feedback texts and an environment state image of a target inpatient area. Therefore, nursing feedback text association semantics related to a plurality of target patient objects and environment state characteristics of the target inpatient area are captured, and nursing quality management suggestions are intelligently generated based on remarkable interaction representation information between two cross-modal data including the nursing feedback semantics and the environment state of the inpatient area. Therefore, more accurate and effective nursing quality management suggestions can be provided, and more scientific and reasonable nursing quality management can be realized.

Description

Technical Field

[0001] The present application relates to the field of intelligent nursing, and more specifically, to a nursing quality control system based on multimodal data. Background Art

[0002] With advances in medical technology and rising patient expectations for healthcare, nursing quality has become a core concern for medical institutions. Effective nursing quality management not only improves patient satisfaction but also optimizes the allocation of medical resources and enhances the overall level of medical services. However, traditional nursing records are primarily stored in paper form across various departments, resulting in fragmented data that is easily lost or damaged. Furthermore, traditional nursing quality management models rely primarily on manual inspections, regular evaluations, and patient feedback. These methods are often time-consuming, labor-intensive, and inefficient, making it difficult to reflect the actual conditions in wards and changes in patient needs in real time. Efficiently and accurately managing nursing quality has become a pressing issue, especially in large hospitals or medical centers.

[0003] Therefore, an optimized nursing quality control system based on multimodal data is desired. Summary of the Invention

[0004] In order to solve the above technical problems, the present application is proposed. The embodiments of the present application provide a nursing quality control system based on multimodal data.

[0005] According to one aspect of the present application, a nursing quality control system based on multimodal data is provided, which includes: a patient nursing feedback module, which is used to collect nursing feedback text information of multiple target patient objects to obtain a set of nursing feedback text descriptions; a ward monitoring module, which is used to obtain a target ward environmental status image captured by a high-definition camera in real time; a nursing-related data transmission module, which is used to transmit the set of nursing feedback text descriptions and the target ward environmental status image to a nursing quality management server; a nursing quality management module, which is used to perform, on the nursing quality management server, a deep analysis of the set of nursing feedback text descriptions and the target ward environmental status image based on cross-domain joint interactive coding to obtain nursing quality management suggestions; and a nursing quality management suggestion display module, which is used to transmit the nursing quality management suggestions to a terminal device of a nursing manager for display.

[0006] Compared with the existing technology, the multimodal data-based nursing quality control system provided by this application uses artificial intelligence-based data processing technology to comprehensively analyze the collection of acquired nursing feedback text descriptions and the target ward environmental status image, thereby capturing the associated semantics of nursing feedback texts related to multiple target patient objects and the environmental status characteristics of the target ward. Based on the significant interactive representation information between the two cross-modal data of nursing feedback semantics and ward environmental status, it intelligently generates nursing quality management suggestions. In this way, more accurate and effective nursing quality management suggestions can be provided, which is conducive to achieving more scientific and reasonable nursing quality management. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings: Figure 1 4 is a system block diagram of a nursing quality control system based on multimodal data according to an embodiment of the present application.

[0008] Figure 2 4 is a block diagram of a nursing quality management module in a nursing quality control system based on multimodal data according to an embodiment of the present application.

[0009] Figure 3 This is a block diagram of a nursing feedback text description semantic encoding unit in a nursing quality control system based on multimodal data according to an embodiment of the present application.

[0010] Figure 4 This is a block diagram of a nursing quality multimodal information interaction representation unit in a nursing quality control system based on multimodal data according to an embodiment of the present application. DETAILED DESCRIPTION

[0011] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described herein.

[0012] With the continuous advancement of medical technology and patients' increasing expectations for medical services, nursing quality has become a key concern for medical institutions. Scientific and effective nursing quality management not only enhances patient satisfaction but also rationally allocates medical resources, thereby improving the overall level of medical services. However, traditional nursing records are currently stored primarily in paper form across various departments, resulting in fragmented data that is prone to loss or damage. Furthermore, traditional nursing quality management methods rely primarily on manual inspections, regular evaluations, and patient feedback. These methods are often time-consuming and labor-intensive, with low efficiency and difficulty in promptly reflecting the actual conditions of wards and changes in patient needs. Especially in large hospitals or medical centers, achieving efficient and accurate management of nursing quality has become a critical issue that needs to be addressed urgently.

[0013] In recent years, the development of artificial intelligence (AI), the Internet of Things (IoT), and 5G communication technologies has provided new solutions for nursing quality management. By integrating high-definition cameras, smart sensors, and other devices, real-time monitoring of ward environments can be achieved, providing more comprehensive and intuitive data. Natural language processing technology can automatically analyze text feedback provided by patients to understand their satisfaction and specific needs. Deep learning algorithms can extract valuable information from large amounts of data, helping managers make scientific and rational decisions.

[0014] Based on this, in the technical solution of this application, a nursing quality control system based on multimodal data is proposed. Figure 1 FIG is a system block diagram of a nursing quality control system based on multimodal data according to an embodiment of the present application. Figure 1 As shown, in the nursing quality control system 100 based on multimodal data, it includes: a patient nursing feedback module 110, which is used to collect nursing feedback text information of multiple target patient objects to obtain a set of nursing feedback text descriptions; a ward monitoring module 120, which is used to obtain the target ward environment status image collected by the high-definition camera in real time monitoring; a nursing-related data transmission module 130, which is used to transmit the set of nursing feedback text descriptions and the target ward environment status image to the nursing quality management server; a nursing quality management module 140, which is used to perform in-depth analysis of the set of nursing feedback text descriptions and the target ward environment status image based on cross-domain joint interactive coding on the nursing quality management server to obtain nursing quality management suggestions; a nursing quality management suggestion display module 150, which is used to transmit the nursing quality management suggestions to the terminal device of the nursing manager for display.

[0015] In an embodiment of the present application, the patient care feedback module 110 is used to collect care feedback text information of multiple target patient objects to obtain a collection of care feedback text descriptions. It should be understood that the care feedback text descriptions of multiple target patient objects will include the service attitudes of caregivers reflected by different patients, such as whether they are patient in solving problems, whether they care about the patient's emotional changes, and will also include an evaluation of the caregivers' professional operations, such as whether they are skilled in giving injections, whether the dressing change process is standardized, etc. In addition, it will also reflect the speed at which caregivers respond to needs and information on ward hygiene needs, item organization, and satisfaction with the catering services provided by the hospital. In general, by analyzing various types of information in the care feedback text descriptions, problems in nursing work can be accurately located, which is conducive to generating targeted care quality management suggestions, thereby improving the overall care quality.

[0016] In an embodiment of the present application, the ward monitoring module 120 is used to obtain the target ward environment status image collected by the high-definition camera in real time. It should be understood that the target ward environment status image obtained in real time contains ward facility status information, sanitary and clean information, etc. Specifically, through the target ward environment status image, it is possible to understand the placement information of various types of equipment in the ward, equipment operation information, and the cleaning information of the ground and object surfaces in the ward. In other words, by analyzing the target ward environment status image, it is possible to understand relevant information on the equipment operation and sanitary environment in the ward. Based on this, targeted nursing quality management suggestions can be made. For example, if the analysis finds that certain areas have been dirty and messy for a long time, the system may suggest increasing the cleaning frequency of the area or adjusting the work distribution of the cleaning staff to improve the cleanliness of the area.

[0017] In an embodiment of the present application, the nursing-related data transmission module 130 is used to transmit the set of nursing feedback text descriptions and the target ward environmental status image to the nursing quality management server. It should be understood that the nursing feedback text descriptions and the target ward environmental status images are important data sources that reflect nursing quality from different dimensions. Uploading them to the nursing quality management server can unify the management and in-depth analysis of scattered data, which helps to examine the nursing quality situation from a global perspective and identify potential problems and improvement points. And because the 5G network has the characteristics of high bandwidth and low latency, this means that both high-definition images and large amounts of text data can be quickly uploaded to the server and processed in a timely manner, which enables the final generated nursing quality management suggestions to reflect the actual situation of the current nursing work more promptly, and help nursing managers solve real-time problems, thereby improving the real-time monitoring and management capabilities of nursing quality.

[0018] In an embodiment of the present application, the nursing quality management module 140 is used to perform an in-depth analysis based on cross-domain joint interactive coding on the set of nursing feedback text descriptions and the target ward environmental status image on the nursing quality management server to obtain nursing quality management suggestions. It should be understood that the nursing feedback text description comes from the patient and reflects the patient's subjective feelings and experience of the nursing service, including the service attitude, professional level, timeliness of nursing operations, etc. of the nursing staff. The ward environmental status image shows the actual environmental conditions of the ward from an objective perspective, such as the cleanliness of the ward, the placement and integrity of the equipment, etc. Through in-depth analysis of these two types of data, the quality of nursing work can be comprehensively evaluated from both subjective and objective dimensions, avoiding the one-sidedness that may arise from relying solely on a single information source for evaluation.

[0019] In particular, in the process of generating corresponding nursing quality management recommendations by deeply analyzing the collection of nursing feedback text descriptions and the target ward environment status images in the nursing quality management server, it is crucial to effectively extract relevant features from the patient nursing feedback text and ward environment images, and effectively fuse the data information from these two different modalities to more accurately understand the patient's needs and ward conditions. This helps ensure that the generated nursing quality management recommendations are both targeted and can quickly respond to changes.

[0020] Based on this, the technical concept of this application is to use artificial intelligence-based image processing and natural language understanding algorithms to comprehensively analyze the collection of nursing feedback text descriptions and the target ward environmental status images, thereby capturing the associated semantics of nursing feedback texts related to multiple target patient objects and the environmental status characteristics of the target ward, and based on the significant interactive representation information between the two cross-modal data of nursing feedback semantics and ward environmental status, perform nursing quality management, and generate corresponding nursing quality management suggestions. In this way, by combining the patient's satisfaction feedback and the ward environmental status information, more accurate and effective nursing quality management suggestions can be automatically provided, which not only helps to improve the efficiency and quality of nursing work, but also helps to enhance the patient's satisfaction and comfort, which is of great significance to promoting the development of smart medical care.

[0021] Figure 2 FIG is a block diagram of a nursing quality management module in a nursing quality control system based on multimodal data according to an embodiment of the present application. Figure 2As shown, the nursing quality management module 140 includes: a nursing feedback text description semantic encoding unit 141, which is used to perform text semantic association encoding on the set of nursing feedback text descriptions to obtain multiple nursing feedback text semantic association encoding features; a ward environment state feature extraction unit 142, which is used to perform feature extraction based on the ward environment state on the target ward environment state image to obtain target ward environment state features; a nursing quality multimodal information interaction representation unit 143, which is used to perform cross-domain joint encoding processing based on modal core features on the multiple nursing feedback text semantic association encoding features and the target ward environment state features to obtain a nursing feedback-ward environment state cross-modal significant interaction representation; a nursing quality management suggestion generation unit 144, which is used to generate the nursing quality management suggestion based on the nursing feedback-ward environment state cross-modal significant interaction representation.

[0022] In the embodiment of the present application, the nursing feedback text description semantic encoding unit 141 is used to perform text semantic association encoding on the set of nursing feedback text descriptions to obtain multiple nursing feedback text semantic association encoding features. Specifically, Figure 3 FIG. 1 is a block diagram of a semantic coding unit for describing nursing feedback text in a nursing quality control system based on multimodal data according to an embodiment of the present application. Figure 3 As shown, the nursing feedback text description semantic encoding unit 141 includes: a nursing feedback text semantic encoding feature generation subunit 1411, which is used to pass each nursing feedback text description in the set of nursing feedback text descriptions through a semantic encoder based on the Bert model to obtain a set of nursing feedback text semantic encoding feature vectors; a nursing feedback text semantic association encoding subunit 1412, which is used to pass the set of nursing feedback text semantic encoding feature vectors through a multiple nursing feedback text semantic association encoder based on LSTM to obtain multiple nursing feedback text semantic association encoding feature vectors as the multiple nursing feedback text semantic association encoding features.

[0023] In an embodiment of the present application, the nursing feedback text semantic coding feature generation subunit 1411 is used to pass each nursing feedback text description in the set of nursing feedback text descriptions through a semantic encoder based on the Bert model to obtain a set of nursing feedback text semantic coding feature vectors. It should be understood that each nursing feedback text description in the set of nursing feedback text descriptions contains a large amount of text content. Through semantic coding, key semantic features can be extracted and irrelevant information can be removed, thereby more efficiently utilizing the useful information in the text. For example, patients may have some emotional expressions or content that is irrelevant to nursing quality in their feedback. Semantic coding can help focus on key content related to nursing quality, such as information on nursing operations, service attitude, etc. Specifically, each nursing feedback text description in the set of nursing feedback text descriptions is semantically encoded through a semantic encoder based on the Bert model to extract word-granularity semantic feature information in each nursing feedback text description, thereby obtaining a set of nursing feedback text semantic coding feature vectors. Those skilled in the art will appreciate that the Bert model utilizes a bidirectional Transformer architecture, which allows it to simultaneously consider contextual information before and after the text. This means it can better understand the meaning of each word based on the other words in the sentence describing the nursing feedback text, helping to capture the nuances of patient feedback. Furthermore, the Bert model is pre-trained based on a large-scale corpus and has already learned a wealth of linguistic knowledge and patterns. When applied to the semantic encoding of nursing feedback text, there is no need to train from scratch. Pre-trained weights and model structures can be leveraged to quickly adapt to specific tasks in the nursing field through fine-tuning and other methods, significantly saving training time and computing resources and improving the model's training efficiency and performance.

[0024] In this embodiment of the present application, the nursing feedback text semantic association encoding subunit 1412 is configured to pass the set of nursing feedback text semantic encoding feature vectors through an LSTM-based multiple nursing feedback text semantic association encoder to obtain multiple nursing feedback text semantic association encoding feature vectors as the multiple nursing feedback text semantic association encoding features. It should be understood that the set of nursing feedback text semantic encoding feature vectors is encoded through the LSTM-based multiple nursing feedback text semantic association encoder to extract the correlation features between the nursing feedback semantics of different patient subjects, thereby obtaining multiple nursing feedback text semantic association encoding feature vectors. This not only considers the nursing feedback content of a single patient subject, but also analyzes the interrelationships between multiple patient feedback. For example, it can help identify whether certain types of feedback frequently appear together, or whether negative feedback indicates subsequent dissatisfaction. This can reveal the commonalities and differences between different patient feedback, providing a basis for improving nursing services. In other words, by learning the correlation features between multiple patient feedback, the system can gain a more comprehensive understanding of the overall needs and preferences of the patient population. This not only helps identify common problems, but also helps identify individual special cases, thereby formulating more precise and effective nursing strategies and service improvement measures.

[0025] The following is a detailed description of a specific implementation process of "passing the set of the semantic encoding feature vectors of the nursing feedback text through the LSTM-based multiple nursing feedback text semantic association encoder to obtain multiple nursing feedback text semantic association encoding feature vectors": Before performing specific encoding operations, thorough data preparation is required. In the previous data processing workflow, a semantic encoder based on the Bert model was used to successfully generate a set of semantically encoded feature vectors for each nursing feedback text. This set serves as the foundational data for subsequent processing, containing the semantic encoding information for each nursing feedback text description. In practical applications, the accuracy and completeness of this set must be ensured, as any missing or incorrect data could affect subsequent analysis results.

[0026] Next, initialize the LSTM model. The performance of an LSTM model depends largely on its parameter settings. First, determine the number of hidden units. This number is crucial, as it determines the expressive power of the LSTM model. If the number of hidden units is too small, the model may not be able to learn the complex semantic relationships between nursing feedback text. Excessive numbers increase the model's computational complexity and training time, and may even lead to overfitting. This number is typically adjusted based on the complexity of the problem and the amount of data. Common values ​​include 64, 128, or 256. In addition to the number of hidden units, the number of layers also requires careful consideration. LSTM networks can be constructed with either single or multi-layers. Multi-layer LSTMs can learn more complex sequential patterns, but this increases training time and computing resource requirements. In practical applications, the optimal number of layers should be determined based on a careful balance of pros and cons. LSTM units typically use sigmoid and tanh activation functions to control information transmission and updates. During initialization, ensure that these activation functions function properly to ensure model accuracy. The weights of the LSTM model also need to be initialized. Common initialization methods include random initialization, such as Xavier initialization or He initialization. These methods can ensure that the model has appropriate initial parameter values ​​at the beginning of training, which helps the model convergence.

[0027] After model initialization, the input data needs to be processed. The set of semantically encoded feature vectors for the nursing feedback text should be considered as a sequence of data, with each feature vector representing a time step in the sequence. The LSTM model processes these time steps sequentially, gradually learning the semantic associations within the sequence. To ensure the LSTM model can correctly process the input data, the set of input feature vectors must be formatted appropriately for its input. Typically, the input to an LSTM model is a three-dimensional tensor with the shape (batch_size, time_steps, input_dim). Here, batch_size represents the number of samples fed into the model at a time, time_steps represents the length of the sequence, i.e., the number of semantically encoded feature vectors for the nursing feedback text, and input_dim represents the dimension of each feature vector. When formatting the input, ensure that the data is properly arranged and dimensioned to ensure smooth input into the LSTM model.

[0028] The forget gate plays a crucial role in the computation of an LSTM unit. At each time step, the LSTM unit first uses the forget gate to determine how much information from the cell state at the previous time step should be forgotten. The forget gate's output is a vector between 0 and 1, calculated using the sigmoid function. Each element of this output vector represents the degree of forgetting of the corresponding information. Values ​​closer to 0 indicate a higher degree of forgetting, while values ​​closer to 1 indicate a higher degree of retention. The forget gate allows the LSTM model to dynamically decide whether to retain or forget the cell state information from the previous time step based on the current input feature vector, thereby avoiding information redundancy and interference. The input gate determines how much new information from the current time step's input should be added to the cell state. The input gate's output is first calculated using the sigmoid function, representing the selectivity of the current input information. The tanh function is then used to create a candidate cell state that incorporates the new information from the current input. The input gate allows the LSTM model to selectively add new information to the cell state, thereby updating the cell state. Updating the cell state is a core operation of the LSTM model. The cell state is updated based on the outputs of the forget and input gates. Specifically, the cell state at the previous time step is multiplied by the output of the forget gate to forget information. The candidate cell state is then multiplied by the output of the input gate to add new information. Finally, the two are added together to obtain the cell state at the current time step. In this way, the LSTM model can effectively retain and update cell state information during sequence processing, thereby learning long-term dependencies within the sequence. The output gate determines how much information in the current cell state is output to the hidden state. The output of the output gate is first calculated using the sigmoid function, representing the degree of output of the cell state information. The cell state is then processed using the tanh function, and the processed result is multiplied by the output of the output gate to obtain the hidden state for the current time step. The hidden state, which contains both the input information of the current time step and historical information from previous time steps, is a key output of the LSTM model.

[0029] After completing the processing of all time steps, the final feature vector needs to be obtained. The hidden state of the last time step can be selected as the semantic association encoding feature vector of multiple nursing feedback texts, because the hidden state of the last time step contains the information of the entire sequence. It is also possible to perform some aggregation operation on the hidden states of all time steps, such as average pooling or maximum pooling, to obtain the final feature vector. Average pooling is to average the corresponding elements of the hidden states of all time steps, while maximum pooling is to take the maximum value of the corresponding elements of the hidden states of all time steps. These aggregation operations can integrate the information of multiple time steps to obtain a more representative feature vector. The final semantic association encoding feature vector of multiple nursing feedback texts will be used for subsequent analysis and processing, such as cross-modal aggregation analysis with the target ward environmental status feature map.

[0030] In order for the LSTM model to accurately learn the semantic associations between nursing feedback text, model training and optimization are necessary. First, an appropriate loss function must be defined. The choice of loss function depends on the requirements of the subsequent task. For classification tasks, the cross-entropy loss function can be used, which measures the difference between the model's predictions and the true labels. For regression tasks, the mean squared error loss function can be used, which measures the error between the model's predictions and the true values. Choosing an appropriate loss function guides model training, allowing model parameters to be adjusted in a way that minimizes the loss function. Furthermore, an optimizer must be selected to update the LSTM model's parameters. Common optimizers include stochastic gradient descent (SGD), Adam, and Adagrad. Different optimizers have different characteristics and applicable scenarios. Stochastic gradient descent is a basic optimization algorithm that updates model parameters in the opposite direction of the gradient by calculating the gradient of the loss function. The Adam optimizer combines momentum and adaptive learning rates to automatically adjust the learning rate during training, improving training efficiency and stability. The Adagrad optimizer adaptively adjusts the learning rate of each parameter based on its historical gradient information. In practical applications, it is necessary to select a suitable optimizer based on the characteristics of the model and the data situation.

[0031] During training, the LSTM model is trained multiple times using labeled training data. During each iteration, the model is fed with input data, the model's output and loss function are calculated, and then the optimizer is used to update the model's parameters based on the gradient of the loss function. This process is repeated until the loss function converges or the preset number of training rounds is reached. To improve the model's generalization and prevent overfitting, techniques such as batch normalization and dropout can be employed. Batch normalization normalizes the input data, accelerating model convergence. Dropout randomly drops a subset of neurons during training to reduce inter-neuron dependencies and improve the model's generalization.

[0032] Through the above series of operations, the set of semantic encoding feature vectors of the nursing feedback text can be passed through the LSTM-based multiple nursing feedback text semantic association encoder to obtain multiple nursing feedback text semantic association encoding feature vectors.

[0033] In this embodiment of the present application, the ward environment state feature extraction unit 142 is configured to perform feature extraction based on the ward environment state on the target ward environment state image to obtain target ward environment state features. Specifically, in this embodiment of the present application, the ward environment state feature extraction unit is configured to pass the target ward environment state image through a ward environment state feature extractor based on a dilated convolutional neural network to obtain a target ward environment state feature map as the target ward environment state feature. It should be understood that in nursing quality management, understanding the actual environmental conditions of the ward is crucial for assessing nursing quality and patient safety. For example, a ward environment state image captured by a high-definition camera may contain information on various aspects, such as ward cleanliness, equipment placement, and accessibility. Traditional convolution operations have a limited receptive field (i.e., the size of the image area that each pixel can see) when extracting image features, which may result in the loss of important contextual information. Dilated convolution, by inserting holes (i.e., intervals not involved in the calculation) in its filter, can expand the receptive field without increasing the number of parameters, thereby capturing a wider range of information while maintaining high resolution. The target ward environment state image is passed through a ward environment state feature extractor based on a dilated convolutional neural network to obtain a target ward environment state feature map. In other words, the feature extractor based on the dilated convolutional neural network can accurately identify these elements of the ward environment state and convert the extracted target ward environment state features into a meaningful feature map, which is very useful for fully understanding the state of the ward environment. It can help the system identify potential safety hazards or non-compliant situations, such as whether emergency exits are blocked or whether there is medical equipment placed in an unregulated manner.

[0034] In the embodiment of the present application, the nursing quality multimodal information interactive representation unit 143 is used to perform cross-domain joint encoding processing based on modal core features on the semantic association coding features of the multiple nursing feedback texts and the target ward environment status features to obtain a nursing feedback-ward environment status cross-modal significant interaction representation. Specifically, Figure 4 FIG. 1 is a block diagram of a multimodal information interactive representation unit for nursing quality in a multimodal data-based nursing quality control system according to an embodiment of the present application. Figure 4 As shown, the nursing quality multimodal information interactive representation unit 143 includes: a nursing feedback text core clue extraction subunit 1431, which is used to extract the core clues of the semantic association encoding feature vectors of the multiple nursing feedback texts based on point convolution encoding to obtain the multiple nursing feedback text semantic core clue encoding vectors; a target ward state core clue extraction subunit 1432, which is used to extract the target ward environment state core clue encoding vector from the target ward environment state feature map; a nursing feedback ward environment state feature association subunit 1433, which is used to extract the target ward environment state core clue encoding vector based on the multiple nursing feedback text semantic core clue encoding feature map. The code vector and the target ward environment state core clue encoding vector are used to construct a core clue weaving template matrix between the nursing feedback and ward environment state modalities; the nursing feedback ward environment state cross-modal significant interaction analysis subunit 1434 is used to perform cross-modal fine-grained feature interaction analysis on the semantic association encoding feature vectors of the multiple nursing feedback texts and the target ward environment state feature map based on the core clue weaving template matrix between the nursing feedback and ward environment state modalities to obtain a nursing feedback-ward environment state cross-modal significant interaction representation vector as the nursing feedback-ward environment state cross-modal significant interaction representation.

[0035] It should be understood that nursing quality includes not only the nursing feedback semantics of the patient object, but also the environmental status monitoring information of the ward. There is a hidden connection and mutual influence between the two, which provides a basis for nursing quality management and suggestion generation. However, due to the different essential attributes of the semantic features of the nursing feedback text and the ward environmental status features, there are significant differences in data form, representation space, information distribution, etc. These differences make direct interaction or joint encoding difficult. Traditional multimodal fusion methods often rely on explicit alignment or simple feature splicing, which makes it difficult to fully capture the deep semantic dependency between different modal data features. Based on this, in the technical solution of the present application, the multiple nursing feedback text semantic association coding features and the target ward environmental status features are further subjected to cross-domain joint encoding processing based on modal core features to obtain a cross-modal significant interaction representation of nursing feedback-ward environmental status. In particular, the cross-domain joint encoding processing based on modal core features aims to achieve joint encoding and multi-level semantic alignment of cross-modal features through the mining and modeling of core information between modalities. In other words, the process of cross-domain joint encoding processing based on modal core features is based on the core clue features of the two modalities, namely the semantic association encoding features of multiple nursing feedback texts and the environmental status features of the target ward. By constructing the correlation template and cross-domain interaction mechanism between modalities, relying on attention mechanism, heterogeneous converter and template constraint learning and other technologies, the semantic relevance and complementarity of multimodal features are integrated to generate a cross-modal significant interaction representation of nursing feedback-ward environmental status with high semantic consistency and significant interactivity.

[0036] Specifically, the semantic association encoding feature vectors of the multiple nursing feedback texts are first subjected to core clue extraction based on point convolution coding to obtain the semantic core clue encoding vectors of the multiple nursing feedback texts. The above process can be expressed as:

[0037] in, is the semantic association encoding feature vector of the multiple nursing feedback texts, It is point convolution processing, is the bias vector, yes activation function, It is the semantic core clue encoding vector of multiple nursing feedback texts.

[0038] It should be understood that the original semantic association encoding feature vectors of multiple nursing feedback texts may contain a large amount of redundant or irrelevant information. In order to focus on the important elements that reflect the essence of nursing quality and help the model quickly grasp the key points and critical issues of nursing work, it is necessary to perform a core clue extraction processing operation based on point convolution coding on the semantic association encoding feature vectors of multiple nursing feedback texts. Through this operation, the original vector can be converted into a more abstract but highly condensed semantic core clue encoding vector of multiple nursing feedback texts. This can present complex nursing feedback information in a more concise and representative form, thereby facilitating subsequent analysis and helping the model quickly understand the core points of the nursing feedback.

[0039] Specifically, in the embodiment of the present application, the target ward state core clue extraction subunit is used to split the target ward environment state feature map to obtain a sequence of target ward environment state local feature maps. This process can be expressed as:

[0040] in, is the target ward environmental state characteristic map, represents feature map splitting, Respectively represent the first, second, and third in the sequence of local feature maps of the target ward environment state and Local characteristic map of the environmental status of each target ward.

[0041] Performing attention-based salient region attention analysis on each target ward environment state local feature map in the sequence of target ward environment state local feature maps to obtain a sequence of salient region attention coefficients. This process can be expressed as:

[0042] in, The first character in the sequence of the local feature map of the target ward environment state Local characteristic map of the target ward environment status, is a convolution layer with a convolution kernel of 3×3, for activation function, For the The local deep semantic feature map of the target ward environment status, To perform global pooling on each feature matrix along the channel dimension in the feature map, For the The local feature vector of the target ward environment state corresponding to the local deep semantic feature map of the target ward environment state, and are the maximum and minimum values ​​in the vector, respectively. 、 and are weight coefficients, for activation function, is the attention coefficient of the significant area.

[0043] Based on the sequence of the salient area attention coefficients, a core clue search is performed on the target ward environment state feature map to obtain the target ward environment state core clue encoding vector. This process can be expressed as:

[0044] in, For the The local feature vector of the target ward environment state corresponding to the local deep semantic feature map of the target ward environment state, is the significant area attention coefficient, is the exponential function value with the natural constant e as the base, is the number of feature maps in the sequence of local feature maps of the target ward environment state, It is the core clue encoding vector of the target ward's environmental status.

[0045] It should be understood that the data characteristics of the nursing feedback text description and the target ward environment status image are significantly different. The semantic association encoding feature vectors of the multiple nursing feedback texts obtained by processing the nursing feedback text descriptions focus more on the semantic level; while the target ward environment status feature map obtained by processing the target ward environment status image carries stronger spatial dimensional information, such as the layout of the ward and the placement of equipment. This difference in data characteristics precludes the use of the same method for processing the semantic association encoding feature vectors of the multiple nursing feedback texts. Therefore, a method specifically suitable for processing spatial information is required, namely, using a regional attention mechanism and convolution operations to process the target ward environment status feature map to obtain the target ward environment status core clue encoding vector. Specifically, by using the regional attention mechanism to reweight the salient areas, it is possible to highlight those parts that are critical to the nursing quality assessment while maintaining the original spatial dimension information of the target ward environmental status feature map. This method enables the system to not only identify the basic situation of the physical environment, but also understand which specific areas or elements are most worthy of attention; and the local receptive field based on convolution operations can deeply explore and enhance the detailed structural information within each small area in the feature map, which means that the system can better understand the specific layout of the ward, equipment placement and other details, thereby providing a basis for formulating specific nursing quality management recommendations.

[0046] Specifically, in this embodiment of the present application, the nursing feedback ward environment state feature association subunit is used to: construct an inter-modal core clue weaving template between the multiple nursing feedback text semantic core clue encoding vectors and the target ward environment state core clue encoding vector to obtain the nursing feedback-ward environment state inter-modal core clue weaving template matrix. The above process can be expressed as:

[0047] in, is the semantic core clue encoding vector of multiple nursing feedback texts, is the core clue encoding vector of the target ward environment state, and are feature mapping functions, such as linear mapping or nonlinear kernel function, represents transpose, It is the core clue weaving template matrix between the nursing feedback and ward environment status modalities.

[0048] It should be understood that the information of the two modalities, the nursing feedback text description and the target ward environment status image, each contains information in different aspects. However, there is an intrinsic connection between the data of these two modalities. For example, a good ward environment may improve the patient's satisfaction with nursing, while the patient's negative feedback on nursing services may imply that there are certain problems with the ward environment. In order to establish an explicit association between the data of these two modalities, so that the two originally independent modal data can be interconnected and complement each other, thereby more comprehensively reflecting the actual situation of nursing quality, it is necessary to construct an inter-modal core clue weaving template between the multiple nursing feedback text semantic core clue encoding vectors and the target ward environment status core clue encoding vectors in this application to obtain the nursing feedback-ward environment status inter-modal core clue weaving template matrix. This template matrix is ​​analogous to the association graph structure established between the two modalities, which clarifies the relationship between the nursing feedback text semantic core clues and the ward environment status core clues. In this graph structure, each core clue can be regarded as a node in the graph, and the edge weight represents the strength of the association between the nodes. This approach allows for a visual display of the interrelationship between the two modal data types, helping the model more clearly understand how various aspects of nursing quality influence each other. Specifically, the core thread weaving template matrix between the nursing feedback and ward environment status modalities integrates data from both nursing feedback and ward environment status, providing a richer and more comprehensive information foundation for generating nursing quality management recommendations. By considering the relationship between the two modal data types, we can avoid analyzing the problem from a single perspective and thus formulate more targeted and comprehensive management recommendations.

[0049] Specifically, in an embodiment of the present application, the nursing feedback ward environment state cross-modal significant interaction analysis subunit is used to: perform feature decoupling along the channel dimension on the target ward environment state feature map to obtain a set of local feature matrices of the target ward environment state. This process can be expressed as:

[0050] in, is the target ward environmental state characteristic map, represents feature decoupling along the channel dimension, Respectively represent each target ward environment state local feature matrix in the sequence of target ward environment state local feature matrices.

[0051] The semantic association encoding feature vectors of the multiple nursing feedback texts are used as query vectors, each target ward environment state local feature matrix in the set of target ward environment state local feature matrices is used as a key matrix, and the core clue weaving template matrix between the nursing feedback and ward environment state modalities is used as a priori information constraint matrix. These are input into a cross-modal template constraint encoder based on a heterogeneous converter to obtain a set of nursing feedback-ward environment state cross-modal fine-grained interaction encoding vectors. The process can be expressed as follows:

[0052] in, It is the core clue weaving template matrix between the nursing feedback and ward environment status modalities. is the semantic association encoding feature vector of the multiple nursing feedback texts, is the first in the sequence of the local feature matrix of the target ward environment state The local feature matrix of the target ward environment state, express The transpose of for The scale of the matrix is ​​the width times the height, is matrix multiplication, express function, The first in the set of cross-modal fine-grained interaction encoding vectors of nursing feedback and ward environment status A cross-modal fine-grained interaction encoding vector of nursing feedback and ward environment status.

[0053] The position-wise mean vector of the set of the nursing feedback-ward environment state cross-modal fine-grained interaction encoding vectors is calculated to obtain the nursing feedback-ward environment state cross-modal significant interaction representation vector. This process can be expressed as:

[0054] in, The first in the set of cross-modal fine-grained interaction encoding vectors of nursing feedback and ward environment status Nursing feedback-ward environment status cross-modal fine-grained interaction encoding vector, is the number of vectors in the set of cross-modal fine-grained interaction encoding vectors of nursing feedback and ward environment status, is the cross-modal significant interaction representation vector between nursing feedback and ward environment status.

[0055] It should be understood that different channels in the target ward environment status feature map may represent different types of information. For example, some channels may be related to the ward's sanitary conditions, while others may be related to the placement of equipment. By decoupling the characteristics of the target ward environment status feature map along the channel dimension, these different types of information can be separated. This allows for more targeted interactive analysis of different local features and the semantic features of the nursing feedback text in subsequent cross-modal template constraint encoding, thus avoiding a decrease in analysis accuracy due to information congestion.

[0056] Accordingly, considering that the textual descriptions of nursing feedback and the target ward environment state images represent different modalities, comprehensive assessment of nursing quality requires effective interaction between the two. Here, by using the semantically associated encoded feature vectors of multiple nursing feedback texts as query vectors and the local feature matrix of the target ward environment state as a key matrix, an attention mechanism is employed to enable the semantic features of the nursing feedback text to actively search for relevant information from the local features of the ward environment state, thereby establishing cross-modal connections and exploring potential relationships between the two modalities. Furthermore, the core clue weaving template matrix between the nursing feedback and ward environment state modalities reflects the global semantic constraints and local fine-grained relationships between the two modalities. Using this as a priori information constraint matrix can provide guidance and standardization for cross-modal interaction. In real-world nursing scenarios, the interaction between two modal data is not arbitrary but rather has inherent logic. For example, patient feedback on the timeliness of care may be related to factors such as the location of the nurse station and the distribution of call devices within the ward environment. The template matrix guides the model to interact between these potentially relevant pieces of information, strengthening the standardization and precision of the interaction and improving the accuracy and reliability of the results. It's important to note that the Heterogeneous Transformer differs from the native Transformer by introducing a template matrix that weaves core clues between the modalities of nursing feedback and ward environment status. This template matrix ensures high coupling and interaction between modal representations, significantly improving the representational capabilities of the results. In other words, the resulting set of fine-grained cross-modal interaction encoding vectors for nursing feedback and ward environment status more comprehensively and accurately represents the complex relationship between the semantics of nursing feedback text and ward environment status, providing a deeper and more accurate information foundation for the subsequent generation of nursing quality management recommendations.

[0057] It should be understood that after completing the cross-modal fine-grained interaction, what is obtained is a set of multiple nursing feedback-ward environment status cross-modal fine-grained interaction coding vectors. The number of vectors in this set is large and relatively complex, which is not conducive to subsequent analysis and processing. In order to use this information more efficiently, it is necessary to further compress and integrate it so as to extract the most representative and critical information. Based on this, in this application, by calculating the positional mean vector of the set of nursing feedback-ward environment status cross-modal fine-grained interaction coding vectors, the key information scattered in each vector is integrated and refined, highlighting the significant features and important information in the cross-modal interaction, so that the final nursing feedback-ward environment status cross-modal significant interaction representation vector can more centrally and concisely represent the cross-modal interaction relationship between nursing feedback and ward environment status. The nursing quality management suggestions generated based on the nursing feedback-ward environment status cross-modal significant interaction representation vector will be more targeted and operational. It can make specific and practical suggestions based on the specific interaction between nursing feedback and ward environmental status, avoiding general and abstract suggestions, and enabling nursing managers to formulate and implement improvement measures more specifically to improve the effectiveness and efficiency of nursing quality management.

[0058] In an embodiment of the present application, the nursing quality management suggestion generation unit 144 is used to generate the nursing quality management suggestion based on the nursing feedback-ward environment status cross-modal significant interaction representation. Specifically, in an embodiment of the present application, the nursing quality management suggestion generation unit is used to: after adding a prompt statement at the end of the nursing feedback-ward environment status cross-modal significant interaction representation vector, input it into the AIGC-based management suggestion generator to obtain the nursing quality management suggestion, wherein the prompt statement is: Generate nursing quality management suggestions based on input corpus. It should be understood that the nursing feedback-ward environment status cross-modal significant interaction representation vector contains a lot of information about nursing services and ward environment, but this information is relatively broad and abstract. Adding the prompt statement "Generate nursing quality management suggestions based on input corpus" can clearly tell the AIGC model the desired output direction, that is, to generate suggestions related to nursing quality management, and avoid the model generating irrelevant or off-topic content. In this way, by combining patient satisfaction feedback and ward environmental status information, more accurate and effective nursing quality management suggestions can be automatically generated. This not only helps to improve the efficiency and quality of nursing work, but also helps to enhance patient satisfaction and comfort, which is of great significance to promoting the development of smart healthcare.

[0059] The following is a detailed description of a specific implementation process of "adding a prompt statement to the tail of the nursing feedback-ward environment status cross-modal significant interaction representation vector, and then inputting it into the AIGC-based management suggestion generator to obtain the nursing quality management suggestion, wherein the prompt statement is: generate nursing quality management suggestions based on input corpus": The first stage is data preparation. This is the foundation of the entire process. Before executing the step of combining the vector with the prompt statement and inputting it into the AIGC model, it is necessary to ensure that an accurate and complete cross-modal significant interaction representation vector of the nursing feedback and ward environmental status has been obtained through a series of preliminary analyses and processing. This vector integrates the semantic information of the nursing feedback text and the characteristic information of the ward environmental status. Its quality directly affects the effectiveness of the subsequent generated suggestions. In actual operation, the data must be strictly checked and verified to prevent missing or incorrect data. At the same time, because the subsequent processing involves combining with the prompt statement and inputting the AIGC model, the vector also needs to undergo necessary format checks and conversions to make it compatible with the requirements of the subsequent links and ensure the smooth progress of the entire process.

[0060] The next step is to add a prompt. The prompt, "Generate nursing quality management recommendations based on the input corpus," is key to guiding the AIGC model to generate recommendations that meet specific needs. When adding it, it must be precisely placed at the end of the vector representing the cross-modal significant interaction between nursing feedback and ward environment status. Because the AIGC model primarily processes text input, the vector must be converted into a suitable text format according to specific encoding rules before adding the prompt. This process must strictly adhere to the formatting specifications required by the model. Any formatting errors may prevent the model from correctly understanding the input information, thereby affecting the quality of the recommendation generation.

[0061] After adding the prompt statement, the AIGC model configuration and input phase begins. This phase begins by selecting the most suitable AIGC-based management suggestion generator model, such as the GPT series, from among the numerous pre-trained language models available on the market, based on actual needs and data characteristics. When selecting a model, factors such as performance and suitability for the task at hand should be comprehensively considered. Once the model is finalized, it is loaded into the computing environment and initialized, including adjusting model parameters and configuring the runtime environment, to ensure proper functioning. Before inputting the vector with the prompt statement into the model, further input preparation and optimization are required. This includes tokenization, stop word removal, and text length adjustment to improve model processing efficiency and accuracy. Furthermore, the input text should be truncated or padded according to the model's input constraints to ensure that the input length is within the model's acceptable range. Finally, the processed and optimized input text is provided to the AIGC management suggestion generator. The model parses and processes the input information, leveraging its internal neural network structure and pre-trained knowledge to attempt to capture key features and semantic relationships within the input information. Combined with guidance from the prompt statement, the generation of nursing quality management suggestions begins.

[0062] During the suggestion generation phase, the AIGC Management Suggestion Generator receives input and performs inference calculations. Based on the input, the model searches for relevant information and patterns within its pre-trained knowledge system and, leveraging its own generative capabilities, gradually generates text content related to nursing quality management. During the inference process, the model continuously adjusts and optimizes the generated content to ensure that it conforms to logical and linguistic standards. However, the initial results generated by the model are often not directly usable and require further processing and optimization. This includes grammar checking, semantic correction, and removal of duplicate or irrelevant content to ensure that the generated suggestions are targeted and actionable, effectively addressing issues in nursing quality management. Post-processing techniques such as rule matching and template filling can be used to refine and improve the generated suggestions.

[0063] Results evaluation and optimization are crucial for ensuring the quality of generated recommendations. To assess the quality of generated nursing quality management recommendations, a series of scientifically sound evaluation metrics are necessary, such as accuracy, relevance, actionability, and completeness. Accuracy requires that recommendations accurately address issues identified in nursing feedback and ward environmental conditions; relevance emphasizes the close connection between recommendations and input information; actionability focuses on whether recommendations are practical and feasible; and completeness ensures that recommendations cover all important aspects without omitting critical information. In practice, generated recommendations often require manual evaluation. Nursing managers or professionals, drawing on their experience and expertise, assess the quality of recommendations and provide feedback. This feedback can help identify issues in the generation process, such as overly general recommendations or a lack of specific measures. Based on this manual evaluation feedback, the management recommendation generator is optimized. This may involve adjusting model parameters, increasing training data, and refining the model structure. Through continuous model optimization, the quality and performance of generated recommendations are improved, making them more tailored to practical needs.

[0064] Finally, the output and application phase occurs. After evaluation and optimization, the resulting nursing quality management recommendations are output to the nursing manager's terminal for display. Based on these recommendations, the manager can formulate specific nursing quality management measures and adjust and improve nursing work. Simultaneously, a corresponding tracking and feedback mechanism should be established to monitor and evaluate the effectiveness of the recommendations' implementation. This will further optimize nursing quality management, form a closed-loop management process, and continuously improve nursing quality and service levels.

[0065] Here, when the multiple nursing feedback text semantic association encoding feature vectors and the target ward environment state feature map represent the nursing feedback text context semantic association features and the target ward environment image semantic encoding features respectively, when performing cross-domain fine-grained aggregation analysis based on modal prior, the insufficient modal prior cross-domain association correspondence caused by the modal differences and dimensional differences between text semantics and image semantics will cause sparse fine-grained correspondence, resulting in a lack of inference degree generated by the aggregation analysis, thereby reducing the corpus quality of the obtained nursing quality management recommendations.

[0066] Therefore, after adding a prompt statement to the tail of the nursing feedback-ward environment status cross-modal significant interaction representation vector and inputting it into the AIGC-based management suggestion generator, the nursing feedback-ward environment status cross-modal significant interaction representation vector is subjected to cross-modal interaction optimization, and the process includes the following steps: The linear correlation matrix and nonlinear correlation matrix of the nursing feedback-ward environment state cross-modal significant interaction representation vector are calculated to obtain the nursing feedback-ward environment state cross-modal fine-grained aggregation linear correlation matrix and the nursing feedback-ward environment state cross-modal fine-grained aggregation nonlinear correlation matrix, which are expressed as:

[0067] in, 、 、 、 represents different weight hyperparameters, and represents the first significant interaction representation vector between the nursing feedback and the ward environment state. and elements, represents the first in the cross-modal fine-grained aggregate linear correlation matrix of nursing feedback and ward environment status. elements, The first in the cross-modal fine-grained aggregation nonlinear correlation matrix of nursing feedback and ward environment status is represented by elements.

[0068] Based on the nursing feedback-ward environment state cross-modal fine-grained aggregation linear association matrix and the nursing feedback-ward environment state cross-modal fine-grained aggregation nonlinear association matrix, the nursing feedback-ward environment state cross-modal significant interaction representation vector is respectively optimized at different scales to obtain the nursing feedback-ward environment state cross-modal fine-grained aggregation linear optimization vector and the nursing feedback-ward environment state cross-modal fine-grained aggregation nonlinear optimization vector, which are expressed as:

[0069] in, represents the cross-modal significant interaction representation vector of the nursing feedback-ward environment state, is matrix multiplication, represents the cross-modal fine-grained aggregate linear correlation matrix of nursing feedback and ward environment status, represents the cross-modal fine-grained aggregation nonlinear correlation matrix of nursing feedback and ward environment status, represents the cross-modal fine-grained aggregation linear optimization vector of nursing feedback-ward environment status, Represents the cross-modal fine-grained aggregation nonlinear optimization vector of nursing feedback-ward environment status.

[0070] The nursing feedback-ward environment state cross-modal fine-grained aggregation linear correlation matrix and the nursing feedback-ward environment state cross-modal fine-grained aggregation nonlinear correlation matrix are subjected to multi-scale mode union to obtain the nursing feedback-ward environment state cross-modal fine-grained aggregation multi-scale union matrix, which is expressed as:

[0071] in, and represents different weight hyperparameters, Indicates point multiplication by position, Represents the cross-modal fine-grained aggregation multi-scale joint matrix of nursing feedback and ward environment status.

[0072] The intrinsic feature extraction of the nursing feedback-ward environment state cross-modal fine-grained aggregation multi-scale joint matrix is ​​performed to obtain the nursing feedback-ward environment state cross-modal fine-grained aggregation multi-scale eigenvector .

[0073] The nursing feedback-ward environment state cross-modal fine-grained aggregation linear optimization vector and the nursing feedback-ward environment state cross-modal fine-grained aggregation nonlinear optimization vector are activated with different activation functions to obtain a first optimization weight vector and a second optimization weight vector, which are expressed as:

[0074] in, express activation function, express activation function, represents the first optimized weight vector, represents the second optimized weight vector.

[0075] Using the first optimized weight vector and the second optimized weight vector, multi-scale aggregation optimization is performed on the nursing feedback-ward environment state cross-modal fine-grained aggregated multi-scale eigenvector to obtain an optimized nursing feedback-ward environment state cross-modal significant interaction representation vector, which is expressed as:

[0076] in, Represents the cross-modal fine-grained aggregation of multi-scale eigenvectors of nursing feedback-ward environment status, represents vector addition, Represents the optimized cross-modal significant interaction representation vector between nursing feedback and ward environment status.

[0077] That is, the linear correlation matrix and the nonlinear correlation matrix of the nursing feedback-ward environment state cross-modal significant interaction representation vector are used as the multi-scale pattern recognition unit of the nursing feedback-ward environment state cross-modal significant interaction representation vector, and the cluster interaction path optimization of the nursing feedback-ward environment state cross-modal significant interaction representation vector and the internal feedback mechanism of the nursing feedback-ward environment state cross-modal significant interaction representation vector are performed on different association clusters to simulate the sparse triggering of the association system based on the node group, and the cluster core feature extraction of the linear correlation matrix and the nonlinear correlation matrix of the nursing feedback-ward environment state cross-modal significant interaction representation vector is used to perform the multi-scale predictability integration of the nursing feedback-ward environment state cross-modal significant interaction representation vector, so as to avoid the lack of association caused by sparsity affecting the generation of reasoning degree loss, and improve the corpus quality of the nursing quality management suggestion obtained by inputting the AIGC-based management suggestion generator.

[0078] In this embodiment of the present application, the nursing quality management suggestion display module 150 is used to transmit the nursing quality management suggestions to the nursing manager's terminal device for display. It should be understood that the nursing manager is responsible for the management and decision-making of the entire nursing team. Transmitting the nursing quality management suggestions to their terminal device for display allows them to fully understand the problems, potential risks, and improvement directions in the nursing work, thereby making more scientific and reasonable decisions based on these accurate and detailed suggestions to improve the quality and efficiency of nursing care.

[0079] In summary, the multimodal data-based nursing quality control system 100 according to the embodiment of the present application is described, which uses artificial intelligence-based data processing technology to comprehensively analyze the collection of acquired nursing feedback text descriptions and the target ward environmental status image, thereby capturing the associated semantics of the nursing feedback texts related to multiple target patient objects and the environmental status characteristics of the target ward, and intelligently generating nursing quality management suggestions based on the significant interactive representation information between the two cross-modal data of nursing feedback semantics and ward environmental status. In this way, more accurate and effective nursing quality management suggestions can be provided, which is conducive to achieving more scientific and reasonable nursing quality management.

Claims

1. A nursing quality control system based on multimodal data, characterized in that: include: A patient nursing feedback module is used to collect nursing feedback text information of multiple target patient objects to obtain a set of nursing feedback text descriptions; The ward monitoring module is used to obtain the target ward environment status image collected by the high-definition camera in real time; A nursing-related data transmission module, configured to transmit the set of nursing feedback text descriptions and the target ward environment status image to a nursing quality management server; A nursing quality management module is configured to perform, on the nursing quality management server, a deep analysis based on cross-domain joint interactive coding on the set of nursing feedback text descriptions and the target ward environment status image to obtain nursing quality management suggestions; The nursing quality management suggestion display module is used to transmit the nursing quality management suggestion to the terminal device of the nursing manager for display.

2. The nursing quality control system based on multimodal data according to claim 1 is characterized in that: The nursing quality management module includes: A nursing feedback text description semantic coding unit is used to perform text semantic association coding on the set of nursing feedback text descriptions to obtain multiple nursing feedback text semantic association coding features; a ward environment state feature extraction unit, configured to extract features based on the ward environment state of the target ward environment state image to obtain target ward environment state features; a nursing quality multimodal information interactive representation unit, configured to perform cross-domain joint encoding processing based on modal core features on the semantic association coding features of the multiple nursing feedback texts and the target ward environment status features to obtain a nursing feedback-ward environment status cross-modal significant interaction representation; A nursing quality management suggestion generating unit is used to generate the nursing quality management suggestion based on the nursing feedback-ward environment state cross-modal significant interaction representation.

3. The nursing quality control system based on multimodal data according to claim 2 is characterized in that: The nursing feedback text description semantic coding unit includes: a nursing feedback text semantic coding feature generating subunit, configured to pass each nursing feedback text description in the set of nursing feedback text descriptions through a semantic encoder based on a Bert model to obtain a set of nursing feedback text semantic coding feature vectors; The nursing feedback text semantic association encoding subunit is used to pass the set of the nursing feedback text semantic encoding feature vectors through the LSTM-based multiple nursing feedback text semantic association encoder to obtain multiple nursing feedback text semantic association encoding feature vectors as the multiple nursing feedback text semantic association encoding features.

4. The nursing quality control system based on multimodal data according to claim 3 is characterized in that: The ward environment state feature extraction unit is used to: pass the target ward environment state image through a ward environment state feature extractor based on an expanded convolutional neural network to obtain a target ward environment state feature map as the target ward environment state feature.

5. The nursing quality control system based on multimodal data according to claim 4 is characterized in that: The nursing quality multimodal information interactive representation unit includes: A nursing feedback text core clue extraction subunit is used to perform core clue extraction based on point convolution coding on the semantic association coding feature vectors of the multiple nursing feedback texts to obtain multiple nursing feedback text semantic core clue coding vectors; A target ward state core clue extraction subunit is used to extract a target ward environment state core clue encoding vector from the target ward environment state feature map; A nursing feedback ward environment state feature association subunit is configured to construct a core clue weaving template matrix between the nursing feedback and ward environment state modalities based on the multiple nursing feedback text semantic core clue encoding vectors and the target ward environment state core clue encoding vectors; The nursing feedback ward environment status cross-modal significant interaction analysis subunit is used to weave a template matrix based on the core clues between the nursing feedback and ward environment status modalities, and perform cross-modal fine-grained feature interaction analysis on the semantic association encoding feature vectors of the multiple nursing feedback texts and the target ward environment status feature map to obtain a nursing feedback-ward environment status cross-modal significant interaction representation vector as the nursing feedback-ward environment status cross-modal significant interaction representation.

6. The nursing quality control system based on multimodal data according to claim 5 is characterized in that: The target ward status core clue extraction subunit is used to: Splitting the target ward environment state characteristic graph to obtain a sequence of target ward environment state local characteristic graphs; Performing attention-based salient region attention analysis on each target ward environment state local feature map in the sequence of target ward environment state local feature maps to obtain a sequence of salient region attention coefficients; Based on the sequence of the salient area attention coefficients, a core clue search is performed on the target ward environment state characteristic map to obtain the target ward environment state core clue encoding vector.

7. The nursing quality control system based on multimodal data according to claim 6 is characterized in that: The nursing feedback ward environment status feature association subunit is used to: construct an inter-modal core clue weaving template between the multiple nursing feedback text semantic core clue encoding vectors and the target ward environment status core clue encoding vector to obtain the nursing feedback-ward environment status inter-modal core clue weaving template matrix.

8. The nursing quality control system based on multimodal data according to claim 7 is characterized in that: The nursing feedback ward environment status cross-modal significant interaction analysis subunit is used to: Performing feature decoupling along the channel dimension on the target ward environment state feature map to obtain a set of target ward environment state local feature matrices; Using the multiple nursing feedback text semantic association encoding feature vectors as query vectors, using each target ward environment state local feature matrix in the set of target ward environment state local feature matrices as a key matrix, and using the nursing feedback-ward environment state inter-modal core clue weaving template matrix as a priori information constraint matrix, the matrix is ​​input into a cross-modal template constraint encoder based on a heterogeneous converter to obtain a set of nursing feedback-ward environment state cross-modal fine-grained interaction encoding vectors; The position-wise mean vector of the set of the nursing feedback-ward environment state cross-modal fine-grained interaction encoding vectors is calculated to obtain the nursing feedback-ward environment state cross-modal significant interaction representation vector.

9. The nursing quality control system based on multimodal data according to claim 8, characterized in that: The nursing quality management suggestion generation unit is used to: add a prompt statement to the tail of the nursing feedback-ward environment status cross-modal significant interaction representation vector, and input it into the AIGC-based management suggestion generator to obtain the nursing quality management suggestion, wherein the prompt statement is: generate nursing quality management suggestions based on the input corpus.

Images