Intelligent nursing service compliance monitoring and optimizing system based on multi-modal analysis

Abstract

The invention relates to the crossing field of medical health information technology and artificial intelligence, and discloses an intelligent nursing service compliance monitoring and optimizing system based on multi-modal analysis, which comprises a data acquisition module, a data transmission and storage module, a core analysis engine and an application and feedback module, interaction audio in the service process is obtained and subjected to encryption and acoustic feature anonymization processing; the core analysis engine executes multi-modal analysis to calculate a risk score in real time, and executes automatic root cause analysis under the triggering of a specific event; and the application and feedback module presents an analysis result and captures a manual disposal conclusion as high-quality annotation data for driving automatic iterative optimization of an algorithm model. The service interaction audio is subjected to multi-modal analysis to calculate the comprehensive risk score, and automatic root cause analysis triggered by a specific event is combined, so that comprehensive and objective automatic monitoring from real-time early warning to deep tracing is realized, and the subjectivity and limitation of traditional manual spot check are overcome.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical health information technology and artificial intelligence, in particular to an intelligent nursing service compliance monitoring and optimization system based on multi-modal analysis. BACKGROUND

[0002] With the increasing demand for professional and refined nursing services in society, how to ensure service quality, improve user satisfaction and effectively manage potential risks has become the core issue of concern in the nursing service industry. To address this challenge, the existing technology has made some explorations. The initial application of technology mainly stops at deploying audio recording devices to facilitate manual playback and spot checks after the service is completed. With the development of automatic speech recognition technology, some solutions attempt to convert service interaction audio into text and apply keyword retrieval or simple rule matching for content review based on this. However, this kind of text-based analysis method has significant defects: its understanding of semantics is superficial and cannot recognize contextual logic, leading to deviations in judgment of complex language phenomena such as irony and metaphor; more importantly, it completely ignores the rich emotional information carried by the voice, such as tone, speed and volume, which are the key to judging communication quality and perceiving emotional state. Therefore, relying solely on keyword matching is prone to false positives and false negatives, and its monitoring effectiveness and reliability are greatly compromised.

[0003] In view of the limitations of the above single technical path, the current industry still highly relies on traditional methods such as user follow-up, satisfaction questionnaire or manual spot check of service recordings by management personnel after the service is completed in practice. However, these methods have their inherent limitations. The mode of manual spot check determines that its coverage is extremely limited and cannot achieve continuous monitoring of all service processes, and the evaluation results are also easily affected by the subjective judgment of the inspectors, making it difficult to unify the evaluation standards. More critically, such supervision means is essentially lagging, only after the service is completed or even after the user complains, it cannot identify and intervene in real time during the risk event, missing the key opportunity to prevent and control the escalation of the situation.

[0004] Massive, unstructured voice interaction data generated in the service process is the most direct and richest basis for judging service quality, but traditional technical means are difficult to effectively analyze and deeply understand the semantics of it. This makes it difficult to trace the root cause of the event and conduct systematic attribution analysis when facing service disputes. Even if some existing technologies try to introduce artificial intelligence models for analysis, these models are usually static, and their decision logic is fixed once deployed. They generally lack an effective mechanism to learn and absorb valuable experience from human intervention in real business scenarios, and cannot continuously optimize their performance based on feedback, leading to the gradual failure of the model in complex and changing real-world environments. SUMMARY

[0005] In view of the deficiencies of the prior art, the present application provides an intelligent nursing service compliance monitoring and optimization system based on multi-modal analysis, which solves the problem that the nursing service quality monitoring in the prior art relies on manual work, has strong subjectivity and low coverage, and is difficult to perform real-time and deep risk analysis and objective attribution on unstructured interaction data in the service process.

[0006] To achieve the above purpose, the present application is implemented by the following technical solutions: an intelligent nursing service compliance monitoring and optimization system based on multi-modal analysis, comprising: A data acquisition module deployed on a mobile terminal device for acquiring interaction audio in the service process; A data transmission and storage module for encrypted transmission and acoustic feature anonymization processing of the interaction audio; A core analysis engine deployed on a cloud server for receiving the anonymized interaction audio and performing multi-modal analysis and risk calculation; An application and feedback module deployed on a cloud server for presenting analysis results and performing iterative optimization of algorithm models based on human intervention results.

[0007] Further, the core analysis engine comprises: A speech-to-text unit for converting the anonymized interaction audio into a text sequence; A multi-modal analysis unit for parallel processing of the text sequence and the anonymized interaction audio; A compliance rule engine unit for risk decision-making based on the output of the multi-modal analysis unit; An automated root cause analysis unit for performing deep causal tracing under certain event triggers.

[0008] Furthermore, the multimodal analysis unit is used to receive the text sequence output by the speech-to-text unit and the anonymized interactive audio, so as to extract the semantic information of the text sequence, the emotional information of the interactive audio, the keyword information of the text sequence, and the speaker role information that distinguishes the two parties in the interaction in parallel; wherein, the speaker role information is generated by applying speaker log analysis technology to the interactive audio to assign a temporary, anonymous role identifier to each independent speaker in this interaction.

[0009] Furthermore, the compliance rule engine unit is used to load a rule set, where rules are defined as logical combinations of queries on the semantic information, sentiment information, keyword information, and speaker role information, and calculate a comprehensive risk score based on the matching results of the logical combination queries. The calculation method for the comprehensive risk score is defined by the following formula: ; In the formula, The overall risk score representing this time segment; It is an index of the rules in the rule base; It is the first The triggering condition for each rule is a logical combination of analysis results such as semantics, sentiment, and keywords; It is the first The preset risk weight values ​​corresponding to each rule; It is an indicator function whose value is determined by a condition. The value is 1 when the condition is met, and 0 when the condition is not met.

[0010] Furthermore, the automated root cause analysis unit is activated upon receiving an instruction, automatically aggregates contextual data related to a specific event, inputs the contextual data into a generative large language model to perform causal inference, and outputs structured causal chain data.

[0011] Furthermore, the application and feedback module includes: Dashboards and alert units for real-time risk warning and multi-dimensional data visualization; A closed-loop feedback and model iteration unit used to capture conclusions from human intervention and drive model self-optimization.

[0012] Furthermore, the closed-loop feedback and model iteration unit is used to capture the results of manual handling of the early warning event, using these results as truth labels, and binding these truth labels with the original data that triggered the early warning to form labeled training samples. These labeled training samples are used for iterative optimization of the algorithm model in the core analysis engine. The core parameter update rules for this iterative optimization are defined as follows: ; wherein: the model at the th iteration updated; the current parameter set of the model at the th iteration; denotes the learning rate, which is a hyperparameter used to control the step size of parameter update; denotes a mini-batch of training samples randomly sampled from the feedback dataset ; denotes the average loss value computed by the model using the current parameters on the data batch ; denotes the gradient vector of the loss function with respect to the model parameters , which indicates the direction in which the loss function value decreases most rapidly at the current parameter point.

[0013] Further, the application and feedback module is further configured to construct a dynamic risk profile for the nursing staff and quantify it as a risk profile feature vector, calculate a matching degree score between the risk profile feature vector and the course skill vector, and generate an individualized ability improvement suggestion according to the matching degree score. The matching degree score is calculated by the following cosine similarity formula: ; wherein: denotes the matching score between the nursing staff vector and the course vector; denotes a feature vector representing the ability or needs of the nursing staff; denotes a feature vector representing the course content or target; denotes the dot product of vector and vector ; denotes the Euclidean norm of vector ; denotes the Euclidean norm of vector . The system recommends the top several courses with the highest scores to the nursing staff and their supervisors, thereby achieving precise training and empowering their professional growth.

[0014] Further, the data transmission and storage module is configured to remove the biometric information in the audio signal that can be used to identify the identity of a specific speaker by modifying the fundamental frequency and formant of the interactive audio signal to achieve the acoustic feature anonymization processing.

[0015] Further, the data acquisition module comprises a recording control logic associated with the service order state machine, and the recording control logic is configured to activate the audio acquisition function only when the service process state machine migrates to the in-service state, and to suspend the audio acquisition function when the service process state machine leaves the in-service state.

[0016] The present application provides an intelligent nursing service compliance monitoring and optimization system based on multi-modal analysis. The present application has the following advantages: 1. The present application calculates a comprehensive risk score by performing multi-modal analysis on service interaction audio, and combines automated root cause analysis triggered by specific events to achieve comprehensive and objective automated monitoring from real-time early warning to deep traceability, overcoming the subjectivity and limitations of traditional manual spot checks and providing data-driven deep insights for management decisions.

[0017] 2. The present application captures the manual handling results of the warning events by the management personnel as high-quality labeled data, and uses the data to drive the automated iteration of the algorithm model in the core analysis engine, thereby building a data-driven system self-optimization closed loop, enabling the analysis and decision-making capabilities of the system to continuously evolve with the accumulation of real business data, and solving the technical problem of AI models being difficult to continuously adapt in complex application scenarios.

[0018] 3. The present application uses recording control logic associated with the service order state machine, and combines acoustic feature anonymization processing technology for audio data to ensure the legality and compliance of data acquisition while achieving effective monitoring, thereby maximizing the privacy and security of both parties. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 The system framework of the present application is shown in the figure; Figure 2 The core analysis engine function diagram of the present application is shown in the figure; Figure 3 The real-time monitoring flowchart of the standard service process of the present application is shown in the figure; Figure 4 The automated root cause analysis flowchart triggered by complaint events of the present application is shown in the figure.

[0020] Among them, 100, data acquisition module; 200, data transmission and storage module; 300, core analysis engine; 400, application and feedback module. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the drawings in the specification of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0022] Referring to the drawings Figure 1 , Figure 1 is a system function architecture diagram according to an embodiment of the present application. The present application provides an intelligent nursing service compliance monitoring and quality optimization system based on multi-modal analysis. The implementation of the system relies on a distributed architecture in which mobile terminal devices and cloud servers work cooperatively.

[0023] At the system function architecture level, the intelligent nursing service compliance monitoring and quality optimization system based on multi-modal analysis includes a data acquisition module 100, a data transmission and storage module 200, a core analysis engine 300, and an application and feedback module 400.

[0024] The data acquisition module 100 is deployed on a mobile terminal device operated by a nursing staff, and is the source of original data of the system. The data acquisition module 100 specifically includes a user interaction interface for obtaining electronic authorization of both parties of a service, a recording control logic associated with a service order state, and a local data cache queue for coping with network uncertainty.

[0025] The data transmission and storage module 200 is distributed in mobile terminals and cloud servers. The data transmission and storage module 200 includes a data segmentation and encryption unit configured in a mobile terminal, and a unit deployed in a cloud server for securely receiving data and performing acoustic feature anonymization processing. The data transmission and storage module 200 ensures that the data is encrypted before leaving the terminal device, and that the personal identity features are removed by technical means before entering the analysis process.

[0026] The core analysis engine 300 is deployed in a cloud server, and is a computing center for performing all intelligent analysis tasks. The core analysis engine 300 specifically includes a speech-to-text unit for realizing audio-to-text conversion; a multi-modal analysis unit for extracting semantics, sentiment and keywords in parallel; a compliance rule engine unit for making risk decisions based on a preset rule base; and an automatic root cause analysis unit for performing deep causal tracing under the triggering of a specific event.

[0027] The application and feedback module 400, as an upper application of the cloud service, is the final presentation of the system analysis result and the business execution endpoint. The application and feedback module 400 specifically includes a board and early warning unit for real-time risk early warning and multi-dimensional data visualization, and a closed-loop feedback and model iteration unit for capturing artificial disposal conclusions and driving model self-optimization.

[0028] The overall workflow of the intelligent nursing service compliance monitoring and quality optimization system based on multi-modal analysis, and the method for realizing intelligent monitoring and optimization, includes the following steps: S101: When the system starts the nursing service process, the data acquisition module 100 acquires the electronic authorization of both parties on the mobile terminal, and activates the audio acquisition function when the service state is switched to in progress, and starts real-time recording of the interactive audio in the service process.

[0029] S102: The data transmission and storage module 200 cuts the collected continuous audio stream into audio data segments with sequence identification and fixed time length on the mobile terminal, and independently performs encryption processing on the load of each data segment, and then uploads it to the cloud server through a secure network transmission protocol. After the cloud server receives the data, it performs decryption and acoustic feature anonymization processing to protect user privacy.

[0030] S103: The core analysis engine 300 receives and processes the preprocessed audio data. The speech-to-text unit in the engine converts the audio signal into a text sequence in real time, and then the multi-modal analysis unit processes the audio and text data in parallel, and outputs structured analysis results, including the semantic vector of the text, the emotion category of the audio, and the Boolean label of the keyword detection.

[0031] S104: The compliance rule engine unit obtains the output of the multi-modal analysis unit, and matches it with the rules in the rule library one by one. For each time slice, the engine calculates the comprehensive risk score according to the matched rules and their preset weights , which is defined by the following formula: ; In the formula, represents the comprehensive risk score of the time slice; is the index of the rule in the rule library; is the trigger condition of the th rule, which is a logical combination of semantic, emotional, keyword and other analysis results; is the preset risk weight value corresponding to the th rule; is an indicator function, whose value is 1 when the condition is met, and 0 when it is not met.

[0032] S105: Application and Feedback Module 400 executes corresponding follow-up actions according to the risk score and other analysis conclusions output by Core Analysis Engine 300. The other analysis conclusions specifically include but are not limited to: structured analysis results output by Multi-modal Analysis Unit, which explicitly indicate the specific rule identifier that triggered the risk calculation, the semantic topics identified in the text sequence, the emotional state categories analyzed in the audio signal, and the list of keywords that hit, if the risk threshold is exceeded, the system automatically triggers an early warning and notifies the manager through multiple channels. All analysis results are aggregated and sedimented to dynamically update the caregiver risk portrait and service quality statistical data in the quality optimization dashboard.

[0033] S106: When a specific event such as user complaint occurs that requires in-depth analysis, the manager can trigger an automated root cause analysis process. At this time, Core Analysis Engine 300 will automatically aggregate relevant data and generate a cause-effect chain map. In addition, the manager's manual handling results of the early warning event will be fed back through Application and Feedback Module 400 as high-quality labeled data for iterative optimization of algorithm models in Core Analysis Engine 300, forming a closed loop of technology and management.

[0034] Specifically, the manual handling results, such as the manager's judgment that a certain early warning event is an effective risk-service attitude problem or an ineffective risk-system false alarm, are considered as the true value label of the event . Application and Feedback Module 400 binds this true value label with the original data packet that triggered the early warning (the data packet contains audio clips, transcribed text, and intermediate features analyzed by the system) to form a complete, labeled training sample . These new samples are merged into a dedicated feedback dataset , which is used to periodically or on-demand retrain and fine-tune the model parameters of the algorithm models in Core Analysis Engine 300 (such as sentiment classifiers or domain semantic understanding models). The iterative optimization process aims to minimize the loss function of the model on the feedback dataset (e.g., cross-entropy loss function), and the core parameter update step can be described by stochastic gradient descent (SGD) or its optimization variants (such as Adam algorithm), with the update rule defined as follows: ; In the formula: represents the model parameter set (e.g., all weights and biases of a neural network) after the th iteration of the model update.​​​ a current parameter set of the model at the i-th iteration; a learning rate, which is a hyperparameter used to control the step size of parameter updates; a mini-batch of training samples randomly sampled from the feedback dataset a mean loss value calculated by the model using the current parameters on the data batch a gradient vector with respect to the model parameters , which indicates the direction in which the loss function value decreases most rapidly at the current parameter point.

[0035] This closed-loop mechanism ensures that the decision-making ability of the algorithm model can be continuously corrected and evolved as the artificial handling data in the real business scenario accumulates.

[0036] In the embodiment of the present application, the data acquisition module 100 and the data transmission and storage module 200 cooperatively realize the data entry pipeline from the service site to the cloud server, which ensures the compliance of data acquisition, the security and integrity of data transmission, and the technical processing of privacy information before entering the analysis. The specific implementation method includes the following steps: S201: The system performs a compliance authorization process. In the client application of the mobile terminal, an electronic authorization agreement is independently presented to the nursing staff and the service user, and the agreement content clearly informs the purpose, scope and privacy protection measures of data acquisition. The system records the electronic confirmation credentials of both parties, which are data structures containing the digital identity of both parties, the service order number, the device unique identifier and the authorization operation server timestamp, and are forcibly associated with the service record, serving as the basis for the legality of subsequent data processing.

[0037] S202: The system performs audio acquisition control accurately bound to the service state. The client application has a built-in service process state machine, which defines multiple states including "to be served", "in service", "service suspended" and "service ended". The audio acquisition function is strictly configured to be started by calling the underlying audio API of the operating system only when the state machine migrates to the "in service" state, and to be immediately terminated when the state machine leaves the "in service" state (for example, migrates to "service suspended" or "service ended"). This implementation ensures that the scope of audio data acquisition is strictly consistent with the service time agreed in the contract, avoiding the collection of irrelevant data.​​​​

[0038] S203: The system locally implements reliable caching and resuming of data in the mobile terminal. The collected audio data segments are first entered into the locally set safe storage queue before uploading. The queue can be a first-in-first-out (FIFO) queue implemented based on a file system or an embedded database (such as SQLite). At the same time, the network state monitor built into the client continuously polls or listens to the network connection state through system broadcast. When it is monitored that the network connection is interrupted or the signal quality is lower than the preset threshold, the data uploading task is suspended, and the data segments are retained in the local queue. When the network connection is restored, the breakpoint resuming unit is activated to resume the uploading task in the order of the queue, and through data packet sequence number verification with the server, the integrity of the data transmission is ensured.

[0039] S204: The system performs data segmentation and encrypted transmission. The continuous audio stream is cut into data segments with sequence numbers and time stamps in the mobile terminal, and the time length of each data segment is fixed, for example, every 30 seconds or 60 seconds. The data segmentation and encryption unit calls the encryption library to encrypt the data payload of each data segment using the Advanced Encryption Standard (AES) algorithm. Specifically, the AES-256-GCM mode can be used, which can generate a message authentication code while encrypting to ensure the confidentiality and integrity of the data. The encrypted data packet, with the non-encrypted metadata (such as sequence number, time stamp) in the packet header and the encrypted audio data in the payload, is then uploaded to the cloud server through the secure channel established by the TLS (Transport Layer Security) protocol.

[0040] S205: The system performs anonymization processing of audio data in the cloud server. After the data is uploaded to the server and decrypted, the audio anonymization processing unit performs acoustic feature transformation before sending it to the core analysis engine 300. Acoustic feature transformation aims to remove the biometric information in the audio signal that can be used to identify the identity of a specific speaker, while maximizing the preservation of the speech content and emotional prosody features. Its technical implementation includes modifying the pitch (Pitch, ) and formants (Formants) of the audio signal. Pitch modification can be achieved by randomly shifting or normalizing the original pitch contour to a specified mean. Formant modification can be achieved by analyzing the vocal tract filter parameters through linear predictive coding (LPC), fine-tuning the filter parameters, and then synthesizing the speech again. After the above processing, the output audio retains the necessary information for speech recognition and emotion analysis, but cannot be directly associated with the original speaker in terms of listening.

[0041] Refer to the attached Figure 2 , Figure 2is a schematic diagram of the core analysis engine function according to an embodiment of the present application. In the embodiment of the present application, the core analysis engine 300 is deployed in a cloud server, responsible for performing all intelligent computing tasks from raw data to high-level semantic understanding and risk decision-making. The specific implementation method of the engine includes the following steps: S301: The engine performs real-time conversion of voice content. The speech-to-text unit receives the anonymized audio data segment transmitted from the data transmission and storage module 200. In order to adapt to the complex noise such as background human voice and appliance running sound in the home care environment, the audio data is first sent to the AI spoken language noise reduction model for preprocessing. This model is based on a deep neural network and can identify and suppress the noise components of non-target human voice, improving the signal-to-noise ratio of subsequent speech recognition. The noise-reduced audio stream is input into the streaming automatic speech recognition (ASR) model. This ASR model uses an advanced architecture based on, for example, Transformer-Transducer, which supports incremental decoding while receiving audio streams, thereby outputting text sequences synchronized with the audio stream in real time with low latency, providing text input for subsequent real-time analysis.

[0042] Preferably, after the speech-to-text unit S301 performs conversion, the core analysis engine 300 further performs a speaker role differentiation step before the multi-modal analysis unit performs parallel processing. Specifically, the system applies speaker diarization analysis technology to the received raw interaction audio (before performing acoustic feature anonymization processing). This technology extracts and compares the acoustic features of different speech segments in the audio stream, and clusters the speech belonging to different speakers without identifying the real identity of the specific person. The system assigns a temporary, anonymous role identifier (e.g., "Speaker A", "Speaker B") to each independent speaker identified in this service interaction. In this way, the system generates speaker role information that is aligned in time with the text sequence. This speaker role information is then transmitted to the multi-modal analysis unit along with the text sequence, sentiment information, and keyword information, allowing the subsequent compliance rule engine to make more accurate and context-related risk decisions based on the speech behavior of a specific role (e.g., only analyzing whether the speech of "Speaker A", the caregiver, is compliant) S302: The engine performs parallel multi-modal feature extraction. The multi-modal analysis unit receives the text sequence output by the speech-to-text unit and the original anonymized audio stream, and simultaneously performs semantic, sentiment, and keyword analysis in three dimensions.

[0043] In terms of semantic understanding, the system calls a large language model fine-tuned on a nursing service domain knowledge base. Specifically, a general pre-trained language model is chosen as the base, and a professional corpus containing a large number of nursing specifications, service standards, historical excellent service dialogue records, and industry terms is used for instruction tuning or parameter-efficient fine-tuning. The fine-tuned model can more accurately understand the specific intent, entity, and context logic in the service dialogue. For the input text sequence , the model generates a high-dimensional semantic vector , i.e. , which is a mathematical representation of the deep meaning of the text.

[0044] In terms of sentiment analysis, the system directly processes audio data to capture non-text prosodic information. The acoustic model extracts a set of acoustic features from the audio signal, including Mel-frequency cepstral coefficients (MFCCs), pitch, energy, zero-crossing rate, and harmonic-to-noise ratio (HNR). These features are input into a pre-trained sentiment classifier, such as a convolutional neural network (CNN) or long short-term memory network (LSTM) model, which outputs a probability distribution vector over pre-defined sentiment categories (e.g., positive, neutral, negative, angry). In terms of keyword detection, the system performs high-speed matching on text sequences based on a keyword library that can be dynamically updated by the background management system. The keyword library not only contains single words but also supports regular expression-based sentence patterns. The system uses an efficient multi-pattern matching algorithm (e.g., Aho-Corasick algorithm) for real-time detection, outputting a Boolean flag set indicating whether the current text segment hits specific words or patterns in the library. The core of the Aho-Corasick algorithm is to construct a finite state automaton, whose state transition process is deterministic and can be precisely described by a state transition function. For the state transition of the automaton at any time, it follows the formula: ; where: represents the current state of the automaton after processing the th character of the input text sequence; represents the state of the automaton after processing the th character, where the initial state is (the root node of the automaton). represents the i-th character in the input text sequence; represents the state transition function, which receives the current state and the current input character , and returns the next state. The state transition function is the core of the algorithm, whose internal logic depends on the pre-constructed goto function and failure function . Function denotes the target state to which the transition occurs if there is a direct path when receiving character in state ; function denotes the state that represents the longest true suffix of the current matched string to which the system should back up when there is a mismatch in state . The state transition function handles the mismatch case by iteratively querying the failure function until a valid transition is found or the initial state is reached. After each state transition to , the system checks whether the state and all states reachable through its failure chain have associated output keywords, and if so, adds the matched keywords to the Boolean tag set for the time step.

[0045] S303: The engine performs the automated compliance adjudication based on multi-modal fusion. The compliance rule engine unit receives the output results from the multi-modal analysis unit, i.e., the semantic vector , the sentiment probability distribution and the keyword tag set . The engine loads and executes a rule set, where each rule is defined as a logical combination query on the above multi-modal analysis results. For example, a rule for detecting potential conflict risks can be defined as: (cosine similarity between semantic vector and user complaint topic > threshold ) AND (probability of sentiment category being anger > threshold ). When the logical condition of any rule is satisfied, the engine determines that a compliance event is triggered, and calculates the risk score according to the previously set risk calculation formula , is the index of the rule in the rule library, and the weight of the corresponding rule is accumulated to calculate the comprehensive risk score of the current time slice, where the indicator function has the condition on , ​and a Boolean function whose truth value determines the accumulation of risk.

[0046] S304: The engine performs automated root cause analysis under certain conditions. When receiving instructions triggered by external events (e.g., a manager enters a user complaint in the system), the automated root cause analysis unit is activated. The automated root cause analysis unit first aggregates contextual data from the database according to event association information (such as service order number, timestamp), and the aggregated data set includes: the complete interaction text and sentiment analysis sequence of this service, the historical risk portrait data of the involved nursing staff, and the relevant standard operation procedure (SOP) document fragments retrieved from the knowledge base. Subsequently, the automated root cause analysis unit constructs these heterogeneous data into a structured prompt, and inputs the prompt into a generative large language model with strong logical reasoning ability. The model is instructed to perform the cause and effect task, analyzes and outputs the direct cause, indirect cause and root cause of the complaint event. The text output of the model is parsed by the system into structured causal chain data, which is then transmitted to the application and feedback module 400 for visual presentation.

[0047] In an embodiment of the present application, the application and feedback module 400 is a bridge connecting intelligent analysis and business management, responsible for converting the insights of the core analysis engine 300 into specific management actions, and establishing a data-driven system self-optimization closed loop. The specific implementation method of the application and feedback module 400 includes the following steps: S401: The system performs visual risk presentation and multi-level early warning. The dashboard and early warning unit continuously receives real-time analysis results from the core analysis engine 300, especially the comprehensive risk score When the score exceeds the preset risk threshold , the system triggers the corresponding warning strategy according to the severity level of the risk. The strategy is configurable, which defines the notification channels (such as push within the client application, SMS, email) and notification object levels (such as general risk notification to direct supervisor, serious risk notification to regional manager) corresponding to different risk levels. At the same time, all service analysis data is continuously aggregated to build a multi-dimensional dynamic risk portrait for each nursing staff. The portrait is visualized and displayed through the dashboard interface, and its dimensions can include: the frequency and trend of various violation events (such as negative sentiment, service taboo), average service sentiment score, high-risk period distribution, etc.

[0048] S402: The system generates personalized capability improvement suggestions based on the risk portrait. This function is realized through the recommendation engine. Specifically, the dynamic risk portrait of each nursing staff is quantized into a feature vector Each dimension of the vector corresponds to an assessable risk indicator. Meanwhile, the system maintains a training course library, each course in the library is also labeled as a feature vector whose dimensions correspond to the skill points that the course can improve. The recommendation engine generates recommendations by calculating the matching degree between the caregiver risk vector and the course skill vector, the matching degree score can be calculated by the following cosine similarity formula: ; wherein: represents the matching score between the caregiver vector and the course vector; represents the feature vector representing the caregiver's ability or needs; represents the feature vector representing the course content or target; represents the vector and the vector dot product; represents the Euclidean norm of the vector ; represents the Euclidean norm of the vector . The system recommends the top courses to the caregiver and his supervisor, thereby achieving precise training and empowering their professional growth.

[0049] S403: The system performs the capture of the treatment feedback and the data closed loop, and the feedback module 400 is deeply integrated with the work order management system inside the enterprise. After the investigation and treatment of the management personnel, the final treatment conclusion of each work order automatically created by the early warning event is captured by the system. These conclusions are structured labels, for example, the management personnel may determine the event as an effective risk-service attitude problem, an effective risk-operation process violation or an invalid risk-system false alarm. These conclusions confirmed by manual review, together with their corresponding original data (including audio clips, transcription text, multi-modal analysis results), are automatically formatted by the system into a high-quality labeled sample with true value labels.

[0050] S404: System automation iteration based on closed-loop data-driven model. The closed-loop feedback and model iteration unit stores the high-quality labeled samples generated in S403 into a dedicated model training database. The system establishes a standardized model iteration workflow (MLOps) that can be configured to be triggered periodically (e.g., weekly) or automatically after accumulating a sufficient number of new samples. After the workflow is started, the system uses the complete dataset containing the newly labeled samples to perform incremental training or parameter fine-tuning on one or more algorithmic models (e.g., sentiment classifier, domain semantic understanding model) in the core analysis engine 300. The newly trained candidate model is evaluated on a reserved validation set, and only when the candidate model outperforms the current model in the production environment in terms of key indicators (e.g., accuracy, recall), the system automatically deploys it online to replace the old model. This mechanism ensures that the recognition and decision accuracy of the system can continuously evolve with the accumulation of real business data.

[0051] Referring to the drawings Figure 3 With the drawings Figure 4 , Figure 3 is a real-time monitoring flowchart of standard service processes according to an embodiment of the present application, Figure 4 is an automated root cause analysis flowchart triggered by a complaint event according to an embodiment of the present application. In embodiments of the present application, the overall functions of the system are achieved through the collaborative workflow between modules, covering typical application scenarios from daily real-time monitoring to in-depth analysis of specific events.

[0052] The specific implementation method of the real-time monitoring flowchart of the standard service process includes the following steps: S501: The process starts with the caregiver starting the service on the mobile terminal device for a specified service order. The user interaction interface of the data acquisition module 100 presents the authorization terms to both parties and obtains their electronic confirmation credentials. When the service status in the client application is switched to "in service", the audio acquisition function is activated to start recording the service interaction audio.

[0053] S502: The collected audio stream is processed by the data transmission and storage module 200. On the mobile terminal, the audio stream is segmented, encrypted, and securely uploaded to the cloud server through the breakpoint resume mechanism. On the cloud side, the server completes data reception, decryption, and acoustic feature anonymization processing.

[0054] S503: The core analysis engine 300 processes the incoming audio data segments in real time. The engine performs speech-to-text, domain-tuned large language model-based semantic understanding, acoustic feature-based sentiment analysis, and dynamic keyword matching in parallel, and calculates the comprehensive risk score in real time according to the pre-set rule library and weights .

[0055] S504: The application executes decisions according to the output of the core analysis engine 300. If the risk threshold is continuously below the preset threshold, the system maintains a silent monitoring state; if any time slice exceeds the threshold, the system immediately sends warning information to the corresponding management level according to the risk level through the preset notification strategy.

[0056] S505: When the service process ends and audio collection stops, the system structures and archives all relevant data of the service, including raw data, text sequences, multi-modal analysis results sequences, and all triggered early warning events, and automatically generates a service summary report containing key quality indicators, and stores it in association with the service order.

[0057] The specific implementation method of the automated root cause analysis process triggered by a complaint event includes the following steps: S506: The process begins with the quality management personnel activating the automated root cause analysis function in the management background interface of the system for a user complaint or a serious risk event that has been recorded.

[0058] S507: After receiving the instruction, the automated root cause analysis unit in the core analysis engine 300 is triggered. This unit automatically retrieves and aggregates all relevant context data from the database according to the index information such as the service order number, timestamp, and the identity of the nursing staff involved. The aggregated data set contains: ; In the formula, is the complete interaction data of the service, including text, sentiment, risk event, etc. time series records; is the historical risk profile data of the nursing staff involved; is the standard operating procedure document related to the complaint content retrieved from the system knowledge base.

[0059] S508: The automated root cause analysis unit constructs the aggregated data set into a structured prompt. This prompt is designed to ask the generative large language model to perform explicit causal reasoning tasks, such as: Based on the following service records, personnel background and standard procedures, analyze the root cause of the user complaint, and output the causal chain in the format of root cause-intermediate cause-direct cause-event.

[0060] S509: The large language model processes the input prompt and performs reasoning, outputting a natural language text describing the causal relationship. The system parses this text and extracts structured causal chain data, such as JSON or XML format, clearly indicating the transmission path and logical relationship from the root cause to the final event.

[0061] S510: The application receives the structured causal chain data with the feedback module 400, and calls the visualization component to render it into an intuitive root cause analysis graph. The graph is presented to the manager in a graphical manner, which helps him quickly understand the whole picture of the event, make fair disposal decisions, and formulate targeted improvement measures.

Claims

1. An intelligent care service compliance monitoring and optimization system based on multi-modal analysis, characterized in that, The system comprises: a data collection module deployed on a mobile terminal device, configured to acquire interactive audio in a service process; a data transmission and storage module, configured to perform encrypted transmission and acoustic feature anonymization processing on the interactive audio; a core analysis engine deployed on a cloud server, configured to receive the interactive audio after the anonymization processing and perform multi-modal analysis and risk calculation; an application and feedback module deployed on the cloud server, configured to present analysis results and perform iterative optimization of an algorithm model based on artificial handling results.

2. The intelligent care service compliance monitoring and optimization system based on multi-modal analysis of claim 1, wherein, The core analysis engine comprises: a speech-to-text unit, configured to convert the interactive audio after the anonymization processing into a text sequence; a multi-modal analysis unit, configured to perform parallel processing on the text sequence and the interactive audio after the anonymization processing; a compliance rule engine unit, configured to make a risk decision based on the output of the multi-modal analysis unit; an automated root cause analysis unit, configured to perform deep causal tracing under the triggering of a specific event.

3. The intelligent care service compliance monitoring and optimization system based on multi-modal analysis of claim 2, wherein, The multi-modal analysis unit is configured to receive the text sequence output by the speech-to-text unit and receive the interactive audio after the anonymization processing, to extract semantic information of the text sequence, sentiment information of the interactive audio, keyword information of the text sequence, and speaker role information distinguishing the two parties of the interaction in parallel.

4. The intelligent care service compliance monitoring and optimization system based on multi-modal analysis of claim 3, wherein, The compliance rule engine unit is configured to load a rule set, wherein the rules in the rule set are defined as logical combination queries on the semantic information, sentiment information, keyword information and speaker role information, and a comprehensive risk score is calculated according to the matching result of the logical combination query.

5. The intelligent care services compliance monitoring and optimization system based on multi-modal analysis as claimed in claim 2, wherein, The automated root cause analysis unit is configured to be activated when receiving an instruction, to automatically aggregate context data related to a specific event, and to input the context data into a generative large language model to perform causal inference and output structured causal chain data.

6. The intelligent care service compliance monitoring and optimization system based on multi-modal analysis of claim 1, wherein, The application and feedback module comprises: a dashboard and early warning unit for real-time risk early warning and multi-dimensional data visualization; a closed-loop feedback and model iteration unit for capturing artificial handling conclusions and driving model self-optimization.

7. The intelligent care services compliance monitoring and optimization system based on multi-modal analysis as claimed in claim 6, wherein, The closed-loop feedback and model iteration unit is configured to capture artificial handling results for early warning events, use the artificial handling results as true value labels, and bind the true value labels with original data triggering the early warning to form labeled training samples, which are used for iterative optimization of the algorithm model in the core analysis engine.

8. The intelligent care services compliance monitoring and optimization system based on multi-modal analysis as claimed in claim 6, wherein, The application and feedback module is further configured to construct a dynamic risk portrait for a caregiver and quantify it into a risk portrait feature vector, calculate a matching degree score between the risk portrait feature vector and a course skill vector, and generate personalized ability improvement suggestions according to the matching degree score.

9. The intelligent care services compliance monitoring and optimization system based on multi-modal analysis as claimed in claim 1, wherein, The data transmission and storage module is configured to remove biological feature information in the audio signal that can be used to identify the identity of a specific speaker by modifying the fundamental frequency and formant of the interactive audio signal, to realize the acoustic feature anonymization processing.

10. The intelligent care service compliance monitoring and optimization system based on multi-modal analysis as claimed in claim 1, wherein, The data collection module comprises a recording control logic associated with the service order state machine, the recording control logic being configured to activate an audio collection function only when the service flow state machine migrates to the "in service" state and to discontinue the audio collection function when the service flow state machine leaves the "in service" state.

Images