Multi-modal nursing behavior recognition method, system and equipment based on nursing worker main view angle and medium

Abstract

The invention relates to the technical field of nursing behavior recognition, in particular to a multi-modal nursing behavior recognition method, system and device based on a nursing worker main view angle and a medium, and the method comprises the steps: obtaining a to-be-recognized sample, and carrying out the preprocessing and modal fusion, and obtaining a multi-modal feature sequence; and inputting the multi-modal feature sequence into a pre-trained nursing behavior recognition model constructed based on a neural semi-Markov model, respectively calculating a segment-level emission probability, a duration probability and a work order conditional transition probability, and outputting a nursing behavior prediction sequence by using segment-level dynamic planning. According to the invention, accurate identification of behavior boundaries and effective utilization of multi-modal information can be realized, and a nursing behavior prediction sequence with a time range is output.

Description

Technical Field

[0001] This application relates to the field of nursing behavior recognition technology, specifically to a multimodal nursing behavior recognition method, system, device and medium based on the caregiver's perspective. Background Technology

[0002] The fields of smart elderly care and home health monitoring have raised clear demands for improving the standardization and efficiency of nursing care. In existing technologies, wearable camera devices used by caregivers to capture their first-person perspective videos provide a crucial data source for analyzing nursing behavior, playing a positive role in improving the efficiency and standardization of nursing care.

[0003] In existing technologies, behavior recognition of caregiver-centric videos mainly adopts a fixed time window slicing classification method, which involves dividing continuous video into equal-length segments, extracting features from each segment independently, and classifying them. Some solutions attempt to introduce audio information and supplement visual analysis through feature splicing or decision fusion.

[0004] However, existing methods for recognizing nursing behaviors based on caregiver-centric videos still have significant shortcomings: fixed window segmentation is difficult to adapt to the large differences in the duration of nursing behaviors, often resulting in the fragmentation of a complete behavior or the merging of multiple short behaviors; in terms of multimodal processing, existing fusion methods fail to fully consider the correspondence between visual events and audio signals in fine temporal sequence, reducing the efficiency of multimodal information utilization; in addition, existing methods usually output discrete behavior labels, failing to provide accurate start and end times and duration information for each behavior. Summary of the Invention

[0005] To address the technical problems of existing nursing behavior recognition methods based on caregiver-centric videos, which employ fixed window slicing and simple multimodal fusion strategies, resulting in inaccurate behavior boundary identification, insufficient utilization of multimodal information, and inability to output behavior sequences with precise time ranges, this application provides a multimodal nursing behavior recognition method, system, device, and medium based on caregiver-centric videos. By using a nursing behavior recognition model based on a neural semi-Markov model for segment-level modeling and temporal deep fusion, it achieves accurate identification of behavior boundaries, effective utilization of multimodal information, and direct output of structured behavior sequences with time ranges.

[0006] Firstly, this application provides a method for recognizing multimodal nursing behaviors based on the caregiver's perspective, including: The video from the caregiver's main perspective, the synchronized audio, and the corresponding nursing work order to be identified are obtained as samples to be identified and preprocessed to extract visual feature vectors, audio feature vectors, and work order vectors. By using linear transformation or cross-attention mechanism, visual feature vectors, audio feature vectors and work order vector codes are modally fused to obtain a multimodal feature sequence; The multimodal feature sequence is input into the pre-trained nursing behavior recognition model, which is built based on a neural semi-Markov model. The segment-level emission probability, duration probability, and work order conditional transition probability are calculated based on the multimodal feature sequence. Using segment-level dynamic programming, the nursing behavior prediction sequence is output. The nursing behavior prediction sequence is a segment sequence containing multiple predicted behavior segments. Each predicted behavior segment contains a nursing behavior category label, the duration of the segment, and the confidence level.

[0007] It should be further explained that the multimodal nursing behavior recognition method based on the caregiver's perspective specifically includes the following steps: S1. A nursing behavior recognition model is constructed based on a neural semi-Markov model. The input of the nursing behavior recognition model is a multimodal feature sequence, and the output is the corresponding nursing behavior prediction sequence. The nursing behavior prediction sequence is a segment sequence containing multiple predicted behavior segments. Each predicted behavior segment contains a nursing behavior category label, the duration of the segment, and the confidence level. S2. Obtain multiple training samples. Each training sample includes a nursing behavior category and a nursing assistant's first-person perspective video with known start and end times for each nursing behavior, synchronized audio, and the corresponding nursing work order. The nursing work order contains all possible nursing behavior categories. S3. Preprocess each training sample to extract the corresponding visual feature vector, audio feature vector and work order vector. Then, use linear transformation or cross-attention mechanism to encode the visual feature vector, audio feature vector and work order vector for modal fusion to obtain the corresponding multimodal feature sequence. S4. Input the multimodal feature sequences of each training sample into the nursing behavior recognition model, and use the nursing behavior category and start and end time of each nursing behavior as the real label for supervised training to obtain the pre-trained nursing behavior recognition model. S5. Obtain the main-view video of the caregiver to be identified, the synchronized audio, and the corresponding nursing work order as the sample to be identified, and perform the same preprocessing and modal fusion as in step S3 to obtain the multimodal feature sequence of the sample to be identified; S6. Input the multimodal feature sequence of the sample to be identified into the pre-trained nursing behavior recognition model, and output the corresponding nursing behavior prediction sequence.

[0008] It should be further explained that the video is captured from the caregiver's own perspective using wearable camera equipment. Synchronous audio is captured using the microphone built into the wearable camera device or an external synchronous recording device.

[0009] It should be further noted that the preprocessing includes: The caregiver's first-person perspective video and synchronized audio are temporally aligned and sampled to extract the visual feature vector of the corresponding video frame and the audio feature vector of the corresponding audio frame for each sampled frame. Encode nursing work orders into work order vectors; Modal fusion of visual feature vectors, audio feature vectors, and work order vector encoding includes: For each sampled frame, the visual feature vector of the corresponding video frame, the audio feature vector of the corresponding audio frame, and the work order vector are fused into the fused feature vector of the sampled frame using a linear transformation or cross-attention mechanism. The fused feature vectors of all sampled frames are arranged in chronological order to form a multimodal feature sequence.

[0010] It should be further explained that extracting the visual feature vectors of video frames specifically includes: Use a hand / tool ​​detection model to extract regional features of hands or tools in the current video frame; Use a global visual encoder to extract global contextual features of the current video frame; The regional features and global context features are fused to form the visual feature vector of the current video frame.

[0011] It should be further noted that the hand / tool ​​detection model is a CNN or Transformer-based object detection or pose estimation model, including YOLOv7n, MediaPipe, DETR, or Faster R-CNN.

[0012] It should be further noted that the global visual encoder is a video understanding model, including Video-SwinTransformer, TimeSformer, MViT, I3D, or SlowFast.

[0013] It should be further explained that the audio feature vectors of audio frames are extracted using a temporal audio feature extraction network. The temporal audio feature extraction network is a cascaded network containing convolutional layers and temporal coding layers, including convolutional recurrent neural networks or PANNs models.

[0014] It should be further explained that the nursing work orders are encoded into work order vectors using an encoding network, which is a combination of Bi-GRU and Attention mechanisms.

[0015] It should be further noted that the nursing behavior recognition model includes a segment-level feature construction network, an emission probability calculation layer, a duration prediction network, a state transition probability layer, and a sequence decoder, wherein: The input to the segment-level feature construction network is a feature subsequence corresponding to any candidate time period extracted from the multimodal feature sequence. The importance weight of each frame feature in the feature subsequence is calculated through the attention pooling mechanism and then weighted and summed. The output is the segment-level feature vector of that time period. The input to the emission probability calculation layer is the segment-level feature vector output by the segment-level feature construction network. The segment-level feature vector is mapped to the logits of each nursing behavior category by a multilayer perceptron. Then, it is normalized by the Softmax function to output the emission probability of each nursing behavior category in the corresponding time period. The input to the duration prediction network includes the segment-level feature vector output by the segment-level feature construction network and the candidate nursing behavior category embedding vector from the learnable category embedding matrix. The parameters of the duration distribution are predicted by a small multilayer perceptron, and the output is the duration probability of the corresponding time period under the candidate nursing behavior category. The category embedding matrix is ​​learned through multimodal feature sequences during training. The input to the state transition probability layer includes the previous nursing behavior category embedding vector and the candidate nursing behavior category embedding vector from the category embedding matrix, as well as the work order vector; the state transition probability layer calculates the transition score between states through linear mapping, and performs bias adjustment in combination with the work order vector, and outputs the transition probability of transitioning from the previous nursing behavior category to the candidate nursing behavior category under the work order condition. The sequence decoder takes the emission probability, duration probability, and transition probability of all candidate time periods as input, solves for the optimal path that maximizes the joint probability through segment-level dynamic programming, and outputs a nursing behavior prediction sequence.

[0016] It should be further noted that the candidate time periods processed by the segment-level feature construction network are generated through enumeration, where for each candidate starting frame... Enumerate all that satisfy Candidate end frame Forming candidate time periods ;in The preset minimum duration, This is the preset maximum duration.

[0017] It should be further noted that the formula for calculating the emission probability in the emission probability calculation layer is as follows:

[0018] in, Indicates candidate time period The nursing behavior category is the candidate nursing behavior category. The probability of launch; Indicates candidate time period Segment-level feature vectors; Indicates the category of candidate nursing behavior; Indicates the total number of nursing behavior categories; Indicates an index of nursing behavior categories; This indicates the corresponding candidate nursing behavior category. Learnable weight vectors; This indicates the corresponding nursing behavior category. Learnable weight vectors; This indicates the corresponding candidate nursing behavior category. Learnable bias parameters; This indicates the corresponding nursing behavior category. Learnable bias parameters.

[0019] It should be further noted that in the duration prediction network, the duration probability adopts a log-normal distribution, and its probability density function is: in, This indicates that the nursing behavior category is a candidate nursing behavior category. Under the condition that the duration is The probability of; Indicates duration; This indicates that the nursing behavior category is a candidate nursing behavior category. ; Indicates candidate nursing behavior categories The mean parameter of the corresponding log-normal distribution; Indicates candidate nursing behavior categories The corresponding standard deviation parameter of the log-normal distribution; and The calculation formula is:

[0020] Indicates candidate time period Segment-level feature vectors; Indicates candidate nursing behavior categories The corresponding candidate nursing behavior category embedding vector, Obtained from the learnable category embedding matrix. , for The dimension; Indicates to and Perform vector concatenation operation; The parameter is Duration prediction network.

[0021] It should be further noted that the formula for calculating the transition probability in the state transition probability layer is:

[0022] in, Represents the nursing work order vector Under the conditions, from the previous nursing behavior category Transfer to candidate care behavior category The transition probability; Indicates the category of candidate nursing behavior; Indicates the category of the previous nursing care behavior; Represents a nursing work order vector; Represents the candidate nursing behavior categories in the nursing work order vector. The indicated value; Represents the nursing behavior category in the nursing work order vector. The indicated value; Indicates based on nursing behavior category Corresponding nursing behavior category embedding vector and candidate nursing behavior category The learnable score is calculated from the embedding vector of the corresponding candidate nursing behavior category; Indicates based on nursing behavior category Corresponding nursing behavior category embedding vector and nursing behavior category The learnable score is calculated from the corresponding nursing behavior category embedding vector; Hyperparameters for controlling the impact intensity of work orders; A smoothing constant used to prevent numerical overflow.

[0023] It should be further noted that the segment-level dynamic programming used in the sequence decoder is specifically the segment-level Viterbi algorithm, and its recursive formula is:

[0024] in, Indicates the first Frames are categorized by candidate nursing behaviors. The score of the optimal path at the end; Indicates the first Frame before a nursing behavior category The score of the optimal path at the end; Indicates duration; Indicates the category of the previous nursing care behavior; Indicates from time arrive Segment-level feature vectors; Indicates from time arrive During the specified time period, the nursing behavior category was a candidate nursing behavior category. The probability of launch; This indicates that the nursing behavior category is a candidate nursing behavior category. Under the condition that the duration is The probability of; Represents the nursing work order vector Under the conditions, from the previous nursing behavior category Transfer to candidate care behavior category The transition probability.

[0025] It should be further noted that the nursing behavior recognition model uses a joint loss function for supervised training, which is expressed as:

[0026] in, Indicates the total training loss; Indicates launch loss; Indicates the loss of duration; This represents the preset duration loss weighting coefficient; Indicates the transfer of losses; This represents the preset transfer loss weighting coefficient.

[0027] It should be further explained that launch loss The cross-entropy loss function is used for calculation; Duration loss The negative log-likelihood loss function is used for calculation; Transfer of losses The negative log-likelihood loss function is used for calculation.

[0028] It should be further noted that the training of the nursing behavior recognition model adopts a phased training strategy, including: First, pre-train the backbone networks for visual feature extraction and audio feature extraction; Then, with the backbone network parameters fixed, the segment-level feature construction network, emission probability calculation layer, duration prediction network, and state transition probability layer in the neural semi-Markov model are trained. Finally, the entire nursing behavior recognition model was fine-tuned end-to-end.

[0029] It should be further noted that the nursing behavior recognition model also supports weakly supervised training. When only nursing work orders are used as labels, a multi-instance learning loss function is employed for optimization. The multi-instance learning loss function is expressed as follows:

[0030] in, This represents the loss from learning multiple instances; Indicates the category of nursing behavior in the nursing work order. The true label; Indicates time The nursing behavior category is the nursing behavior category The probability of; This indicates the total number of nursing behavior categories; This refers to the entire video segment from the caregiver's perspective.

[0031] It should be further noted that this also includes: calculating the uncertainty estimate of each predicted behavior segment in the nursing behavior prediction sequence, and merging the uncertainty estimate into the corresponding predicted behavior segment.

[0032] It should be further noted that the uncertainty estimate is the sum of accidental uncertainty and cognitive uncertainty, where: Random uncertainty The calculation is based on the emission probability entropy, and the formula is as follows:

[0033] Cognitive uncertainty The prediction variance is calculated from multiple forward propagations, using the following formula:

[0034] in, Indicates candidate time period The nursing behavior category is the nursing behavior category The probability of launch; Indicates an index of nursing behavior categories; Indicates the total number of nursing behavior categories; Indicates variance calculation; This indicates the preset number of forward propagation samples. The value range is 10-1000; Indicates the index of the number of forward propagation samples; Indicates the first During the next forward propagation sampling, the candidate time period The nursing behavior category is the nursing behavior category The probability of emission.

[0035] It should be further noted that this also includes post-processing of the nursing behavior prediction sequence output by the nursing behavior recognition model. Post-processing includes time nonmaximum suppression and work order consistency verification.

[0036] It should be further noted that time nonmaximum suppression includes: Calculate the crossover ratio (CRR) between any two predicted behavior segments in the nursing behavior prediction sequence, and remove the predicted behavior segment with lower confidence from the two predicted behavior segments whose CRR exceeds a preset CRR threshold. The CRR is the ratio of the length of the overlap time of two predicted behavior segments to the sum of the lengths of the two predicted behavior segments. Delete predicted behavior segments whose duration is shorter than a preset duration threshold.

[0037] It should be further noted that the intersection-union ratio (IU) threshold ranges from 0.5 to 0.9. The duration threshold ranges from 30 to 90 seconds.

[0038] It should be further noted that the work order consistency check includes: Identify anomalous predicted behavior segments in the nursing behavior prediction sequence. Anomalous predicted behavior segments include: predicted behavior segments whose nursing behavior category labels do not belong to the categories listed in the nursing work order. Predicted behavior segments with a confidence level lower than a preset confidence threshold; Mark the abnormal prediction behavior segment as pending manual confirmation.

[0039] It should be further noted that the confidence threshold ranges from 0.6 to 0.8.

[0040] Secondly, this application provides a multimodal nursing behavior recognition system based on the caregiver's perspective, used to implement the aforementioned multimodal nursing behavior recognition method, including: The sample acquisition module is used to acquire the caregiver's first-person perspective video, synchronized audio, and corresponding nursing work order as samples to be identified. The sample preprocessing module is used to preprocess the samples to be identified, extracting visual feature vectors, audio feature vectors, and work order vectors. The multimodal feature sequence generation module is used to perform modal fusion of visual feature vectors, audio feature vectors, and work order vector codes using linear transformation or cross-attention mechanisms to obtain multimodal feature sequences. The nursing behavior prediction module is used to input multimodal feature sequences into a pre-trained nursing behavior recognition model. Based on the multimodal feature sequences, it calculates the segment-level emission probability, duration probability, and work order conditional transition probability, and outputs the nursing behavior prediction sequence using segment-level dynamic programming.

[0041] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described multimodal nursing behavior recognition method.

[0042] Fourthly, this application provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described multimodal nursing behavior recognition method.

[0043] As can be seen from the above technical solutions, this application has the following advantages: 1. This application uses a nursing behavior recognition model based on a neural semi-Markov model to perform segment-level modeling of nursing behavior, which can directly handle behavior segments of variable length, effectively solve the behavior fragmentation problem caused by fixed window division, maintain the temporal integrity of nursing behavior, and significantly improve the accuracy and boundary precision of long-term behavior recognition.

[0044] 2. This application achieves the collaborative utilization of visual, audio, and work order information at a fine-grained temporal level by constructing and deeply fusing temporally aligned multimodal feature sequences. This fully captures the temporal correlation of different modal events, enhances the nursing behavior recognition model's understanding and adaptability to complex scenarios, and improves the robustness and accuracy of the recognition results.

[0045] 3. This application encodes nursing work order information into work order vectors and integrates them into the recognition process, enabling the nursing behavior recognition model to make full use of domain prior knowledge. This effectively guides the model to focus on work order-related behavior categories and maintains good recognition performance even with limited labeled data. This improves the generalization ability and recall rate of the nursing behavior recognition model in practical scenarios. Attached Figure Description

[0046] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart of a multimodal nursing behavior recognition method based on the caregiver's perspective in one embodiment of this application.

[0048] Figure 2 This is a schematic block diagram of a multimodal nursing behavior recognition system based on the caregiver's perspective in one embodiment of this application.

[0049] Figure 3 This is a schematic diagram of the hardware structure of an electronic device in one embodiment of this application. Detailed Implementation

[0050] To make the purpose, features, and advantages of this application more apparent and understandable, specific embodiments and accompanying drawings will be used to clearly and completely describe the technical solution protected by this application. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] The multimodal nursing behavior recognition method of this application will be described in detail below. Specific details such as particular system structures and technologies are presented for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details.

[0052] In the multimodal nursing behavior recognition method disclosed in this application, the term "comprising" indicates the presence of the described feature, whole, step, operation, element, and / or component, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or sets thereof. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0053] To facilitate a clear description of the technical solutions of this application, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" do not necessarily imply that they are different.

[0054] The terms "one embodiment" or "some embodiments" used in this application mean that one or more embodiments of this application include the specific features, structures, or characteristics described in that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this application do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0055] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0056] The multimodal nursing behavior recognition method provided in this application embodiment is executed by a computer device, and correspondingly, the multimodal nursing behavior recognition system based on the caregiver's perspective runs in the computer device.

[0057] This application provides a method for recognizing multimodal nursing behaviors based on the caregiver's perspective, including: The video from the caregiver's main perspective, the synchronized audio, and the corresponding nursing work order to be identified are obtained as samples to be identified and preprocessed to extract visual feature vectors, audio feature vectors, and work order vectors. By using linear transformation or cross-attention mechanism, visual feature vectors, audio feature vectors and work order vector codes are modally fused to obtain a multimodal feature sequence; The multimodal feature sequence is input into the pre-trained nursing behavior recognition model, which is built based on a neural semi-Markov model. The segment-level emission probability, duration probability, and work order conditional transition probability are calculated based on the multimodal feature sequence. Using segment-level dynamic programming, the nursing behavior prediction sequence is output. The nursing behavior prediction sequence is a segment sequence containing multiple predicted behavior segments. Each predicted behavior segment contains a nursing behavior category label, the duration of the segment, and the confidence level.

[0058] Figure 1 This is a flowchart of a multimodal nursing behavior recognition method based on the caregiver's perspective, according to an embodiment of this application. Figure 1 The implementing entity can be a multimodal nursing behavior recognition system. Depending on different needs, the order of the steps in this flowchart can be changed, and some can be omitted.

[0059] like Figure 1 As shown, this multimodal nursing behavior recognition method based on the caregiver's perspective includes: Step S1: Construct a nursing behavior recognition model based on a neural semi-Markov model. The input of the nursing behavior recognition model is a multimodal feature sequence, and the output is the corresponding nursing behavior prediction sequence. The nursing behavior prediction sequence is a segment sequence containing multiple predicted behavior segments. Each predicted behavior segment contains a nursing behavior category label, the duration of the segment, and the confidence level.

[0060] By constructing a nursing behavior recognition model based on a neural semi-Markov model, and defining the input as a multimodal feature sequence and the output as a nursing behavior prediction sequence, a probabilistic modeling of behavior segments is realized. This model can directly output segment sequences containing category, duration, and confidence, thereby improving the continuity and structure of the recognition results.

[0061] In some specific embodiments, the nursing behavior recognition model includes a segment-level feature construction network, an emission probability calculation layer, a duration prediction network, a state transition probability layer, and a sequence decoder, wherein: The input to the segment-level feature construction network is a feature subsequence corresponding to any candidate time period extracted from the multimodal feature sequence. The importance weight of each frame feature in the feature subsequence is calculated through the attention pooling mechanism and then weighted and summed. The output is the segment-level feature vector of that time period. The input to the emission probability calculation layer is the segment-level feature vector output by the segment-level feature construction network. The segment-level feature vector is mapped to the logits of each nursing behavior category by a multilayer perceptron. Then, it is normalized by the Softmax function to output the emission probability of each nursing behavior category in the corresponding time period. The input to the duration prediction network includes the segment-level feature vector output by the segment-level feature construction network and the candidate nursing behavior category embedding vector from the learnable category embedding matrix. The parameters of the duration distribution are predicted by a small multilayer perceptron, and the output is the duration probability of the corresponding time period under the candidate nursing behavior category. The category embedding matrix is ​​learned through multimodal feature sequences during training. The input to the state transition probability layer includes the previous nursing behavior category embedding vector and the candidate nursing behavior category embedding vector from the category embedding matrix, as well as the work order vector; the state transition probability layer calculates the transition score between states through linear mapping, and performs bias adjustment in combination with the work order vector, and outputs the transition probability of transitioning from the previous nursing behavior category to the candidate nursing behavior category under the work order condition. The sequence decoder takes the emission probability, duration probability, and transition probability of all candidate time periods as input, solves for the optimal path that maximizes the joint probability through segment-level dynamic programming, and outputs a nursing behavior prediction sequence.

[0062] By constructing a nursing behavior recognition model that includes a segment-level feature construction network, an emission probability calculation layer, a duration prediction network, a state transition probability layer, and a sequence decoder, hierarchical probability modeling of behavior segments is achieved. This model can jointly optimize category, duration, and transition relationship, outputting a structured prediction sequence, thereby improving the accuracy and interpretability of the recognition results.

[0063] In some specific embodiments, the candidate time periods processed by the segment-level feature construction network are generated by enumeration, wherein for each candidate start frame... Enumerate all that satisfy Candidate end frame Forming candidate time periods ;in The preset minimum duration, This is the preset maximum duration.

[0064] By generating candidate time periods through enumeration and presetting minimum and maximum duration constraints on candidate end frames, the comprehensiveness and rationality of segment division are ensured, covering behavioral segments of different lengths. This provides rich candidate regions for segment-level feature extraction and reduces the risk of omissions or incorrect segmentation.

[0065] In some specific embodiments, the formula for calculating the transmission probability in the transmission probability calculation layer is as follows:

[0066] in, Indicates candidate time period The nursing behavior category is the candidate nursing behavior category. The probability of launch; Indicates candidate time period Segment-level feature vectors; Indicates the category of candidate nursing behavior; Indicates the total number of nursing behavior categories; Indicates an index of nursing behavior categories; This indicates the corresponding candidate nursing behavior category. Learnable weight vectors; This indicates the corresponding nursing behavior category. Learnable weight vectors; This indicates the corresponding candidate nursing behavior category. Learnable bias parameters; This indicates the corresponding nursing behavior category. Learnable bias parameters.

[0067] By using learnable weights and bias parameters to calculate emission probabilities and normalizing them based on segment-level feature vectors and the Softmax function, the probabilistic output of behavior categories is achieved. This enables the model to quantify the confidence level of each segment belonging to each category, improving the accuracy of category prediction and the reliability of uncertainty assessment.

[0068] In some specific embodiments, the duration prediction network uses a log-normal distribution for the duration probability, with the following probability density function: in, This indicates that the nursing behavior category is a candidate nursing behavior category. Under the condition that the duration is The probability of; Indicates duration; This indicates that the nursing behavior category is a candidate nursing behavior category. ; Indicates candidate nursing behavior categories The mean parameter of the corresponding log-normal distribution; Indicates candidate nursing behavior categories The corresponding standard deviation parameter of the log-normal distribution; and The calculation formula is:

[0069] Indicates candidate time period Segment-level feature vectors; Indicates candidate nursing behavior categories The corresponding candidate nursing behavior category embedding vector, Obtained from the learnable category embedding matrix. , for The dimension; Indicates to and Perform vector concatenation operation; The parameter is Duration prediction network.

[0070] By parameterizing the duration probability using a log-normal distribution and using the mean and standard deviation parameters of the duration prediction network output, probabilistic modeling of behavior duration is achieved, which can adapt to the duration changes of different behaviors and improve the flexibility and statistical rationality of duration prediction.

[0071] In some specific embodiments, the formula for calculating the transition probability in the state transition probability layer is as follows:

[0072] in, Represents the nursing work order vector Under the conditions, from the previous nursing behavior category Transfer to candidate care behavior category The transition probability; Indicates the category of candidate nursing behavior; Indicates the category of the previous nursing care behavior; Represents a nursing work order vector; Represents the candidate nursing behavior categories in the nursing work order vector. The indicated value; Represents the nursing behavior category in the nursing work order vector. The indicated value; Indicates based on nursing behavior category Corresponding nursing behavior category embedding vector and candidate nursing behavior category The learnable score is calculated from the embedding vector of the corresponding candidate nursing behavior category; Indicates based on nursing behavior category Corresponding nursing behavior category embedding vector and nursing behavior category The learnable score is calculated from the corresponding nursing behavior category embedding vector; Hyperparameters for controlling the impact intensity of work orders; A smoothing constant used to prevent numerical overflow.

[0073] By calculating the transition probability based on the category embedding vector and the work order vector, and by introducing the work order indication value and control parameters for bias adjustment, soft constraint modeling of behavior transition is realized. This enables the model to optimize the sequence logic using work order priors, thereby improving the accuracy and context relevance of behavior sequence prediction.

[0074] In some specific embodiments, the segment-level dynamic programming used in the sequence decoder is specifically the segment-level Viterbi algorithm, whose recursive formula is:

[0075] in, Indicates the first Frames are categorized by candidate nursing behaviors. The score of the optimal path at the end; Indicates the first Frame before a nursing behavior category The score of the optimal path at the end; Indicates duration; Indicates the category of the previous nursing care behavior; Indicates from time arrive Segment-level feature vectors; Indicates from time arrive During the specified time period, the nursing behavior category was a candidate nursing behavior category. The probability of launch; This indicates that the nursing behavior category is a candidate nursing behavior category. Under the condition that the duration is The probability of; Represents the nursing work order vector Under the conditions, from the previous nursing behavior category Transfer to candidate care behavior category The transition probability.

[0076] By using the recursive formula of the segment-level Viterbi algorithm to solve for the optimal path, the maximum joint probability search of the global sequence is achieved. This can efficiently find the most likely behavior segment sequence, improve the decoding accuracy and computational efficiency, and is suitable for the reasoning needs of long-term sequences.

[0077] Step S2: Obtain multiple training samples. Each training sample includes a nursing behavior category and a nursing assistant's first-person perspective video with known start and end times for each nursing behavior, synchronized audio, and the corresponding nursing work order. The nursing work order contains all possible nursing behavior categories.

[0078] By acquiring a rich and accurately labeled multimodal data source, including caregiver-centric videos, synchronized audio, and care work orders for each training sample with known care behavior categories and start and end times, the model training can learn behavioral features and multimodal associations based on real spatiotemporal information. This improves the accuracy and generalization ability of subsequent recognition models and reduces performance degradation caused by insufficient data or labeling errors. In some specific embodiments, wearable camera devices worn by caregivers are used to capture video from the caregiver's own perspective; Synchronous audio is captured using the microphone built into the wearable camera device or an external synchronous recording device.

[0079] By using wearable cameras worn by caregivers to collect video and synchronized audio from their own perspective, the consistency of perspective and temporal synchronization of data collection were ensured. This provided a high-quality and realistic input source for subsequent multimodal feature extraction and fusion, enhanced the model's adaptability and reliability in real-world complex environments, and reduced recognition errors caused by device differences or temporal misalignments.

[0080] Step S3: Preprocess each training sample to extract the corresponding visual feature vector, audio feature vector and work order vector. Then, use linear transformation or cross-attention mechanism to encode the visual feature vector, audio feature vector and work order vector for modal fusion to obtain the corresponding multimodal feature sequence.

[0081] By preprocessing and modal fusion of each training sample to obtain a multimodal feature sequence, the transformation from raw data to structured features is realized, ensuring the temporal alignment and fusion of multimodal information, providing a unified and efficient representation for model input, and improving feature quality and model processing speed.

[0082] In some specific embodiments, the preprocessing includes: The caregiver's first-person perspective video and synchronized audio are temporally aligned and sampled to extract the visual feature vector of the corresponding video frame and the audio feature vector of the corresponding audio frame for each sampled frame. Encode nursing work orders into work order vectors; Modal fusion of visual feature vectors, audio feature vectors, and work order vector encoding includes: For each sampled frame, the visual feature vector of the corresponding video frame, the audio feature vector of the corresponding audio frame, and the work order vector are fused into the fused feature vector of the sampled frame using a linear transformation or cross-attention mechanism. The fused feature vectors of all sampled frames are arranged in chronological order to form a multimodal feature sequence.

[0083] By temporally aligning frame sampling of the caregiver's first-person perspective video and synchronized audio, and extracting visual feature vectors, audio feature vectors, and encoding nursing work orders into work order vectors, a multimodal feature sequence is fused together. This achieves a unified representation of multimodal data at a fine temporal level, providing structured and aligned input for subsequent model processing, and improving the completeness of feature expression and the efficiency of model inference.

[0084] In some specific embodiments, extracting the visual feature vectors of video frames specifically includes: Use a hand / tool ​​detection model to extract regional features of hands or tools in the current video frame; Use a global visual encoder to extract global contextual features of the current video frame; The regional features and global context features are fused to form the visual feature vector of the current video frame.

[0085] By using a hand / tool ​​detection model to extract regional features of the hand or tool, and combining them with a global visual encoder to extract global contextual features, the model is fused into a visual feature vector. This achieves a collaborative representation of local details and the global scene, enhances the model's ability to perceive key actions and the environment in nursing behavior, and improves the robustness and discriminability of visual features under complex perspectives.

[0086] In some specific embodiments, the hand / tool ​​detection model is a CNN or Transformer-based object detection or pose estimation model, including YOLOv7n, MediaPipe, DETR, or Faster R-CNN.

[0087] By employing CNN or Transformer-based hand / tool ​​detection models such as YOLOv7n, MediaPipe, DETR, or Faster R-CNN, a flexible and efficient regional feature extraction scheme is provided, enabling the model to adapt to different computing resources and accuracy requirements, enhancing the accuracy and speed of hand or tool detection, and laying the foundation for subsequent visual feature fusion.

[0088] In some specific embodiments, the global visual encoder is a video understanding model, including Video-SwinTransformer, TimeSformer, MViT, I3D, or SlowFast.

[0089] By using video understanding models such as Video-Swin Transformer, TimeSformer, MViT, I3D, or SlowFast as global visual encoders, deep spatiotemporal feature extraction of video frames is achieved, capturing long-term dependencies and dynamic changes in nursing behaviors, improving the richness of global contextual features and the model's ability to understand behavioral semantics.

[0090] In some specific embodiments, a temporal audio feature extraction network is used to extract audio feature vectors of audio frames. The temporal audio feature extraction network is a cascaded network containing convolutional layers and temporal coding layers, including convolutional recurrent neural networks or PANNs models.

[0091] By using temporal audio feature extraction networks such as convolutional recurrent neural networks or PANNs to extract audio feature vectors, temporal modeling and feature abstraction of audio events are achieved. This effectively captures key sound cues in nursing scenarios, provides complementary audio information for multimodal fusion, and enhances the model's sensitivity to behavioral boundaries and recognition accuracy.

[0092] In some specific embodiments, an encoding network is used to encode nursing work orders into work order vectors. The encoding network is a combination of Bi-GRU and Attention mechanism.

[0093] By using a combined network of Bi-GRU and Attention mechanisms to encode nursing work orders into work order vectors, sequence modeling and key information focusing of work order texts are achieved. This allows task priors to be incorporated into the model in vector form, guiding the behavior recognition process and improving the model's generalization ability and task relevance under sparse annotation.

[0094] Step S4: Input the multimodal feature sequences of each training sample into the nursing behavior recognition model, and use the nursing behavior category and start and end time of each nursing behavior as the real label for supervised training to obtain the pre-trained nursing behavior recognition model.

[0095] By inputting the multimodal feature sequences of training samples into the model and using real labels for supervised training, the optimal learning of model parameters was achieved. This enabled the model to accurately capture behavioral patterns from labeled data, improving recognition accuracy and robustness, and providing a reliable training foundation for practical applications.

[0096] In some specific embodiments, a joint loss function is used for supervised training, which is expressed as:

[0097] in, Indicates the total training loss; Indicates launch loss; Indicates the loss of duration; This represents the preset duration loss weighting coefficient; Indicates the transfer of losses; This represents the preset transfer loss weighting coefficient.

[0098] By using a joint loss function that combines emission loss, duration loss, and transition loss for supervised training, multi-task collaborative optimization is achieved, enabling the model to balance the learning of category, duration, and transition relationships, thereby improving training stability and overall recognition performance.

[0099] In some specific embodiments, launch loss The cross-entropy loss function is used for calculation; Duration loss The negative log-likelihood loss function is used for calculation; Transfer of losses The negative log-likelihood loss function is used for calculation.

[0100] By employing cross-entropy loss to calculate emission loss and negative log-likelihood loss to calculate duration loss and transition loss, a standardized loss calculation method is provided, ensuring the effectiveness and convergence of gradient descent, optimizing model parameters, and improving training efficiency.

[0101] In some specific embodiments, a staged training strategy is employed, including: First, pre-train the backbone networks for visual feature extraction and audio feature extraction; Then, with the backbone network parameters fixed, the segment-level feature construction network, emission probability calculation layer, duration prediction network, and state transition probability layer in the neural semi-Markov model are trained. Finally, the entire nursing behavior recognition model was fine-tuned end-to-end.

[0102] By adopting a phased training strategy, including pre-training the backbone network, training some modules with fixed parameters, and end-to-end fine-tuning, the training process was gradually optimized, reducing the risk of overfitting and improving the model's generalization ability and deployment adaptability.

[0103] In some specific embodiments, the nursing behavior recognition model also supports weakly supervised training. When only nursing work orders are used as labels, a multi-instance learning loss function is employed for optimization. The multi-instance learning loss function is expressed as follows:

[0104] in, This represents the loss from learning multiple instances; Indicates the category of nursing behavior in the nursing work order. The true label; Indicates time The nursing behavior category is the nursing behavior category The probability of; This indicates the total number of nursing behavior categories; This refers to the entire video segment from the caregiver's perspective.

[0105] By supporting weakly supervised training and optimizing using a multi-instance learning loss function, the model can still learn effectively even with only nursing work order labels, reducing the reliance on fine annotation, expanding application scenarios, and improving data utilization efficiency.

[0106] Step S5: Obtain the main-view video of the caregiver to be identified, the synchronized audio, and the corresponding nursing work order as the sample to be identified. Perform the same preprocessing and modal fusion as in step S3 to obtain the multimodal feature sequence of the sample to be identified.

[0107] By acquiring the caregiver's first-person perspective video, synchronized audio, and nursing work orders as samples to be identified, and performing the same preprocessing as during training, the consistency of data processing in the inference stage is ensured, enabling the model to efficiently process new samples and maintaining the continuity of the identification process and the comparability of results.

[0108] Step S6: Input the multimodal feature sequence of the sample to be identified into the pre-trained nursing behavior recognition model, and output the corresponding nursing behavior prediction sequence.

[0109] By inputting the multimodal feature sequence of the sample to be identified into a pre-trained nursing behavior recognition model and outputting a predicted sequence, automatic reasoning for unknown samples is realized, which can quickly generate structured behavior recognition results, improving the practicality and deployment efficiency of the method.

[0110] In some specific embodiments, step S6 further includes: calculating the uncertainty estimate of each predicted behavior segment in the nursing behavior prediction sequence, and merging the uncertainty estimate into the corresponding predicted behavior segment.

[0111] By calculating the uncertainty estimate of each predicted behavior segment in the nursing behavior prediction sequence and merging them into the corresponding segment, the reliability of the identification results is quantified, enabling users to assess the prediction confidence and improving the practicality and decision support capabilities of the output results.

[0112] In some specific embodiments, the uncertainty estimate is the sum of accidental uncertainty and cognitive uncertainty, wherein: Random uncertainty The calculation is based on the emission probability entropy, and the formula is as follows:

[0113] Cognitive uncertainty The prediction variance is calculated from multiple forward propagations, using the following formula:

[0114] in, Indicates candidate time period The nursing behavior category is the nursing behavior category The probability of launch; Indicates an index of nursing behavior categories; Indicates the total number of nursing behavior categories; Indicates variance calculation; This indicates the preset number of forward propagation samples. The value range is 10-1000; Indicates the index of the number of forward propagation samples; Indicates the first During the next forward propagation sampling, the candidate time period The nursing behavior category is the nursing behavior category The probability of emission.

[0115] By distinguishing between accidental uncertainty and cognitive uncertainty, and using emission probability entropy and variance from multiple forward propagation to calculate respectively, a detailed uncertainty measure is provided, which can identify data noise and cognitive limitations of the model, thereby enhancing the accuracy and interpretability of uncertainty estimation.

[0116] In some specific embodiments, step S7 is also included: post-processing the nursing behavior prediction sequence output by S6, which includes time non-maximum suppression and work order consistency verification.

[0117] By post-processing the output nursing behavior prediction sequence, including time nonmaximum suppression and work order consistency verification, the prediction results are optimized and verified, overlapping segments and outliers are reduced, and the cleanliness and reliability of the final output sequence are improved.

[0118] In some specific embodiments, time nonmaximum suppression includes: Calculate the crossover ratio (CRR) between any two predicted behavior segments in the nursing behavior prediction sequence, and remove the predicted behavior segment with lower confidence from the two predicted behavior segments whose CRR exceeds a preset CRR threshold. The CRR is the ratio of the length of the overlap time of two predicted behavior segments to the sum of the lengths of the two predicted behavior segments. Delete predicted behavior segments whose duration is shorter than a preset duration threshold.

[0119] By calculating the intersection-union ratio between predicted behavior segments and removing low-confidence segments, as well as deleting short-duration segments, temporal deduplication and filtering are achieved, ensuring the simplicity and accuracy of the output sequence and reducing false alarms and redundant information.

[0120] In some specific embodiments, the cross-union ratio (CUI) threshold ranges from 0.5 to 0.9; The duration threshold ranges from 30 to 90 seconds.

[0121] By setting specific ranges for the intersection-union ratio (IU) and duration thresholds, adjustable parameter control is provided, enabling post-processing to adapt to different scenario requirements, balancing recall and precision, and enhancing the flexibility and applicability of the method.

[0122] In some specific embodiments, work order consistency verification includes: Identify anomalous predicted behavior segments in the nursing behavior prediction sequence. Anomalous predicted behavior segments include: predicted behavior segments whose nursing behavior category labels do not belong to the categories listed in the nursing work order. Predicted behavior segments with a confidence level lower than a preset confidence threshold; Mark the abnormal prediction behavior segment as pending manual confirmation.

[0123] By identifying and labeling segments of abnormal predicted behavior, such as segments whose categories do not belong to work orders or have low confidence, automated quality checks are achieved, enabling users to quickly locate suspicious results and improving the reliability of output results and the efficiency of manual review.

[0124] In some specific embodiments, the confidence threshold ranges from 0.6 to 0.8.

[0125] By setting the range of confidence threshold values, a standardized basis for anomaly judgment is provided, making work order consistency verification operable, optimizing the post-processing workflow, and reducing subjective bias.

[0126] The following are embodiments of the multimodal nursing behavior recognition system based on the caregiver's perspective provided in this application. This multimodal nursing behavior recognition system based on the caregiver's perspective belongs to the same inventive concept as the multimodal nursing behavior recognition methods in the above embodiments. For details not described in detail in the embodiments of the multimodal nursing behavior recognition system, please refer to the embodiments of the multimodal nursing behavior recognition method based on the caregiver's perspective described above.

[0127] like Figure 2 As shown, the multimodal nursing behavior recognition system based on the caregiver's perspective includes: The sample acquisition module is used to acquire the caregiver's first-person perspective video, synchronized audio, and corresponding nursing work order as samples to be identified. The sample preprocessing module is used to preprocess the samples to be identified, extracting visual feature vectors, audio feature vectors, and work order vectors. The multimodal feature sequence generation module is used to perform modal fusion of visual feature vectors, audio feature vectors, and work order vector codes using linear transformation or cross-attention mechanisms to obtain multimodal feature sequences. The nursing behavior prediction module is used to input multimodal feature sequences into a pre-trained nursing behavior recognition model. Based on the multimodal feature sequences, it calculates the segment-level emission probability, duration probability, and work order conditional transition probability, and outputs the nursing behavior prediction sequence using segment-level dynamic programming.

[0128] The multimodal nursing behavior recognition system in this embodiment is used to implement a multimodal nursing behavior recognition method based on the caregiver's perspective.

[0129] This application also provides an electronic device for implementing the various embodiments of this application. Figure 3 To illustrate the hardware structure of an electronic device according to various embodiments of this application, as shown in the following diagram... Figure 3 As shown, the electronic device includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor.

[0130] Those skilled in the art will understand that the electronic device structure involved in the embodiments of this application does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0131] In embodiments of this application, electronic devices include, but are not limited to, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of this application described and / or claimed herein.

[0132] In this application embodiment, the processor can be implemented using at least one of an Application-Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a processor, a controller, a microcontroller, a microprocessor, or an electronic unit designed to perform the functions described herein. In some cases, such implementations can be implemented within a controller. For software implementations, implementations such as processes or functions can be implemented with separate software modules that allow the performance of at least one function or operation. The software code can be implemented by a software application (or program) written in any suitable programming language, and the software code can be stored in memory and executed by the controller.

[0133] In addition, the electronic device includes some functional modules not shown, which will not be described in detail here.

[0134] Those skilled in the art will understand that the various aspects of the electronic device provided in this application can be implemented as a system, method, or program product. Therefore, the various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0135] This application also provides a storage medium storing a program product capable of implementing a multimodal nursing behavior recognition method based on the caregiver's perspective. In some possible implementations, various aspects of this application can also be implemented as a program product including program code that, when run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this application.

[0136] The storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example,, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0137] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for recognizing multimodal nursing behaviors based on the caregiver's perspective, characterized in that, include: The video from the caregiver's main perspective, the synchronized audio, and the corresponding nursing work order to be identified are obtained as samples to be identified and preprocessed to extract visual feature vectors, audio feature vectors, and work order vectors. By using linear transformation or cross-attention mechanism, visual feature vectors, audio feature vectors and work order vector codes are modally fused to obtain a multimodal feature sequence; The multimodal feature sequence is input into the pre-trained nursing behavior recognition model, which is built based on a neural semi-Markov model. The segment-level emission probability, duration probability, and work order conditional transition probability are calculated based on the multimodal feature sequence. Using segment-level dynamic programming, the nursing behavior prediction sequence is output. The nursing behavior prediction sequence is a segment sequence containing multiple predicted behavior segments. Each predicted behavior segment contains a nursing behavior category label, the duration of the segment, and the confidence level.

2. The multimodal nursing behavior recognition method as described in claim 1, characterized in that, preprocessing include: The caregiver's first-person perspective video and synchronized audio are temporally aligned and sampled to extract the visual feature vector of the corresponding video frame and the audio feature vector of the corresponding audio frame for each sampled frame. Encode nursing work orders into work order vectors; Modal fusion of visual feature vectors, audio feature vectors, and work order vector encoding includes: For each sampled frame, the visual feature vector of the corresponding video frame, the audio feature vector of the corresponding audio frame, and the work order vector are fused into the fused feature vector of the sampled frame using a linear transformation or cross-attention mechanism. The fused feature vectors of all sampled frames are arranged in chronological order to form a multimodal feature sequence.

3. The multimodal nursing behavior recognition method as described in claim 1, characterized in that, The nursing behavior recognition model includes a segment-level feature construction network, an emission probability calculation layer, a duration prediction network, a state transition probability layer, and a sequence decoder, wherein: The input to the segment-level feature construction network is a feature subsequence corresponding to any candidate time period extracted from the multimodal feature sequence. The importance weight of each frame feature in the feature subsequence is calculated through the attention pooling mechanism and then weighted and summed. The output is the segment-level feature vector of that time period. The input to the emission probability calculation layer is the segment-level feature vector output by the segment-level feature construction network. The segment-level feature vector is mapped to the logits of each nursing behavior category by a multilayer perceptron. Then, it is normalized by the Softmax function to output the emission probability of each nursing behavior category in the corresponding time period. The input to the duration prediction network includes the segment-level feature vector output by the segment-level feature construction network and the candidate nursing behavior category embedding vector from the learnable category embedding matrix. The parameters of the duration distribution are predicted by a small multilayer perceptron, and the output is the duration probability of the corresponding time period under the candidate nursing behavior category. The category embedding matrix is ​​learned through multimodal feature sequences during training. The input to the state transition probability layer includes the previous nursing behavior category embedding vector and the candidate nursing behavior category embedding vector from the category embedding matrix, as well as the work order vector; the state transition probability layer calculates the transition score between states through linear mapping, and performs bias adjustment in combination with the work order vector, and outputs the transition probability of transitioning from the previous nursing behavior category to the candidate nursing behavior category under the work order condition. The sequence decoder takes the emission probability, duration probability, and transition probability of all candidate time periods as input, solves for the optimal path that maximizes the joint probability through segment-level dynamic programming, and outputs a nursing behavior prediction sequence.

4. The multimodal nursing behavior recognition method as described in claim 1, characterized in that, The nursing behavior recognition model is trained under supervision using a joint loss function, which is expressed as follows: in, Indicates the total training loss; Indicates launch loss; Indicates the loss of duration; This represents the preset duration loss weighting coefficient; Indicates the transfer of losses; This represents the preset transfer loss weighting coefficient.

5. The multimodal nursing behavior recognition method as described in claim 4, characterized in that, Launch loss The cross-entropy loss function is used for calculation; Duration loss The negative log-likelihood loss function is used for calculation; Transfer of losses The negative log-likelihood loss function is used for calculation.

6. The multimodal nursing behavior recognition method as described in claim 1, characterized in that, The training of the nursing behavior recognition model adopts a phased training strategy, including: First, pre-train the backbone networks for visual feature extraction and audio feature extraction; Then, with the backbone network parameters fixed, the segment-level feature construction network, emission probability calculation layer, duration prediction network, and state transition probability layer in the neural semi-Markov model are trained. Finally, the entire nursing behavior recognition model was fine-tuned end-to-end.

7. The multimodal nursing behavior recognition method as described in claim 1, characterized in that, After outputting the nursing behavior prediction sequence, the process also includes: calculating the uncertainty estimate of each predicted behavior segment in the nursing behavior prediction sequence, and merging the uncertainty estimates into the corresponding predicted behavior segment.

8. A multimodal nursing behavior recognition system based on the caregiver's perspective, characterized in that, To implement the multimodal nursing behavior recognition method as described in any one of claims 1-7, the method includes: The sample acquisition module is used to acquire the caregiver's first-person perspective video, synchronized audio, and corresponding nursing work order as samples to be identified. The sample preprocessing module is used to preprocess the samples to be identified, extracting visual feature vectors, audio feature vectors, and work order vectors. The multimodal feature sequence generation module is used to perform modal fusion of visual feature vectors, audio feature vectors, and work order vector codes using linear transformation or cross-attention mechanisms to obtain multimodal feature sequences. The nursing behavior prediction module is used to input multimodal feature sequences into a pre-trained nursing behavior recognition model. Based on the multimodal feature sequences, it calculates the segment-level emission probability, duration probability, and work order conditional transition probability, and outputs the nursing behavior prediction sequence using segment-level dynamic programming.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes a computer program, it implements the steps of the multimodal nursing behavior recognition method as described in any one of claims 1-7.

10. A storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the multimodal nursing behavior recognition method as described in any one of claims 1-7.

Images