A nursing behavior recognition method in a privacy scene

Abstract

The application discloses a nursing behavior recognition method in a privacy scene, and extracts action features of a nursing person and a person to be nursed in a nursing behavior on the basis of a depth image, so as to construct a trajectory feature matrix in a time sequence and a spatial range, analyze the trajectory feature matrix by using a machine learning algorithm, and realize nursing behavior recognition and classification of multiple persons in the privacy scene. The nursing behavior recognition method provided by the application is not dependent on traditional image features, is not easily affected by environmental and light changes, and only depends on a depth image sequence, which is different from audio and video collection of a traditional camera, and can effectively protect the privacy of a person to be recognized. The application comprehensively analyzes the identity, behavior and image features of multiple persons in a depth image, gives a nursing behavior recognition mode based on joint analysis of behaviors of participants, and effectively makes up for the defect that a traditional model cannot recognize joint behaviors in special scenes.

Description

TECHNICAL FIELD

[0001] The application relates to a nursing behavior recognition method in a privacy scene and belongs to the technical field of depth image processing. BACKGROUND

[0002] Nursing behaviors exist in private scenes such as old-age care, patient care, infant care, etc. The nursing behaviors are service behaviors such as daily life care, safety guarantee, rehabilitation training, and mind and body counseling, clinical medical care, etc. provided by caregivers such as nursing staff, medical staff, and managers to the cared-for such as the elderly, patients, and infants. In the nursing implementation process, there are complex interactions between the caregivers and the cared-for, and with the increasing demand for effective monitoring and supervision of these complex nursing behaviors in the private field, how to effectively recognize the above nursing behaviors while protecting privacy is a problem to be solved.

[0003] Using a traditional camera for behavior recognition has a serious privacy leakage problem. With the development of depth image sensors, the use of infrared light reflection imaging makes it possible to collect image data that does not contain human faces and body details, thus eliminating the collection of sensitive information from the source and protecting user privacy, which provides a possibility for behavior monitoring and supervision in a private scene. However, common behavior recognition methods are based on the analysis and processing of images collected by traditional cameras, which has great limitations in a private scene. In addition to being unable to effectively analyze depth images, it is also difficult to effectively distinguish between caregivers and cared-for, making it difficult to model complex nursing behaviors in detail.

[0004] Although some related technical methods have been used to solve the problem of behavior recognition in similar private scenes, the scenes involved and the methods used are limited, and there is a lack of comprehensive analysis of nursing behavior participant data, making it difficult to meet the needs of nursing behavior recognition in a private scene. For example, Chinese patent CN109086659A provides a method for behavior classification in depth images combined with skeleton features, which proves the feasibility of behavior recognition in depth images, but it only emphasizes behavior classification and does not define specific behaviors or model specific scenes. Chinese patent CN111860434B proposes a behavior recognition method for intelligent home scenes and achieves good results in intelligent home scenes, but this method uses traditional image modeling and does not use private and secure depth image data, which does not eliminate the collection of private data from the source. SUMMARY

[0005] The technical problem to be solved by the present application is that there is currently a lack of effective means for recognizing nursing behaviors in a private scene.

[0006] To address the aforementioned technical problems, the present invention provides a method for recognizing nursing behaviors in privacy scenarios, characterized by comprising the following steps:

[0007] Step 1: For the input raw depth image sequence x1, x2, ..., x... N Process the data and extract the human detection bounding boxes B1, B2, ..., B1 for all participants. N And the human body ROI regions M1, M2, ..., M N Among them, B N M represents the detection bounding boxes for multiple human bodies in the original depth image of frame N. N The ROI region segmentation results for multiple human bodies in the original depth image of frame N.

[0008] Step 2: Analyze the multi-person human detection bounding boxes B1, B2, ..., B in the original depth image sequence. N Track the movement trajectories of different individuals and obtain multi-person human tracking information sequences BT1, BT2, ..., BT1 for all participants. N , among which, BT N Let BT be the tracking information of multiple human bodies in the Nth frame. N = [tn1, tn2, ..., tn] k ], tn k BT is the tracking number of the kth person in the current frame's human detection bounding box. N The definition is shown in equation (1):

[0009] BT N =f t (f bv (B N ...B N-i ), f gv (VG N , ...VG N-i ), f mv (VM N , ...VM N-i )) (1)

[0010] In equation (1), i is the number of frames associated with the current frame and the preceding frame, f bv f is the cross-union ratio calculation function for human body detection. gv f is a function for calculating pixel feature changes in computer graphics. mv f is the momentum-related direction calculation function. t (f bv f gv f mv ) is the function for calculating the human tracking number, VG N To utilize the graphics feature analysis algorithm f g (BN The calculated graphical eigenvectors, VM N To utilize machine learning algorithm f m (B N PS N The calculated motion feature vector VM of the human detection box N PS1, PS2, ..., PS N To be used with human detection frames B1, B2, ..., B N The pixel sequence of the corresponding region;

[0011] Step 3: Based on the tracking results, analyze the key point features of different human bodies and construct the coordinates of key points for all participants. This includes the following steps:

[0012] Step 3-1: Combine the human body tracking information sequence BT1, BT2, ..., BT N And the ROI regions M1, M2, ..., M of multiple human bodies in the image sequence. N Using deep learning algorithm f k (BT N M N Calculate the set of human keypoint coordinates KC for different participants on each frame of the 2D depth image. N This yields the sequence of two-dimensional human body keypoint coordinate sets KC1, KC2, ..., KC N Among them, the set of human body key point coordinates KC N It can be expressed as the following formula (2):

[0013]

[0014] In equation (2), k represents the number of people in the multi-person human detection box in the current frame. This represents the set of x and y coordinates of the q-th keypoint of the k-th person in the current frame, calculated using a deep learning algorithm, denoted as q / x.

[0015] Step 3-2: Combine the human body ROI regions M1, M2, ..., M N And the sequence of two-dimensional human body keypoint coordinate sets KC1, KC2, ..., KC N Using spatial coordinate transformation algorithm f z (M N KC N Calculate the three-dimensional coordinate set KC′ of key points on the human body in space. N This yields the sequence of three-dimensional coordinate sets of key points: KC′1, KC′2, ... KC′. N Among them, the three-dimensional coordinate set KC′ N It can be expressed as the following formula (3):

[0016]

[0017] In formula (3), k represents the number of people in the current frame multi-person human detection frame, represents the x, y, z coordinate set of the qth key point of the kth human body in the current frame calculated by the spatial coordinate conversion algorithm in the three-dimensional coordinate system, wherein x, y represent the two-dimensional coordinates of the current pixel on the image, and z represents the gray value of the current pixel, denoted as

[0018] Step 4: In combination with the key point coordinates, analyze the human body displacement change between the image related frames, and calculate the motion vitality features of all participants, including the following steps:

[0019] Step 4-1: In combination with the voxel information vi of the unit pixel in the depth image in the three-dimensional space x，y，z and the key point three-dimensional coordinate set sequence KC'1, KC'2,..., KC' N , calculate the actual key point coordinate set KCR1, KCR2,..., KCR N of each frame;

[0020] Step 4-2: Calculate the average actual coordinate set KCA1, KCA2,..., KCA N of all key points in each limb region;

[0021] Step 4-3: On the basis of the average actual coordinate set KCA1, KCA2,..., KCA N , use the human vitality value algorithm f hv (KCA N ,..., KCA N-i ) to calculate the multi-person motion vitality feature HV N , and obtain the motion vitality feature sequence HV1, HV2,..., HV N , wherein f hv (KCA N ,..., KCA N-i ) is calculated according to the following formula (4):

[0022]

[0023] In formula (4), p is the number of the current frame and the related frame above, i is the number of the current frame and the maximum frame above, q is the current limb region number, b is the maximum number of limb regions, d is the actual distance calculation function of the average coordinates of the limb between two frames, w q is the limb region vitality value weight operator, is the actual coordinate of the specific limb region;

[0024] Step 5: Combine the multi-person human detection frames B1, B2, ..., B N Multi-person human tracking information sequence BT1, BT2, ..., BT N And the multi-person motion vitality feature sequences HV1, HV2, ..., HV N The activity characteristics and related thresholds of each participating role in the nursing scenario are constructed and defined, and then a human identification algorithm f is constructed. id (B N BT N HV N ), calculate the identity vector ID of multiple human bodies in each frame. N This yields a vector sequence of multiple individuals' identity information: ID1, ID2, ..., ID1. N ;

[0025] Step 6: Use deep learning algorithms to build a participant behavior recognition algorithm model f pb (VG N KC′ N KCA N After training the model, extract the human region graphics feature vector sequence VG1, VG2, ..., VG from the image sequence to be predicted. N A sequence of three-dimensional coordinates of key points on the human body in space, KC′1, KC′2, ... KC′ N The set of average actual coordinates of all key points in various limb regions of the human body after transformation based on depth values: KCA1, KCA2, ..., KCA N Using participant behavior recognition algorithm model f pb (VG N KC′ N KCA N The intermediate process of HAR algorithm derivation before predicting image sequences and obtaining the final classification result. N As participant behavioral characteristics, the participant behavioral characteristic analysis result sequence HAR1, HAR2, ..., HAR was obtained. N ;

[0026] Step 7: Construct a nursing behavior recognition algorithm model using deep learning algorithms. nursingbehavior (BT N HV N ID N HAR N Extract the contextual multi-person human tracking information sequence BT1, BT2, ..., BT from the image sequence to be predicted. N Multi-person motion activity feature sequences HV1, HV2, ..., HV N, multi-person human identity information vector sequence ID1, ID2,..., ID N , participant behavior feature analysis result sequence HAR1, HAR2,..., HAR N , using a nursing behavior recognition algorithm model f nursingbehavior (BT N , HV N , ID N , HAR N ) to predict the image sequence, calculate the nursing behavior recognition result NBAR N in each frame, and obtain the nursing behavior recognition result sequence NBAR1, NBAR2,..., NBAR N ; then use a deep learning algorithm to construct a nursing behavior sequence post-processing algorithm f final (NBAR1, NBAR2,..., NBAR N ), further analyze the nursing behavior recognition result sequence NBAR1, NBAR2,..., NBAR N according to the time window, aggregate the nursing behavior recognition results, take the k nursing behavior types with the highest prediction probability in a single window as the effective nursing behavior of the window, and obtain the Top k nursing behavior prediction results in the image sequence according to the probability prediction.

[0027] Preferably, the step 1 specifically comprises the following steps:

[0028] Step 1-1: Obtain the original depth image sequence x1, x2,..., x N , wherein N represents the Nth original depth image.

[0029] Step 1-2: Process the original depth image, calculate and obtain the human body detection frame B1, B2,..., B N frame by frame. N , wherein B N is the detection frame result of multiple human bodies in the Nth original depth image.

[0030] Step 1-3: From the human body detection frame area, segment the human body ROI area M1, M2,..., M N frame by frame. N , wherein M N is the ROI area segmentation result of multiple human bodies in the Nth original depth image.

[0031] Preferably, the step 2 specifically comprises the following steps:

[0032] Step 2-1: For the multiple human body detection frames B1, B2,..., B N in the original depth image sequence, use a graph feature analysis algorithm f g (BN )calculating the graph feature vector VG of each frame N , to obtain the feature vector sequence VG1, VG2,..., VG N , wherein the graph feature vector VG N is expressed as the following formula (5):

[0033]

[0034] In formula (5), k represents the number of people in the multi-person body detection frame of the current frame, represents the mth feature obtained by analyzing the kth body region of the current frame by the graph algorithm;

[0035] Step 2-2: calculating the body detection frame B1, B2,..., B N corresponding to the pixel sequence PS1, PS2,..., PS N , using the machine learning algorithm f m (B N , PS N ) to calculate the motion feature vector VM N of each frame of the body detection frame, and constructing the feature vector sequence VM1, VM2,..., VM N , wherein the motion feature vector VM N of the body detection frame is expressed as the following formula (6):

[0036]

[0037] In formula (6), k represents the number of people in the multi-person body detection frame of the current frame, represents the momentum of the pth direction obtained by analyzing the kth body region of the current frame by combining context analysis, and in the embodiment, the momentum of the pth direction can be two directions in a two-dimensional plane;

[0038] Step 2-3: combining the body detection frame B1, B2,..., B N , the feature vector sequence VG1, VG2,..., VG N , and the feature vector sequence VM1, VM2,..., VM N , to construct the context multi-person body tracking information sequence BT1, BT2,..., BT N , wherein BT N is the tracking information of the multi-person in the Nth frame, denoted as BT N = [tn1, tn2,..., tn k ], tn k is the tracking number of the kth person in the body detection frame of the current frame, and BT N is defined as shown in the following formula (7):

[0039] BT N =f t (f bv (B N ...B N-i ), f gv (VG N , ...VG N-i ), f mv (VM N , ...VM N-i (7)

[0040] In equation (7), i is the number of frames associated with the current frame and the preceding frame, and f bv f is the cross-union ratio calculation function for human body detection. gv f is a function for calculating pixel feature changes in computer graphics. mv f is the momentum-related direction calculation function. t (f bv f gv f mv ) is a function for calculating the human body tracking number.

[0041] Preferably, in step 6, the participant behavior recognition algorithm model f is constructed in the following manner. pb (VG N KC′ N KCA N The model training dataset is as follows:

[0042] Using a self-labeled depth image dataset, and taking participant behavior within a time window as the gold standard, we extract the graphical feature vector sequence VG1, VG2, ..., VG of the human body region within the time window. N A sequence of three-dimensional coordinates of key points on the human body in space, KC′1, KC′2, ... KC′ N The set of average actual coordinates of all key points in various limb regions of the human body after transformation based on depth values: KCA1, KCA2, ..., KCA N This is used to construct the model training dataset.

[0043] Preferably, in step 7, the nursing behavior recognition algorithm model f is constructed in the following manner. nursingbehavior (BT N HV N ID N HAR N The model training dataset is as follows:

[0044] Step 7-1: defining i kinds of nursing behaviors, distinguishing the care and the cared in the nursing behaviors, defining the duration window of the nursing activities, and defining the activity characteristics of the participants in the window;

[0045] Step 7-2: labeling the self-owned depth image dataset, taking the nursing behaviors in a time window interval as the gold standard, extracting the context multi-person body tracking information sequence BT1, BT2,..., BT N , the multi-person motion activity feature sequence HV1, HV2,..., HV N , the multi-person body identity information vector sequence ID1, ID2,..., ID N , the participant behavior feature analysis result sequence HAR1, HAR2,..., HAR N , and constructing the model training dataset based on the above data.

[0046] The application extracts the action characteristics of the nursing person and the cared person in the nursing behaviors based on the depth image, constructs the trajectory feature matrix in the time sequence and the spatial range, analyzes the trajectory feature matrix by using the machine learning algorithm, and realizes the multi-person participating nursing behavior recognition and classification in the privacy scene.

[0047] Compared with the prior art, the application has the following advantages:

[0048] 1) The nursing behavior recognition method provided by the application does not depend on the characteristics of the traditional image, is not easily affected by the environment and light changes, and only depends on the depth image sequence, which is different from the audio and video collection of the traditional camera, and can effectively protect the privacy of the identified person.

[0049] 2) The application comprehensively analyzes the identity, behavior and image features of multiple persons in the depth image, gives a nursing behavior recognition mode based on the joint analysis of the participant behaviors, and effectively makes up for the defect that the traditional model cannot recognize the joint behaviors in special scenes.

[0050] 3) The multi-person nursing behavior recognition process provided by the application does not need the cooperation of the identified person, and is easy to realize the non-conscious recognition and continuous recognition. DETAILED DESCRIPTION

[0051] Figure 1 It is the overall flowchart of the nursing behavior recognition method in the privacy scene of the application;

[0052] Figure 2 It is the flowchart of the method for processing the input depth image sequence and extracting the body region information of all participants of the application;

[0053] Figure 3 It is the flowchart of the method for analyzing the body region image, tracking the action trajectories of different bodies, and obtaining the body tracking results of all participants of the application.

[0054] Figure 4 The flow chart of the method for analyzing the skeleton characteristics of different human bodies and constructing the human key point coordinate of all participants is shown in the following figure:

[0055] Figure 5 The flow chart of the method for analyzing the human displacement change between the image related frames and calculating the motion activity characteristics of all participants is shown in the following figure:

[0056] Figure 6 The flow chart of the method for confirming the care and cared role identity of all participants by combining the human tracking result and the motion activity characteristics is shown in the following figure:

[0057] Figure 7 The flow chart of the method for constructing the participant behavior characteristic extraction model and performing the behavior characteristic analysis on all participants in the input image sequence is shown in the following figure:

[0058] Figure 8 The flow chart of the method for constructing the nursing behavior recognition model by combining the participant behavior characteristic analysis result and performing the nursing behavior recognition and classification on the input image sequence is shown in the following figure. DETAILED DESCRIPTION

[0059] The application will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate but not to limit the scope of the application. Furthermore, it should be understood that those skilled in the art can make various modifications or changes to the application after reading the content of the application, and these equivalent forms also fall within the scope of the appended claims of the application.

[0060] The nursing behavior recognition method in a private scene disclosed in the embodiment comprises the following steps:

[0061] Step 1: processing the input depth image sequence to extract the human region image of all participants, specifically comprising the following steps:

[0062] Step 1-1: obtaining the original depth image sequence x1, x2,..., xN, wherein N represents the Nth frame of original depth image;

[0063] Step 1-2: processing the original depth image, frame by frame calculating and obtaining the human detection frame B1, B2,..., BN; N , wherein BN represents the detection frame result of multiple human bodies in the Nth frame of original depth image; N

[0064] Step 1-3: segmenting the human ROI region M1, M2,..., MN from the human detection frame region, frame by frame; N , wherein MN represents the ROI region of the Nth frame of original depth image; N ​The ROI region segmentation result of multiple human bodies in the Nth original depth image.

[0065] Step 2: Analyzing the human body region image, tracking the action trajectories of different human bodies, and obtaining the human body tracking results of all participants, specifically including the following steps:

[0066] Step 2-1: For the multiple human body detection boxes B1, B2,...,B N , calculate the graph feature vector VG g of each frame by using the graph feature analysis algorithm f N (B N ), to obtain the feature vector sequence VG1, VG2,...,VG N , wherein the graph feature vector VG N is represented as formula (1) as follows:

[0067]

[0068] In formula (1), k represents the number of personnel in the current frame of multiple human body detection boxes, represents the mth feature obtained by analyzing the kth human body region in the current frame by using the graph algorithm;

[0069] Step 2-2: Calculate the pixel sequence PS1, PS2,...,PS N of the corresponding region of the human body detection boxes B1, B2,...,B N , and calculate the human body detection box motion feature vector VM N of each frame by using the machine learning algorithm f m (B N , PS N ), to construct the feature vector sequence VM1, VM2,...,VM N , wherein the human body detection box motion feature vector VM N is represented as formula (2) as follows:

[0070]

[0071] In formula (2), k represents the number of personnel in the current frame of multiple human body detection boxes, represents the pth direction momentum obtained by analyzing the kth human body region in the current frame by combining context analysis, and in the embodiment, the pth direction momentum can be two directions in a two-dimensional plane;

[0072] Step 2-3: Combine the human body detection boxes B1, B2,...,B N , the feature vector sequence VG1, VG2,...,VG N , and the feature vector sequence VM1, VM2,...,VMN , build context multi-person body tracking information sequence BT1, BT2,..., BT N , where BT N is the tracking information of the Nth frame of multiple persons, denoted as BT N = [tn1, tn2,...tn k ], tn k is the tracking number of the kth person in the body detection frame, and BT N is defined as shown in the following formula (3):

[0073] BT N = f t (f bv (B N ,...B N-i ), f gv (VG N ,...VG N-i ), f mv (VM N ,...VM N-i )) (3)

[0074] In formula (3), i is the number of the current frame and the context associated frame, f bv is the body detection intersection over union calculation function, f gv is the pixel feature change calculation function of graphics, f mv is the momentum correlation direction calculation function, and f t (f bv , f gv , f mv ) is the body tracking number calculation function.

[0075] Step 3: combine the tracking results, analyze the key point features of different bodies, and build the body key point coordinates of all participants, which includes the following steps:

[0076] Step 3-1: combine the body tracking information sequence BT1, BT2,..., BT N , and the multi-person ROI region M1, M2,..., M N in the image sequence, and use the deep learning algorithm f k (BT N , M N ) to calculate the body key point coordinate set KC N of different participants on each two-dimensional depth image, and obtain the two-dimensional body key point coordinate set sequence KC1, KC2,..., KC N , where the body key point coordinate set KC N is represented as the following formula (4):

[0077]

[0078] In equation (4), k represents the number of people in the multi-person human detection box in the current frame. This represents the set of x and y coordinates of the q-th keypoint of the k-th person in the current frame, calculated using a deep learning algorithm, denoted as q / x.

[0079] Step 3-2: Combine the human body ROI regions M1, M2, ..., M N And the sequence of two-dimensional human body keypoint coordinate sets KC1, KC2, ..., KC N Using spatial coordinate transformation algorithm f z (M N KC N Calculate the three-dimensional coordinate set KC′ of key points on the human body in space. N This yields the sequence of three-dimensional coordinate sets of key points: KC′1, KC′2, ... KC′. N Among them, the three-dimensional coordinate set KC′ N It can be expressed as the following formula (5):

[0080]

[0081] In equation (5), k represents the number of people in the multi-person human detection box in the current frame. Let represent the set of x, y, z coordinates of the q-th keypoint of the k-th person in the current frame, calculated using a spatial coordinate transformation algorithm, in a three-dimensional spatial coordinate system. Here, x and y represent the two-dimensional coordinates of the current pixel in the image, and z represents the grayscale value of the current pixel. This is denoted as .

[0082] Step 4: Combine key point coordinates to analyze the changes in human body displacement between related image frames and calculate the motion activity characteristics of all participants. This includes the following steps:

[0083] Step 4-1: Combine the voxel information vi corresponding to each unit pixel in the depth image in 3D space. x，y，z And the sequence of 3D coordinate sets of key points KC′1, KC′2, ... KC′ N Calculate the actual coordinate set of human keypoints in each frame: KCR1, KCR2, ..., KCR N ;

[0084] Step 4-2: Divide the human body limb region into 6 parts: trunk, limbs, head, etc. Calculate the average actual coordinate set KCA1, KCA2, ..., KCA of all keypoints in each limb region in each frame. N ;

[0085] Step 4-3: Calculate the multi-person motion vitality feature HV based on the average actual coordinate set KCA1, KCA2, …, KCA N , and the human vitality value algorithm f hv (KCA N , …, KCA N-i ). N , to obtain the motion vitality feature sequence HV1, HV2, …, HV N , where f hv (KCA N , …, KCA N-i ) is calculated according to the following formula (6):

[0086]

[0087] In formula (6), p is the number of the current frame and the associated frame above, i is the number of the current frame and the maximum number of associated frames above, q is the current limb region number, b is the maximum number of limb regions, d is the actual distance calculation function of the limb average coordinates between two frames, w q is the limb region vitality value weight operator, is the actual coordinates of the specific limb region, and the vitality value is positively correlated with the human displacement distance.

[0088] Step 5: Combine the motion vitality feature to confirm the care and care recipient role identity of all participants, which includes the following steps:

[0089] Step 5-1: Combine the multi-person body detection box B1, B2, …, B N , the multi-person body tracking information sequence BT1, BT2, …, BT N , and the multi-person motion vitality feature sequence HV1, HV2, …, HV N to construct and define the vitality feature and related threshold of each participant role in the care scene, where the participant role in the care scene can be a caregiver and a care recipient.

[0090] Step 5-2: Construct the human identity recognition algorithm f id (B N , BT N , HV N ), calculate the identity vector ID N of each person in each frame, and obtain the multi-person body identity information vector sequence ID1, ID2, …, ID N .

[0091] Step 6: Construct a participant behavior feature extraction model to analyze the behavior features of all participants in the input image sequence, which includes the following steps:

[0092] Step 6-1: Label the proprietary depth image dataset, using participant behavior within a time window as the gold standard, and extract the graphical feature vector sequence VG1, VG2, ..., VG of the human body region within the time window. N A sequence of three-dimensional coordinates of key points on the human body in space, KC′1, KC′2, ... KC′ N The set of average actual coordinates of all key points in various limb regions of the human body after transformation based on depth values: KCA1, KCA2, ..., KCA N This is used to construct the model training dataset;

[0093] Step 6-2: Construct a participant behavior recognition algorithm model f using deep learning algorithms. pb (VG N KC′ N KCA N The model is trained using the training set described in step 6-1 to obtain the available participant behavior recognition algorithm model weights. The deep learning algorithm used can be a model based on convolutional neural network (CNN) such as ResNet and DenseNet, or a deep learning model based on recurrent neural network (RNN) and its variants such as LSTM and GRU such as Transformer.

[0094] Step 6-3: Extract the human region graphical feature vector sequence VG1, VG2, ..., VG from the image sequence to be predicted. N A sequence of three-dimensional coordinates of key points on the human body in space, KC′1, KC′2, ... KC′ N The set of average actual coordinates of all key points in various limb regions of the human body after transformation based on depth values: KCA1, KCA2, ..., KCA N The participant behavior recognition algorithm f constructed in step 6-2 pb (VG N KC′ N KCA N The intermediate process of HAR algorithm derivation before predicting image sequences and obtaining the final classification result. N As participant behavioral characteristics, the participant behavioral characteristic analysis result sequence HAR1, HAR2, ..., HAR was obtained. N .

[0095] Step 7: Based on the analysis results of participant behavioral characteristics, construct a nursing behavior recognition model to classify and identify nursing behaviors in the input image sequences. This includes the following steps:

[0096] Step 7-1: define i kinds of care behaviors, distinguish care and care receivers in care behaviors, define care activity duration window, and define participant activity characteristics in the window;

[0097] Step 7-2: annotate the self-owned depth image dataset with care behaviors in a time window interval as the gold standard, extract context multi-person body tracking information sequence BT1, BT2,..., BT N , multi-person motion activity feature sequence HV1, HV2,..., HV N , multi-person identity information vector sequence ID1, ID2,..., ID N , participant behavior feature analysis result sequence HAR1, HAR2,..., HAR N , and build model training dataset based on the above data;

[0098] Step 7-3: use deep learning algorithm to build care behavior recognition algorithm model f nursingbehavior (BT N , HV N , ID N , HAR N ), and train the model using the training set described in step 7-2 to obtain the available care behavior recognition algorithm model weight, wherein the deep learning algorithm used can be convolutional neural network, recurrent neural network, and deep learning models such as ResNet, DenseNet, and Transformer based on them;

[0099] Step 7-4: extract context multi-person body tracking information sequence BT1, BT2,..., BT N , multi-person motion activity feature sequence HV1, HV2,..., HV N , multi-person identity information vector sequence ID1, ID2,..., ID N , participant behavior feature analysis result sequence HAR1, HAR2,..., HAR N , and use the care behavior recognition algorithm f nursingbehavior (BT N , HV N , ID N , HAR N ) built in step 7-3 to predict the image sequence, calculate the care behavior recognition result NBAR N in each frame, and obtain the care behavior recognition result sequence NBAR1, NBAR2,..., NBAR N ;

[0100] Step 7-5: use deep learning algorithm to build care behavior sequence post-processing algorithm ffinal (NBAR1, NBAR2,..., NBAR N ), a nursing behavior recognition result sequence NBAR1, NBAR2,..., NBAR N Further analysis is performed according to a time window, the nursing behavior recognition results are aggregated, the k nursing behavior types with the highest prediction probability in a single window are taken as the effective nursing behaviors of the window, and the Top k nursing behavior prediction results in the image sequence according to the highest probability prediction are obtained, wherein the deep learning algorithm used can be a convolutional neural network, a recurrent neural network, and a deep learning model based on ResNet, DenseNet, Transformer, etc.

Claims

1. A method for recognizing a care behavior in a privacy scenario, characterized by, Comprising the following steps: Step 1: Process the input raw depth image sequence x1, x2, ..., x N Process the data and extract the human detection bounding boxes B1, B2, ..., B for all participants. N and the human body ROI regions M1, M2, ..., M N Among them, B N M represents the detection bounding boxes for multiple human bodies in the original depth image of frame N. N The ROI region segmentation results for multiple human bodies in the original depth image of frame N. Step 2: Analyze the original depth image sequence to detect multiple human body bounding boxes B1, B2, …, BN N , track the movement trajectories of different human bodies, and obtain a multi-human tracking information sequence BT1, BT2, …, BT N of all participants, where BT N is the tracking information of multiple human bodies in the Nth frame, denoted as BT N = [tn1, tn2, …, tn k ], tn k is the tracking number of the kth human body in the current frame human body bounding box, and BT N is defined as shown in the following formula (1): BT N = f t (f bv (B N ,…B N-i ),f gv (VG N ,…VG N-i ),f mv (VM N ,…VM N-i )) (1) In formula (1), i is the current frame and the number of frames associated above, f bv is a human body detection intersection over union calculation function, f gv is a graphics pixel feature change calculation function, f mv is a momentum correlation direction calculation function, f t (f bv ,f gv ,f mv ) is a human body tracking number calculation function, VG N is a graphics feature vector calculated by using a graphics feature analysis algorithm f g (B N ) N is a human body detection frame motion feature vector VM calculated by using a machine learning algorithm f m (B N ,PS N ) N ,PS1,PS2,…,PS N is a pixel sequence corresponding to the human body detection frame B1,B2,…,B N ​ Step 3: Combine the tracking results, analyze the key point features of different human bodies, and construct the human body key point coordinates of all participants, specifically comprising the following steps: Step 3-1: Combine human tracking information sequence BT1, BT2, …, BT N , and multiple human ROI regions M1, M2, …, M N in the image sequence, and use the deep learning algorithm f k (BT N , M N ) to calculate the human key point coordinate set KC N of different participants on each two-dimensional depth image, to obtain the two-dimensional human key point coordinate set sequence KC1, KC2, …, KC N , wherein the human key point coordinate set KC N is expressed as formula (2) as follows: In formula (2), k represents the number of persons in the current frame multi-person human body detection frame, represents the set of x, y coordinates of the qth key point of the kth human body in the current frame calculated by using a deep learning algorithm, and is recorded as Step 3-2: Combine the human body ROI regions M1, M2, ..., M N And the sequence of two-dimensional human body keypoint coordinate sets KC1, KC2, ..., KC N Using spatial coordinate transformation algorithm f z (M N KC N Calculate the three-dimensional coordinate set KC′ of key points on the human body in space. N The sequence of three-dimensional coordinates of key points is obtained as KC′1, KC′2, ... KC′. N Among them, the three-dimensional coordinate set KC′ N It can be expressed as the following formula (3): In formula (3), k represents the number of persons in the current frame of the multi-person body detection frame, represents the x, y, z coordinate set of the qth key point of the kth person in the current frame calculated by using a spatial coordinate conversion algorithm in a three-dimensional coordinate system, wherein x and y represent the two-dimensional coordinates of the current pixel on the image, and z represents the gray value of the current pixel, denoted as Step 4: Combine the key point coordinates, analyze the human body displacement changes between the image related frames, and calculate the motion activity features of all participants, specifically comprising the following steps: Step 4-1: combine the voxel information vi corresponding to the unit pixel in the depth image in the three-dimensional space x,y,z and the key point three-dimensional coordinate set sequence KC'1, KC'2,..., KC' N , calculate the actual key point coordinate set KCR1, KCR2,..., KCR N of each frame Step 4-2: Calculate the average actual coordinate set of all key points of each limb region in each frame KCA1, KCA2, …, KCA N ; Step 4-3: Calculate the average real coordinate set KCA1, KCA2, …, KCA N On this basis, the human vitality value algorithm f hv (KCA N ,…,KCA N-j ) is used to calculate the multi-person motion vitality feature HV N , and the motion vitality feature sequence HV1, HV2, …, HV N is obtained, wherein f hv (KCA N ,…,KCA N-j ) is calculated according to the following formula (4): In formula (4), p is the current frame and the above-mentioned associated frame number, j is the current frame and the above-mentioned associated maximum frame number, g is the current limb region number, b is the maximum number of limb regions, d is an actual distance calculation function of the average coordinates of the limbs between two frames, w g is a weight operator of the activity value of each limb region, is an actual coordinate of a specific limb region. Step 5: combine multi-person human detection boxes B1, B2, …, B N , multi-person human tracking information sequence BT1, BT2, …, BT N , and multi-person motion vitality feature sequence HV1, HV2, …, HV N , construct and define the vitality features of each participant role in the nursing scene and the related threshold, and then construct a human identity recognition algorithm f id (B N , BT N , HV N ), calculate the identity vector ID N of each multi-person in each frame, and obtain the multi-person identity information vector sequence ID1, ID2, …, ID N ; Step 6: constructing a participant behavior recognition algorithm model f using a deep learning algorithm pb (KC N ,KC′ N ,KCA N After training the model, the graph feature vector sequence VG1, VG2, …, VG N , the key point three-dimensional coordinate set sequence KC'1, KC'2, … KC' N , and the average actual coordinate set KCA1, KCA2, …, KCA N of all key points of each limb region of the human body according to the depth value conversion are extracted from the image sequence to be predicted. pb (VG N ,KC′ N ,KCA N ) to predict the image sequence, and take the algorithm intermediate process HAR N before the final classification result as the participant behavior feature to obtain the participant behavior feature analysis result sequence HAR1, HAR2, …, HAR N ; Step 7: Constructing a nursing behavior recognition algorithm model f using a deep learning algorithm nursingbehavior (BT N ,HV N ,ID N ,HAR N ), extract the context multi-person tracking information sequence BT1, BT2, …, BT N , multi-person motion vigor feature sequence HV1, HV2, …, HV N , multi-person identity information vector sequence ID1, ID2, …, ID N , participant behavior feature analysis result sequence HAR1, HAR2, …, HAR N , use the nursing behavior recognition algorithm model f nursingbehavior (BT N ,HV N ,ID N ,HAR N ) to predict the image sequence, calculate the nursing behavior recognition result NBAR N in each frame, and obtain the nursing behavior recognition result sequence NBAR1, NBAR2, …, NBAR N ; then use a deep learning algorithm to construct a nursing behavior sequence post-processing algorithm f final (NBAR1, NBAR2, …, NBAR N ), further analyze the nursing behavior recognition result sequence NBAR1, NBAR2, …, NBAR N according to the time window, aggregate the nursing behavior recognition results, take the k nursing behavior types with the highest prediction probability in a single window as the effective nursing behavior of the window, and obtain the Top k nursing behavior prediction results in the image sequence according to the highest probability prediction.

2. The method of claim 1, wherein, The step 1 specifically comprises the following steps: Step 1-1: Obtain original depth image sequence x1, x2, …, x N where N represents the Nth original depth image. Step 1-2: Processing the original depth image, frame by frame, to calculate and obtain the human body detection frame B1, B2, …, B N , wherein B N is the detection frame result of multiple human bodies in the Nth frame of the original depth image; Step 1-3: From the human body bounding box region, segment the human body ROI region M1, M2, …, M frame by frame N wherein M N is the ROI region segmentation result of multiple human bodies in the Nth original depth image.

3. The method of claim 1, wherein, The step 2 specifically comprises the following steps: Step 2-1: Multi-person bounding boxes B1, B2, …, Bn are detected in the original depth image sequence for each frame. N The graph feature analysis algorithm f g (B N ) is used to calculate the graph feature vector VG N of each frame, and the feature vector sequence VG1, VG2, …, VGn is obtained. N The graph feature vector VG N is expressed as formula (5) as follows: In formula (5), k represents the number of persons in the current frame multi-person human body detection frame, represents the mth feature obtained by analyzing the kth human body region of the current frame through a graph algorithm. Step 2-2: Calculate the human body detection frame B1, B2, …, B N The pixel sequence PS1, PS2, …, PS N corresponding to the region is calculated using a machine learning algorithm f m (B N , PS N ) to calculate the human body detection frame motion feature vector VM N of each frame, and a feature vector sequence VM1, VM2, …, VM N is constructed, wherein the human body detection frame motion feature vector VM N is expressed as formula (6) as follows: In formula (6), k represents the number of persons in the current frame multi-person human body detection frame, represents the momentum of the pth direction obtained by combining the context analysis and the analysis of the kth human body region in the current frame. In this embodiment, the momentum of the pth direction can be two directions in a two-dimensional plane. Step 2-3: Combine human detection boxes B1, B2, …, B N , feature vector sequences VG1, VG2, …, VG N , and feature vector sequences VM1, VM2, …, VM N , to construct context multi-human tracking information sequence BT1, BT2, …, BT N , where BT N is the tracking information of the multi-human in the Nth frame, denoted as BT N = [tn1, tn2, …, tn k ], tn k is the tracking number of the kth human in the current frame human detection box in order, and the definition of BT N is shown in the following formula (7): BT N = f t (f bv , B N ,..., B N-i ), f gv (VG N ,..., VG N-i ), f mv (VM N ,..., VM N-i )) (7) In formula (7), i is the current frame and the number of frames associated above, f bv is a human detection intersection over union calculation function, f gv is a graphics pixel feature change calculation function, f mv is a momentum correlation direction calculation function, f t (f bv , f gv , f mv ) is a human tracking number calculation function.

4. The method of claim 1, wherein, In step 6, the model training dataset for training the participant behavior recognition algorithm model f is constructed in the following way pb (VG N ,KC′ N ,KCA N ) of the model training dataset: The self-owned depth image dataset is labeled, and the participant behavior in a time window interval is taken as a gold standard to extract a human body region graphics feature vector sequence VG1, VG2, …, VG in the time window N A key point three-dimensional coordinate set sequence KC'1, KC'2, …, KC' of each key point of the human body in space N An average actual coordinate set KCA1, KCA2, …, KCA of all key points of each limb region of the human body converted according to a depth value N A model training dataset is constructed in this way.

5. The method of claim 1, wherein, In step 7, the model training dataset for training the care behavior recognition algorithm model f is constructed in the following way nursingbehavior (BT N ,HV N ,ID N ,HAR N ) Step 7-1: Define l kinds of nursing behaviors, distinguish between caregivers and care recipients in nursing behaviors, define the duration window of nursing activities, and define the activity features of participants within the window; Step 7-2: Label the self-owned depth image dataset, take the care behavior in a time window interval as the gold standard, extract the context multi-person body tracking information sequence BT1, BT2, …, BT in the time window N , multi-person motion activity feature sequence HV1, HV2, …, HV N , multi-person body identity information vector sequence ID1, ID2, …, ID N , participant behavior feature analysis result sequence HAR1, HAR2, …, HAR N , and build a model training data set based on the above data.

Images