Human body interactive behavior recognition method, device, equipment and readable storage medium

Through multi-source data fusion processing and a preset human interaction behavior recognition model, the interaction behavior between the human body and other objects is solved, and the problem of low recognition accuracy in the prior art is achieved, achieving higher recognition accuracy and efficiency.

CN114067424BActive Publication Date: 2024-11-08CHINA MOBILE COMM LTD RES INST +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010748455.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-07-30
Publication Date
2024-11-08
Estimated Expiration
2040-07-30

AI Technical Summary

Technical Problem

The prior art cannot accurately identify the interactive objects of human body behavior and actions, resulting in low recognition accuracy when identifying the interactive behavior of the human body and other objects.

Method used

The multi-source data fusion processing method is adopted to obtain keyframes that meet preset conditions through at least two channels of image data to be processed, and the keyframes are identified based on the preset human interaction behavior recognition model to generate the interactive behavior of the target human body.

Benefits of technology

It improves the accuracy of human interaction behavior recognition, reduces the redundant calculation of recognition through multi-source data fusion processing, and improves the recognition speed and accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114067424B_ABST
    Figure CN114067424B_ABST
Patent Text Reader

Abstract

The present invention provides a method, device, equipment and readable storage medium for identifying human interaction behaviors, and relates to the technical field of behavior recognition and detection, wherein the method for identifying human interaction behaviors comprises: obtaining at least two channels of image data to be processed; obtaining key frames that meet preset conditions based on at least two channels of image data to be processed; identifying key frames based on a preset human interaction behavior recognition model to generate the interaction behaviors of a target human body; the interaction behaviors are the interaction behaviors of the target human body and other objects, and other objects include other people or objects other than the target human body. In an embodiment of the present invention, key frames that meet preset conditions are obtained based on at least two channels of image data to be processed, and a preset human interaction behavior recognition model is used to identify key frames to generate the interaction behaviors of the target human body. In this way, the interaction behaviors of the target human body and other objects can be identified, and the recognition results can be made more accurate through multi-source data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of behavior recognition detection technology, and in particular to a method, device, equipment and readable storage medium for recognizing human interactive behavior. Background Art

[0002] At present, when human behavior recognition is performed, human posture information is usually obtained based on wearable devices to recognize human behavior. However, this recognition method can only recognize human actions, but cannot recognize the interactive objects of human behavior actions. When it is necessary to recognize the interactive behavior of the human body and other objects, the recognition accuracy is low. Summary of the invention

[0003] The embodiments of the present invention provide a method, device, equipment and readable storage medium for identifying human interactive behavior to solve the problem that the interactive objects of human behavior actions cannot be identified in the prior art, and the recognition accuracy is low when the interactive behavior between the human body and other objects needs to be identified.

[0004] To solve the above technical problems, the present invention is achieved as follows:

[0005] In a first aspect, an embodiment of the present invention provides a method for identifying human interaction behavior, comprising:

[0006] Acquire at least two channels of image data to be processed;

[0007] Acquire a key frame that meets a preset condition based on the at least two channels of image data to be processed;

[0008] The key frames are identified based on a preset human interaction behavior recognition model to generate an interaction behavior of a target human body; the interaction behavior is an interaction behavior between the target human body and other objects, and the other objects include other people or objects other than the target human body.

[0009] Optionally, acquiring at least two paths of image data to be processed includes:

[0010] Acquire first image data to be processed based on the first camera, and acquire second image data to be processed based on the second camera;

[0011] Data fusion processing is performed on the first image data to be processed and the second image data to be processed to generate the image data to be processed.

[0012] Optionally, the first image data to be processed is used to indicate skeleton node information, and the second image data to be processed is used to indicate behavior speed information;

[0013] The fusing the first to-be-processed image data and the second to-be-processed image data to generate the to-be-processed image data comprises:

[0014] Acquire a human body structure feature vector based on the skeleton node information;

[0015] Acquire a behavior speed feature vector of a human body based on the behavior speed information, wherein the behavior speed feature vector includes an acceleration vector and an angular velocity vector;

[0016] Data fusion processing is performed on the human body structure feature vector, the acceleration vector and the angular velocity vector to generate a first feature vector of the image data to be processed.

[0017] Optionally, the skeletal node information includes joint angle information and joint vector modulus;

[0018] The step of acquiring a human body structure feature vector based on the skeleton node information includes:

[0019] Acquire first joint angle information whose usage frequency exceeds a first set frequency threshold and first joint vector modulus whose usage frequency exceeds a second set frequency threshold;

[0020] The human body structure feature vector is generated based on the first joint angle information and the first joint vector modulus.

[0021] Optionally, acquiring a key frame satisfying a preset condition based on the at least two channels of image data to be processed includes:

[0022] The key frame satisfying a preset condition is acquired based on the first feature vector.

[0023] Optionally, the preset human interaction behavior recognition model is obtained in the following manner:

[0024] Historical image data and human interaction behaviors corresponding to the historical image frame data are acquired as a sample set, and iterative training is performed based on the sample set to generate the preset human interaction behavior recognition model.

[0025] Optionally, the identifying the key frame based on a preset human body interaction behavior recognition model to generate the target human body interaction behavior includes:

[0026] Acquire at least one first region in the key frame;

[0027] Performing object classification processing on the first area to obtain first information of a target human body and second information of other objects;

[0028] Generate a plurality of human body interaction behaviors based on the first information and the second information using a preset human body interaction behavior recognition model;

[0029] Calculate the confidence of each human interaction behavior;

[0030] The human body interaction behavior with the highest confidence is selected as the target human body interaction behavior.

[0031] In a second aspect, an embodiment of the present invention provides a human interaction behavior recognition device, comprising:

[0032] A first acquisition module, used for acquiring at least two channels of image data to be processed;

[0033] A second acquisition module, used for acquiring key frames meeting preset conditions based on the at least two channels of image data to be processed;

[0034] The recognition module is used to recognize the key frames based on a preset human interaction behavior recognition model to generate the target human interaction behavior; the interaction behavior is the interaction behavior of the target human body with other objects, and the other objects include other people or objects other than the target human body.

[0035] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method for identifying human body interactive behavior as described in the first aspect.

[0036] In a fourth aspect, an embodiment of the present invention provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method for identifying human interactive behavior as described in the first aspect are implemented.

[0037] In an embodiment of the present invention, key frames that meet preset conditions are obtained based on at least two channels of image data to be processed, and a preset human interaction behavior recognition model is used to recognize the key frames to generate the interaction behavior of the target human body. In this way, the interaction behavior of the target human body with other objects can be recognized, and the recognition result can be made more accurate through multi-source data. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0039] Figure 1 is a flow chart of a method for identifying human interaction behavior provided by an embodiment of the present invention;

[0040] Figure 2 is a schematic diagram of 25 skeletal nodes of a human body provided by an embodiment of the present invention;

[0041] Figure 3 It is a schematic diagram of the skeletal joints involved in the waving action provided in an embodiment of the present invention.

[0042] Figure 4 is a flow chart of a method for identifying key frames based on a preset human interaction behavior recognition model provided by an embodiment of the present invention;

[0043] Figure 5 is a module diagram of a human body interactive behavior recognition device provided by an embodiment of the present invention;

[0044] Figure 6 is a schematic diagram of a module structure of an electronic device provided by an embodiment of the present invention;

[0045] Figure 7 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] In the embodiments of the present invention, a method, device, equipment and readable storage medium for identifying human interactive behavior are proposed to solve the problem that the existing interactive objects of human behavior actions cannot be identified, and the recognition accuracy is low when the interactive behavior between the human body and other objects needs to be identified.

[0048] See also Figure 1 , Figure 1 is a flow chart of a method for identifying human interaction behavior provided by an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0049] Step 101: Obtain at least two channels of image data to be processed.

[0050] In this embodiment, at least two channels of image data to be processed are acquired by deploying at least two channels of sensors, so that the acquired data is multi-sourced to avoid single data affecting the recognition accuracy. In this embodiment, two channels of image data are used as an example for explanation, wherein one channel of image data is image data of the human body and the environment in which the human body is located, and the other channel of image data is image data of the human body. Alternatively, in other feasible embodiments, data from three or more sources may also be acquired, which is only used as an example and is not limited here.

[0051] Step 102: Acquire key frames that meet preset conditions based on at least two channels of image data to be processed.

[0052] Among them, by obtaining key frames, redundant data in the image data can be removed, the amount of calculation is reduced, and the recognition speed is further improved.

[0053] Step 103: recognize the key frames based on a preset human interaction behavior recognition model to generate the target human interaction behavior; the interaction behavior is the interaction behavior of the target human with other objects, and the other objects include other people or objects other than the target human.

[0054] In the above steps, key frames that meet preset conditions are obtained based on at least two channels of image data to be processed, and a preset human interaction behavior recognition model is used to recognize the key frames to generate the target human interaction behavior. In this way, the target human interaction behavior with other objects can be recognized, and the recognition result can be made more accurate through multi-source data.

[0055] Here, it should be understood that, in the method of the specific embodiment of the present invention, the above step 101 specifically includes the following.

[0056] Acquire first image data to be processed based on the first camera, and acquire second image data to be processed based on the second camera;

[0057] Data fusion processing is performed on the first image data to be processed and the second image data to be processed to generate image data to be processed.

[0058] In this embodiment, the first sensor is a kinect sensor, and the second sensor is a wearable device, wherein, when deploying the above sensors, the kinect sensor is deployed in the environment where the human body is located, such as a room or other indoor or outdoor places. It should be noted that, since the acquisition range of the kinect sensor is limited, the environmental places referred to in this embodiment are all environmental places with a certain range. In addition, since the human body's behavioral actions are mainly based on the main trunk of the human body, such as the waist or arm, when deploying the wearable device, it is deployed on the main trunk. For example, a 6-axis posture sensor can be worn on the wrist and waist of a person, including a 3-axis acceleration sensor and a 3-axis angular velocity sensor. Optionally, the above-mentioned deployment method of the sensor is only an example and is not limited, and its specific deployment method can be adjusted according to the specific environment.

[0059] Furthermore, by performing data fusion on the first image data collected by the kinect sensor and the second image data collected by the wearable device, image data integrating more features can be obtained, which can reflect the features of the image data in many aspects.

[0060] Optionally, the first image data to be processed is used to indicate the skeleton node information, and the second image data to be processed is used to indicate the behavior speed information;

[0061] The above-mentioned fusing the first image data to be processed and the second image data to be processed to generate the image data to be processed includes:

[0062] Obtaining the human body structure feature vector of the target human body based on the skeleton node information;

[0063] Acquire a behavior speed feature vector of the target human body based on the behavior speed information, where the behavior speed feature vector includes an acceleration vector and an angular velocity vector;

[0064] Data fusion processing is performed on the human body structure feature vector, the acceleration vector and the angular velocity vector to generate a first feature vector of the image data to be processed.

[0065] Specifically, the kinect sensor collects the skeleton node information and video stream data information of the human body as the first image data to be processed, and the wearable device collects the behavior speed information corresponding to the human posture as the second image data to be processed.

[0066] Among them, Figure 2 As shown, there are several skeletal nodes of the human body. In this embodiment, 25 main joints are taken as an example to illustrate, and 25 skeletal nodes of the human body are extracted from each frame image collected by the kinect sensor, wherein the 25 skeletal nodes are head node A, neck node B, shoulder spine node M, left shoulder node C, right shoulder node D, left elbow node E, right elbow node F, spine center point N, spine node O, left wrist node G, right wrist node H, left hand node I, right hand node J, left hand tip node K, right hand tip node L, left thumb node P, right thumb node Q, left hip node R, right hip node S, left knee node T, right knee node U, left ankle node V, right ankle node W, left foot node X, and right foot node Y.

[0067] Select the human spine base node 0 as the origin of the coordinate system to establish a three-dimensional coordinate system. Then, each bone node has a corresponding coordinate (X, Y, Z) in the three-dimensional coordinate system. For example, the coordinates of the bone joint point A are (X A , Y A , Z A ), then the vector of each bone joint point is ﹛(X A , Y A , Z A ),(X B ,Y B ,Z B )…(X O ,Y O ,Z O)﹜. The human body structure feature vector can be obtained by obtaining the three-dimensional coordinates of each bone node of the human body.

[0068] In addition, in this embodiment, a 6-axis posture sensor, including a 3-axis acceleration sensor and a 3-axis angular velocity sensor, is worn on a person's wrist and waist as an example to illustrate the method of obtaining the behavior velocity characteristic vector of the human body. It should be noted that when the human body is performing an action, the skeletal nodes associated with the action will undergo corresponding changes, and the angles between the limbs related to the action and the relative positions between the joints will change. Therefore, when the human body is performing an action, the 3-axis acceleration sensor and the 3-axis angular velocity sensor worn will generate corresponding acceleration and angular velocity, wherein the acceleration collected by the 3-axis acceleration sensor is expressed as follows.

[0069]

[0070] In the formula, Indicates the 3-axis acceleration, a x represents the acceleration of the X axis, a y represents the acceleration of the Y axis, a z Indicates the acceleration of the Z axis.

[0071] The angular velocity collected by the 3-axis angular velocity sensor is expressed as follows.

[0072]

[0073] In the formula, Represents the angular velocity of the three axes, ω x represents the angular velocity of the X axis, ω y represents the angular velocity of the Y axis, ω z Indicates the angular velocity of the Z axis.

[0074] In order to better analyze the collected acceleration and angular velocity, in this embodiment, the average value of the total acceleration and the total angular velocity is calculated to construct the feature vector of the 6-axis attitude sensor as the human behavior velocity feature vector. The calculation formula is as follows.

[0075]

[0076] The average value of the total angular velocity The calculation formula is as follows.

[0077]

[0078] The calculation formula of the eigenvector W of the 6-axis attitude sensor is as follows.

[0079]

[0080] Among them, data fusion processing of human body structure feature vector, acceleration vector and angular velocity vector is performed to generate the first feature vector of the image data to be processed, which can reflect the characteristics of the image to be processed from many aspects, and facilitate the subsequent analysis of human body behavior in more aspects.

[0081] Optionally, the skeleton node information includes joint angle information and joint vector modulus;

[0082] The human body structure feature vectors obtained based on the bone node information include:

[0083] Acquire first joint angle information whose usage frequency exceeds a first set frequency threshold and first joint vector modulus whose usage frequency exceeds a second set frequency threshold;

[0084] A human body structure feature vector is generated based on the first joint angle information and the first joint vector modulus.

[0085] It should be noted that when the human body performs behavioral actions, most of them are actually related to daily habits, and the skeletal joints involved in the daily actions of the human body are mostly the same. Therefore, in this embodiment, 16 joint angles that are more closely related to daily actions are selected to form the first joint angle information, and 6 vector moduli that are more closely related to daily actions are selected as the first joint vector moduli. For example, the daily usage frequency of a certain joint angle can be set to 20 times, and the information of the joint angle with a usage frequency of more than 20 times is used as the first joint angle information. Here, the usage frequency of a certain joint angle is only used as an example and is not limited. Alternatively, in other feasible embodiments, other frequency thresholds can also be used.

[0086] In this embodiment, 16 joint angles that are most closely related to daily movements are extracted to form a joint angle feature vector P, which is expressed as follows.

[0087] P=(P Neck_ShoulderRight_ElbowRight , P ShoulderRight_ElbowRight_WristRight , P Neck_ShoulderLeft_ElbowLeft , P ShoulderLeft_ElbowLeft_WristLeft ,P Neck_ShoulderLeft_Spine, P Neck_ShoulderLeft_Spine ,P ShoulderRight_ElbowRight_Spine ,P ShoulderLeft_ElbowLeft_Spine ,P ElbowRight_WristRight_Spine ,P ElbowLeft_WristLeft_Spine ,P ShoulderRight_Spine_HipRight ,P ShoulderLeft_Spine_HipLeft ,P Spine_HipRight_KneeRight ,P Spine_HipLeft_KneeLeft ,P HipRight_KneeRight_FootRight ,P HipLeft_KneeLeft_FootLeft ).

[0088] Where P Neck_ShoulderRight_ElbowRight represents the joint angles between the neck, right shoulder, and right elbow, PShoulderRight_ElbowRight_WristRight P represents the joint angle between the shoulder-right elbow and the elbow-right wrist. Neck_ShoulderLeft_ElbowLeft represents the joint angles between the neck, left shoulder, and left elbow, P ShoulderLeft_ElbowLeft_WristLeft represents the joint angle between the left shoulder, elbow and left wrist, P Neck_ShoulderLeft_Spine represents the joint angle between the neck, shoulder and left spine, P Neck_ShoulderLeft_Spine represents the joint angle between the neck, shoulder and right spine, P ShoulderRight_ElbowRight_Spine represents the joint angle between the right shoulder, elbow and spine, P ShoulderLeft_ElbowLeft_Spine represents the joint angle between the left shoulder elbow spine, P ElbowRight_WristRight_Spine represents the joint angle between the elbow and right wrist spine, P ElbowLeft_WristLeft_Spine represents the joint angle between the elbow and left wrist spine, P ShoulderRight_Spine_HipRight represents the joint angle between the right shoulder and spine, P ShoulderLeft_Spine_HipLeft represents the joint angle between the left shoulder and spine, P Spine_HipRight_KneeRight represents the joint angles between the spine, left hip, and right knee, P Spine_HipLeft_KneeLeft represents the joint angles between the spine, left hip, and left knee, P HipRight_KneeRight_FootRight represents the joint angle between the right foot spine, P HipLeft_KneeLeft_FootLeft Indicates the joint angle between the spine of the left foot.

[0089] In this embodiment, a method for calculating the joint angle feature vector is described as follows, taking a human waving hand as an example.

[0090] For example, Figure 3 In the waving action, the skeletal joints D, F, and H are involved, and the limb vectors and limb vector Among them, the limb vector Note: B ShoulderRight_ElbowRight , limb vector Note: B ElbowRight_WristRight , the angle between the two is α, and the corresponding joint angle feature vector is P ShoulderRight_ElbowRight_WristRight Then, the calculation formula is as follows.

[0091]

[0092] In the formula, a1 represents the vector The difference between the x-axis coordinate of the starting point F and the x-axis coordinate of the end point D, b1 represents the vector The difference between the Y-axis coordinate of the starting point F and the x-axis coordinate of the end point D, c1 represents the vector The difference between the x-axis coordinate of the starting point F and the z-axis coordinate of the end point D.

[0093] Among them, let the coordinates of point F be (x1, y1, z1) and the coordinates of point D be (x2, y2, z2), then the vector It can be expressed as (x1-x2, y1-y2, z1-z2), where a1=x1-x2, b1=y1-y2, c1=z1-z2.

[0094]

[0095] In the formula, a2 represents the vector The difference between the x-axis coordinate of the starting point F and the x-axis coordinate of the end point D, b2 represents the vector The difference between the Y-axis coordinate of the starting point F and the x-axis coordinate of the end point D, c2 represents the vector The difference between the x-axis coordinate of the starting point F and the z-axis coordinate of the end point D.

[0096]

[0097] In the formula, Representation vector The modulus value, Representation vector The modulus value of .

[0098] Among them, the joint angle feature vector P ShoulderRight_ElbowRight_WristRight It is the joint angle feature vector of human body waving.

[0099] In addition, combined with the analysis of human motion behavior, six vector moduli with greater correlation with daily motions are extracted as joint vector moduli, among which the human body structure vectors from head to hands, spine to hands, and spine to feet are selected respectively. The modulus from head to right hand is used as an example for calculation. The calculation formula of the vector modulus from head to right hand is as follows.

[0100]

[0101] Where M Head_HandRight Represents the magnitude of the vector from the head to the right hand.

[0102] Furthermore, the above-mentioned joint angle feature vectors and joint vector moduli are combined to form a human body structure feature vector, which is represented by the following formula.

[0103] V=(P,M).

[0104] Where V represents the 22-dimensional human joint angle feature vector, P represents the joint angle feature vector, and M represents the feature vector composed of modulus vectors of different joint points.

[0105] Specifically, in this embodiment, 16 joint angle feature vectors and 6 joint vector moduli are selected to form a 22-dimensional human body structure feature vector. Alternatively, in other feasible embodiments, other numbers of joint angle feature vectors and joint vector moduli can also be used to form human body structure feature vectors of other dimensions. Here, it is only used as an example and is not limited, but no matter how it is transformed, it is within the scope of protection of this embodiment.

[0106] In this embodiment, the feature vectors of the skeleton nodes and sensors are combined into a fusion data feature vector F, and F can be expressed as F = (V, W), a 24-dimensional vector. Each feature vector has a timestamp t, which is used to mark the time sequence of the feature vector F as follows:

[0107] F=(P Neck_ShoulderRight_ElbowRight , P ShoulderRight_ElbowRight_WristRight , P Neck_ShoulderLeft_ElbowLeft , P ShoulderLeft_ElbowLeft_WristLeft , P Neck_ShoulderLeft_Spine , P Neck_ShoulderLeft_Spine , P ShoulderRight_ElbowRight_Spine , P ShoulderLeft_ElbowLeft_Spine , P ElbowRight_WristRight_Spine , P ElbowLeft_WristLeft_Spine , P ShoulderRight_Spine_HipRight , P ShoulderLeft_Spine_HipLeft P Spine_HipRight_KneeRight , P Spine_HipLeft_KneeLeft , P HipRight_KneeRight_FootRight , P HipLeft_KneeLeft_FootLeft ,M Head_HandRight , M Head_HandLeft ,M spine_HandRight ,M Spine_HandLeft , M Spine_FootLeft , M Spine_FootRight , ).

[0108] Optionally, acquiring a key frame satisfying a preset condition based on at least two channels of image data to be processed includes:

[0109] A key frame satisfying a preset condition is obtained based on the first feature vector.

[0110] In this implementation, the key frame is obtained based on the first feature vector that fuses the human body structure feature vector and the human body behavior speed feature vector, so that the obtained key frame can reflect the human body's behavior characteristics in more dimensions.

[0111] In this embodiment, K-means is used to extract the key frames of the image corresponding to the image data to be processed. Specifically, if an action sequence consists of N frames, then the N frames S N It is expressed as follows.

[0112] S N ={S1,S2,S3,...,S N},

[0113] Si is the i-th frame in N frames, and the feature vector of the i-th frame is F i , where i is 1…N. The calculation formula is as follows.

[0114] F N ={F1,F2,F3,...,F N}.

[0115] The vector F N Clustering is divided into K categories, C i (i=1,2,...,K) represents, the generated cluster centers are C1, C2,..., C K Clustering is performed based on each cluster center, wherein the steps of clustering are as follows.

[0116] For each human behavior feature sample i, calculate the state class it should belong to (timestamp is not involved in the calculation):

[0117]

[0118] Where F represents the 24-dimensional fused feature vector.

[0119] For each human state class j, recalculate the centroid C of the class j :

[0120]

[0121] Where N is the number of frames, d i It indicates the class to which each sample i should belong, and j indicates a class after clustering.

[0122] Repeat the above two steps until the centroid remains unchanged or changes very little, then K category video key frames are obtained, I = {I1, I2, I3, ..., I K}, extract the K category video key frames, and sort the extracted key frames in the order of timestamps, so as to perform behavior recognition based on the key frames later. In this implementation, human behavior interaction recognition based on key frames can avoid the complexity brought by redundant data, reduce the computational workload, and improve recognition efficiency.

[0123] Optionally, the preset human interaction behavior recognition model is obtained in the following manner:

[0124] The human interaction behaviors of historical image data and corresponding historical image frame data are obtained as a sample set, and iterative training is performed based on the sample set to generate a preset human interaction behavior recognition model.

[0125] It should be noted that the training of the human body interaction behavior recognition model can be performed offline or online. For better explanation, in this embodiment, the human body interaction behavior recognition model is pre-trained offline.

[0126] For example, at least two channels of historical image data are obtained, and data fusion processing is performed on the two channels of historical image data in the above manner to obtain image data that fuses the features of multi-source data, and the image data is used as the input of the model, and the human interaction behavior corresponding to the image data is used as the output, and iterative training is performed repeatedly until the model meets the preset convergence. The human interaction behavior recognition model that meets the preset convergence is used as the preset human interaction behavior recognition model obtained by the final training.

[0127] like Figure 4 As shown, optionally, identifying key frames based on a preset human body interaction behavior recognition model to generate the target human body interaction behavior includes:

[0128] Acquire at least one first region in a key frame;

[0129] Performing object classification processing on the first area to obtain first information of a target human body and second information of other objects;

[0130] Generate a plurality of human body interaction behaviors based on the first information and the second information using a preset human body interaction behavior recognition model;

[0131] Calculate the confidence of each human interaction behavior;

[0132] The human interaction behavior with the highest confidence is selected as the target human interaction behavior.

[0133] Furthermore, the relationship detection and interactive behavior recognition between people and other objects in the image are performed based on the key frames extracted as above. Among them, other objects can be other people or other objects other than the target human body, that is, all objects that interact with the target human body. In this embodiment, objects are used as examples for illustration. Among them, a human-centered triple is used to describe the relationship between the interactive actions of people and objects. The representation of the triple can be <person, action, object>. As a transformable implementation method, other forms of triples can also be used to describe the relationship between people and objects. Here, it is only used as an example and is not limited. However, no matter how it is transformed, it is within the scope of protection of this embodiment.

[0134] Furthermore, in this embodiment, the Faster R-CNN (Towards Real-Time Object Detection with Region Proposal Networks, convolutional neural network) detection framework is used to detect the key frames extracted above, and obtain several first regions including the detection object, wherein the detection object refers to a person and an object. Then, the RoiAlign method is used to extract features from the first region, and the first region is subjected to object classification processing based on the extracted features to obtain several bounding box information about the human body and several bounding boxes about the object, wherein the bounding box information about the human body is used as the first information, and the information about the object is used as the second information. In this embodiment, the person is recorded as b h , let object be b o . And for b h and b o There is a confidence s for the corresponding category h (b h The category confidence of o (b o The confidence level of the category is s. h Refers to the confidence of human action, confidence s o Refers to the confidence level of an object.

[0135] For example, for each person b in the first message h Detect action a. h Use the RoiAlign method to extract features and obtain the confidence s of action a a h The calculation formula is as follows.

[0136] s a h ,a∈{a1,a2,...a H};

[0137] In the formula, a H Indicates the Hth action.

[0138] Furthermore, the interactive object is positioned with the person as the center. In this embodiment, the determination of the interactive object depends on the position of the person, for example, from b h Extract features and predict the distribution of possible locations of target objects. Model the target object using a Gaussian function. The confidence g of the possible location of the target object is a h,o It can be represented as a 4-D Gaussian distribution, which calculates the possible relative position of the target object relative to the person. The calculation formula is shown below.

[0139]

[0140] Where b o|h represents the coded coordinates of bo relative to bh, represents the predicted position of the target object, and σ represents the hyperparameter value.

[0141]

[0142] In the formula, x o and o They represent the X-axis and Y-axis coordinates of people and objects respectively, h represents the frame height of the first area, and w represents the frame width.

[0143] According to the feature vector of the person, the position of the target object that may be associated with the person is predicted. The predicted position is matched with the second information about the object obtained above to obtain the object that actually interacts with the person.

[0144] It is worth noting that there may be multiple objects that interact with people. In this implementation, a person, an action, and each object that is actually associated with a person are grouped into a triple, and multiple triples may be grouped. Then, a triple score is assigned to each triple based on the preset human interaction behavior recognition model, that is, the confidence of each person's interaction behavior with the object is calculated, wherein the calculation formula for the confidence of each person's interaction behavior with the object is as follows.

[0145] S a h,o =s h *s o *s a h *g a h,o ;

[0146] In the formula, S a h,o Indicates the confidence of the interaction between people and objects, s a h represents the confidence of action a, g a h,o Indicates the confidence of the object that may be associated with action a.

[0147] The triplet with the highest confidence is selected as the final recognized interaction behavior between people and objects.

[0148] The human body interaction behavior recognition method in this embodiment obtains key frames that meet preset conditions based on two channels of image data to be processed, and uses a preset human body interaction behavior recognition model to recognize the key frames to generate the interaction behavior of the target human body. In this way, the interaction behavior between the target human body and other objects can be identified, and the recognition result can be made more accurate through multi-source data.

[0149] See also Figure 5 . Figure 5 A human interaction behavior recognition device 500 provided in this embodiment includes:

[0150] A first acquisition module 501 is used to acquire at least two channels of image data to be processed;

[0151] A second acquisition module 502, configured to acquire a key frame satisfying a preset condition based on the at least two channels of image data to be processed;

[0152] The recognition module 503 is used to recognize the key frames based on a preset human interaction behavior recognition model to generate the target human interaction behavior; the interaction behavior is the interaction behavior of the target human with other objects, and the other objects include other people or objects other than the target human.

[0153] Optionally, the first acquisition module 501 includes:

[0154] A third acquisition module, used to acquire first image data to be processed based on the first camera, and acquire second image data to be processed based on the second camera;

[0155] The first fusion module is used to perform data fusion processing on the first image data to be processed and the second image data to be processed to generate the image data to be processed.

[0156] Optionally, the first image data to be processed is used to indicate skeleton node information, and the second image data to be processed is used to indicate behavior speed information;

[0157] The first fusion module includes:

[0158] A fourth acquisition module, used for acquiring a human body structure feature vector of a human body based on the skeletal node information;

[0159] A fifth acquisition module, configured to acquire a behavior speed feature vector of a human body based on the behavior speed information, wherein the behavior speed feature vector includes an acceleration vector and an angular velocity vector;

[0160] The second fusion module is used to perform data fusion processing on the human body structure feature vector, the acceleration vector and the angular velocity vector to generate a first feature vector of the image data to be processed.

[0161] Optionally, the skeletal node information includes joint angle information and joint vector modulus;

[0162] The fourth acquisition module includes:

[0163] A sixth acquisition module, used to acquire first joint angle information whose usage frequency exceeds a first set frequency threshold and first joint vector modulus whose usage frequency exceeds a second set frequency threshold;

[0164] The first generating module is used to generate the human body structure feature vector based on the first joint angle information and the first joint vector modulus.

[0165] Optionally, when acquiring, the first acquisition module 501 acquires the key frame that meets a preset condition based on the first feature vector.

[0166] Optionally, the preset human interaction behavior recognition model is obtained in the following manner:

[0167] Historical image data and human interaction behaviors corresponding to the historical image frame data are acquired as a sample set, and iterative training is performed based on the sample set to generate the preset human interaction behavior recognition model.

[0168] Optionally, the identification module 503 includes:

[0169] A seventh acquisition module, used to acquire at least one first area in the key frame;

[0170] A processing module, configured to perform object classification processing on the first area to obtain first information of a target human body and second information of other objects;

[0171] A second generating module, configured to generate a plurality of human body interaction behaviors based on the first information and the second information by using a preset human body interaction behavior recognition model;

[0172] A calculation module, used to calculate the confidence of each human interaction behavior;

[0173] A selection module is used to select the human body interaction behavior with the highest confidence as the target human body interaction behavior.

[0174] The human interaction behavior recognition device in the embodiment of the present application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and the non-mobile electronic device can be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0175] The human body interaction behavior recognition device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0176] The human body interaction behavior recognition device provided in the embodiment of the present application can achieve Figures 1 to 4 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0177] Optional, such as Figure 6 As shown, an embodiment of the present application also provides an electronic device 600, including a processor 601, a memory 602, and a program or instruction stored in the memory 602 and executable on the processor 601. When the program or instruction is executed by the processor 601, each process of the above-mentioned method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0178] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0179] Figure 7 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0180] The electronic device 700 includes but is not limited to: a radio frequency unit 701, a network module 702, an audio output unit 703, an input unit 704, a sensor 705, a display unit 706, a user input unit 707, an interface unit 708, a memory 709, and a processor 710.

[0181] Those skilled in the art will appreciate that the electronic device 700 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 710 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 7 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0182] The processor 710 is used to: obtain at least two channels of image data to be processed;

[0183] Acquire a key frame that meets a preset condition based on the at least two channels of image data to be processed;

[0184] The key frames are identified based on a preset human interaction behavior recognition model to generate an interaction behavior of a target human body; the interaction behavior is an interaction behavior between the target human body and other objects, and the other objects include other people or objects other than the target human body.

[0185] Optionally, acquiring at least two paths of image data to be processed includes:

[0186] Acquire first image data to be processed based on the first camera, and acquire second image data to be processed based on the second camera;

[0187] Data fusion processing is performed on the first image data to be processed and the second image data to be processed to generate the image data to be processed.

[0188] Optionally, the first image data to be processed is used to indicate skeleton node information, and the second image data to be processed is used to indicate behavior speed information;

[0189] The fusing the first to-be-processed image data and the second to-be-processed image data to generate the to-be-processed image data comprises:

[0190] Acquire a human body structure feature vector based on the skeleton node information;

[0191] Acquire a behavior speed feature vector of a human body based on the behavior speed information, wherein the behavior speed feature vector includes an acceleration vector and an angular velocity vector;

[0192] Data fusion processing is performed on the human body structure feature vector, the acceleration vector and the angular velocity vector to generate a first feature vector of the image data to be processed.

[0193] Optionally, the skeletal node information includes joint angle information and joint vector modulus;

[0194] The step of acquiring a human body structure feature vector based on the skeleton node information includes:

[0195] Acquire first joint angle information whose usage frequency exceeds a first set frequency threshold and first joint vector modulus whose usage frequency exceeds a second set frequency threshold;

[0196] The human body structure feature vector is generated based on the first joint angle information and the first joint vector modulus.

[0197] Optionally, acquiring a key frame satisfying a preset condition based on the at least two channels of image data to be processed includes:

[0198] The key frame satisfying a preset condition is acquired based on the first feature vector.

[0199] Optionally, the preset human interaction behavior recognition model is obtained in the following manner:

[0200] Historical image data and human interaction behaviors corresponding to the historical image frame data are acquired as a sample set, and iterative training is performed based on the sample set to generate the preset human interaction behavior recognition model.

[0201] Optionally, the identifying the key frame based on a preset human body interaction behavior recognition model to generate the target human body interaction behavior includes:

[0202] Acquire at least one first region in the key frame;

[0203] Performing object classification processing on the first area to obtain first information of a target human body and second information of other objects;

[0204] Generate a plurality of human body interaction behaviors based on the first information and the second information using a preset human body interaction behavior recognition model;

[0205] Calculate the confidence of each human interaction behavior;

[0206] Select the human interaction behavior with the highest confidence as the target human interaction behavior

[0207] In an embodiment of the present application, the electronic device 700 obtains key frames that meet preset conditions based on at least two channels of image data to be processed, and uses a preset human interaction behavior recognition model to recognize the key frames to generate the interaction behavior of the target human body. In this way, the interaction behavior between the target human body and other objects can be identified, and the recognition result can be made more accurate through multi-source data.

[0208] It should be understood that in the embodiment of the present application, the input unit 704 may include a graphics processor (Graphics Processing Unit, GPU) 7041 and a microphone 7042, and the graphics processor 7041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 706 may include a display panel 7061, and the display panel 7061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 707 includes a touch panel 7071 and other input devices 7072. The touch panel 10071 is also called a touch screen. The touch panel 7071 may include two parts: a touch detection device and a touch controller. Other input devices 7072 may include, but are not limited to, a physical keyboard, a function key (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here. The memory 709 can be used to store software programs and various data, including but not limited to applications and operating systems. The processor 710 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 710.

[0209] The embodiment of the present application also provides a readable storage medium on which a program or instruction is stored. When the program or instruction is executed by a processor, the above Figures 1 to 4 The various processes of the human body interaction behavior recognition method embodiment can achieve the same technical effect. To avoid repetition, they will not be described here.

[0210] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0211] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A method for identifying human interaction behavior, characterized in that: include: Acquire at least two channels of image data to be processed; Acquire a key frame that meets a preset condition based on the at least two channels of image data to be processed; The key frames are identified based on a preset human interaction behavior recognition model to generate an interaction behavior of a target human body; the interaction behavior is an interaction behavior of the target human body with other objects, and the other objects include other people or objects other than the target human body; The acquiring of at least two channels of image data to be processed comprises: Acquire first image data to be processed based on the first camera, and acquire second image data to be processed based on the second camera; Performing data fusion processing on the first image data to be processed and the second image data to be processed to generate the image data to be processed; The first image data to be processed is used to indicate the skeleton node information, and the second image data to be processed is used to indicate the behavior speed information; The fusing the first to-be-processed image data and the second to-be-processed image data to generate the to-be-processed image data comprises: Acquire a human body structure feature vector based on the skeleton node information; Acquire a behavior speed feature vector of a human body based on the behavior speed information, wherein the behavior speed feature vector includes an acceleration vector and an angular velocity vector; Performing data fusion processing on the human body structure feature vector, the acceleration vector and the angular velocity vector to generate a first feature vector of the image data to be processed; The skeleton node information includes joint angle information and joint vector modulus value; The step of acquiring a human body structure feature vector based on the skeleton node information includes: Acquire first joint angle information whose usage frequency exceeds a first set frequency threshold and first joint vector modulus whose usage frequency exceeds a second set frequency threshold; The human body structure feature vector is generated based on the first joint angle information and the first joint vector modulus.

2. The human body interaction behavior recognition method according to claim 1, characterized in that: The step of acquiring a key frame satisfying a preset condition based on the at least two channels of image data to be processed comprises: The key frame satisfying a preset condition is acquired based on the first feature vector.

3. The human body interactive behavior recognition method according to claim 1, characterized in that: The preset human interaction behavior recognition model is obtained as follows: Historical image data and human interaction behaviors corresponding to the historical image frame data are acquired as a sample set, and iterative training is performed based on the sample set to generate the preset human interaction behavior recognition model.

4. The human body interactive behavior recognition method according to claim 1, characterized in that: The identifying the key frames based on the preset human body interaction behavior recognition model to generate the target human body interaction behavior includes: Acquire at least one first region in the key frame; Performing object classification processing on the first area to obtain first information of a target human body and second information of other objects; Generate a plurality of human body interaction behaviors based on the first information and the second information using a preset human body interaction behavior recognition model; Calculate the confidence of each human interaction behavior; The human body interaction behavior with the highest confidence is selected as the target human body interaction behavior.

5. A human interaction behavior recognition device, characterized in that: include: A first acquisition module, used for acquiring at least two channels of image data to be processed; A second acquisition module, used for acquiring key frames meeting preset conditions based on the at least two channels of image data to be processed; A recognition module, used to recognize the key frames based on a preset human interaction behavior recognition model to generate the target human interaction behavior; The interactive behavior refers to the interactive behavior of the target human body and other objects, and the other objects include other persons or objects other than the target human body; The first acquisition module includes: A third acquisition module, used to acquire first image data to be processed based on the first camera, and acquire second image data to be processed based on the second camera; A first fusion module, used for performing data fusion processing on the first to-be-processed image data and the second to-be-processed image data to generate the to-be-processed image data; The first image data to be processed is used to indicate the skeleton node information, and the second image data to be processed is used to indicate the behavior speed information; The first fusion module includes: A fourth acquisition module, used for acquiring a human body structure feature vector of a human body based on the skeletal node information; A fifth acquisition module, configured to acquire a behavior speed feature vector of a human body based on the behavior speed information, wherein the behavior speed feature vector includes an acceleration vector and an angular velocity vector; A second fusion module is used to perform data fusion processing on the human body structure feature vector, the acceleration vector and the angular velocity vector to generate a first feature vector of the image data to be processed; The skeleton node information includes joint angle information and joint vector modulus value; The fourth acquisition module includes: A sixth acquisition module, used to acquire first joint angle information whose usage frequency exceeds a first set frequency threshold and first joint vector modulus whose usage frequency exceeds a second set frequency threshold; The first generating module is used to generate the human body structure feature vector based on the first joint angle information and the first joint vector modulus.

6. An electronic device, characterized in that: It includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor. When the program or instruction is executed by the processor, the steps of the human body interactive behavior recognition method as described in any one of claims 1 to 4 are implemented.

7. A readable storage medium, characterized in that: The readable storage medium stores programs or instructions, and when the programs or instructions are executed by the processor, the steps of the human body interactive behavior recognition method as described in any one of claims 1-4 are implemented.