A method and related system for collaborative behavior detection and recognition

Abstract

The present invention discloses a method and related system for collaborative behavior detection and recognition. The method includes: selecting a video data source for which collaborative behavior detection is to be performed; obtaining detection box information of objects and people by using an object detection algorithm; obtaining skeleton key point data corresponding to people by using a multi-person pose estimation algorithm; detecting the collaborative behavior of people interacting with each other according to the human skeleton key point data and a predefined formula for human-human interaction behavior; detecting the collaborative behavior of people interacting with devices by using a predefined calculation formula for human-device collaborative behavior; calculating the information entropy of the collaborative behavior according to the collaborative confidence of the user's collaborative behavior, and obtaining the feedback results of the recognition of two types of collaborative behaviors. This method can effectively detect the interaction behavior between people and the collaborative behavior of people interacting with devices in a collaborative scenario, and can push the collaborative behavior recognition and analysis results to the collaborative members in real time, ultimately improving the office efficiency of the collaborative working environment.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and is particularly applicable to a method and related system for collaborative behavior detection and recognition. Background Art

[0002] Human Action Recognition (HAR), that is, understanding and recognizing human actions, is crucial in many practical applications. It can be used in visual surveillance systems to identify dangerous human activities; it can also be used in autonomous navigation systems to understand human actions and work collaboratively with humans; in addition, HAR is also important in aspects such as video retrieval and human-computer interaction. Generally, the data patterns utilized for human action recognition are diverse and complex, and are classified into two major categories according to different modalities: visual modality and sensor modality. These data patterns encode different information sources to achieve action recognition, and each has its own advantages and applicable application scenarios.

[0003] Although great success has been achieved in the design of human body action recognition under some common frameworks, there are still the following problems in research and application in specific scenarios:

[0004] For complex scenarios such as collaborative scenarios with specific behaviors: Human body action recognition in collaborative scenarios is still challenging. For example, in a meeting scenario, it not only includes human-human interaction but also human-device interaction, and the detection of these two types of interaction behaviors is still a quite challenging task; the research on human body action recognition technology with overlapping detection regions of multiple targets existing in specific scenarios is not deep enough. It will render the application of existing human body action recognition models ineffective and cause unreasonable reconstruction of human body postures.

[0005] Most of the current technologies for group behavior recognition have not been studied and implemented too much. As a result, it is difficult for users to comprehensively obtain and analyze this information in complex scenarios in a short time: at what moment in the collaborative scenario and what are they doing. However, analyzing the human body postures in collaborative scenarios is of great significance for us to understand collaborative behaviors, understand the activity levels of people in collaborative scenarios, and ultimately effectively improve collaborative efficiency.

[0006] Therefore, how to accurately recognize human body actions in complex collaborative scenarios has become an urgent problem for practitioners in the same field to solve. Summary of the Invention

[0007] The purpose of the present invention is to provide a method and related system for collaborative behavior detection and recognition that at least partially solves the above technical problems.

[0008] To achieve the above purpose, the technical solutions adopted by the present invention are as follows:

[0009] In a first aspect, the present invention provides a method for collaborative behavior detection and recognition, comprising the following steps: selecting a video data source for which collaborative behavior detection is to be performed; the collaborative behaviors including: interaction behaviors between humans and devices and interaction behaviors between humans;

[0010] Using an object detection algorithm to obtain detection box information of objects and humans;

[0011] Using a multi-person pose estimation algorithm to obtain bone key point data corresponding to humans;

[0012] Detecting collaborative behaviors of human-to-human interaction according to the human bone key point data and a predefined formula for human-to-human interaction behaviors;

[0013] Detecting collaborative behaviors of human-to-device interaction through a predefined calculation formula for human-to-device collaborative behaviors;

[0014] Calculating the information entropy of the collaborative behavior according to the collaborative confidence of the user's collaborative behavior, and obtaining a feedback result for the recognition of two types of collaborative behaviors.

[0015] Further, selecting a video data source for which collaborative behavior detection is to be performed includes:

[0016] Obtaining video data in a collaborative scenario by using a preset method;

[0017] Preprocessing the video data to obtain key frame data;

[0018] Judging whether the key frame data meets a preset requirement;

[0019] When the preset requirement is met, outputting data in a format that meets the preset requirement.

[0020] Further, selecting a video data source for which collaborative behavior detection is to be performed further includes:

[0021] When the preset requirement is not met, sending an error report; and recording the error-processed data in a database.

[0022] Further, using an object detection algorithm to obtain detection box information of objects and humans includes:

[0023] Using a YOLOV5 or SSD object detection algorithm to detect data in a format that meets the preset requirement, and outputting video key frame detection box information including detected personnel or devices.

[0024] Further, using a multi-person pose estimation algorithm to obtain bone key point data corresponding to humans includes:

[0025] Extracting coordinate data of 14 key points of the human body skeleton through the OpenPose algorithm.

[0026] Further, based on the human skeletal key point data and a predefined formula for human - to - human interaction behavior, the collaborative behavior of human - to - human interaction is detected, including:

[0027] Determine the human key point parts corresponding to the predefined human - to - human interaction behavior; human - to - human interaction behaviors include: passing, receiving, joining, and leaving;

[0028] Construct a human part displacement feature vector from the displacements of the coordinates of the corresponding human key point parts at two different times, and estimate the motion state of the human part;

[0029] Based on determining the motion state of the human part, judge whether the change in the included angle of the human part vectors is within a preset range; and judge whether the head orientation and the motion direction of the human part are within a second preset range;

[0030] When the change in the included angle of the human part vectors is within the preset range and the head orientation and the motion direction of the human part are within the second preset range, it is determined that a collaborative behavior of human - to - human interaction has occurred.

[0031] Further, through a predefined calculation formula for human - to - device collaborative behavior, the collaborative behavior of human - to - device interaction is detected, including:

[0032] Determine the human key point parts corresponding to the predefined human - to - device interaction behavior; human - to - device interaction behaviors include: operating the keyboard, operating the screen, and operating the mouse;

[0033] Judge whether the coordinate data of the corresponding human key point parts fall within the keyboard, screen, and mouse detection frames corresponding to the identified device;

[0034] When it falls within the detection frame, it is determined that a collaborative behavior of human - to - device interaction has occurred.

[0035] Further, according to the collaborative confidence of the user's collaborative behavior, calculate the information entropy of the collaborative behavior to obtain the feedback results of the two - type collaborative behavior recognition, including:

[0036] Statistically calculate the frequencies of simultaneous human part movements, the frequencies of changes in vector included angles, and the determination frequencies of the same head orientation and human part motion direction within two successive preset time periods, as well as the frequency counts of sitting, standing, and arm - stretching behaviors, and calculate their probabilities as the collaborative confidence of human - to - human interaction behavior;

[0037] Take the area overlap rate between the bounding rectangle of the human key point part corresponding to the human - to - device interaction behavior and the keyboard, screen, or mouse detection frame corresponding to the device as the collaborative confidence of human - to - device interaction;

[0038] Statistically classify collaborative behavior categories, corresponding confidence levels, and collaborative activity levels, and generate the information entropy of collaborative behavior;

[0039] Based on the information entropy of collaborative behavior, obtain the feedback results of the recognition of two types of collaborative behavior.

[0040] In a second aspect, the present invention also provides a system for collaborative behavior detection and recognition, including:

[0041] A selection and acquisition module, configured to select a video data source for which collaborative behavior detection is required; the collaborative behavior includes: the interaction behavior between people and devices and the interaction behavior between people;

[0042] A collaborative behavior detection module, configured to use an object detection algorithm to obtain the detection frame information of objects and people; use a multi-person pose estimation algorithm to obtain the skeletal key point data corresponding to people; based on the human skeletal key point data and a predefined formula for the interaction behavior between people, detect the collaborative behavior of the interaction between people; detect the collaborative behavior of the interaction between people and devices through a predefined calculation formula for the collaborative behavior between people and devices;

[0043] A data storage module, configured to store video frame information, collaborative behavior recognition results, their corresponding collaborative confidence levels, and collaborative behavior information entropy;

[0044] A recognition and feedback module, configured to calculate the information entropy of collaborative behavior based on the collaborative confidence level of the user's collaborative behavior, and obtain the feedback results of the recognition of two types of collaborative behavior.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] A method for collaborative behavior detection and recognition provided by an embodiment of the present invention includes: selecting a video data source for which collaborative behavior detection is required; the collaborative behavior includes: the interaction behavior between people and devices and the interaction behavior between people; using an object detection algorithm to obtain the detection frame information of objects and people; using a multi-person pose estimation algorithm to obtain the skeletal key point data corresponding to people; based on the human skeletal key point data and a predefined formula for the interaction behavior between people, detect the collaborative behavior of the interaction between people; detect the collaborative behavior of the interaction between people and devices through a predefined calculation formula for the collaborative behavior between people and devices; calculate the information entropy of collaborative behavior based on the collaborative confidence level of the user's collaborative behavior, and obtain the feedback results of the recognition of two types of collaborative behavior. This method can effectively detect the interaction behavior between people and the collaborative behavior of the interaction between people and devices in a collaborative scenario, and can push the collaborative behavior recognition and analysis results to collaborative members in real time, ultimately improving the office efficiency of the collaborative working environment. Description of the Drawings

[0047] Figure 1Flowchart of the method for collaborative behavior detection and recognition provided by an embodiment of the present invention;

[0048] Figure 2 Schematic diagram of the video source selection and data preprocessing provided by an embodiment of the present invention.

[0049] Figure 3 Schematic diagram of the object detection and human pose estimation provided by an embodiment of the present invention.

[0050] Figure 4 Schematic diagram of the key points of the human skeleton provided by an embodiment of the present invention.

[0051] Figure 5 Schematic diagram of the calculation and recognition process of human and interaction behaviors and human and device interaction behaviors provided by an embodiment of the present invention.

[0052] Figure 6 Schematic diagram of the calculation unit of the human transfer behavior provided by an embodiment of the present invention.

[0053] Figure 7 Schematic diagram of the calculation unit of a person operating a laptop keyboard provided by an embodiment of the present invention.

[0054] Figure 8 Schematic diagram of the analysis of the recognition result of the collaborative behavior and the system feedback provided by an embodiment of the present invention.

[0055] Figure 9 Schematic diagram of the system communication for collaborative behavior detection and recognition provided by an embodiment of the present invention. Detailed implementation manners

[0056] To make the technical means, creative features, achieved purposes and functions of the present invention easy to understand, the present invention will be further described below in conjunction with specific implementation manners.

[0057] In the description of the present invention, it should be noted that the orientation or positional relationship indicated by the terms "upper", "lower", "inner", "outer", "front end", "rear end", "both ends", "one end", "the other end", etc. is the orientation or positional relationship based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance.

[0058] In the description of the present invention, it should be noted that unless otherwise clearly specified and defined, terms such as "installation", "provided with", "connection", etc. should be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the communication inside two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0059] Embodiment 1:

[0060] A method for collaborative behavior detection and recognition provided by the present invention includes:

[0061] Step 1: Select a video data source for which collaborative behavior detection is required; the collaborative behavior includes: the interaction behavior between people and devices and the interaction behavior between people.

[0062] Step 2: Use an object detection algorithm to obtain the detection box information of objects and people.

[0063] Step 3: Use a multi-person pose estimation algorithm to obtain the skeletal key point data corresponding to people.

[0064] Step 4: Detect the collaborative behavior of people interacting with each other according to the human skeletal key point data and the predefined formula for the interaction behavior between people.

[0065] Step 5: Detect the collaborative behavior of people interacting with devices through the predefined calculation formula for the collaborative behavior between people and devices.

[0066] Step 6: Calculate the information entropy of the collaborative behavior according to the collaborative confidence of the user's collaborative behavior to obtain the feedback results of the recognition of the two types of collaborative behavior.

[0067] In the above step 1, the collaborative behavior is specifically subdivided into: the collaborative behavior of people interacting with each other and the collaborative behavior of people interacting with devices. In a collaborative scenario, the collaborative behavior refers to the process of using the way of people interacting with each other or people interacting with devices to do something or a sequence of actions to achieve a common goal. To clearly illustrate the technical solution of the present invention, 7 kinds of collaborative behaviors are specifically taken as examples for detailed description. As shown in Table 1 below, it is the definition of the key collaborative behavior categories in the collaborative scenario.

[0068]

[0069] Table 1 Key collaborative behavior categories in the collaborative scenario

[0070] Steps 1-5, by selecting different video sources, preprocess the video data as the input data for object detection algorithms (such as YOLOV5 object detection algorithm, SSD, etc.), and output the video key frame information containing detected personnel or devices. Next, extract the key point information of human skeletal points through a human pose estimation algorithm (such as OpenPose algorithm). The above object detection algorithm and human pose estimation algorithm can also adopt other existing relevant methods, which are not limited in the embodiments of the present disclosure, and the specific detection process will not be elaborated in detail.

[0071] Then, for the determination of each type of collaborative behavior, through a calculation unit, a collaborative behavior discrimination is given. For example, as shown in Table 1 above, there are 4 collaborative behaviors in the collaborative behavior analysis of human-to-human interaction; and 3 collaborative behavior categories in the collaborative behavior analysis of human-to-device interaction, totaling 7 types of collaborative behaviors.

[0072] In step 6, the categories of collaborative behaviors of human-to-human interaction and human-to-device interaction obtained through analysis, their corresponding collaborative confidence levels, and the information entropy of collaborative behaviors are used as important quantitative indicators for the collaborative participation activity in the collaborative scenario. Finally, the analysis information is transmitted to the end user in real time to improve the collaboration efficiency of the collaborative personnel in the collaborative scenario.

[0073] The principle is as Figure 1 shown. It is necessary to analyze the collaborative behaviors in the collaborative scenario. Specifically, the collaborative behaviors can be divided into human-to-device interaction behaviors and human-to-human interaction behaviors. After selecting a video data source, first use an object detection algorithm to obtain the detection box information of objects and people; then use a multi-person pose estimation algorithm to obtain the skeletal key point data. Based on the preprocessed human skeletal key points and the defined human-to-human interaction behavior formula, detect the collaborative behaviors of human-to-human interaction; apply the defined calculation formula for human-to-device collaborative behaviors to the detection of human-to-device interaction behaviors. Then, based on the collaborative confidence level of the user's collaborative behaviors, calculate the information entropy of the collaborative behaviors, and give the feedback results of the recognition of the two types of collaborative behaviors. Finally, provide a more intuitive understanding for the user to improve the efficiency of collaborative work.

[0074] The following will describe each of the above steps in detail:

[0075] In step 1 above, the specific process is as Figure 2 shown:

[0076] Step S201 refers to the ability to obtain the video in the collaborative scenario from a monocular camera, which is used to shoot the collaborative behavior video of the collaborative personnel in the collaborative scenario; it is also possible to select the camera of a personal notebook to shoot, or upload the video through the method of uploading a local video file.

[0077] Step S202 is to output the key frame data after video preprocessing. Generally, video preprocessing here means: cutting the video into individual image frames. The specific cutting can be processed at one frame per second or several frames per second.

[0078] Step S203 is to check whether the video data format meets the system's specification requirements. The system's specification requirements mainly refer to that the size of the image frames cut should meet the requirements of the selected object detection module and the selected human pose estimation module for the frame size.

[0079] Step S204 means that if the data format does not meet the system requirements, an error report will be sent.

[0080] Step S205 is to record the error-processed data in the database for later verification.

[0081] Step S206 outputs the data in the format that meets the system's regulations.

[0082] As Figure 3 shown, in Steps 2 and 3, the main tasks are to obtain the detection box information of the target and the person and the coordinate information of the human body bone points, preparing the necessary data for the calculation of the collaborative behavior in the next sub-process. Specifically, it includes:

[0083] Step S301 is to complete object detection on the video frame data;

[0084] Step S302 detects whether the current video frame contains a person. If there is a person, Step S303 is executed; otherwise, Step S308 is executed.

[0085] Step S303 performs human pose estimation on the person in the current frame;

[0086] Step S304 obtains information such as the coordinate points of the human body bone points in the current frame;

[0087] Step S305 continues to detect whether the current frame contains an object. If the current frame contains an object, it branches to Step S306; otherwise, it branches to Step S307.

[0088] Step S306 predicts to obtain the detection box information bbox of the object or device.

[0089] Step S307 outputs the boundary box information of the target or the person and the coordinate information of the human body bone points.

[0090] Step S308 detects whether there is the next frame of data. If there is the next frame, it continues to branch to Step S301; otherwise, this process ends.

[0091] Among them, the above-mentioned bone key points are as Figure 4As shown, the information of 14 key points of different parts of the human body obtained by the human pose estimation algorithm, and each key point corresponds to a two-dimensional coordinate (x i , y i ) and a confidence level conf i , where i represents the number of the human key point, and the value ranges from the number 0 to 13.

[0092] Specifically, nose: (x0, y0), conf0; neck: (x1, y1), conf1; right shoulder: (x2, y2), conf2; right elbow: (x3, y3), conf3; right wrist: (x4, y4), conf4; left shoulder: (x5, y5), conf5; left elbow: (x6, y6), conf6; left wrist: (x7, y7), conf7; right hip: (x8, y8), conf8; right knee: (x9, y9), conf9; right ankle: (x 10 , y 10 ), conf 10 ; left hip: (x 11 , y 11 ), conf 11 ; left knee: (x 12 , y 12 ), conf 12 ; left ankle: (x 13 , y 13 ), conf 13 .

[0093] Steps 4 and 5 above, as Figure 5 shown, are the flow diagrams of the calculation and recognition of the interaction behavior between people and equipment and between people in the present invention, including:

[0094] Step S501, input the data information obtained from the sub-processes of object detection and human pose estimation.

[0095] Step S502, used to verify whether the data contains the detection box information of the team members. If it contains the detection box and skeleton point information of the team members, then go to Step S503; otherwise, this process ends.

[0096] Step S503 is the preprocessing of information such as human skeleton point coordinates, and completes the data preparation for the recognition of human-human interaction.

[0097] Step S504, complete the calculations of each calculation unit for human-human interaction behavior.

[0098] Step S505, used to verify whether the input data contains device information. If it also contains the detection box information of the device, then execute Step S506; otherwise, this process ends.

[0099] Step S506: Separate the information of the device detection frame in the input data to complete the data preparation for human-device interaction recognition.

[0100] Step S507: Complete the calculations of each computing unit for human-human interaction behavior.

[0101] Step S508: Calculate the frequencies of each atomic behavior recognition.

[0102] Step S509: Output the recognition results of various collaborative behaviors, and this process ends.

[0103] Among them, the above S504 involves the analysis of 4 computing units, namely the computing unit for transfer behavior, the computing unit for receiving behavior, the computing unit for joining behavior, and the computing unit for leaving behavior. Since the steps of these 4 types of computing units are similar, one of the computing units will be described in detail below:

[0104] As Figure 6 shown, it is a schematic diagram of the computing unit for human transfer behavior of the present invention. When passing something to others, atomic actions such as arm stretching and head tilting forward generally occur. Taking the collaborative behaviors and coordinate data involved in Table 1 above and Figure 4 as an example, the specific calculation is divided into the following 3 steps:

[0105] (1) Specifically, the displacement of the elbow and wrist key points at two different times constitutes the arm displacement feature vector {D elbow , D wrist} to estimate the motion state of the human arm:

[0106]

[0107] Among them, Threshold is the threshold. According to prior empirical knowledge, it is set to 0.1 in the present invention.

[0108] (2) Based on the arm movement, judge whether the arm is lifted, and then judge the change in the included angle of the arm vector, and whether the included angle is getting larger, and whether the head orientation is within a certain range of the arm extension direction.

[0109] 1) Judge whether the arm is lifted

[0110]

[0111] 2) Taking the left shoulder, left elbow, and left wrist as an example, then judge the change in the included angle of the arm vector, and whether the included angle is first getting larger and then smaller:

[0112]

[0113]

[0114]

[0115] Satisfy α1(t + t τ ) > α1(t) and α1(t + 2t τ ) < α1(t + t τ ) (6 - 3)

[0116] α1(t) represents the included angle of the arm vector at time t; t τ represents a certain fixed time value; α1(t + t τ ) represents the included angle of the arm vector after an additional time t τ from time t; α1(t + 2t τ ) represents the included angle of the arm vector after an additional time 2t τ from time t;

[0117] 3) Whether the head orientation and the arm extension direction are maintained within a certain range at the same time.

[0118]

[0119]

[0120] And satisfy α2(t) ∈ [0°, 90°] (6 - 4)

[0121] Only when the determination conditions corresponding to the above two steps are satisfied can it be determined that a collaborative behavior of "transfer" (to others) in human - to - human interaction has occurred. Define time t1, t2 = t1 + t δ , t3 = t1 + 2t δ ; Here, count the number of frequencies fre1 of arm - raising, fre2 of the change in the included angle of the arm vector, fre3 of the determination that the head orientation is the same as the arm extension direction, and fre n of other atomic behaviors, such as sitting, standing, and arm - stretching behaviors, occurring simultaneously within the time periods [t1, t2] and [t2, t3]; calculate their probabilities as the collaborative confidence level CC j1 (t):

[0122]

[0123] Among them, the subscript j represents the category of human - to - human interaction behavior. According to Table 1 above, the value of j is 1, 2, 3, 4; the subscript number 1 represents the number of the "transfer" collaborative behavior, ccj1 (t) represents that the collaborative confidence is a function of time t.

[0124] That is: in step 4, through the detected collaborative behavior of human-human interaction, the collaborative confidence of the transfer behavior in the collaborative behavior of human-human interaction is further calculated.

[0125] Since the calculation units of other collaborative behaviors of human-human interaction are similar to the above, only this example is taken and will not be elaborated further.

[0126] And the above step S507 involves the analysis of a total of 3 calculation units, namely the calculation unit for operating the keyboard of a (laptop) computer, the calculation unit for operating the screen of a (mobile phone or tablet), and the calculation unit for operating a mouse. Since the steps of these 3 types of calculation units are similar, similarly, the present invention will elaborate in detail on one of the calculation units:

[0127] As Figure 7 shown, it is a schematic diagram of the calculation unit for a person to operate the keyboard of a (laptop) computer. The key points involved in its calculation are the right elbow, right wrist, left elbow, and left wrist. Generally, if the left wrist or right wrist falls within the laptop frame, it is considered that the laptop is being operated. It is specifically divided into the following 4 calculation steps:

[0128] (1) Through human pose estimation, for example, the two-dimensional coordinates of the right elbow, right wrist, left elbow, and left wrist obtained by the OpenPose algorithm are (x3, y3), (x4, y4), (x6, y6), and (x7, y7) respectively; calculate the upper left corner coordinates (x a1 , y a1 )) and the lower right corner coordinates (x a2 , y a2 ) of the bounding rectangle where the key points of the elbow and wrist are located:

[0129]

[0130]

[0131] (2) Through an object detection algorithm, such as the YOLOV5 algorithm, obtain (b x , b y , b w , b h ), the rectangular frame information of the laptop computer: (b x , b y , b w , b h ) is the predicted bounding box bbox. Among them, b w , b h are the predicted values (width and height of the border) relative to the entire image, b x , b yRepresent two two-dimensional coordinates, which respectively correspond to the upper left corner coordinates (x b1 , y b1 ) and the lower right corner coordinates (x b2 , y b2 ) of the boundary rectangle where the laptop keyboard is located in sequence.

[0132] (3) Calculate the area overlap rate between the boundary rectangles where the elbow and wrist are located and the rectangle of the laptop:

[0133]

[0134] (4) And define the obtained IOU i as the collaboration confidence cc i1 (t) of "operating (the laptop) keyboard":

[0135]

[0136] where the subscript i represents the category of human-device interaction behavior. According to Table 1 above, the value of i is 1, 2, 3; the subscript number 1 represents the number of the collaboration behavior of "operating (the laptop) keyboard", and the variable t represents that the collaboration confidence cc i1 (t) is a function of time t; Intersection i represents the intersection, and Union i represents the union.

[0137] That is: through the collaboration behavior of human-device interaction detected in step 5, further calculate the collaboration confidence of the "operating (the laptop) keyboard" behavior in the collaboration behavior of this human-device interaction.

[0138] In step 6, the information entropy cie based on collaboration behavior is defined as in formula (8-1) below. Calculate the collaboration activity metric cie for collaboration behavior detection, and feedback the cie metrics that change over time and the statistical situations of different collaboration behavior categories to the end users in the collaboration scenario, ultimately improving the collaboration office efficiency in the collaboration scenario. The specific statistical analysis is as follows:

[0139] Statistically analyze 7 collaboration behavior categories and their corresponding confidence levels and collaboration activities. Based on the information entropy of information theory, define the information entropy of human-human interaction based on collaboration behavior:

[0140]

[0141] From formula (8-1), the obtained cie j(t) refers to the information entropy of the collaborative behavior of all detectable human - to - human interactions of the user varying with time t. Where q refers to the number of users involved in human - to - human interaction behaviors, j is the unique category number of human - to - human interaction behaviors, and c n is the weight parameter of the human - to - human interaction behavior category, and satisfies cc jn (t) refers to the collaborative confidence of the human - to - human interaction collaborative behavior at time t. n refers to the unique sequential number of the human - to - human interaction collaborative behavior category, and its range of variation is {1, 2, 3, 4}.

[0142] Define the information entropy of human - to - device interaction based on collaborative behavior:

[0143]

[0144] From formula (8 - 2), the obtained cie i (t) refers to the information entropy of the collaborative behavior of all detectable human - to - device interactions of the user varying with time t. Where p refers to the number of users involved in human - to - device interaction behaviors, i is the unique category number of human - to - device interaction behaviors, and c m is the weight parameter of different categories of human - to - device interaction behaviors, and satisfies cc im (t) refers to the collaborative confidence of the human - to - device interaction collaborative behavior at time t. m refers to the unique sequential number of the human - to - device interaction collaborative behavior category, and its range of variation is {1, 2, 3}.

[0145] Define the information entropy based on two types of collaborative behaviors:

[0146] cie(t) = α·cie i (t)+β·cie j (t) (8 - 3)

[0147] From formula (8 - 3), the obtained cie(t) refers to the information entropy of the collaborative behavior of all detectable human - to - device interactions and human - to - human interactions of the user varying with time t. Here, α refers to the weight of the collaborative behavior information entropy of human - to - device interaction behaviors, β refers to the weight of the collaborative behavior information entropy of human - to - human interaction behaviors, and α + β = 1.

[0148] As Figure 8 shown, it is a schematic flowchart of the analysis of the recognition result of the collaborative behavior of the present invention and the system feedback. In this final sub - process, it mainly completes the statistics of the collaborative confidence cc of 7 types of collaborative behaviors of 2 different types of collaborative behaviors and the calculation of the information entropy cie of the collaborative behavior, and finally gives the visualization display result. Among them:

[0149] Step S601: Input data such as the recognition results of various collaboration behaviors in the previous sub - process.

[0150] Step S602: Calculate the information entropy cie of the collaboration behavior according to formulas (8 - 1), (8 - 2) and (8 - 3).

[0151] Step S603: Transmit the recognition results and statistical analysis data to the database in real - time.

[0152] Step S604: Judge by the user whether to store in the database. If so, go to Step S605; otherwise, go to Step S606.

[0153] Step S605: Complete the storage of the data analysis results in the database.

[0154] Step S606: It is a processing process for the system to give feedback or response.

[0155] Step S607: Complete the visual display of the final result.

[0156] The method for detecting and recognizing collaboration behaviors provided by the embodiments of the present invention can detect 7 collaboration behaviors of personnel in the collaboration scenario and give feedback and evaluation on the activity level of personnel in the collaboration scenario, aiming at the problem that the research on collaboration behaviors in the current collaboration scenario is not deep enough. It enables participants to more intuitively understand the behaviors that occur when personnel and devices share the physical space in the collaboration scenario, and helps participants adjust collaboration strategies and improve collaboration methods.

[0157] It can effectively detect the interaction behaviors between people and the interaction behaviors between people and devices in the collaboration scenario, identify the collaboration behavior categories, push the collaboration behavior recognition results to the collaboration members, and ultimately improve the office efficiency of the collaborative working environment.

[0158] Embodiment 2:

[0159] The present invention also provides a system for detecting and recognizing collaboration behaviors, including:

[0160] A selection and acquisition module, used to select the video data source that needs to be detected for collaboration behaviors; the collaboration behaviors include: the interaction behaviors between people and devices and the interaction behaviors between people.

[0161] A collaboration behavior detection module, used to obtain the detection frame information of objects and people by using the object detection algorithm; obtain the bone key - point data corresponding to people by using the multi - person pose estimation algorithm; detect the collaboration behaviors of people - to - people interaction according to the human bone key - point data and the predefined formula for people - to - people interaction behaviors; detect the collaboration behaviors of people - to - device interaction through the predefined calculation formula for people - to - device collaboration behaviors.

[0162] A data storage module for storing video frame information, cooperative behavior recognition results, their corresponding cooperative confidence levels, and cooperative behavior information entropy;

[0163] An identification feedback module for calculating the information entropy of cooperative behaviors based on the cooperative confidence levels of the user's cooperative behaviors to obtain feedback results for two types of cooperative behavior recognition.

[0164] Among them, in the selection and acquisition module, for example, 3 different video input methods can be used to collect and verify whether the video data meets the system requirements. Its main function is to output video frame data information for subsequent processing;

[0165] In the cooperative behavior detection module, it mainly realizes the detection of human-human interaction cooperative behaviors and the cooperative behaviors between humans and devices, and transmits the calculation and recognition result information to the database for storage, real-time feedback, and response-end processing through socket communication in real time.

[0166] In the data storage module, the main task is to store analysis information such as preprocessed video frame information, cooperative behavior recognition results, their corresponding cooperative confidence levels, and cooperative behavior information entropy.

[0167] In the identification feedback module, it mainly pushes the detection results after cooperative behavior recognition and other analysis information results to the terminal users in real time, and finally provides a scientific reference basis for the personnel in the cooperative scenario to improve the cooperation efficiency in the cooperative scenario.

[0168] Such as Figure 9 shown, is a communication schematic diagram of the system for cooperative behavior detection and recognition of the present invention and other sub-processes. After starting the system service, first obtain real-time video data through the camera and complete the preprocessing of video frames, and then through the socket communication mechanism, hand over the video frame data to the database to complete the calculation and recognition of cooperative behaviors, and save the detected cooperative behavior recognition results to the database and buffer through the socket communication mechanism; then the identification feedback module with real-time feedback and response functions will listen to the data updates of the buffer and the database through the socket communication mechanism, and push the required data to each terminal. Finally, multiple users of the terminal will transmit the details data such as cooperative behavior detection in the database to each terminal for rendering and viewing by multiple users after receiving the terminal user confirmation instruction, ultimately improving the efficiency of collaborative office. For other matters not covered, refer to the specific relevant description of the above-mentioned Embodiment 1.

[0169] In this embodiment, the definition of collaborative behavior in a collaborative scenario is given. Specifically, it is divided into collaborative behaviors of human-device interaction and human-human interaction. For each type of collaborative behavior, its collaborative atomic actions and the involved human key points are analyzed and explanations are given. On this basis, a system prototype is built. The pre-processed video stream is input into the system, and the relevant information of the object detection box and human key points is obtained through popular object detection algorithms and human pose estimation algorithm models. Then, relevant interpretability analysis is carried out on the calculation processes for discriminating two representative types of collaborative behaviors, realizing the detection of collaborative behaviors, and finally providing a scientific reference basis for personnel in the collaborative scenario and improving the collaborative efficiency in the collaborative scenario.

[0170] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these changes and modifications.

Claims

1. A method for collaborative behavior detection and recognition, characterized in that, It includes the following steps: Select a video data source for which collaborative behavior detection is to be performed; The collaborative behaviors include: the interaction behaviors between humans and devices and the interaction behaviors between humans; Use an object detection algorithm to obtain the detection box information of objects and humans; Use a multi-person pose estimation algorithm to obtain the skeletal key point data corresponding to humans; Detect the collaborative behaviors of human-to-human interaction based on the human skeletal key point data and the predefined human-to-human interaction behavior formula; specifically including: determining the human key point parts corresponding to the predefined human-to-human interaction behaviors; human-to-human interaction behaviors include: passing, receiving, joining, and leaving; constructing a human body part displacement feature vector from the displacement of the coordinate data of the corresponding human body key point parts at two different times to estimate the motion state of the human body parts; based on determining the motion state of the human body parts, judging whether the change in the included angle of the human body part vectors is within a preset range; and judging whether the head orientation and the human body part motion direction remain within a second preset range; when the change in the included angle of the human body part vectors is within the preset range and the head orientation and the human body part motion direction remain within the second preset range, it is determined that a collaborative behavior of human-to-human interaction has occurred; Detect the collaborative behaviors of human-to-device interaction through the predefined calculation formula of human-to-device collaborative behaviors; specifically including: determining the human key point parts corresponding to the predefined human-to-device interaction behaviors; human-to-device interaction behaviors include: operating the keyboard, operating the screen, and operating the mouse; judging whether the coordinate data of the corresponding human key point parts fall within the keyboard, screen, and mouse detection boxes corresponding to the identified devices; when it falls within the detection box, it is determined that a collaborative behavior of human-to-device interaction has occurred; Calculate the information entropy of the collaborative behavior according to the collaborative confidence of the user's collaborative behavior to obtain the feedback results of the two types of collaborative behavior recognition; specifically including: counting the frequencies of simultaneous human body part movements, changes in vector included angles, and determination frequencies of the same head orientation and human body part motion directions within two preset time periods before and after, as well as the frequency of sitting, standing, and arm stretching behaviors, and calculating their probabilities as the collaborative confidence of human-to-human interaction behaviors; taking the area overlap rate between the bounding rectangle of the human key point parts corresponding to the human-to-device interaction behavior and the keyboard, screen, or mouse detection box corresponding to the device as the collaborative confidence of human-to-device interaction; counting the collaborative behavior categories, corresponding confidences, and collaborative activity levels, generating the information entropy of the collaborative behavior; and obtaining the feedback results of the two types of collaborative behavior recognition according to the information entropy of the collaborative behavior.

2. The method for collaborative behavior detection and recognition according to claim 1, wherein, Selecting a video data source for which collaborative behavior detection is to be performed includes: Obtain video data in the collaborative scene by using a preset method; Preprocess the video data to obtain key frame data; Judge whether the key frame data meets the preset requirements; When the preset requirements are met, output formatted data that meets the preset requirements.

3. A method for collaborative behavior detection and recognition according to claim 2, characterized in that, Selecting a video data source for which collaborative behavior detection is to be performed further includes: When the preset requirements are not met, send an error report; and record the error-processed data in the database.

4. A method for collaborative behavior detection and recognition according to claim 2, characterized in that Using an object detection algorithm to obtain the detection box information of objects and humans includes: Use the YOLOV5 or SSD object detection algorithm to detect data in a format that meets preset requirements, and output the detection frame information of the key video frames containing the detected personnel or devices.

5. A method for collaborative behavior detection and recognition according to claim 1, characterized in that, Use the multi-person pose estimation algorithm to obtain the skeletal key point data corresponding to people, including: Extract the coordinate data of 14 key points of the human skeleton through the OpenPose algorithm.

6. A system for collaborative behavior detection and recognition, characterized in that, Use the method for collaborative behavior detection and recognition as described in any one of claims 1-5. The system includes: A selection acquisition module for selecting the video data source that needs to be detected for collaborative behavior; the collaborative behaviors include: the interaction behaviors between people and devices and the interaction behaviors between people. A collaborative behavior detection module for using the object detection algorithm to obtain the detection frame information of objects and people. Use the multi-person pose estimation algorithm to obtain the skeletal key point data corresponding to people; according to the human skeletal key point data and the predefined formula for the interaction behavior between people, detect the collaborative behavior of the interaction between people; through the predefined calculation formula for the collaborative behavior between people and devices, detect the collaborative behavior of the interaction between people and devices. A data storage module for storing video frame information, the recognition results of collaborative behaviors, their corresponding collaborative confidence levels, and the information entropy of collaborative behaviors. A recognition feedback module for calculating the information entropy of the collaborative behavior according to the collaborative confidence level of the user's collaborative behavior, and obtaining the feedback results of the recognition of two types of collaborative behaviors.

Images