Online behavior recognition method, device, electronic device and storage medium

By acquiring the correlation judgment of the human body skeleton information of the video image and the confidence timing array of the start and end point, the existing online behavior recognition methods have been solved, and the accurate identification and adaptation of complex behaviors has been achieved. Ability to differ in behavior time.

CN114299619BActive Publication Date: 2024-11-29JIHUA LAB
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202210055484.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-18
Publication Date
2024-11-29
Estimated Expiration
2042-01-18

AI Technical Summary

Technical Problem

The existing online behavior recognition methods are poor in flexibility and low in recognition accuracy, and cannot effectively deal with complex behaviors performed intersecting and parallel, as well as the problems of the time and length of time for different individuals to complete the same behavior.

Method used

By obtaining the latest N-frame video information of the human body skeleton information, inputting a pre-trained classification model, detecting the start and ending action confidence of the behavior, and using the correlation of the start and ending point confidence timing array to judge the start and ending flags of the behavior , adapt to behavioral time differences and complex scenarios of different individuals.

Benefits of technology

It achieves higher recognition accuracy and flexibility, and can correctly identify the cross-simultaneous execution of multiple behaviors, which is suitable for multiple scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114299619B_ABST
    Figure CN114299619B_ABST
Patent Text Reader

Abstract

The present application belongs to the field of behavior recognition technology, and discloses an online behavior recognition method, device, electronic device and storage medium, which obtains the human skeleton information of the latest N frames of video images; identifies the M behavior types with the highest probability of the behavior of each frame of video images; detects the starting point confidence of the starting action and the end point confidence of the ending action of each frame of video images belonging to various behavior types; detects whether the start mark of the corresponding behavior type appears according to the correlation between the starting point confidence timing array of various behavior types and the standard starting point confidence timing array of the corresponding behavior type; detects whether the end mark of the corresponding behavior type appears according to the correlation between the end point confidence timing array of various behavior types and the standard end point confidence timing array of the corresponding behavior type; and outputs the recognition result of the behavior type according to the detection results of the start mark and the end mark; thereby having good flexibility and high recognition accuracy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of behavior recognition technology, and in particular to an online behavior recognition method, device, electronic device and storage medium. Background Art

[0002] Behavior recognition technology can be roughly divided into two categories: offline behavior recognition methods and online behavior recognition methods. Offline behavior recognition methods require the input of a complete video sequence and then output the corresponding behavior category recognition results. The current offline behavior recognition methods are relatively mature, and their feature extraction mainly includes two-stream method and 3D convolution. However, offline behavior recognition has many constraints, such as the inability to monitor the output results in real time and lack of flexibility. Therefore, it is of great significance to study online real-time behavior recognition methods.

[0003] Existing online behavior recognition methods generally use a sliding window method to locate the temporal position of the behavior. The sliding window method requires a large amount of calculation, and is based on the following two assumptions: 1. It is assumed that human behavior is performed sequentially, that is, people will perform the next behavior only after completing a certain behavior; 2. It is assumed that the time lengths of different individuals performing the same behavior are basically the same. However, these two assumptions are oversimplifications of human behavior in some cases. Actual daily behaviors often involve complex situations such as cross-cutting and parallel execution, and different individuals sometimes have large differences in the length of time to complete the same behavior. Therefore, existing online behavior recognition methods have poor flexibility and low recognition accuracy. Summary of the invention

[0004] The purpose of the present application is to provide an online behavior recognition method, device, electronic device and storage medium, which have good flexibility and high recognition accuracy.

[0005] In a first aspect, the present application provides an online behavior recognition method for online recognition of the behavior type of human behavior, comprising the steps of:

[0006] A1. Obtain the human skeleton information of the latest N frames of video images; N is a preset positive integer;

[0007] A2. Input the human skeleton information of each frame of the video image into a pre-trained classification model, and identify the M behavior types with the highest probability of each frame of the video image; M is a preset positive integer;

[0008] A3. Based on the human skeleton information of each frame of the video image, according to the pre-trained start and end point confidence regression model, the starting point confidence of the starting action and the end point confidence of the ending action of each frame of the video image belonging to the various types of behavior are detected;

[0009] A4. According to the correlation between the starting point confidence timing array of various behavior types and the standard starting point confidence timing array of the corresponding behavior type, detect whether the start mark of the corresponding behavior type appears;

[0010] A5. According to the correlation between the endpoint confidence timing array of the various behavior types and the standard endpoint confidence timing array of the corresponding behavior type, detect whether the end mark of the corresponding behavior type appears;

[0011] A6. Output the recognition result of the behavior type according to the detection results of the start mark and the end mark.

[0012] This online behavior recognition method can detect behavior types through the human skeleton information of the latest N frames of video images, and has strong real-time performance. It determines whether the corresponding behavior type occurs by detecting the appearance of the start and end marks of various behavior types, and can adapt to the difference in the length of time it takes for different individuals to complete the same behavior, and has a higher recognition accuracy. At the same time, it independently detects and processes multiple behavior types with a high probability of occurrence, and can correctly identify complex situations where multiple behaviors are crossed and executed in parallel. The algorithm is flexible and can be applied to a variety of scenarios.

[0013] Preferably, step A1 comprises:

[0014] Get the latest N frames of video images;

[0015] Human skeleton information is extracted from each frame of the video image.

[0016] Preferably, before step A1, the method further comprises the following steps:

[0017] Generate M starting point confidence time series arrays of length N, where the initial value of each data in the starting point confidence time series array is 0;

[0018] Step A4 includes:

[0019] A401. According to the behavior type of the behavior of the video image in each frame, insert the corresponding starting point confidence into the end of the corresponding starting point confidence time series array, and delete the first data of the starting point confidence time series array;

[0020] A402. Calculate the first correlation between each of the starting point confidence time series arrays and the standard starting point confidence time series array of the corresponding behavior type according to the following formula:

[0021] ;

[0022] in, is the first correlation between the j-th starting point confidence time series array and the standard starting point confidence time series array of the corresponding behavior type, is the value of the i-th data of the j-th starting point confidence time series array, is the value of the i-th data of the standard starting point confidence time series array of the behavior type corresponding to the j-th starting point confidence time series array;

[0023] A403. Compare the first correlation of each of the starting point confidence time series arrays with a first preset correlation threshold to determine whether a start mark of a corresponding behavior type appears.

[0024] Preferably, after step A401 and before step A402, the method further includes the following steps:

[0025] A404. Calculate the following formula for each starting point confidence time series array: The first part of the confidence time series array of the standard starting point of the corresponding behavior type The second correlation of data:

[0026] ;

[0027] in, is the second correlation between the j-th starting point confidence time series array and the standard starting point confidence time series array of the corresponding behavior type, is a preset positive integer less than N, is the jth starting point confidence time series array The value of the data;

[0028] A405. Compare the second correlation of each of the starting point confidence time series arrays with a second preset correlation threshold to determine whether a sign indicating that a corresponding behavior type is about to start appears;

[0029] Step A403 and the following steps are also included:

[0030] A406. Modify the judgment result of whether the start mark of the corresponding behavior type appears according to the appearance of the about-to-start mark in each of the starting point confidence timing arrays.

[0031] Therefore, compared with the method of only relying on the first correlation to determine whether the start mark appears, the determination result of whether the start mark appears by considering both the second correlation and the first correlation is more accurate, which can further improve the recognition accuracy of the online behavior recognition method.

[0032] Preferably, before step A1, the method further comprises the following steps:

[0033] Generate M endpoint confidence time series arrays of length N, where the initial value of each data in the endpoint confidence time series array is 0;

[0034] Step A5 includes:

[0035] A501. Insert the corresponding endpoint confidence into the end of the corresponding endpoint confidence timing array according to the behavior type of the behavior of each frame of the video image and delete the first data of the endpoint confidence timing array;

[0036] A502. Calculate the third correlation between each of the endpoint confidence time series arrays and the standard endpoint confidence time series array of the corresponding behavior type according to the following formula:

[0037] ;

[0038] in, is the third correlation between the j-th endpoint confidence time series array and the standard endpoint confidence time series array of the corresponding behavior type, is the value of the ith data in the jth endpoint confidence time series array, is the value of the i-th data of the standard endpoint confidence time series array of the behavior type corresponding to the j-th endpoint confidence time series array;

[0039] A503. Compare the third correlation of each of the endpoint confidence time series arrays with the first preset correlation threshold to determine whether an end mark of a corresponding behavior type appears.

[0040] Preferably, after step A501 and before step A502, the method further includes the following steps:

[0041] A504. Calculate the back of each endpoint confidence time series array according to the following formula: The first part of the data and the standard endpoint confidence time series array of the corresponding behavior type The fourth correlation of data:

[0042] ;

[0043] in, is the fourth correlation between the j-th endpoint confidence time series array and the standard endpoint confidence time series array of the corresponding behavior type, is a preset positive integer less than N, is the jth endpoint confidence time series array The value of the data;

[0044] A505. Compare the fourth correlation of each of the endpoint confidence time series arrays with the second preset correlation threshold to determine whether a sign indicating that the corresponding behavior type is about to end appears;

[0045] Step A503 and the following steps are also included:

[0046] A506. Modify the judgment result of whether the end mark of the corresponding behavior type appears according to the appearance of the end mark of each end point confidence timing array.

[0047] Therefore, compared with the method of relying solely on the third correlation to determine whether the end mark appears, the determination result of determining whether the end mark appears by considering both the fourth correlation and the third correlation is more accurate, and can further improve the recognition accuracy of the online behavior recognition method.

[0048] Preferably, step A6 comprises:

[0049] When an end mark of a behavior type is detected, if a start mark of the corresponding behavior type has appeared within a second preset time period before the current moment, a recognition result indicating that a behavior of the corresponding behavior type is detected is output.

[0050] In a second aspect, the present application provides an online behavior recognition device for online recognition of the behavior type of human behavior, including:

[0051] The first acquisition module is used to acquire the human skeleton information of the latest N frames of video images, where N is a preset positive integer;

[0052] A classification module, used to input the human skeleton information of each frame of the video image into a pre-trained classification model, and identify M behavior types with the highest probability of the behavior of each frame of the video image; M is a preset positive integer;

[0053] A confidence recognition module is used to detect the starting point confidence of the starting action and the ending point confidence of the ending action of each frame of the video image, based on the human skeleton information of each frame of the video image and a pre-trained starting and ending point confidence regression model.

[0054] A first detection module, configured to detect whether a start mark of a corresponding behavior type appears according to the correlation between the starting point confidence time series arrays of various behavior types and the standard starting point confidence time series arrays of the corresponding behavior type;

[0055] A second detection module, configured to detect whether an end mark of a corresponding behavior type appears according to the correlation between the endpoint confidence time series arrays of the various behavior types and the standard endpoint confidence time series arrays of the corresponding behavior types;

[0056] The recognition module is used to output a recognition result of the behavior type according to the detection results of the start mark and the end mark.

[0057] The online behavior recognition device can detect behavior types through the human skeleton information of the latest N frames of video images, with strong real-time performance. It can determine whether a behavior of the corresponding behavior type occurs by detecting the appearance of the start and end marks of various behavior types, and can adapt to the difference in the length of time taken by different individuals to complete the same behavior, with a higher recognition accuracy. At the same time, it independently detects and processes multiple behavior types with a high probability of occurrence, and can correctly identify complex situations where multiple behaviors are crossed and executed in parallel. The algorithm is flexible and can be applied to a variety of scenarios.

[0058] In a third aspect, the present application provides an electronic device, including a processor and a memory, wherein the memory stores a computer program executable by the processor, and when the processor executes the computer program, it runs the steps in the online behavior recognition method described above.

[0059] In a fourth aspect, the present application provides a storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in the online behavior recognition method described above are executed.

[0060] Beneficial effects:

[0061] The online behavior recognition method, device, electronic device and storage medium provided by the present application obtain the human skeleton information of the latest N frames of video images; N is a preset positive integer; the human skeleton information of each frame of the video image is respectively input into a pre-trained classification model to identify the M behavior types with the highest probability of the behavior of each frame of the video image; M is a preset positive integer; based on the human skeleton information of each frame of the video image, according to a pre-trained start and end point confidence regression model, the starting point confidence of the starting action and the end point confidence of the ending action of the behavior of each frame of the video image belonging to various behavior types are detected; according to the correlation between the starting point confidence timing array of various behavior types and the standard starting point confidence timing array of the corresponding behavior type, whether the start mark of the corresponding behavior type appears; according to the correlation between the end point confidence timing array of various behavior types and the standard end point confidence timing array of the corresponding behavior type, whether the end mark of the corresponding behavior type appears; according to the detection results of the start mark and the end mark, the recognition result of the behavior type is output; thereby having good flexibility and high recognition accuracy.

[0062] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent from the description, or may be understood through implementation of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 A flowchart of an online behavior recognition method provided in an embodiment of the present application.

[0064] Figure 2 A schematic diagram of the structure of an online behavior recognition device provided in an embodiment of the present application.

[0065] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0066] Figure 4 A schematic diagram of an exemplary updating process of a starting point confidence timing array.

[0067] Figure 5 A schematic diagram of another exemplary updating process of the starting point confidence timing array. DETAILED DESCRIPTION

[0068] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0069] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0070] Please refer to Figure 1 , Figure 1 An online behavior recognition method in some embodiments of the present application is used to online recognize the behavior type of human behavior, including the steps of:

[0071] A1. Obtain the human skeleton information of the latest N frames of video images; N is a preset positive integer;

[0072] A2. Input the human skeleton information of each frame of video image into the pre-trained classification model, and identify the M behavior types with the highest probability of the behavior of each frame of video image (specifically, the behavior of the person in the video image); M is a preset positive integer;

[0073] A3. Based on the human skeleton information of each frame of video image, according to the pre-trained start and end point confidence regression model, the starting point confidence of the starting action and the end point confidence of the ending action of each frame of video image belonging to various behavior types are detected;

[0074] A4. According to the correlation between the starting point confidence timing array of various behavior types and the standard starting point confidence timing array of the corresponding behavior type, detect whether the start mark of the corresponding behavior type appears;

[0075] A5. According to the correlation between the endpoint confidence timing array of various behavior types and the standard endpoint confidence timing array of the corresponding behavior type, detect whether the end mark of the corresponding behavior type appears;

[0076] A6. Output the recognition result of the behavior type according to the detection results of the start mark and the end mark.

[0077] This online behavior recognition method can detect behavior types through the human skeleton information of the latest N frames of video images, and has strong real-time performance. It determines whether a behavior of the corresponding behavior type occurs by detecting the appearance of the start and end marks of various behavior types, and can adapt to the difference in the length of time it takes for different individuals to complete the same behavior, and has a higher recognition accuracy. At the same time, it independently detects and processes multiple behavior types with a high probability of occurrence, and can correctly identify complex situations where multiple behaviors are crossed and executed in parallel. The algorithm is flexible and can be applied to a variety of scenarios.

[0078] Since the online behavior recognition method is used to recognize the behavior type of human behavior online in real time, the latest N frames of video images refer to a frame of video image acquired at the current moment and N frames of video images consisting of the penultimate frame to the penultimate N-1 frame of video image acquired before the current moment. Among them, N can be set according to actual needs. In this embodiment, N=9, but it is not limited thereto.

[0079] Wherein, step A1 comprises:

[0080] A101. Get the latest N frames of video images;

[0081] A102. Extract human skeleton information from each frame of video image.

[0082] The human skeleton information mainly includes the coordinate information of the human head and joints. The existing human skeleton information extraction method can be used to extract the human skeleton information. The human skeleton information extraction method used is not limited here.

[0083] In some preferred embodiments, the human skeleton information includes absolute coordinate information and relative coordinate information of the head and joints of the person, wherein the absolute coordinate information is the coordinate information relative to the origin of the image coordinate system, and the relative coordinate information is the coordinate information relative to the neck joint. By coordinating the two sets of coordinates, the online behavior recognition method can adapt to situations where the person is located at different positions in the picture, the camera is inverted, etc., and improve generalization.

[0084] Furthermore, after step A102, the method further includes:

[0085] A103. Obtain height information based on human skeleton information;

[0086] A104. Normalize the human skeleton information according to the height information.

[0087] Among them, the height information is calculated according to the length of the connection line between each joint. The specific calculation method is existing technology and will not be described in detail here.

[0088] The normalization method is to divide the coordinate information of the head and each joint by the height information. Using the normalized human skeleton information for behavior recognition can make the online behavior recognition method suitable for behavior recognition of people of different body shapes and can adapt to cameras and video streams with different resolutions without the need for forced resolution scaling.

[0089] In some embodiments, the classification model classifies the behavior types of the behavior by using methods such as Random Forest or Gradient Boosting Decision Tree. Among them, the M behavior types with the highest probability of the behavior of the video image refer to the M behavior types that rank in the top M when the probability of the behavior of the video image belonging to various behavior types is sorted from large to small. Among them, M can be set according to actual needs. In this embodiment, M=3, but it is not limited to this. In the case where the input human skeleton information is normalized human skeleton information, when training the classification model, the human skeleton information samples in the training set need to be normalized.

[0090] For example, M=3, and the types of human behaviors include falling, asking for help, climbing, walking, and jumping. After the human skeleton information of a frame of video image is input into the classification model, the classification model recognizes that the probability of the person's behavior in the video image being falling is 30%, the probability of asking for help is 20%, the probability of climbing is 25%, the probability of walking is 15%, and the probability of jumping is 10%. Then the recognition results of the three behavior types of falling, asking for help, and climbing will be output.

[0091] Among them, the start-end point confidence regression model is a neural network model including two LSTM (Long Short-Term Memory) layers and one fully connected layer. When training, the start-end point confidence regression model uses a video image sequence including a start action and an end action of a behavior type as a sample, and calibrates the frame image of the start action and the frame image of the end action. The start-end point confidence regression model calculates the start point confidence of the start action and the end point confidence of the end action of each frame of the video image in the video that the behavior belongs to the behavior type through the following formula:

[0092] ;

[0093] ;

[0094] in, For the video The behavior of the frame video image belongs to the starting point confidence of the starting action of this behavior type, is the frame number of the frame image marked as the starting action in the video, For the video The behavior of the frame video image belongs to the endpoint confidence of the termination action of this behavior type, The frame number of the frame image marked as the termination action in the video.

[0095] When the input human skeleton information is normalized human skeleton information, the human skeleton information of the samples in the training set needs to be normalized when training the start-end point confidence regression model.

[0096] Since the human skeleton information of each frame of video image can identify M types of behaviors, in step A3, the human skeleton information of each frame of video image is detected to obtain M starting point confidence data and M end point confidence data. For example, assuming M=3, the three behavior types identified by the human skeleton information of a frame of video image are falling, asking for help and climbing; the behavior type "falling" and the human skeleton information of the frame of video image are input into the start-end point confidence regression model to obtain the starting point confidence of the starting action of the behavior of the frame of video image belonging to "falling" and the end point confidence of the ending action belonging to "falling"; the behavior type "asking for help" and the human skeleton information of the frame of video image are input into the start-end point confidence regression model to obtain the starting point confidence of the starting action of the behavior of the frame of video image belonging to "asking for help" and the end point confidence of the ending action belonging to "asking for help"; the behavior type "climbing" and the human skeleton information of the frame of video image are input into the start-end point confidence regression model to obtain the starting point confidence of the starting action of the behavior of the frame of video image belonging to "climbing" and the end point confidence of the ending action belonging to "climbing".

[0097] In some embodiments, before step A1, the method further comprises the following steps:

[0098] Generate M starting point confidence time series arrays of length N, where the initial value of each data in the starting point confidence time series array is 0;

[0099] Thus, step A4 includes:

[0100] A401. According to the behavior type of each frame of the video image (referring to the M types of behavior identified), the corresponding starting point confidence is inserted into the end of the corresponding starting point confidence time series array, and the first data of the starting point confidence time series array is deleted;

[0101] A402. Calculate the first correlation between each starting point confidence time series array and the standard starting point confidence time series array of the corresponding behavior type according to the following formula:

[0102] ;

[0103] in, is the first correlation between the j-th starting point confidence time series array and the standard starting point confidence time series array of the corresponding behavior type, is the value of the i-th data in the j-th starting point confidence time series array, is the value of the i-th data of the standard starting point confidence time series array of the behavior type corresponding to the j-th starting point confidence time series array;

[0104] A403. Compare the first correlation of each starting point confidence time series array with the first preset correlation threshold to determine whether a start mark of a corresponding behavior type appears.

[0105] For example Figure 4 The figure is an exemplary updating process of a starting point confidence time series array, wherein M=3, N=9, and the data in the three initialized starting point confidence time series arrays are all 0; the three actions with the top three probabilities obtained from the first frame of video image recognition are falling, asking for help and climbing, and the corresponding starting point confidences are 0.1, 0.1 and 0.1, respectively, which are inserted into the end of the corresponding starting point confidence time series array and the first data of the corresponding starting point confidence time series array is deleted; the three actions with the top three probabilities obtained from the second frame of video image recognition are falling, asking for help and climbing, and the corresponding starting point confidences are 0.2, 0.2 and 0.2, respectively, which are inserted into the end of the corresponding starting point confidence time series array and the first data of the corresponding starting point confidence time series array is deleted, and so on.

[0106] It should be noted that, when the M behavior types identified by a certain frame of video image are different from the M behavior types identified by the previous frame of video image, the behavior type that no longer appears (i.e., the behavior type that belongs to one of the M behavior types of the previous frame of video image but does not belong to one of the M behavior types of the current frame of video image) is replaced by the newly appeared behavior type (i.e., the behavior type that does not belong to one of the M behavior types of the previous frame of video image), the starting point confidence timing array of the replaced behavior type is initialized (its data are all 0), and then the starting point confidence of the newly appeared behavior type is inserted to the end of the starting point confidence timing array, and the first number of the starting point confidence timing array is deleted.

[0107] For example Figure 5 The following is an exemplary updating process of a starting point confidence timing array. The three actions identified in the first and second video frames are falling, asking for help and climbing, respectively. The three actions identified in the third video frame are falling, asking for help and staying, respectively. At this time, the newly appearing behavior type is staying, and the no longer appearing behavior type is climbing. Thus, the starting point confidence timing array of "staying" is used to replace the starting point confidence timing array of "climbing". After the starting point confidence timing array of "staying" is initialized, the starting point confidence (0.1) corresponding to "staying" of the third video frame is inserted at the end of the starting point confidence timing array, and the first number (0) of the starting point confidence timing array is deleted.

[0108] Each behavior type has a standard starting point confidence time series array of length N, and the standard starting point confidence time series array can be measured in advance. The closer the first correlation is to 1, the greater the probability that the starting action of the corresponding behavior type appears in the video image. When the first correlation is greater than the first preset correlation threshold, it can be considered that the start mark of the corresponding behavior type appears. Therefore, step A403 includes:

[0109] If a first correlation between the starting point confidence time series array and the standard starting point confidence time series array of the corresponding behavior type is greater than a first preset correlation threshold, it is determined that a start mark of the corresponding behavior type appears.

[0110] The first preset correlation threshold may be set according to actual needs. For example, the first preset correlation threshold is 0.9.

[0111] In some preferred embodiments, after step A401 and before step A402, the following steps are further included:

[0112] A404. Calculate the back of each starting point confidence time series array according to the following formula The first part of the confidence time series array of the standard starting point of the corresponding behavior type The second correlation of data:

[0113] ;

[0114] in, is the second correlation between the j-th starting point confidence time series array and the standard starting point confidence time series array of the corresponding behavior type, is a preset positive integer less than N, is the jth starting point confidence time series array The value of the data;

[0115] A405. Compare the second correlation of each starting point confidence time series array with the second preset correlation threshold to determine whether a corresponding behavior type is about to start a sign;

[0116] Step A403 and the following steps are also included:

[0117] A406. Modify the judgment result of whether the start mark of the corresponding behavior type appears according to the appearance of the about-to-start mark of each starting point confidence timing array.

[0118] Therefore, compared with the method of only relying on the first correlation to determine whether the start mark appears, the determination result of whether the start mark appears by considering both the second correlation and the first correlation is more accurate, which can further improve the recognition accuracy of the online behavior recognition method.

[0119] in, The value can be determined according to actual needs. In this embodiment, The value is 5, but not limited to this.

[0120] Wherein, step A405 includes: if the back of the starting point confidence time series array The first part of the confidence time series array of the standard starting point of the corresponding behavior type If the second correlation of the data is greater than the second preset correlation threshold, it is determined that the start-about sign appears. The second preset correlation threshold can be set according to implementation needs. In this embodiment, the second preset correlation threshold is 0.9, but is not limited thereto.

[0121] Wherein, step A406 includes:

[0122] If it is determined based on the first correlation that a start mark of a behavior type appears, then it is determined whether a start mark of the behavior type has appeared within a first preset time period (which can be set according to actual needs) before the current moment. If so, it is finally determined that the start mark of the behavior type appears. If not, it is finally determined that the start mark of the behavior type does not appear.

[0123] In some embodiments, before step A1, the method further comprises the following steps:

[0124] Generate M endpoint confidence time series arrays of length N, where the initial value of each data in the endpoint confidence time series array is 0;

[0125] Step A5 includes:

[0126] A501. Insert the corresponding endpoint confidence into the end of the corresponding endpoint confidence timing array according to the behavior type of each frame of the video image and delete the first data of the endpoint confidence timing array;

[0127] A502. Calculate the third correlation between each endpoint confidence time series array and the standard endpoint confidence time series array of the corresponding behavior type according to the following formula:

[0128] ;

[0129] in, is the third correlation between the j-th endpoint confidence time series array and the standard endpoint confidence time series array of the corresponding behavior type, is the value of the i-th data in the j-th endpoint confidence time series array, is the value of the i-th data of the standard endpoint confidence time series array of the behavior type corresponding to the j-th endpoint confidence time series array;

[0130] A503. Compare the third correlation of each endpoint confidence time series array with the first preset correlation threshold to determine whether an end mark of the corresponding behavior type appears.

[0131] The update process of the endpoint confidence time series array is similar to that of the starting point confidence time series array. For details, please refer to Figure 4 The process can be completed by changing the starting point confidence level to the end point confidence level.

[0132] It should be noted that when the M behavior types identified in a certain frame of video image are different from the M behavior types identified in the previous frame of video image, the behavior type that no longer appears (i.e., the behavior type that belongs to one of the M behavior types in the previous frame of video image but does not belong to one of the M behavior types in the current frame of video image) is replaced by the newly appeared behavior type (i.e., the behavior type that does not belong to one of the M behavior types in the previous frame of video image), the endpoint confidence timing array of the replaced behavior type is initialized (its data are all 0), and then the endpoint confidence of the newly appeared behavior type is inserted at the end of the endpoint confidence timing array, and the first number of the endpoint confidence timing array is deleted. For details, please refer to Figure 5 The updating process of the starting point confidence time series array is completed, and the starting point confidence is replaced by the end point confidence.

[0133] Each behavior type has a standard endpoint confidence time series array of length N, and the standard endpoint confidence time series array can be measured in advance. The closer the third correlation is to 1, the greater the probability that the termination action of the corresponding behavior type appears in the video image. When the third correlation is greater than the first preset correlation threshold, it can be considered that the end mark of the corresponding behavior type appears. Therefore, step A503 includes:

[0134] If the third correlation between the endpoint confidence timing array and the standard endpoint confidence timing array of the corresponding behavior type is greater than the first preset correlation threshold, it is determined that the end mark of the corresponding behavior type appears.

[0135] In some preferred embodiments, after step A501 and before step A502, the following steps are further included:

[0136] A504. Calculate the confidence level of each endpoint time series array according to the following formula: The first part of the data and the standard endpoint confidence time series array of the corresponding behavior type The fourth correlation of data:

[0137] ;

[0138] in, is the fourth correlation between the j-th endpoint confidence time series array and the standard endpoint confidence time series array of the corresponding behavior type, is a preset positive integer less than N, is the first value of the jth endpoint confidence time series array The value of the data;

[0139] A505. Compare the fourth correlation of each endpoint confidence time series array with the second preset correlation threshold to determine whether a corresponding behavior type is about to end the mark;

[0140] Step A503 and the following steps are also included:

[0141] A506. According to the appearance of the end mark of each endpoint confidence timing array, correct the judgment result of whether the end mark of the corresponding behavior type appears.

[0142] Therefore, compared with the method of relying solely on the third correlation to determine whether the end mark appears, the determination result of determining whether the end mark appears by considering both the fourth correlation and the third correlation is more accurate, and can further improve the recognition accuracy of the online behavior recognition method.

[0143] Wherein, step A505 includes: if the back of an endpoint confidence time series array The first part of the data and the standard endpoint confidence time series array of the corresponding behavior type If the fourth correlation of the data is greater than the second preset correlation threshold, it is determined that the end mark appears.

[0144] Wherein, step A506 includes:

[0145] If it is determined based on the third correlation that an end mark of a behavior type appears, then it is determined whether an end mark of the behavior type has appeared within a first preset time period (which can be set according to actual needs) before the current moment. If so, then it is finally determined that an end mark of the behavior type appears. If not, then it is finally determined that an end mark of the behavior type does not appear.

[0146] In this embodiment, step A6 includes:

[0147] When an end mark of a behavior type is detected, if a start mark of the corresponding behavior type has appeared within a second preset time period (which can be set according to actual needs) before the current moment, a recognition result indicating that a behavior of the corresponding behavior type has been detected is output.

[0148] That is, only when the end mark of a certain behavior type appears and the start mark of the behavior type has appeared within the previous second preset time period, is this type of behavior considered to be detected; the behavior type is identified through the appearance of the start mark and the end mark, and when multiple behaviors are executed crosswise or in parallel, the identification result can be output correctly, and the interference of invalid behaviors (i.e., behaviors that only have an end mark but no start mark within the previous second preset time period) can be avoided.

[0149] As can be seen from the above, the online behavior recognition method obtains the human skeleton information of the latest N frames of video images; N is a preset positive integer; the human skeleton information of each frame of video images is respectively input into a pre-trained classification model to identify the M behavior types with the highest probability of the behavior of each frame of video images; M is a preset positive integer; based on the human skeleton information of each frame of video images, according to the pre-trained start and end point confidence regression model, the starting point confidence of the starting action and the end point confidence of the ending action of each frame of video images belonging to various behavior types are detected; according to the correlation between the starting point confidence time series array of various behavior types and the standard starting point confidence time series array of the corresponding behavior type, whether the start mark of the corresponding behavior type appears; according to the correlation between the end point confidence time series array of various behavior types and the standard end point confidence time series array of the corresponding behavior type, whether the end mark of the corresponding behavior type appears; according to the detection results of the start mark and the end mark, the recognition result of the behavior type is output; thereby having good flexibility and high recognition accuracy.

[0150] Please refer to Figure 2The present application provides an online behavior recognition device for online recognition of the behavior type of human behavior, including:

[0151] The first acquisition module 1 is used to acquire the human skeleton information of the latest N frames of video images, where N is a preset positive integer;

[0152] Classification module 2, used to input the human skeleton information of each frame of video image into a pre-trained classification model, and identify the M behavior types with the highest probability of each frame of video image; M is a preset positive integer;

[0153] The confidence recognition module 3 is used to detect the starting point confidence of the starting action and the ending point confidence of the ending action of each frame of video image, based on the human skeleton information of each frame of video image and the pre-trained starting and ending point confidence regression model;

[0154] A first detection module 4, configured to detect whether a start mark of a corresponding behavior type appears according to the correlation between the start point confidence time series arrays of various behavior types and the standard start point confidence time series arrays of the corresponding behavior type;

[0155] The second detection module 5 is used to detect whether the end mark of the corresponding behavior type appears according to the correlation between the endpoint confidence time series array of each behavior type and the standard endpoint confidence time series array of the corresponding behavior type;

[0156] The recognition module 6 is used to output the recognition result of the behavior type according to the detection results of the start mark and the end mark.

[0157] The online behavior recognition device detects behavior types through the human skeleton information of the latest N frames of video images, and has strong real-time performance. It determines whether the corresponding behavior type occurs by detecting the appearance of the start and end marks of various behavior types, and can adapt to the difference in the length of time taken by different individuals to complete the same behavior, and has a higher recognition accuracy. At the same time, it detects and processes multiple behavior types with a high probability of occurrence separately, and can correctly identify complex situations where multiple behaviors are crossed and executed in parallel. The algorithm is flexible and can be applied to a variety of scenarios.

[0158] Since the online behavior recognition device is used to recognize the behavior type of human behavior online in real time, the latest N frames of video images refer to a frame of video image acquired at the current moment and N frames of video images consisting of the penultimate frame to the penultimate N-1 frame of video image acquired before the current moment. Among them, N can be set according to actual needs. In this embodiment, N=9, but it is not limited thereto.

[0159] Wherein, the first acquisition module 1 is used to execute, when acquiring the latest N frames of human skeleton information of the video image:

[0160] Get the latest N frames of video images;

[0161] Extract human skeleton information from each frame of video image.

[0162] The human skeleton information mainly includes the coordinate information of the human head and joints. The existing human skeleton information extraction method can be used to extract the human skeleton information. The human skeleton information extraction method used is not limited here.

[0163] In some preferred embodiments, the human skeleton information includes absolute coordinate information and relative coordinate information of the head and joints of the person, wherein the absolute coordinate information is the coordinate information relative to the origin of the image coordinate system, and the relative coordinate information is the coordinate information relative to the neck joint. By coordinating the two sets of coordinates, the online behavior recognition device can adapt to situations where the person is located at different positions in the picture, the camera is inverted, etc., thereby improving generalization.

[0164] Furthermore, the first acquisition module 1 is used to, when acquiring the latest N frames of human skeleton information of the video image, also execute:

[0165] Obtain height information based on human skeleton information;

[0166] The human skeleton information is normalized according to the height information.

[0167] Among them, the height information is calculated according to the length of the connection line between each joint. The specific calculation method is existing technology and will not be described in detail here.

[0168] The normalization method is to divide the coordinate information of the head and each joint by the height information. Using the normalized human skeleton information for behavior recognition can make the online behavior recognition device suitable for behavior recognition of people of different body shapes and can adapt to cameras and video streams with different resolutions without the need for forced resolution scaling.

[0169] In some embodiments, the classification model classifies the behavior types of the behavior by using methods such as Random Forest or Gradient Boosting Decision Tree. Among them, the M behavior types with the highest probability of the behavior of the video image refer to the M behavior types that rank in the top M when the probability of the behavior of the video image belonging to various behavior types is sorted from large to small. Among them, M can be set according to actual needs. In this embodiment, M=3, but it is not limited to this. In the case where the input human skeleton information is normalized human skeleton information, when training the classification model, the human skeleton information samples in the training set need to be normalized.

[0170] For example, M=3, and the types of human behaviors include falling, asking for help, climbing, walking, and jumping. After the human skeleton information of a frame of video image is input into the classification model, the classification model recognizes that the probability of the person's behavior in the video image being falling is 30%, the probability of asking for help is 20%, the probability of climbing is 25%, the probability of walking is 15%, and the probability of jumping is 10%. Then the recognition results of the three behavior types of falling, asking for help, and climbing will be output.

[0171] Among them, the start-end point confidence regression model is a neural network model including two LSTM (Long Short-Term Memory) layers and one fully connected layer. When training, the start-end point confidence regression model uses a video image sequence including a start action and an end action of a behavior type as a sample, and calibrates the frame image of the start action and the frame image of the end action. The start-end point confidence regression model calculates the start point confidence of the start action and the end point confidence of the end action of each frame of the video image in the video that the behavior belongs to the behavior type through the following formula:

[0172] ;

[0173] ;

[0174] in, For the video The behavior of the frame video image belongs to the starting point confidence of the starting action of this behavior type, is the frame number of the frame image marked as the starting action in the video, For the video The behavior of the frame video image belongs to the endpoint confidence of the termination action of this behavior type, The frame number of the frame image marked as the termination action in the video.

[0175] When the input human skeleton information is normalized human skeleton information, the human skeleton information of the samples in the training set needs to be normalized when training the start-end point confidence regression model.

[0176] Since the human skeleton information of each frame of video image can identify M types of behaviors, the confidence recognition module 3 detects the human skeleton information of each frame of video image to obtain M starting point confidence data and M end point confidence data. For example, assuming M=3, the three behavior types identified by the human skeleton information of a frame of video image are falling, asking for help and climbing; the behavior type "falling" and the human skeleton information of the frame of video image are input into the start-end point confidence regression model to obtain the starting point confidence of the starting action of the behavior of the frame of video image belonging to "falling" and the end point confidence of the ending action belonging to "falling"; the behavior type "asking for help" and the human skeleton information of the frame of video image are input into the start-end point confidence regression model to obtain the starting point confidence of the starting action of the behavior of the frame of video image belonging to "asking for help" and the end point confidence of the ending action belonging to "asking for help"; the behavior type "climbing" and the human skeleton information of the frame of video image are input into the start-end point confidence regression model to obtain the starting point confidence of the starting action of the behavior of the frame of video image belonging to "climbing" and the end point confidence of the ending action belonging to "climbing".

[0177] In some embodiments, the online behavior recognition device further includes:

[0178] A first array generation module is used to generate M starting point confidence time series arrays of length N, where the initial value of each data in the starting point confidence time series array is 0;

[0179] Thus, the first detection module 4 is used to perform the following when detecting whether the start mark of the corresponding behavior type appears according to the correlation between the starting point confidence time series array of various behavior types and the standard starting point confidence time series array of the corresponding behavior type:

[0180] According to the behavior type of the behavior of each frame of the video image (referring to the M recognized behavior types), the corresponding starting point confidence is inserted into the end of the corresponding starting point confidence time series array and the first data of the starting point confidence time series array is deleted;

[0181] The first correlation between each starting point confidence timing array and the standard starting point confidence timing array of the corresponding behavior type is calculated according to the following formula:

[0182] ;

[0183] in, is the first correlation between the j-th starting point confidence time series array and the standard starting point confidence time series array of the corresponding behavior type, is the value of the i-th data in the j-th starting point confidence time series array, is the value of the i-th data of the standard starting point confidence time series array of the behavior type corresponding to the j-th starting point confidence time series array;

[0184] The first correlation of each starting point confidence time series array is compared with a first preset correlation threshold to determine whether a start mark of a corresponding behavior type appears.

[0185] For example Figure 4 The figure is an exemplary updating process of a starting point confidence time series array, wherein M=3, N=9, and the data in the three initialized starting point confidence time series arrays are all 0; the three actions with the top three probabilities obtained from the first frame of video image recognition are falling, asking for help and climbing, and the corresponding starting point confidences are 0.1, 0.1 and 0.1, respectively, which are inserted into the end of the corresponding starting point confidence time series array and the first data of the corresponding starting point confidence time series array is deleted; the three actions with the top three probabilities obtained from the second frame of video image recognition are falling, asking for help and climbing, and the corresponding starting point confidences are 0.2, 0.2 and 0.2, respectively, which are inserted into the end of the corresponding starting point confidence time series array and the first data of the corresponding starting point confidence time series array is deleted, and so on.

[0186] It should be noted that, when the M behavior types identified by a certain frame of video image are different from the M behavior types identified by the previous frame of video image, the behavior type that no longer appears (i.e., the behavior type that belongs to one of the M behavior types of the previous frame of video image but does not belong to one of the M behavior types of the current frame of video image) is replaced by the newly appeared behavior type (i.e., the behavior type that does not belong to one of the M behavior types of the previous frame of video image), the starting point confidence timing array of the replaced behavior type is initialized (its data are all 0), and then the starting point confidence of the newly appeared behavior type is inserted to the end of the starting point confidence timing array, and the first number of the starting point confidence timing array is deleted.

[0187] For example Figure 5 The following is an exemplary updating process of a starting point confidence timing array. The three actions identified in the first and second video frames are falling, asking for help and climbing, respectively. The three actions identified in the third video frame are falling, asking for help and staying, respectively. At this time, the newly appearing behavior type is staying, and the no longer appearing behavior type is climbing. Thus, the starting point confidence timing array of "staying" is used to replace the starting point confidence timing array of "climbing". After the starting point confidence timing array of "staying" is initialized, the starting point confidence (0.1) corresponding to "staying" of the third video frame is inserted at the end of the starting point confidence timing array, and the first number (0) of the starting point confidence timing array is deleted.

[0188] Each behavior type has a standard starting point confidence time series array of length N, and the standard starting point confidence time series array can be measured in advance. The closer the first correlation is to 1, the greater the probability that the starting action of the corresponding behavior type appears in the video image. When the first correlation is greater than the first preset correlation threshold, it can be considered that the start mark of the corresponding behavior type appears. Therefore, when the first detection module 4 compares the first correlation of each starting point confidence time series array with the first preset correlation threshold to determine whether the start mark of the corresponding behavior type appears, it executes:

[0189] If a first correlation between the starting point confidence time series array and the standard starting point confidence time series array of the corresponding behavior type is greater than a first preset correlation threshold, it is determined that a start mark of the corresponding behavior type appears.

[0190] The first preset correlation threshold may be set according to actual needs. For example, the first preset correlation threshold is 0.9.

[0191] In some preferred embodiments, the first detection module 4 is used to detect whether the start mark of the corresponding behavior type appears according to the correlation between the starting point confidence time series array of various behavior types and the standard starting point confidence time series array of the corresponding behavior type, and further execute:

[0192] The following formula is used to calculate the confidence time series array of each starting point: The first part of the confidence time series array of the standard starting point of the corresponding behavior type The second correlation of data:

[0193] ;

[0194] in, is the second correlation between the j-th starting point confidence time series array and the standard starting point confidence time series array of the corresponding behavior type, is a preset positive integer less than N, is the jth starting point confidence time series array The value of the data;

[0195] Comparing the second correlation of each starting point confidence time series array with a second preset correlation threshold to determine whether a sign of the corresponding behavior type is about to start appears;

[0196] According to the appearance of the about-to-start flag of each starting point confidence time series array, the judgment result of whether the start flag of the corresponding behavior type appears is corrected.

[0197] Therefore, compared with the method of only relying on the first correlation to determine whether the start mark appears, the determination result of whether the start mark appears by considering both the second correlation and the first correlation is more accurate, which can further improve the recognition accuracy of the online behavior recognition device.

[0198] in, The value can be determined according to actual needs. In this embodiment, The value is 5, but not limited to this.

[0199] Among them, when the first detection module 4 compares the second correlation of each starting point confidence time series array with the second preset correlation threshold to determine whether the corresponding behavior type is about to start, the first detection module 4 executes: if the back of the starting point confidence time series array The first part of the confidence time series array of the standard starting point of the corresponding behavior type If the second correlation of the data is greater than the second preset correlation threshold, it is determined that the start-about sign appears. The second preset correlation threshold can be set according to implementation needs. In this embodiment, the second preset correlation threshold is 0.9, but is not limited thereto.

[0200] Wherein, when the first detection module 4 corrects the judgment result of whether the start mark of the corresponding behavior type appears according to the appearance of the about-to-start mark of each starting point confidence time series array, it executes:

[0201] If it is determined based on the first correlation that a start mark of a behavior type appears, then it is determined whether a start mark of the behavior type has appeared within a first preset time period (which can be set according to actual needs) before the current moment. If so, it is finally determined that the start mark of the behavior type appears. If not, it is finally determined that the start mark of the behavior type does not appear.

[0202] In some embodiments, the online behavior recognition device further includes:

[0203] The second array generation module is used to generate M endpoint confidence time series arrays of length N, where the initial value of each data in the endpoint confidence time series array is 0;

[0204] The second detection module 5 is used to perform the following when detecting whether the end mark of the corresponding behavior type appears based on the correlation between the endpoint confidence time series array of various behavior types and the standard endpoint confidence time series array of the corresponding behavior type:

[0205] According to the behavior type of the behavior of each frame of the video image, the corresponding endpoint confidence is inserted into the end of the corresponding endpoint confidence timing array and the first data of the endpoint confidence timing array is deleted;

[0206] The third correlation between each endpoint confidence time series array and the standard endpoint confidence time series array of the corresponding behavior type is calculated according to the following formula:

[0207] ;

[0208] in, is the third correlation between the j-th endpoint confidence time series array and the standard endpoint confidence time series array of the corresponding behavior type, is the value of the i-th data in the j-th endpoint confidence time series array, is the value of the i-th data of the standard endpoint confidence time series array of the behavior type corresponding to the j-th endpoint confidence time series array;

[0209] The third correlation of each endpoint confidence time series array is compared with the first preset correlation threshold to determine whether an end mark of a corresponding behavior type appears.

[0210] The update process of the endpoint confidence time series array is similar to that of the starting point confidence time series array. For details, please refer to Figure 4 The process can be completed by changing the starting point confidence level to the end point confidence level.

[0211] It should be noted that when the M behavior types identified in a certain frame of video image are different from the M behavior types identified in the previous frame of video image, the behavior type that no longer appears (i.e., the behavior type that belongs to one of the M behavior types in the previous frame of video image but does not belong to one of the M behavior types in the current frame of video image) is replaced by the newly appeared behavior type (i.e., the behavior type that does not belong to one of the M behavior types in the previous frame of video image), the endpoint confidence timing array of the replaced behavior type is initialized (its data are all 0), and then the endpoint confidence of the newly appeared behavior type is inserted at the end of the endpoint confidence timing array, and the first number of the endpoint confidence timing array is deleted. For details, please refer to Figure 5 The updating process of the starting point confidence time series array is completed, and the starting point confidence is replaced by the end point confidence.

[0212] Each behavior type has a standard endpoint confidence timing array of length N, and the standard endpoint confidence timing array can be measured in advance. The closer the third correlation is to 1, the greater the probability that the termination action of the corresponding behavior type appears in the video image. When the third correlation is greater than the first preset correlation threshold, it can be considered that the end mark of the corresponding behavior type appears. Therefore, when the second detection module 5 compares the third correlation of each endpoint confidence timing array with the first preset correlation threshold to determine whether the end mark of the corresponding behavior type appears, it executes:

[0213] If the third correlation between the endpoint confidence timing array and the standard endpoint confidence timing array of the corresponding behavior type is greater than the first preset correlation threshold, it is determined that the end mark of the corresponding behavior type appears.

[0214] In some preferred embodiments, the second detection module 5 is used to detect whether the end mark of the corresponding behavior type appears according to the correlation between the endpoint confidence time series array of various behavior types and the standard endpoint confidence time series array of the corresponding behavior type, and further execute:

[0215] The following formula is used to calculate the confidence level of each endpoint time series array: The first part of the data and the standard endpoint confidence time series array of the corresponding behavior type The fourth correlation of data:

[0216] ;

[0217] in, is the fourth correlation between the j-th endpoint confidence time series array and the standard endpoint confidence time series array of the corresponding behavior type, is a preset positive integer less than N, is the first value of the jth endpoint confidence time series array The value of the data;

[0218] Comparing the fourth correlation of each endpoint confidence time series array with the second preset correlation threshold to determine whether a sign indicating that the corresponding behavior type is about to end appears;

[0219] According to the appearance of the imminent end mark of each endpoint confidence timing array, the judgment result of whether the end mark of the corresponding behavior type appears is corrected.

[0220] Therefore, compared with the method of only relying on the third correlation to determine whether the end mark appears, the judgment result of considering both the fourth correlation and the third correlation to determine whether the end mark appears is more accurate, which can further improve the recognition accuracy of the online behavior recognition device.

[0221] The second detection module 5 compares the fourth correlation of each endpoint confidence time series array with the second preset correlation threshold to determine whether the corresponding behavior type is about to end. The first part of the data and the standard endpoint confidence time series array of the corresponding behavior type If the fourth correlation of the data is greater than the second preset correlation threshold, it is determined that the end mark appears.

[0222] Wherein, when the second detection module 5 corrects the judgment result of whether the end mark of the corresponding behavior type appears according to the appearance of the end mark of each endpoint confidence time series array, it executes:

[0223] If it is determined based on the third correlation that an end mark of a behavior type appears, then it is determined whether an end mark of the behavior type has appeared within a first preset time period (which can be set according to actual needs) before the current moment. If so, then it is finally determined that an end mark of the behavior type appears. If not, then it is finally determined that an end mark of the behavior type does not appear.

[0224] In this embodiment, the recognition module 6 is used to perform the following when outputting the recognition result of the behavior type according to the detection results of the start mark and the end mark:

[0225] When an end mark of a behavior type is detected, if a start mark of the corresponding behavior type has appeared within a second preset time period (which can be set according to actual needs) before the current moment, a recognition result indicating that a behavior of the corresponding behavior type has been detected is output.

[0226] That is, only when the end mark of a certain behavior type appears and the start mark of the behavior type has appeared within the previous second preset time period, is the behavior of this behavior type considered to be detected; the behavior type is identified through the appearance of the start mark and the end mark. When multiple behaviors are executed crosswise or in parallel, the identification result can be output correctly and the interference of invalid behaviors (i.e., behaviors that only have an end mark but no start mark within the previous second preset time period) can be avoided.

[0227] As can be seen from the above, the online behavior recognition device obtains the human skeleton information of the latest N frames of video images; N is a preset positive integer; the human skeleton information of each frame of video images is input into a pre-trained classification model to identify the M behavior types with the highest probability of the behavior of each frame of video images; M is a preset positive integer; based on the human skeleton information of each frame of video images, according to the pre-trained start and end point confidence regression model, the starting point confidence of the starting action and the end point confidence of the ending action of each frame of video images belonging to various behavior types are detected; according to the correlation between the starting point confidence timing array of various behavior types and the standard starting point confidence timing array of the corresponding behavior type, whether the start mark of the corresponding behavior type appears; according to the correlation between the end point confidence timing array of various behavior types and the standard end point confidence timing array of the corresponding behavior type, whether the end mark of the corresponding behavior type appears; according to the detection results of the start mark and the end mark, the recognition result of the behavior type is output; thereby having good flexibility and high recognition accuracy.

[0228] Please refer to Figure 3 , Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, the present application provides an electronic device, including: a processor 301 and a memory 302, the processor 301 and the memory 302 are interconnected and communicate with each other through a communication bus 303 and / or other forms of connection mechanisms (not shown), the memory 302 stores a computer program executable by the processor 301, when the electronic device is running, the processor 301 executes the computer program to execute the online behavior recognition method in any optional implementation of the above embodiment, to achieve the following functions: obtain the latest N frames of human skeleton information of video images; N is a preset positive integer; input the human skeleton information of each frame of video images into a pre-trained classification model, and identify each frame of video images The M behavior types with the largest probability of the behavior of the image are detected; M is a preset positive integer; based on the human skeleton information of each frame of video image, according to the pre-trained start and end point confidence regression model, the starting point confidence of the starting action and the end point confidence of the ending action of each frame of video image are detected to belong to various behavior types; according to the correlation between the starting point confidence timing array of various behavior types and the standard starting point confidence timing array of the corresponding behavior type, whether the start mark of the corresponding behavior type appears; according to the correlation between the end point confidence timing array of various behavior types and the standard end point confidence timing array of the corresponding behavior type, whether the end mark of the corresponding behavior type appears; according to the detection results of the start mark and the end mark, the recognition result of the behavior type is output.

[0229] The embodiment of the present application provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, the online behavior recognition method in any optional implementation method of the above embodiment is executed to achieve the following functions: obtain the human skeleton information of the latest N frames of video images; N is a preset positive integer; input the human skeleton information of each frame of video image into a pre-trained classification model respectively, and identify the M behavior types with the highest probability of the behavior of each frame of video image; M is a preset positive integer; based on the human skeleton information of each frame of video image, according to a pre-trained start and end point confidence regression model, detect the starting point confidence of the starting action and the end point confidence of the ending action of each frame of video image belonging to various behavior types; detect whether the start mark of the corresponding behavior type appears according to the correlation between the starting point confidence time series array of various behavior types and the standard starting point confidence time series array of the corresponding behavior type; detect whether the end mark of the corresponding behavior type appears according to the correlation between the end point confidence time series array of various behavior types and the standard end point confidence time series array of the corresponding behavior type; output the recognition result of the behavior type according to the detection results of the start mark and the end mark. Among them, the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable red-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0230] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0231] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0232] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0233] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0234] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An online behavior recognition method for online recognition of the behavior type of human behavior, It is characterized in that Includes steps: A1. Obtain the human skeleton information of the latest N frames of video images; N is a preset positive integer; A2. Input the human skeleton information of each frame of the video image into a pre-trained classification model, and identify the M behavior types with the highest probability of each frame of the video image; M is a preset positive integer; A3. Based on the human skeleton information of each frame of the video image, according to the pre-trained start and end point confidence regression model, the starting point confidence of the starting action and the end point confidence of the ending action of each frame of the video image belonging to the various types of behavior are detected; A4. According to the correlation between the starting point confidence timing array of various behavior types and the standard starting point confidence timing array of the corresponding behavior type, detect whether the start mark of the corresponding behavior type appears; A5. According to the correlation between the endpoint confidence timing array of the various behavior types and the standard endpoint confidence timing array of the corresponding behavior type, detect whether the end mark of the corresponding behavior type appears; A6. Outputting the recognition result of the behavior type according to the detection result of the start mark and the end mark; Before step A1, the method further includes the following steps: Generate M starting point confidence time series arrays of length N, where the initial value of each data in the starting point confidence time series array is 0; Step A4 includes: A401. According to the behavior type of the behavior of the video image in each frame, insert the corresponding starting point confidence into the end of the corresponding starting point confidence time series array, and delete the first data of the starting point confidence time series array; A402. Calculate the first correlation between each of the starting point confidence time series arrays and the standard starting point confidence time series array of the corresponding behavior type according to the following formula: ; in, is the first correlation between the j-th starting point confidence time series array and the standard starting point confidence time series array of the corresponding behavior type, is the value of the i-th data of the j-th starting point confidence time series array, is the value of the i-th data of the standard starting point confidence time series array of the behavior type corresponding to the j-th starting point confidence time series array; A403. Compare the first correlation of each of the starting point confidence time series arrays with a first preset correlation threshold to determine whether a start mark of a corresponding behavior type appears; After step A401 and before step A402, the method further includes the following steps: A404. Calculate the back of each starting point confidence time series array according to the following formula The first part of the confidence time series array of the standard starting point of the corresponding behavior type The second correlation of data: ; in, is the second correlation between the j-th starting point confidence time series array and the standard starting point confidence time series array of the corresponding behavior type, is a preset positive integer less than N, is the jth starting point confidence time series array The value of the data; A405. Compare the second correlation of each of the starting point confidence time series arrays with a second preset correlation threshold to determine whether a sign indicating that a corresponding behavior type is about to start appears; Step A403 and the following steps are also included: A406. Modify the judgment result of whether the start mark of the corresponding behavior type appears according to the appearance of the about-to-start mark in each of the starting point confidence timing arrays.

2. The online behavior recognition method according to claim 1, It is characterized in that Step A1 includes: Get the latest N frames of video images; Human skeleton information is extracted from each frame of the video image.

3. The online behavior recognition method according to claim 1, It is characterized in that Before step A1, the method further includes the following steps: Generate M endpoint confidence time series arrays of length N, where the initial value of each data in the endpoint confidence time series array is 0; Step A5 includes: A501. According to the behavior type of the behavior of the video image in each frame, the corresponding endpoint confidence is inserted into the end of the corresponding endpoint confidence timing array, and the first data of the endpoint confidence timing array is deleted; A502. Calculate the third correlation between each of the endpoint confidence time series arrays and the standard endpoint confidence time series array of the corresponding behavior type according to the following formula: ; in, is the third correlation between the j-th endpoint confidence time series array and the standard endpoint confidence time series array of the corresponding behavior type, is the value of the ith data in the jth endpoint confidence time series array, is the value of the i-th data of the standard endpoint confidence time series array of the behavior type corresponding to the j-th endpoint confidence time series array; A503. Compare the third correlation of each of the endpoint confidence time series arrays with the first preset correlation threshold to determine whether an end mark of a corresponding behavior type appears.

4. The online behavior recognition method according to claim 3, It is characterized in that After step A501 and before step A502, the method further includes the following steps: A504. Calculate the back of each endpoint confidence time series array according to the following formula: The first part of the data and the standard endpoint confidence time series array of the corresponding behavior type The fourth correlation of data: ; in, is the fourth correlation between the j-th endpoint confidence time series array and the standard endpoint confidence time series array of the corresponding behavior type, is a preset positive integer less than N, is the jth endpoint confidence time series array The value of the data; A505. Compare the fourth correlation of each of the endpoint confidence time series arrays with the second preset correlation threshold to determine whether a sign indicating that the corresponding behavior type is about to end appears; Step A503 and the following steps are also included: A506. Modify the judgment result of whether the end mark of the corresponding behavior type appears according to the appearance of the end mark of each end point confidence timing array.

5. The online behavior recognition method according to claim 1, It is characterized in that Step A6 includes: When an end mark of a behavior type is detected, if a start mark of the corresponding behavior type has appeared within a second preset time period before the current moment, a recognition result indicating that a behavior of the corresponding behavior type is detected is output.

6. An online behavior recognition device for online recognition of the behavior type of human behavior, It is characterized in that include: The first acquisition module is used to acquire the human skeleton information of the latest N frames of video images; N is a preset positive integer; A classification module, used to input the human skeleton information of each frame of the video image into a pre-trained classification model, and identify M behavior types with the highest probability of the behavior of each frame of the video image; M is a preset positive integer; A confidence recognition module is used to detect the starting point confidence of the starting action and the ending point confidence of the ending action of each frame of the video image, based on the human skeleton information of each frame of the video image and a pre-trained starting and ending point confidence regression model. A first detection module, configured to detect whether a start mark of a corresponding behavior type appears according to the correlation between the starting point confidence time series arrays of various behavior types and the standard starting point confidence time series arrays of the corresponding behavior type; A second detection module, configured to detect whether an end mark of a corresponding behavior type appears according to the correlation between the endpoint confidence time series arrays of the various behavior types and the standard endpoint confidence time series arrays of the corresponding behavior types; An identification module, configured to output an identification result of the behavior type according to the detection results of the start mark and the end mark; Also includes: A first array generation module is used to generate M starting point confidence time series arrays of length N, wherein the initial value of each data of the starting point confidence time series array is 0; When the first detection module detects whether a start mark of a corresponding behavior type appears based on the correlation between the starting point confidence time series arrays of various behavior types and the standard starting point confidence time series arrays of the corresponding behavior type, the first detection module executes: A401. According to the behavior type of the behavior of the video image in each frame, insert the corresponding starting point confidence into the end of the corresponding starting point confidence time series array, and delete the first data of the starting point confidence time series array; A402. Calculate the first correlation between each of the starting point confidence time series arrays and the standard starting point confidence time series array of the corresponding behavior type according to the following formula: ; in, is the first correlation between the j-th starting point confidence time series array and the standard starting point confidence time series array of the corresponding behavior type, is the value of the i-th data of the j-th starting point confidence time series array, is the value of the i-th data of the standard starting point confidence time series array of the behavior type corresponding to the j-th starting point confidence time series array; A403. Compare the first correlation of each of the starting point confidence time series arrays with a first preset correlation threshold to determine whether a start mark of a corresponding behavior type appears; After step A401 and before step A402, the method further includes the following steps: A404. Calculate the back of each starting point confidence time series array according to the following formula The first part of the confidence time series array of the standard starting point of the corresponding behavior type The second correlation of data: ; in, is the second correlation between the j-th starting point confidence time series array and the standard starting point confidence time series array of the corresponding behavior type, is a preset positive integer less than N, is the jth starting point confidence time series array The value of the data; A405. Compare the second correlation of each of the starting point confidence time series arrays with a second preset correlation threshold to determine whether a sign indicating that a corresponding behavior type is about to start appears; Step A403 and the following steps are also included: A406. Modify the judgment result of whether the start mark of the corresponding behavior type appears according to the appearance of the about-to-start mark in each of the starting point confidence timing arrays.

7. An electronic device, It is characterized in that It comprises a processor and a memory, wherein the memory stores a computer program executable by the processor, and when the processor executes the computer program, it runs the steps in the online behavior recognition method according to any one of claims 1 to 5.

8. A storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the online behavior recognition method according to any one of claims 1 to 5 are executed.

Citation Information

Patent Citations

  • Action recognition method based on neural network and action recognition device based on neural network

    CN107766839A

  • Behavior detection method, apparatus, and computer-readable storage medium

    WO2021208735A1