Behavior detection method and device based on video sequence, equipment and medium

Abstract

The embodiment of the invention discloses a behavior detection method and device based on a video sequence, equipment and a medium. The method comprises the following steps: acquiring multiple key video frames in a video sequence to be detected, and determining action classification feature information, key point feature information and key point coordinate information of a target human body in the key video frames; fusing the action classification feature information of the multiple key video frames to obtain fused action classification feature information; key point feature information in the key video frames is converted into time sequence feature information according to key point coordinate information in the key video frames, and the time sequence feature information of the multiple key video frames is fused to obtain fused time sequence feature information; and obtaining a behavior detection result of the target human body according to the fused action classification feature information and the fused time sequence feature information. According to the scheme, behavior detection is carried out based on the action classification features and the time sequence features, the problem of behavior ambiguity caused by lack of time sequence information is solved, and the behavior detection precision is improved.

Description

Technical Field

[0001] The present invention relates to the field of video detection technology, and in particular to a behavior detection method, device, equipment and medium based on video sequences. Background Art

[0002] With the rapid development of social economy, the construction of smart cities is also in full swing. Among them, the monitoring of pedestrian behavior is one of the important links in the construction of smart cities. Usually, a large number of surveillance cameras are installed in public places to monitor the behavior of pedestrians. However, as the number of surveillance cameras increases, the number of videos that staff need to monitor also increases, making the efficiency and real-time performance of monitoring poor. Therefore, using deep networks to automatically identify abnormal target behavior has become the preferred solution.

[0003] In the related scheme, target detection and recognition are performed on video frame images to obtain the image features and key point features of the target object, and then the image features and key point features are connected to multiple frames respectively, and the two types of information are fused in a cascade manner, and finally action recognition is achieved through a classification network.

[0004] However, in the above scheme, both the image features and key point features are spatial dimension information, and the connection between the two lacks temporal feature information. However, many human behaviors have temporal features. In the prior art, when performing behavior recognition based on video sequences, human behaviors are determined only by feature fusion. This can easily cause behavior ambiguity due to the lack of temporal features, resulting in a lower behavior recognition rate and poor behavior detection accuracy. Summary of the invention

[0005] The present invention provides a behavior detection method, device, equipment and medium based on video sequences, which performs behavior detection based on the action classification features of key video frames and the timing features after the key point features are converted, and can effectively overcome the behavior ambiguity problem caused by the lack of timing information, thereby improving the accuracy of behavior detection.

[0006] According to one aspect of the present invention, a method for detecting behavior based on a video sequence is provided, the method comprising:

[0007] Acquire multiple key video frames in the video sequence to be detected, and determine the action classification feature information, key point feature information and key point coordinate information of the target human body in the key video frames;

[0008] Fusing the action classification feature information of the multiple key video frames to obtain fused action classification feature information;

[0009] Converting key point feature information in the key video frame into time sequence feature information according to key point coordinate information in the key video frame, and fusing the time sequence feature information of multiple key video frames to obtain fused time sequence feature information;

[0010] The behavior detection result of the target human body is obtained according to the fused action classification feature information and the fused temporal feature information.

[0011] According to another aspect of the present invention, there is provided a behavior detection device based on a video sequence, comprising:

[0012] A key frame information determination module is used to obtain multiple key video frames in the video sequence to be detected, and determine the action classification feature information, key point feature information and key point coordinate information of the target human body in the key video frames;

[0013] An action classification feature fusion module is used to fuse the action classification feature information of the multiple key video frames to obtain fused action classification feature information;

[0014] A temporal feature fusion module, used for converting key point feature information in the key video frame into temporal feature information according to key point coordinate information in the key video frame, and fusing the temporal feature information of the multiple key video frames to obtain fused temporal feature information;

[0015] The behavior detection result determination module is used to obtain the behavior detection result of the target human body according to the fused action classification feature information and the fused temporal feature information.

[0016] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the video sequence-based behavior detection method described in any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the video sequence-based behavior detection method described in any embodiment of the present invention when executed.

[0021] The technical solution of the embodiment of the present invention obtains multiple key video frames in the video sequence to be detected, and determines the action classification feature information, key point feature information and key point coordinate information of the target human body in the key video frames; fuses the action classification feature information of the multiple key video frames to obtain fused action classification feature information; converts the key point feature information in the key video frames into timing feature information according to the key point coordinate information in the key video frames, and fuses the timing feature information of the multiple key video frames to obtain fused timing feature information; obtains the behavior detection result of the target human body according to the fused action classification feature information and the fused timing feature information. This technical solution performs behavior detection based on the action classification features of the key video frames and the timing features after the key point features are converted, which can effectively overcome the behavior ambiguity problem caused by the lack of timing information and improve the accuracy of behavior detection.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a flow chart of a behavior detection method based on a video sequence provided according to Embodiment 1 of the present invention;

[0025] Figure 2 is a schematic diagram of a conversion convolution kernel provided according to Embodiment 1 of the present invention;

[0026] Figure 3 is a schematic diagram of multi-dimensional and multi-scale feature fusion provided according to Embodiment 1 of the present invention;

[0027] Figure 4 is a flow chart of a behavior detection method based on video sequences provided according to Embodiment 2 of the present invention;

[0028] Figure 5 is a flow chart of a behavior detection method based on a video sequence provided according to Embodiment 2 of the present invention;

[0029] Figure 6 is a structural diagram of a behavior detection device based on a video sequence provided according to Embodiment 3 of the present invention;

[0030] Figure 7 The present invention is a schematic diagram of the structure of an electronic device for implementing a behavior detection method based on a video sequence according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first", "second", "target", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] Embodiment 1

[0034] Figure 1 The flowchart of a behavior detection method based on a video sequence provided in the first embodiment of the present invention is applicable to the situation of accurately detecting behaviors in a video sequence. For example, the behaviors may be running, falling, standing up, sitting down, and other behaviors with time sequence characteristics. The method may be executed by a behavior detection device based on a video sequence. The behavior detection device based on a video sequence may be implemented in the form of hardware and / or software. The behavior detection device based on a video sequence may be configured in an electronic device with data processing capabilities. Figure 1 As shown, the method includes:

[0035] S110, obtaining multiple key video frames in the video sequence to be detected, and determining the action classification feature information, key point feature information and key point coordinate information of the target human body in the key video frames.

[0036] Among them, the video sequence to be detected may refer to a video sequence waiting to be detected. The key video frame may refer to a key frame in the video sequence to be detected. Exemplarily, a frame extraction method can be used to obtain multiple key video frames in the video sequence to be detected. For example, one frame is selected as a key video frame every 12 frames from the video sequence to be detected. The action classification feature information can be used to characterize the action classification features of the target human body in the key video frame. The key points can be used to calibrate the target human body. For example, a certain number (such as 15) of points can be selected from the head, torso and limbs as key points. The number of key points can be set according to actual needs. The key point feature information can be used to characterize the characteristics of the key points. The key point coordinate information may include the key point coordinate value and the key point confidence.

[0037] In this embodiment, a plurality of key video frames are first obtained from the video sequence to be detected by frame extraction, and then the key video frames are input into a pre-trained target recognition model, so that the action classification feature information, key point feature information and key point coordinate information of the target human body in the key video frames can be obtained. Among them, the selection of key points can be pre-set during the training of the target recognition model. It should be noted that the target recognition model is not specifically limited in this embodiment and can be set according to actual application requirements. Exemplarily, the target recognition model can adopt the yolov5_landmarks model, and the action classification results and key point feature information of each frame can be obtained by decoupling the detection head output by the yolov5_landmarks model.

[0038] S120, fusing the action classification feature information of multiple key video frames to obtain fused action classification feature information.

[0039] In this embodiment, after determining the action classification feature information of the target human body in the key video frame, the action classification feature information can be input into a pre-trained action classification feature fusion model, and the action classification feature information of multiple key video frames can be fused, and the fused action classification feature information can be obtained according to the output result of the action classification feature fusion model. Among them, the fused action classification feature information can refer to the fused information obtained by fusing the action classification feature information of multiple key video frames. It should be noted that in this embodiment, the action classification feature fusion model is not specifically limited, as long as the action classification feature fusion function can be realized.

[0040] S130, converting key point feature information in the key video frame into temporal feature information according to key point coordinate information in the key video frame, and fusing the temporal feature information of multiple key video frames to obtain fused temporal feature information.

[0041] It should be noted that if spatial features are used alone for behavior analysis based on video sequences, it will cause ambiguity in behavior. For example, the multi-frame fusion features of standing up (from sitting to standing) and sitting down (from standing to sitting) are the same. Therefore, the two actions can only be distinguished by judging the time sequence of the actions. The key video frame is input into the pre-trained target recognition model, and the key point feature information output is a feature matrix of H×W×n. Among them, H and W represent the height and width of the key video frame (determined by the target recognition model), and n represents the number of key points. Assume n=15, because 15 key points of the human body are calibrated during data calibration, each layer of features represents the information of a key point. If the 15 key points can be connected, a posture can be represented. Connecting multiple postures in series in the order of the X axis represents an action. At this time, it has the characteristics of sequence, and the feature matrix is ​​converted into a time sequence feature. Among them, the X axis is the direction of W.

[0042] In this embodiment, after determining the key point feature information and key point coordinate information of the target human body in the key video frame, the key point feature information in the key video frame can be converted into time series feature information according to the key point coordinate information in the key video frame. Optionally, the key point feature information in the key video frame is converted into time series feature information according to the key point coordinate information in the key video frame, including: constructing a conversion convolution kernel according to the key point coordinate information and the key point feature information in the key video frame; performing dimensionality reduction processing on the key point feature information according to the conversion convolution kernel to obtain a dimensionality reduction matrix of the key point feature information; performing an X-direction feature extraction operation on the dimensionality reduction matrix according to a preset feature extraction matrix to obtain an X-direction feature of the key video frame as the time series feature information.

[0043] In this embodiment, a conversion convolution kernel is first constructed based on the key point coordinate information and key point feature information in the key video frame. Optionally, the conversion convolution kernel is constructed based on the key point coordinate information and key point feature information in the key video frame, including: determining the size of the conversion convolution kernel based on the key point feature information; determining the convolution kernel position corresponding to the key point in the conversion convolution kernel based on the key point coordinate value in the key point coordinate information; determining the parameter value of the convolution kernel position based on the key point confidence in the key point coordinate information; traversing all key points in the key video frame, and constructing the conversion convolution kernel based on the obtained parameter values ​​of each convolution kernel position.

[0044] When constructing the conversion convolution kernel, first determine the size of the conversion convolution kernel based on the key point feature information. Exemplarily, assuming that the key point feature information is a feature matrix of H×W×n, the size of the conversion convolution kernel can be determined as H×W, that is, the height of the conversion convolution kernel is H and the width is W. Assuming that the X-axis is the direction where W is located and the Y-axis is the direction where H is located, it can be determined that the width of each grid on the X-axis in the conversion convolution kernel is weight / W, and the height of each grid on the Y-axis is height / H. Among them, weight*height represents the image resolution of the key video frame.

[0045] After determining the size of the conversion convolution kernel, the convolution kernel position corresponding to the key point in the conversion convolution kernel can be determined according to the key point coordinate value in the key point coordinate information. Specifically, based on the width and height of each grid in the conversion convolution kernel, the grid corresponding to the key point is marked in the conversion convolution kernel as its convolution kernel position according to the key point coordinate value in the key point coordinate information. Furthermore, the grids around the grid corresponding to the key point (in eight directions) can also be marked, and all marked grids are used as the convolution kernel position corresponding to the key point.

[0046] After determining the convolution kernel position corresponding to the key point, the parameter value of the convolution kernel position can be determined according to the key point confidence in the key point coordinate information. Exemplarily, the parameter value of the convolution kernel position can be initialized to the key point confidence. Among them, the size of the key point confidence can be determined by the target recognition model. Then all the key points in the key video frame are traversed, and the conversion convolution kernel is constructed according to the obtained parameter values ​​of each convolution kernel position. Exemplarily, assuming that there are 15 key points, the convolution kernel positions corresponding to the 15 key points are determined respectively, and the parameter values ​​of the convolution kernel positions are initialized to the confidence of the key points. If there is overlap, they are added. After back propagation adjustment, the non-convolution kernel positions are filled with 0, and they are restricted to back propagation optimization.

[0047] Figure 2 FIG. 1 is a schematic diagram of a conversion convolution kernel provided in Embodiment 1 of the present invention. Figure 2 As shown in the figure, H×W=10×10, weight*height=500*500, so the width and height of each grid in the conversion convolution kernel are both 50. Taking one key point as an example, assuming that the coordinate value corresponding to the key point A is (275,380) and the confidence is 0.91, then Figure 2 The convolution kernel position and parameter value of the key point A are shown in Figure 1. The shaded part is the grid where the coordinate value of the key point A is located. Here, the grid where the coordinate value of the key point A is located and the 8 grids around it are taken as the convolution kernel position corresponding to the key point A.

[0048] After the conversion convolution kernel is constructed, the key point feature information can be reduced in dimension according to the conversion convolution kernel to obtain a reduced dimension matrix of the key point feature information. Exemplarily, the key point feature information of size H×W×n can be convolved with a conversion convolution kernel of size H×W and compressed into a reduced dimension matrix of size H×W to achieve the reduced dimension processing of the key point feature information. Since time is a one-dimensional feature and the spatial feature of the key point is a two-dimensional feature, further dimensionality reduction processing is required to obtain the time feature.

[0049] Specifically, after obtaining the dimension reduction matrix of the key point feature information, the dimension reduction matrix can be subjected to an X-direction feature extraction operation according to a preset feature extraction matrix to obtain the X-direction features of the key video frame as the temporal feature information. The preset feature extraction matrix can be a sobel operator, for example, which can be expressed as Specifically, the Sobel operator is a discrete differential operator that can be used to calculate the approximate gradient of the grayscale of an image. The operator combines Gaussian blur and differential derivation. By taking derivatives in the horizontal and vertical directions, the gradient images of the image in the X and Y directions can be obtained. Exemplarily, by performing matrix multiplication of the Sobel operator and the dimensionality reduction matrix corresponding to the current key video frame, the X-direction feature of the current key video frame can be obtained. Then, the X-direction features of multiple key video frames are connected together to obtain a feature matrix containing a one-dimensional time feature as the temporal feature information.

[0050] Through such a setting, this scheme uses the spatiotemporal feature conversion algorithm based on the Sobel operator to convert the key point feature information into time series feature information, which can not only solve the problem of low effective information content of key point coordinates and mismatch with spatial feature dimensions, but also solve the behavior ambiguity problem caused by the lack of time series information.

[0051] After converting the key point feature information in the key video frame into the timing feature information, the timing feature information of multiple key video frames can be fused to obtain fused timing feature information. The fused timing feature information may refer to fused information obtained by fusing the timing feature information of multiple key video frames. Exemplarily, the timing feature information of multiple key video frames can be input into a pre-trained timing feature fusion model, and the timing feature information of multiple key video frames can be fused, and the fused timing feature information can be obtained according to the output result of the timing feature fusion model. It should be noted that this embodiment does not specifically limit the timing feature fusion model, as long as the timing feature fusion function can be realized.

[0052] In this scheme, action classification feature information is used as spatial features with a small number of parameters, and key point information is used to assist the weighted fusion algorithm to fuse the action classification feature information of multiple frames, which can avoid the influence of erroneous action classification features on behavior recognition.

[0053] S140, obtaining a behavior detection result of the target human body according to the fused action classification feature information and the fused temporal feature information.

[0054] It should be noted that the action classification feature information can be used as the spatial input feature of behavior recognition after multi-frame fusion, and the time series feature information converted from the key point feature information can be used as the time input feature. However, the information contained in the two is different, one contains information in two dimensions, and the other only contains information in the X dimension. The fusion method of the features determines the utilization rate of the information, so it is crucial to realize the fusion of the two feature information.

[0055] In this embodiment, after obtaining the fused action classification feature information and the fused timing feature information, the behavior detection result of the target human body can be determined according to the fused action classification feature information and the fused timing feature information. Optionally, obtaining the behavior detection result of the target human body according to the fused action classification feature information and the fused timing feature information includes: performing multi-dimensional and multi-scale feature fusion on the fused action classification feature information and the fused timing feature information to obtain comprehensive fusion feature information; obtaining the behavior detection result of the target human body according to the comprehensive fusion feature information.

[0056] It should be noted that the general connection operation of two features is to perform matrix splicing in the depth dimension, but due to the particularity of spatiotemporal features, they can also be connected in the H dimension, so that each time point (feature map) of the temporal feature will correspond to a spatial feature, thereby forming a one-to-one spatiotemporal correspondence. Among them, the H dimension refers to the direction along the X-axis. In this scheme, a multi-dimensional and multi-scale feature fusion method is used to fuse spatiotemporal features.

[0057] Specifically, when determining the behavior detection result of the target human body, it is first necessary to perform multi-dimensional and multi-scale feature fusion on the fused action classification feature information and the fused temporal feature information to obtain comprehensive fusion feature information. Optionally, the fused action classification feature information and the fused temporal feature information are subjected to multi-dimensional and multi-scale feature fusion to obtain comprehensive fusion feature information, including: respectively connecting the fused action classification feature information and the fused temporal feature information in at least two dimensions to obtain at least two intermediate fusion feature information; respectively performing at least two convolution operations on the at least two intermediate fusion feature information to obtain spatiotemporal fusion feature information; wherein the convolution kernel sizes of at least two convolution operations are different; connecting and flattening the spatiotemporal fusion feature information of the same scale in the spatiotemporal fusion feature information corresponding to the at least two intermediate fusion feature information to obtain multiple flattened spatiotemporal fusion feature information; connecting multiple flattened spatiotemporal fusion feature information to obtain comprehensive fusion feature information.

[0058] Figure 3A schematic diagram of multi-dimensional and multi-scale feature fusion provided in Example 1 of the present invention.

[0059] The two cubes represent the fused action classification feature information and the fused temporal feature information respectively. Figure 3 As shown in the figure, the fused action classification feature information and the fused temporal feature information are first connected in the depth dimension and H dimension respectively, and the matrix M ( Figure 3 The right cuboid) and the matrix N( Figure 3 The left rectangular block in the middle). Among them, the matrix M and the matrix N are the intermediate fusion feature information. Then perform convolution operations on the matrix M and the matrix N with convolution kernels of 1*1, 5*5, 13*13 and H*W respectively to generate spatiotemporal feature fusion information of different scales. Among them, the convolution kernel sizes of the convolution operation are different, which can be set according to actual needs, but it is necessary to ensure that the convolution kernel of H*W is included. Then, the generated spatiotemporal fusion feature information of the same size is connected, and the connected spatiotemporal fusion feature information is flattened to obtain 4 flattened spatiotemporal fusion feature information, which is one-dimensional feature information. Finally, these 4 flattened spatiotemporal fusion feature information are connected to obtain the comprehensive fusion feature information, thereby completing the multi-dimensional and multi-scale spatiotemporal feature fusion. It should be noted that in order to ensure the same scale, it is necessary to upsample the spatiotemporal feature fusion information corresponding to the matrix N before connecting the generated spatiotemporal fusion feature information of the same size. Among them, Figure 3 In it, upsample means upsampling, concat means concatenation, and Flatten means flattening.

[0060] After obtaining the comprehensive fusion feature information, the behavior detection result of the target person can be obtained according to the comprehensive fusion feature information. Exemplarily, the comprehensive fusion feature information can be input into a pre-trained behavior classification model, and the behavior detection result of the target person can be determined according to the output result of the behavior classification model. Among them, this embodiment does not specifically limit the behavior classification model, as long as the behavior classification function can be realized.

[0061] Through such a setting, this scheme realizes the fusion of action classification features and time series features based on the multi-dimensional and multi-scale feature fusion method, which can effectively solve the problem of fusing different spatial feature information, thereby improving the accuracy of behavior recognition.

[0062] The technical solution of the embodiment of the present invention obtains multiple key video frames in the video sequence to be detected, and determines the action classification feature information, key point feature information and key point coordinate information of the target human body in the key video frames; fuses the action classification feature information of the multiple key video frames to obtain fused action classification feature information; converts the key point feature information in the key video frames into timing feature information according to the key point coordinate information in the key video frames, and fuses the timing feature information of the multiple key video frames to obtain fused timing feature information; obtains the behavior detection result of the target human body according to the fused action classification feature information and the fused timing feature information. This technical solution performs behavior detection based on the action classification features of the key video frames and the timing features after the key point features are converted, which can effectively overcome the behavior ambiguity problem caused by the lack of timing information and improve the accuracy of behavior detection.

[0063] Embodiment 2

[0064] Figure 4 A flowchart of a behavior detection method based on a video sequence is provided for the second embodiment of the present invention. This embodiment is optimized based on the above embodiment. The specific optimization is: before fusing the action classification feature information of multiple key video frames, it also includes: determining whether the key video frame is an error frame based on the key point coordinate information and / or image quality in the key video frame; if so, determining a replacement frame from the adjacent frame group of the key video frame to replace the key video frame.

[0065] like Figure 4 As shown, the method of this embodiment specifically includes the following steps:

[0066] S210, obtaining multiple key video frames in the video sequence to be detected, and determining the action classification feature information, key point feature information and key point coordinate information of the target human body in the key video frames.

[0067] S220: Determine whether the key video frame is an error frame according to key point coordinate information and / or image quality in the key video frame.

[0068] It should be noted that the image recognition process is easily disturbed by external factors, and it is easy to lose frames, blur frames or unstable model recognition. If the wrong key video frames are fused, it will inevitably interfere with the accuracy of behavior recognition. Therefore, in order to ensure the correctness of the key video frames, it is necessary to first determine whether the key video frames are wrong frames before fusing the action classification feature information of multiple key video frames, and replace the key video frames if the key video frames are determined to be wrong frames. Assuming that the video sequence to be detected shows a video of a pedestrian falling, each frame is in an ideal state, then the difference between the previous and next frame images will not be too large, and the movement trend of the target pedestrian is traceable, then the movement direction (key point deflection angle) and movement distance of the key points of the human body are also regular. Therefore, based on the key point coordinate information and / or image quality (such as picture blur information) in the key video frame, appropriate parameters can be selected to determine whether the key video frame is an error frame.

[0069] In this embodiment, optionally, determining whether a key video frame is an error frame based on key point coordinate information and / or image quality in the key video frame includes: determining multiple candidate video frames based on adjacent frame groups of the key video frame, and determining key point coordinate information in the candidate video frame; determining key point position differences and key point rotation angle differences of the candidate video frame based on the key point coordinate information in the candidate video frame and the key point coordinate information in the next video frame corresponding to the candidate video frame; determining key point average position differences and key point average rotation angle differences based on key point position differences and key point rotation angle differences of multiple candidate video frames; determining whether the key point coordinate information of the key video frame includes error coordinates based on the key point position differences and key point rotation angle differences of the key video frame, as well as the key point average position differences and key point average rotation angle differences; determining picture blur information based on RGB information in the key video frame, and determining whether the key video frame is a blurry frame based on the picture blur information; if the key video frame includes error coordinates, or is a blurry frame, the key video frame is an error frame.

[0070] Among them, the candidate video frame can refer to any video frame adjacent to the key video frame, and the number of the candidate video frames can be set according to actual needs. For example, if one frame is extracted from the video sequence to be detected as a key video frame every 12 frames, the 12 frames before and after the key video frame (a total of 24 frames) can be selected as candidate video frames. Since the camera position is fixed, the coordinates of the key points in the 25 video frames are located in the same world coordinates.

[0071] For example, the position difference S of the key points of the i-th frame and the i+1-th frame can be calculated according to the first formula: i , where the first formula is expressed as x i+1 andi+1 The horizontal and vertical coordinates of the key point of the i+1th frame are respectively represented. And the rotation angle difference cosα of the key point of the i-th frame and the i+1th frame relative to the X axis can be calculated according to the second formula i (α i ∈[0°,90°]), where the second formula is expressed as

[0072] The 12 candidate video frames before and after the key video frame are used for calculation, and a total of 22 key point position differences and 22 key point rotation angle differences can be obtained. In order to reduce the impact of bad frames in the candidate video frames, it is necessary to remove a maximum position difference S from the calculation results before and after the key video frame. imax , minimum position difference S imin , the maximum rotation angle difference cosα imax And the minimum rotation angle difference cosα imin , which leaves a total of 20 key point position differences and 20 key point rotation angle differences. Then average the remaining key point position differences and key point rotation angle differences to get the key point average position difference The difference between the average rotation angle of the key points If the key point position difference between the key video frame and the previous frame is in a closed interval And the key point rotation angle difference is in a closed interval It can be determined that the key point coordinate information of the key video frame does not include erroneous coordinates. Among them, p1<p2, and q1<q2, the specific values ​​of p1, p2, q1 and q2 can be set according to actual needs. For example, p1=0.8, p2=1.2, q1=0.8, q2=1.2.

[0073] Since the clarity of the image will directly affect the feature extraction and key point detection results, the image blurriness can be used to determine whether the key video frame is an error frame. For example, the image blurriness information m of the key video frame can be calculated by the third formula: i , if m i <1, then the key video frame can be judged as a blurry frame. The third formula can be expressed as Among them, f(x,y) represents the pixel value at the (x,y) position, x represents the horizontal coordinate, and y represents the vertical coordinate. Represents the grayscale average of all pixels in the image, N x ×N y Represents the total number of all pixels. The smaller m is, the more blurred the image is. If the key video frame includes erroneous coordinates or is a blurred frame, it can be determined that the key video frame is an erroneous frame.

[0074] S230: If yes, determine a replacement frame from a group of adjacent frames of the key video frame, and replace the key video frame.

[0075] In this embodiment, if it is determined that the key video frame is an error frame, it is necessary to determine a replacement frame from the adjacent frame group of the key video frame, and use the replacement frame to replace the key video frame. Optionally, determining the replacement frame from the adjacent frame group of the key video frame includes: determining a position difference parameter based on the key point position difference and the key point average position difference of the candidate video frame; determining a rotation angle difference parameter based on the key point rotation angle difference and the key point average rotation angle difference of the candidate video frame; determining whether the candidate video frame is a replacement frame based on the position difference parameter, the rotation angle difference parameter and the image blur information of the candidate video frame.

[0076] Specifically, first calculate the difference between the key point position difference of each candidate video frame and the average key point position difference As the position difference parameter, the difference between the key point rotation angle difference of each candidate video frame and the average rotation angle difference of the key points is calculated. As the rotation angle difference parameter. Then the fourth formula can be used to calculate the y of each candidate video frame i value, and select y from it i The candidate video frame with the smallest value is used as the replacement frame. The fourth formula is expressed as

[0077] In this scheme, the key point coordinate information and image blur information are used to calculate the key video frame image with a higher error probability, and the error-free previous and next frames are used to replace it, which can improve the accuracy of multi-frame feature fusion.

[0078] S240, fusing the action classification feature information of multiple key video frames to obtain fused action classification feature information.

[0079] S250, converting key point feature information in the key video frame into temporal feature information according to key point coordinate information in the key video frame, and fusing the temporal feature information of multiple key video frames to obtain fused temporal feature information.

[0080] S260, obtaining a behavior detection result of the target human body according to the fused action classification feature information and the fused temporal feature information.

[0081] Among them, the specific implementation method of S240-S260 can be found in the detailed description of S120-S140, which will not be repeated here.

[0082] Figure 5 The flowchart of a behavior detection method based on video sequence provided by the second embodiment of the present invention is as follows. Figure 5As shown, firstly, the video sequence to be detected is subjected to a frame extraction operation to obtain multiple key video frames, and the multiple key video frames are input into the target recognition model to obtain the action classification feature information, key point feature information and key point coordinate information of the target human body in the key video frames. Then, it is determined whether the key video frame is an error frame. If so, a replacement frame is determined from the adjacent frame group of the key video frame to replace the key video frame. When the number of key video frames reaches a preset number, the action classification feature information of the multiple key video frames is fused to obtain fused action classification feature information. At the same time, the key point feature information in the key video frame is converted into time sequence feature information. When the number of key video frames reaches a preset number, the time sequence feature information of the multiple key video frames is fused to obtain fused time sequence feature information. Then, the fused action classification feature information and the fused time sequence feature information are fused to obtain the behavior detection result of the target human body.

[0083] The technical solution of the embodiment of the present invention determines whether the key video frame is an error frame based on the key point coordinate information and / or image quality in the key video frame before fusing the action classification feature information of multiple key video frames; if so, determine a replacement frame from the adjacent frame group of the key video frame to replace the key video frame. This technical solution performs behavior detection based on the timing features after the action classification features of the key video frame and the key point features are converted, effectively overcoming the behavior ambiguity problem caused by the lack of timing information and improving the accuracy of behavior detection. In addition, it can also use the key point coordinate information and / or image quality to determine the key video frame image with a higher error probability, and use the error-free previous and next frames for replacement, further improving the accuracy of multi-frame feature fusion.

[0084] Embodiment 3

[0085] Figure 6 This is a schematic diagram of the structure of a video sequence-based behavior detection device provided in the third embodiment of the present invention. The device can execute the video sequence-based behavior detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. Figure 6 As shown, the device comprises:

[0086] The key frame information determination module 310 is used to obtain multiple key video frames in the video sequence to be detected, and determine the action classification feature information, key point feature information and key point coordinate information of the target human body in the key video frames;

[0087] An action classification feature fusion module 320 is used to fuse the action classification feature information of the multiple key video frames to obtain fused action classification feature information;

[0088] A temporal feature fusion module 330 is used to convert the key point feature information in the key video frame into temporal feature information according to the key point coordinate information in the key video frame, and fuse the temporal feature information of the multiple key video frames to obtain fused temporal feature information;

[0089] The behavior detection result determination module 340 is used to obtain the behavior detection result of the target human body according to the fused action classification feature information and the fused temporal feature information.

[0090] Optionally, the temporal feature fusion module 330 includes:

[0091] A conversion convolution kernel construction unit, used to construct a conversion convolution kernel according to the key point coordinate information and the key point feature information in the key video frame;

[0092] A dimension reduction matrix determination unit, used for performing dimension reduction processing on the key point feature information according to the conversion convolution kernel to obtain a dimension reduction matrix of the key point feature information;

[0093] The temporal feature information determining unit is used to perform an X-direction feature extraction operation on the dimension reduction matrix according to a preset feature extraction matrix to obtain the X-direction feature of the key video frame as the temporal feature information.

[0094] Optionally, the conversion convolution kernel construction unit is specifically used to:

[0095] Determine the size of the conversion convolution kernel according to the key point feature information;

[0096] Determine the convolution kernel position corresponding to the key point in the conversion convolution kernel according to the key point coordinate value in the key point coordinate information;

[0097] Determining the parameter value of the convolution kernel position according to the key point confidence in the key point coordinate information;

[0098] All key points in the key video frame are traversed, and a conversion convolution kernel is constructed according to the obtained parameter values ​​of each convolution kernel position.

[0099] Optionally, the behavior detection result determination module 340 includes:

[0100] A comprehensive fusion feature determination unit, used for performing multi-dimensional and multi-scale feature fusion on the fused action classification feature information and the fused temporal feature information to obtain comprehensive fusion feature information;

[0101] A behavior detection result determination unit is used to obtain the behavior detection result of the target human body according to the comprehensive fusion feature information.

[0102] Optionally, the comprehensive fusion feature determination unit is specifically used to:

[0103] Respectively connect the fused action classification feature information and the fused temporal feature information in at least two dimensions to obtain at least two intermediate fused feature information;

[0104] Performing at least two convolution operations on the at least two intermediate fusion feature information respectively to obtain spatiotemporal fusion feature information; wherein the convolution kernel sizes of the at least two convolution operations are different;

[0105] Connecting and flattening the spatiotemporal fusion feature information with the same scale in the spatiotemporal fusion feature information corresponding to the at least two intermediate fusion feature information to obtain a plurality of flattened spatiotemporal fusion feature information;

[0106] The plurality of smoothed spatio-planar fusion feature information are connected to obtain the comprehensive fusion feature information.

[0107] Optionally, the device further comprises:

[0108] An error frame judgment module is used to determine whether the key video frame is an error frame according to key point coordinate information and / or image quality in the key video frame before fusing the action classification feature information of the multiple key video frames;

[0109] The key frame replacement module is used to determine a replacement frame from a group of adjacent frames of the key video frame to replace the key video frame.

[0110] Optionally, the error frame judgment module is used to:

[0111] Determine a plurality of candidate video frames according to a group of adjacent frames of the key video frame, and determine key point coordinate information in the candidate video frames;

[0112] Determine the key point position difference and the key point rotation angle difference of the candidate video frame according to the key point coordinate information in the candidate video frame and the key point coordinate information in the next video frame corresponding to the candidate video frame;

[0113] Determine the average position difference of the key points and the average rotation angle difference of the key points according to the key point position differences and the key point rotation angle differences of the multiple candidate video frames;

[0114] Determine whether the key point coordinate information of the key video frame includes erroneous coordinates according to the key point position difference and the key point rotation angle difference of the key video frame, and the key point average position difference and the key point average rotation angle difference;

[0115] Determine image blur information according to the RGB information in the key video frame, and determine whether the key video frame is a blurry frame according to the image blur information;

[0116] If the key video frame includes erroneous coordinates or is a blurred frame, the key video frame is an erroneous frame;

[0117] Accordingly, the key frame replacement module is used to:

[0118] Determine a position difference parameter according to the key point position difference of the candidate video frame and the key point average position difference;

[0119] Determine a rotation angle difference parameter according to the key point rotation angle difference of the candidate video frame and the key point average rotation angle difference;

[0120] Determine whether the candidate video frame is a replacement frame according to the position difference parameter, the rotation angle difference parameter and the picture blur information of the candidate video frame.

[0121] The video sequence-based behavior detection device provided in the embodiment of the present invention can execute the video sequence-based behavior detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0122] Embodiment 4

[0123] Figure 7 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0124] like Figure 7As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0125] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0126] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as a behavior detection method based on a video sequence.

[0127] In some embodiments, the behavior detection method based on video sequences may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the behavior detection method based on video sequences described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the behavior detection method based on video sequences in any other appropriate manner (e.g., by means of firmware).

[0128] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0129] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0130] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0131] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0132] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0133] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0134] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0135] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A behavior detection method based on video sequences, characterized in that: The method comprises: Acquire multiple key video frames in the video sequence to be detected, and determine the action classification feature information, key point feature information and key point coordinate information of the target human body in the key video frames; Fusing the action classification feature information of the multiple key video frames to obtain fused action classification feature information; Converting key point feature information in the key video frame into time sequence feature information according to key point coordinate information in the key video frame, and fusing the time sequence feature information of multiple key video frames to obtain fused time sequence feature information; The behavior detection result of the target human body is obtained according to the fused action classification feature information and the fused temporal feature information.

2. The method according to claim 1, characterized in that Converting the key point feature information in the key video frame into time sequence feature information according to the key point coordinate information in the key video frame includes: Constructing a conversion convolution kernel according to the key point coordinate information and the key point feature information in the key video frame; Performing dimensionality reduction processing on the key point feature information according to the conversion convolution kernel to obtain a dimensionality reduction matrix of the key point feature information; An X-direction feature extraction operation is performed on the dimension reduction matrix according to a preset feature extraction matrix to obtain X-direction features of the key video frame as temporal feature information.

3. The method according to claim 2, characterized in that Constructing a conversion convolution kernel according to the key point coordinate information and the key point feature information in the key video frame, including: Determine the size of the conversion convolution kernel according to the key point feature information; Determine the convolution kernel position corresponding to the key point in the conversion convolution kernel according to the key point coordinate value in the key point coordinate information; Determining the parameter value of the convolution kernel position according to the key point confidence in the key point coordinate information; All key points in the key video frame are traversed, and a conversion convolution kernel is constructed according to the obtained parameter values ​​of each convolution kernel position.

4. The method according to claim 1, characterized in that: Obtaining a behavior detection result of the target human body according to the fused action classification feature information and the fused temporal feature information, including: Performing multi-dimensional and multi-scale feature fusion on the fused action classification feature information and the fused temporal feature information to obtain comprehensive fused feature information; The behavior detection result of the target human body is obtained according to the comprehensive fusion feature information.

5. The method according to claim 4, characterized in that The fused action classification feature information and the fused temporal feature information are subjected to multi-dimensional and multi-scale feature fusion to obtain comprehensive fused feature information, including: Respectively connect the fused action classification feature information and the fused temporal feature information in at least two dimensions to obtain at least two intermediate fused feature information; Performing at least two convolution operations on the at least two intermediate fusion feature information respectively to obtain spatiotemporal fusion feature information; wherein the convolution kernel sizes of the at least two convolution operations are different; Connecting and flattening the spatiotemporal fusion feature information with the same scale in the spatiotemporal fusion feature information corresponding to the at least two intermediate fusion feature information to obtain a plurality of flattened spatiotemporal fusion feature information; The plurality of smoothed spatio-planar fusion feature information are connected to obtain the comprehensive fusion feature information.

6. The method according to any one of claims 1 to 5, characterized in that: Before fusing the action classification feature information of the multiple key video frames, the method further includes: Determining whether the key video frame is an error frame according to key point coordinate information and / or image quality in the key video frame; If so, a replacement frame is determined from a group of adjacent frames of the key video frame, and the key video frame is replaced.

7. The method according to claim 6, characterized in that Determining whether the key video frame is an error frame according to key point coordinate information and / or image quality in the key video frame includes: Determine a plurality of candidate video frames according to a group of adjacent frames of the key video frame, and determine key point coordinate information in the candidate video frames; Determine the key point position difference and the key point rotation angle difference of the candidate video frame according to the key point coordinate information in the candidate video frame and the key point coordinate information in the next video frame corresponding to the candidate video frame; Determine the average position difference of the key points and the average rotation angle difference of the key points according to the key point position differences and the key point rotation angle differences of the multiple candidate video frames; Determine whether the key point coordinate information of the key video frame includes erroneous coordinates according to the key point position difference and the key point rotation angle difference of the key video frame, and the key point average position difference and the key point average rotation angle difference; Determine image blur information according to the RGB information in the key video frame, and determine whether the key video frame is a blurry frame according to the image blur information; If the key video frame includes erroneous coordinates or is a blurred frame, the key video frame is an erroneous frame; Accordingly, determining a replacement frame from a group of adjacent frames of the key video frame includes: Determine a position difference parameter according to the key point position difference of the candidate video frame and the key point average position difference; Determine a rotation angle difference parameter according to the key point rotation angle difference of the candidate video frame and the key point average rotation angle difference; Determine whether the candidate video frame is a replacement frame according to the position difference parameter, the rotation angle difference parameter and the picture blur information of the candidate video frame.

8. A behavior detection device based on video sequence, characterized in that: The device comprises: A key frame information determination module is used to obtain multiple key video frames in the video sequence to be detected, and determine the action classification feature information, key point feature information and key point coordinate information of the target human body in the key video frames; An action classification feature fusion module is used to fuse the action classification feature information of the multiple key video frames to obtain fused action classification feature information; A temporal feature fusion module, used for converting key point feature information in the key video frame into temporal feature information according to key point coordinate information in the key video frame, and fusing the temporal feature information of the multiple key video frames to obtain fused temporal feature information; The behavior detection result determination module is used to obtain the behavior detection result of the target human body according to the fused action classification feature information and the fused temporal feature information.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the video sequence-based behavior detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the video sequence-based behavior detection method according to any one of claims 1 to 7 when executed.

Images