Action recognition method and device, electronic equipment and storage medium

Abstract

The invention provides an action recognition method and device, electronic equipment and a storage medium, and the method comprises the steps: processing a first image in a monitoring video based on a pre-trained first model, and obtaining the image features of the first image and a human body region; determining spatial-temporal characteristics of the monitoring video based on a pre-trained second model, wherein the spatial-temporal characteristics are used for representing an association relationship between adjacent video frames in the monitoring video; determining fusion features according to the image features and the spatial-temporal features; and determining an action recognition result of the monitoring video according to the human body region and the fusion feature. The invention relates to the technical field of artificial intelligence, and can improve the accuracy of action recognition.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a motion recognition method, device, electronic device and storage medium. Background Art

[0002] At present, with the development of artificial intelligence technology, more and more organizations or individuals tend to use artificial intelligence models to assist production in order to improve productivity. Some organizations or individuals use edge computing devices to run artificial intelligence models, and use artificial intelligence models to process surveillance videos collected by cameras, so as to identify and analyze the movements of targets in surveillance videos. For example, in a production environment, artificial intelligence models can be used to analyze surveillance videos of factory areas or production lines to identify the degree of standardization of workers' movements during the production process, thereby improving the safety of the production process; for another example, in public areas, artificial intelligence models can be used to analyze the movements of targets in surveillance videos to identify conflicting movements of targets in public areas, thereby promptly identifying movements that endanger public safety.

[0003] Since the current scenarios where artificial intelligence models are used for action recognition usually involve dense targets and small individual targets, the target actions may not be accurately identified, resulting in low accuracy in identifying the actions of targets in surveillance videos. Summary of the invention

[0004] In view of the above, it is necessary to propose an action recognition method, device, electronic device and storage medium to solve the technical problem of low accuracy in action recognition on videos.

[0005] The present application provides an action recognition method, which is applied to an electronic device. The method includes: processing a first image in a surveillance video based on a pre-trained first model to obtain image features and a human body area of ​​the first image; determining spatiotemporal features of the surveillance video based on a pre-trained second model, wherein the spatiotemporal features are used to characterize the association relationship between adjacent video frames in the surveillance video; determining fusion features based on the image features and the spatiotemporal features; and determining an action recognition result of the surveillance video based on the human body area and the fusion features.

[0006] An embodiment of the present application also provides an action recognition device, which includes: a first recognition module, used to process a first image in a surveillance video based on a pre-trained first model to obtain image features and a human body area of ​​the first image; a second recognition module, used to determine the spatiotemporal features of the surveillance video based on the pre-trained second model, and the spatiotemporal features are used to characterize the association relationship between adjacent video frames in the surveillance video; a fusion module, used to determine fusion features based on the image features and the spatiotemporal features; and a third recognition module, used to determine the action recognition result of the surveillance video based on the human body area and the fusion features.

[0007] An embodiment of the present application also provides an electronic device, which includes: a memory storing at least one instruction; and a processor executing the instruction stored in the memory to implement the action recognition method.

[0008] An embodiment of the present application also provides a computer-readable storage medium, in which at least one instruction is stored. The at least one instruction is executed by a processor in an electronic device to implement the action recognition method.

[0009] It can be seen from the above technical solutions that when the embodiment of the present application performs action recognition on the surveillance video, the human body area and image features in the video frame are first determined, and the image features and spatiotemporal features are fused to obtain fused features. In this way, in the process of performing action recognition on the surveillance video, the fused features can be first determined based on the image features and spatiotemporal features of any frame in the surveillance video, and the fused features are used to characterize the image information in the surveillance video and the correlation between the contents in different video frames in the time dimension. Finally, the regional features related to the human body in the fused features are determined based on the human body area, and action recognition is performed based on the regional features, which can increase the probability of including human information in the target area, and thus can improve the accuracy of action recognition. And it can narrow the scope of action recognition, thereby improving the efficiency of action recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is an application scenario diagram of an action recognition method provided in one embodiment of the present application.

[0011] Figure 2 This is a flow chart of an action recognition method provided by an embodiment of the present application.

[0012] Figure 3 is a schematic diagram of a first image provided by an embodiment of the present application.

[0013] Figure 4 This is a flowchart of a method for training a first recognition model provided in one embodiment of the present application.

[0014] Figure 5 It is a flowchart of a method for determining fusion features provided by an embodiment of the present application.

[0015] Figure 6 It is a flowchart of a method for determining action recognition results provided by an embodiment of the present application.

[0016] Figure 7 This is a flowchart of a method for determining a loss value of an initial recognition model provided by an embodiment of the present application.

[0017] Figure 8 It is a flowchart of a method for determining weight parameters provided in one embodiment of the present application.

[0018] Fig. 9 It is a functional module diagram of a motion recognition device provided in one embodiment of the present application.

[0019] Fig.10 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to more clearly understand the purpose, features and advantages of the present application, the present application is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other. In the following description, many specific details are set forth to facilitate a full understanding of the present application, and the embodiments described are only a part of the embodiments of the present application, rather than all of the embodiments.

[0021] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by those skilled in the art to which the present application belongs. The terms used herein in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0023] The embodiment of the present application provides a motion recognition method, which can be applied to one or more electronic devices. The electronic device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a programmable gate array (Field-Programmable Gate Array, FPGA), a digital processor (Digital Signal Processor, DSP), an embedded device, etc.

[0024] An electronic device can be any electronic product that can interact with customers, such as personal computers, tablet computers, smart phones, personal digital assistants (PDAs), game consoles, Internet Protocol Television (IPTV), smart wearable devices, etc.

[0025] The electronic device may also include a network device and / or a client device, wherein the network device includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud consisting of a large number of hosts or network servers based on cloud computing.

[0026] The networks where electronic devices are located include but are not limited to the Internet, wide area network, metropolitan area network, local area network, virtual private network (VPN), etc.

[0027] like Figure 1 As shown, an action recognition method provided by the present application can be applied to an electronic device 100, and the electronic device 100 is communicatively connected to a camera device 200, and the electronic device 100 is used to receive a surveillance video sent by the camera device 200. Among them, the camera device 200 is used to shoot a target (for example, a pedestrian, a vehicle, etc.) within the imaging range to obtain a surveillance video including the target. A first recognition model for target recognition of a certain video frame in the surveillance video and a second recognition model for extracting the spatiotemporal features of the surveillance video are run in the electronic device 100. After the electronic device 100 receives the surveillance video input by the camera device 200, the first recognition model and the second recognition model are used to process multiple video frames in the surveillance video, respectively, to obtain the image features of a certain video frame in the surveillance video and the human body area therein, as well as the spatiotemporal features of multiple video frames, and the image features and the spatiotemporal features are fused to obtain the fused features; and the action recognition result of the target is determined according to the human body area and the fused features.

[0028] Exemplarily, when the surveillance video captured by the camera device 200 includes a person, the first recognition model in the electronic device 100 receives any video frame in the surveillance video (for example, the video frame with the timestamp in the center of the surveillance video), and identifies the human body region where the person in the video frame is located and the image features corresponding to the video frame. The second recognition model in the electronic device 100 also identifies the spatiotemporal features in multiple video frames, wherein the spatiotemporal features are used to characterize the association relationship between the targets in the multiple video frames in the time dimension and the space dimension. The electronic device 100 also performs a fusion operation on the image features and the spatiotemporal features to obtain a fusion feature. In this way, in the process of performing action recognition on the surveillance video, the fusion feature can be first determined according to the image features and spatiotemporal features of any frame in the surveillance video, and the fusion feature is used to characterize the image information in the surveillance video and the association relationship between the content in different video frames in the time dimension. Finally, the regional features related to the human body in the fusion feature are determined according to the human body region, and action recognition is performed according to the regional features, which can increase the probability that the target area includes human information, and thus can improve the accuracy of action recognition. And it can narrow the scope of action recognition, thereby improving the efficiency of action recognition.

[0029] like Figure 2 , is a flow chart of a method for motion recognition provided in an embodiment of the present application. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted. A method for motion recognition provided in an embodiment of the present application includes the following steps.

[0030] S20: Process a first image in the surveillance video based on a pre-trained first model to obtain image features and human body areas of the first image.

[0031] In one embodiment of the present application, when identifying human body movements in a surveillance video, in order to improve the accuracy of movement recognition, the human body region in the video frame of the surveillance video may be first identified. The first image may be any video frame in the surveillance video, and the first recognition model may be a recognition model with a target detection function. Exemplarily, the first recognition model may be the V3 version of the YOLO model, the V4 version of the YOLO model, the V8S version of the YOLO model, or a convolutional neural network model, etc., which is not limited in the present application.

[0032] In one embodiment of the present application, a video frame with a timestamp in the center of the surveillance video can be used as a representative frame of the surveillance video. The video frame with a timestamp in the center is located in the middle of the surveillance video in terms of time sequence, and can better represent the time midpoint of the video, so as to characterize the overall situation in the surveillance video. Since the surveillance video usually records continuous images within a period of time, the video frame with a timestamp in the center contains important information within the period of time, such as the appearance of characters, the occurrence of actions, etc. In addition, selecting a single video frame as a representative frame can reduce the complexity of subsequent image processing and feature extraction, and improve processing efficiency. By extracting and analyzing the features of the representative frame, rapid retrieval, classification and recognition of the surveillance video can be achieved. Using the representative frame can reduce the processing and analysis of the entire surveillance video, thereby simplifying the processing flow and improving the processing speed. And only storing the representative frame instead of the entire video can significantly reduce the storage space occupied and reduce the storage cost. By extracting the features of the representative frame and establishing an index, rapid retrieval and positioning of the surveillance video can be achieved, and the retrieval efficiency can be improved. The representative frame contains the key information of the video, which can be used for subsequent tasks such as action recognition, behavior analysis, and anomaly detection, providing strong support for intelligent monitoring.

[0033] Among them, the YOLO (You Only Look Once) model is an artificial intelligence model with target detection function. Exemplarily, the specific steps of using the YOLO model to identify the human body area in the first image are as follows: pre-set the parameters such as the size of the input data and the size of the output data of the first recognition model. For example, the size of the input data of the first recognition model can be 416×416 or 448×448, and the size of the output data can be 1×6. Image preprocessing, reading the surveillance video to be analyzed from the database, segmenting the surveillance video, and the image processing library (for example, OpenCV) reads the image to be detected. Scale the image to the input size required by the model, convert the image data into a tensor, and perform normalization. The first recognition model encodes the input first image to obtain an output tensor, and the size and size of the output tensor depend on the configuration of the model and the size of the input image. The decoder of the first recognition model extracts the detection result from the output tensor, specifically, extracts the predicted bounding box information of each grid from the output tensor, including the center point coordinates, width, height and confidence of the bounding box. According to the category probability information in the output tensor, determine the category to which each bounding box belongs. The extracted bounding boxes are processed with non-maximum suppression, and the bounding boxes are screened by comparing the confidence and overlap rate of the bounding boxes to eliminate overlapping bounding boxes and retain the best detection results. The final detection box position and category are determined based on the bounding box information and category probability. Figure 3The schematic diagram of the first image 300 is shown, wherein the category corresponding to the target in the detection frame 310 may be "tree", the category corresponding to the target in the detection frame 320 may be "flower", the category corresponding to the target in the detection frame 330 may be "animal", and the category corresponding to the target in the detection frame 340 may be "person". Therefore, the range covered by the detection frame 340 in the first image 300 is the human body area. In this way, the area containing the target (for example, person) in the video frame can be obtained, thereby narrowing the scope of action recognition and improving the accuracy and efficiency of action recognition.

[0034] In one embodiment of the present application, when the pixel area occupied by the target in the surveillance video accounts for a small proportion of the area in the video frame, the accuracy of determining the human body area based on the overlap rate is low. In order to improve the accuracy of identifying the human body area in the first image based on the first recognition model, the loss value of the first recognition model can be determined based on the position and area of ​​the detection frame of the human body area during the training of the first recognition model, thereby improving the performance of the first recognition model. For details, please refer to the method for training the first recognition model. Figure 4 The corresponding detailed description.

[0035] S21, determining the spatiotemporal features of the surveillance video based on a pre-trained second model, wherein the spatiotemporal features are used to characterize the association relationship between adjacent video frames in the surveillance video.

[0036] In one embodiment of the present application, when performing action recognition on a surveillance video, the corresponding spatiotemporal features can also be determined based on multiple video frames in the surveillance video. Subsequently, the spatiotemporal features and the image features of the first image can be used to identify whether the target in the surveillance video has made an action, thereby improving the accuracy of action recognition. Specifically, the spatiotemporal features of multiple video frames in the surveillance video are the characteristics of the video data in the two dimensions of time and space. The spatial features can be used to describe the static properties of the object in the video frame, such as the shape, size, color, and texture. In surveillance videos, these features are usually used to identify and track target objects, such as people and vehicles. For example, different objects in the video can be distinguished by shape and color features; the material of the object can be identified by texture features. The temporal features mainly describe the motion information of the object between consecutive video frames, including the object's motion trajectory, speed, acceleration and other dynamic properties. These features are crucial for understanding the object's motion pattern, predicting its future position, etc. In surveillance videos, temporal features are often used to detect abnormal behaviors, such as sudden acceleration, deceleration, and change of direction. The spatiotemporal features combine spatial features and temporal features to describe the spatiotemporal changes of objects in consecutive video frames. This combination enables the model to more accurately understand dynamic scenes in videos and identify complex spatiotemporal patterns. For example, through spatiotemporal features, continuous motion patterns such as pedestrians walking and vehicles driving in videos can be identified; abnormal behaviors in videos can also be detected, such as people entering restricted areas and vehicles driving in the wrong direction.

[0037] Among them, the spatiotemporal fusion feature can be in the form of a multidimensional tensor. A multidimensional tensor is a generalized form of a multidimensional array, which is used to represent data in multiple dimensions. In the process of monitoring video processing, a multidimensional tensor can be used to represent the spatiotemporal features of multiple video frames in the monitoring video, wherein the information represented by the spatiotemporal features includes the color of the video frame, the texture in the video frame, the shape of the target in the video frame, the movement trajectory of the target in multiple different video frames, etc. In the multidimensional tensor corresponding to the spatiotemporal features, different dimensions correspond to different spatiotemporal features. For example, the first dimension is used to represent the information of the time dimension, and the value of the first dimension can be used to represent the sequence number of the video frame; the second and third dimensions can be used to represent the information of the video frame of the monitoring video in the spatial dimension, and the values ​​of the second and third dimensions can be used to represent the width and height of the video frame; and higher dimensions may represent other features (such as color channel, movement speed, etc.). Among them, the value on the time dimension can represent the sequence number or timestamp of the video frame at different time points, which is used to record the position of the video frame on the time axis, thereby reflecting the timing information of the video data. The values ​​in the spatial dimension can represent the brightness, color or other visual features of the pixels in the image, and are used to describe the spatial structure of the video frame, thereby reflecting the spatial information of the video data. The values ​​in higher dimensions can represent more complex spatiotemporal features, such as the trajectory, speed, acceleration, etc. of an object. They are obtained by calculating and analyzing the spatiotemporal relationship between video frames, and are used to capture the dynamic information of the video data.

[0038] In one embodiment of the present application, the second recognition model can be a three-dimensional convolutional neural network, a recurrent neural network, a transfer model, a dual-stream network model, or a generative pre-trained model, and the present application does not limit this.

[0039] S22, determining a fusion feature according to the image feature and the spatiotemporal feature.

[0040] In one embodiment of the present application, when performing feature recognition on the content in the surveillance video, in order to avoid the problem of low accuracy of the action recognition result when using image features alone for action recognition or using the spatiotemporal features of multiple video frames alone for action recognition, a fusion feature can be determined based on the image features and spatiotemporal features, and the action recognition result of the surveillance video can be determined based on the fusion feature in the subsequent. Among them, the image feature can be a multidimensional tensor obtained after feature extraction of a video frame with a timestamp in the center of the surveillance video, which is used to characterize the spatial information in the video frame, such as the shape, color, texture of the object and the spatial relationship between them. The spatiotemporal feature can be obtained after feature extraction of different video frames in the surveillance video. The spatiotemporal feature contains the time information and motion information between the video frames, and can capture the dynamic information such as the motion trajectory, speed and interaction with other objects of the object in the video. The fusion feature determined based on the image features and spatiotemporal features can more comprehensively describe the information in the surveillance video. The fusion feature contains both the spatial information in the surveillance video and the dynamic time information and motion information, so it can more accurately identify the action in the video. Through reasonable fusion strategies, such as weighted average, feature concatenation or deep learning models, image features and spatiotemporal features can be effectively combined to form fusion features with stronger representation capabilities. Specifically, recognizing actions in surveillance videos based on fusion features can improve the accuracy of action recognition. Since fusion features combine the advantages of image features and spatiotemporal features, they can more comprehensively describe actions in surveillance videos, thereby improving the accuracy of recognition. And recognizing actions in surveillance videos based on fusion features can enhance the stability of action recognition results. Since fusion features contain multiple types of information, fusion features are less affected by interference factors such as illumination changes and occlusions. And recognizing actions in surveillance videos based on fusion features can improve computational efficiency. Through reasonable fusion strategies, the computational efficiency of action recognition can be improved without increasing too much computational complexity. Therefore, recognizing actions in surveillance videos based on fusion features can avoid the possibility of ambiguity in actions due to camera movement, rapid movement of targets or changes in lighting conditions. Fusion features can capture more action details, help reduce this ambiguity and improve recognition accuracy. In surveillance scenes, targets may be occluded by other objects. By fusing spatiotemporal features, the temporal continuity and spatial consistency of the action can be used to infer the information of the occluded part, thereby more accurately identifying the action. It can also avoid changes in lighting: changes in lighting conditions may affect the quality and clarity of the image, and thus affect the accuracy of action recognition. Fusion features can combine information from multiple video frames to reduce the impact of lighting changes on the recognition results. It can also avoid the problem of action complexity: actions in surveillance videos may involve multiple objects and complex motion patterns. Fusion features can provide richer information to help identify these complex actions.

[0041] In one embodiment of the present application, the method of determining the fusion feature according to the image feature and the spatiotemporal feature is shown in Figure 5 The corresponding detailed description.

[0042] S23, determining the action recognition result of the surveillance video according to the human body area and the fusion feature.

[0043] In one embodiment of the present application, the spatiotemporal features include all image information of multiple video frames within the imaging range of the camera device, and in the process of motion recognition, only the image information of a partial area (for example, the human body area) in the video frame needs to be paid attention to. Therefore, the human body area in the first image and the fusion features can be aligned, thereby ensuring that in the process of work recognition, the regional features related to the partial area (for example, the human body) in the fusion features can be paid attention to. In this way, when performing motion recognition on the surveillance video based on the regional features, the accuracy of motion recognition can be improved.

[0044] In one embodiment of the present application, the regional features can be used to characterize the image information of the areas where the target (e.g., human body) exists and is interrelated in multiple video frames of the surveillance video, so the action recognition result can be determined based on the regional features. Specifically, the image information in the target area can be input into the third recognition model to obtain the action recognition result output by the third recognition model. Among them, the third recognition model can be any artificial intelligence model with action recognition function. For example, the third recognition model can be a fully connected neural network model, or a multi-layer perceptron model, which is not limited in the present application.

[0045] In one embodiment of the present application, the method for determining the action recognition result is shown in Figure 6 The corresponding detailed description.

[0046] It can be seen from the above technical solutions that when the embodiment of the present application performs action recognition on the surveillance video, the human body area and image features in the video frame are first determined, and the image features and spatiotemporal features are fused to obtain fused features. In this way, in the process of performing action recognition on the surveillance video, the fused features can be first determined based on the image features and spatiotemporal features of any frame in the surveillance video, and the fused features are used to characterize the image information in the surveillance video and the correlation between the contents in different video frames in the time dimension. Finally, the regional features related to the human body in the fused features are determined based on the human body area, and action recognition is performed based on the regional features, which can increase the probability of including human information in the target area, and thus can improve the accuracy of action recognition. And it can narrow the scope of action recognition, thereby improving the efficiency of action recognition.

[0047] like Figure 4, is a flow chart of a method for determining a preset duration provided by an embodiment of the present application. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted. The method for determining a preset duration provided by an embodiment of the present application includes the following steps.

[0048] S30, performing image enhancement processing on the pre-stored first sample image to obtain a second sample image.

[0049] In one embodiment of the present application, in order to improve the quality of the image input to the first recognition model, the pre-stored first sample image may be enhanced to obtain a second sample image. The second sample image has a higher resolution than the first sample image. The image enhancement process includes interpolation, super-resolution reconstruction, image transformation, etc., which are not limited in the present application.

[0050] Exemplarily, processing the first image according to the interpolation algorithm includes: obtaining new pixel values ​​by calculation or prediction based on the continuity or correlation of the image pixel values ​​in space, and improving the resolution of the first image by inserting new pixel values ​​between the pixels of the first image. For example, the interpolation algorithm can be a nearest neighbor interpolation algorithm, and the value of the new pixel directly adopts the value of the known pixel closest to it. The bilinear interpolation algorithm linearly interpolates the values ​​of the upper and lower and left and right pixels of the target pixel and takes the weighted average. The bicubic interpolation algorithm considers the grayscale values ​​of the 16 pixels around the target pixel and performs interpolation calculations through a cubic polynomial. The spline interpolation algorithm performs interpolation calculations by constructing a piecewise polynomial function so that the interpolation curve smoothly transitions at the known pixel points. Through the processing of the interpolation algorithm, the first image with a lower resolution can be enlarged to obtain a second sample image with a higher resolution, and it can ensure that the clarity and image information of the first image are maintained in the second sample image.

[0051] In one embodiment of the present application, when improving the resolution of a first image, the first image may be first transformed to improve the quality of the first image; then an interpolation algorithm or a super-resolution reconstruction algorithm may be used to enlarge the first image and restore its details to obtain a high-resolution second sample image.

[0052] S31, based on the second sample image, using a pre-built initial recognition model, obtain prediction data in the second sample image, the prediction data including a prediction detection box, a prediction category of the prediction detection box, and a prediction probability of the prediction category.

[0053] In an embodiment of the present application, the pre-built initial recognition model may be a convolutional neural network model, a YOLO model, etc., and the present application does not limit this. Exemplarily, when the initial recognition model is a YOLO model, using the pre-built initial recognition model, obtaining the prediction data of the second sample image includes: dividing the second sample image into multiple grids. For example, the second sample image can be divided into multiple 7x7 grids, and the second sample image can also be divided into multiple grids of different sizes (for example, 13×13, 26×26, 52×52) to identify targets of different sizes in the second sample image. Predict multiple predicted detection frames in the grid according to the pixel values ​​of multiple pixels in each grid, and the position information of each predicted detection frame is used to characterize the position information of the target in the predicted detection frame (for example, the position information of the predicted detection frame can be [x, y, w, h]), the predicted category of the predicted detection frame, and the predicted probability of the predicted category. For example, when the position information of the predicted detection frame is [x, y, w, h], x and y are the coordinates of the center point of the predicted detection frame, and w and h are the width and height of the predicted detection frame. For example, when the position information of the predicted detection frame is [10,9,8,7], it indicates that the center point of the predicted detection frame is the pixel at the 10th row and 9th column in the second sample image, and the width of the predicted detection frame is 8 pixels, and the height of the predicted detection frame is 7 pixels. The predicted category of the predicted detection frame is used to characterize the category of the target in the predicted detection frame predicted by the initial recognition model. For example, the category of the target predicted by the initial recognition model can be "person", "flower", "tree", or "animal". The probability of the predicted category is used to characterize the probability that the image information in the predicted detection frame predicted by the initial recognition model belongs to a certain category. For example, the predicted category can be in the form of a vector [person: 0.7; flower: 0.05; tree: 0.05; animal: 0.2], indicating that the probability of the target in the predicted detection frame belonging to "person" is 0.7, the probability of belonging to "flower" is 0.05, the probability of belonging to "tree" is 0.05, and the probability of belonging to "animal" is 0.2.

[0054] S32, determining the loss value of the initial identification model according to the predicted data and the label data in the second sample image; the label data includes a label detection box and a corresponding label category.

[0055] In one embodiment of the present application, the second sample image corresponds to at least one label data. The label data includes a label detection frame and a corresponding label category. The position information of the label detection frame is used to characterize the position information of the target in the label detection frame, and the label category is used to characterize the category of the target in the label detection frame. The loss value of the initial recognition model is determined based on the predicted data and the label data, and the model performance can be evaluated. The loss value can characterize the gap between the predicted result output by the initial recognition model and the actual label.

[0056] In one embodiment of the present application, the method for determining the loss value is shown in Figure 7 The corresponding detailed description.

[0057] S33, using a back propagation algorithm to update the initial recognition model until the loss value meets a preset condition, thereby obtaining a trained first recognition model.

[0058] In one embodiment of the present application, the preset condition may be that the loss value of the initial recognition model is less than or equal to a preset termination threshold. When the loss value is small, it means that the gap between the predicted detection frame and the label detection frame is small, and the overlap between the predicted detection frame and the label detection frame is high, that is, the result predicted by the initial recognition model is more accurate. According to the change in the loss value, the performance of the initial recognition model at different training stages can be judged, and the optimization strategy (for example, adjusting the learning rate, batch size, etc.) can be adjusted accordingly. During the model training process, the loss value is used as the input of the gradient descent algorithm to guide the update direction of the model parameters. By continuously reducing the loss value through the back propagation algorithm, the initial recognition model can gradually learn the ability to extract effective features from the input image and make accurate predictions. The calculation of the loss value prompts the model to pay more attention to the features related to target recognition in the image, thereby improving the generalization ability and robustness of the model. During the training process, the first recognition model can dynamically adjust the attention to targets of different scales according to the change in the loss value, thereby improving the robustness of detecting targets in complex scenes.

[0059] like Figure 5 , is a flow chart of a method for determining fusion features provided in an embodiment of the present application. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted. The method for determining fusion features provided in an embodiment of the present application includes the following steps.

[0060] S40: Determine corresponding weight parameters according to the image features.

[0061] In one embodiment of the present application, since the image feature is the image information in the video frame with the timestamp centered in the surveillance video, it can characterize the image information corresponding to the target and background that account for a relatively large proportion in the imaging range of the surveillance video. Therefore, the corresponding weight parameter can be determined according to the image feature, and the weight parameter can be used to characterize the importance of the image information corresponding to the target and background that account for a relatively large proportion in the imaging range of the surveillance video. Subsequently, the more important part can be determined in the spatiotemporal feature according to the weight parameter to obtain the fusion feature.

[0062] Exemplarily, the weight parameter may be in the form of a vector, and the dimension of the weight parameter may be [1, 1, C], which is used to characterize that the weight parameter is a vector of C channels in one row and one column.

[0063] In one embodiment of the present application, the method for determining the weight parameter is shown in Figure 8 The corresponding detailed description.

[0064] S41, determining a fusion feature according to the weight parameter and the spatiotemporal feature.

[0065] In one embodiment of the present application, since the weight parameter can be used to characterize the importance of the image information corresponding to the target and background that occupy a relatively large proportion within the imaging range of the surveillance video, the more important part can be determined in the spatiotemporal feature according to the weight parameter to obtain the fusion feature. Specifically, determining the fusion feature according to the weight parameter and the spatiotemporal feature includes: determining the product of the weight parameter and the spatiotemporal feature as the fusion feature.

[0066] Exemplarily, the spatiotemporal features may be in the form of a multidimensional tensor, and the form of the spatiotemporal features may be [B, T, H, W, C], where B represents that the surveillance video includes B video frames, T is a preset constant (for example, T may be 16), H and W represent the width and height of the video frame, and C represents the number of channels for each pixel in the video frame.

[0067] like Figure 6 , is a flow chart of a method for determining an action recognition result provided in an embodiment of the present application. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted. The method for determining an action recognition result provided in an embodiment of the present application includes the following steps.

[0068] S50: Determine a corresponding region feature from the fused features according to the position of the human body region in the first image.

[0069] In an embodiment of the present application, since the actions in the surveillance video are all performed by the human body within the imaging range of the surveillance video, the human body region can be mapped to the fusion feature according to the dimension of the fusion feature and the dimension of the first image, and then the regional feature corresponding to the human body region can be determined from the fusion feature. Thus, the regional feature is used to characterize the image information associated with the human body in the surveillance video in the fusion feature.

[0070] Exemplarily, the dimension of the fusion feature may be [B, T, H, W, C], which is used to characterize that the fusion feature includes T batches, each batch includes B matrices, each matrix has a width of W and a height of H, and the number of channels of each element in each matrix is ​​C; the dimension of the first image may be [H×W], which is used to characterize that the height of the first image is H and the width is W; the position of the human body region in the first image may be characterized as [x, y, w, h], which is used to characterize that the center point of the human body region is the pixel point of the xth row and yth column in the first image, and the width of the human body region is w pixels, and the height of the human body region is h pixels. Since the dimension of any matrix in the fusion feature is the same as the dimension of the first image, the center point of the target region related to the human body in any matrix of the fusion feature can be determined as the feature value of the xth row and yth column, and the width of the target region can be determined as w and the height is h. In this way, the eigenvalues ​​in the target area in each matrix of the fused features can be determined as regional features, so that the regional features can be used to characterize the eigenvalues ​​related to the human body in the fused features, and then the information related to the human body can be focused on during the action recognition process, thereby improving the accuracy of action recognition.

[0071] S51, determining the action recognition result of the surveillance video according to the regional features.

[0072] In one embodiment of the present application, the regional features can be used to characterize the image information of the areas where the target (e.g., human body) exists and is interrelated in multiple video frames of the surveillance video, so the action recognition result can be determined based on the regional features. Specifically, the image information in the target area can be input into the third recognition model to obtain the action recognition result output by the third recognition model. Among them, the third recognition model can be any artificial intelligence model with action recognition function. For example, the third recognition model can be a fully connected neural network model, or a multi-layer perceptron model, which is not limited in the present application.

[0073] like Figure 7 As shown, it is a flow chart of a method for determining the loss value of an initial prediction model provided by an embodiment of the present application. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted. The method for determining the loss value of an initial prediction model provided by an embodiment of the present application includes the following steps.

[0074] S60, determining a position difference between the label detection frame and the predicted detection frame according to the position of the label detection frame and the position of the predicted detection frame.

[0075] In one embodiment of the present application, the position difference between the label detection frame and the predicted detection frame is used to evaluate the deviation between the model predicted detection frame and the label detection frame. The method for determining the position difference can be to determine the intersection-and-union ratio between the predicted detection frame and the label detection frame; it can also be a complete intersection-and-union ratio loss; it can also be a shape attention intersection-and-union ratio or a generative intersection-and-union ratio loss.

[0076] Exemplarily, when the position difference is calculated using the intersection-over-union ratio, the calculation method of the position difference satisfies the following relationship: ; Among them, A represents the set of pixels occupied by the predicted detection box, and B represents the set of pixels occupied by the label detection box.

[0077] S61, determining a category difference according to the predicted category, the label category, and the predicted probability.

[0078] In one embodiment of the present application, the class difference is used to measure the accuracy of the model's prediction of the target class. Exemplarily, the class difference can be calculated based on a binary cross entropy loss function, which is suitable for processing two-class label classification problems; the class difference can also be calculated based on a multivariate cross entropy loss function, which is suitable for processing multi-class label classification problems. For example, when calculating the class difference based on a multivariate cross entropy loss function, the calculation method of the class difference satisfies the following relationship: ; Among them, when the label category of the i-th detection box is the j-th category, is 1, otherwise is 0; when the predicted category of the i-th sample is the j-th category, is the predicted probability; J is the total number of categories; N is the number of predicted detection boxes.

[0079] S62: Determine the loss value according to the position difference and the category difference.

[0080] In one embodiment of the present application, the sum of the position difference and the category difference can be determined as the loss value of the initial recognition model. Specifically, the calculation method of the loss value of the initial recognition model satisfies the following relationship: ; like Figure 8 , is a flow chart of a method for determining weight parameters provided in an embodiment of the present application. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted. The method for determining weight parameters provided in an embodiment of the present application includes the following steps.

[0081] S70, performing a pooling operation on the image features to obtain corresponding compression features.

[0082] In one embodiment of the present application, the pooling operation is used to reduce the size of the image feature, improve the computational efficiency, and prevent overfitting to a certain extent. For example, when the dimension of the image feature is m×n, the number of elements covered by the pooling window in the image feature during each pooling operation is determined according to a preset pooling window (e.g., a 2×2 pooling window) and a preset step size (e.g., a step size of 2), and the step size determines the distance the pooling window moves on the image feature. When the pooling operation is average pooling, all elements in the pooling window are averaged, and the average value is used as an element in the result vector. The window moves by the step size until the entire matrix is ​​covered. When the pooling operation is maximum pooling, the maximum value is selected in the pooling window, and the maximum value is used as an element in the result vector, and the window moves by the step size until the entire matrix is ​​covered. After the pooling operation, a compressed feature with a smaller dimension is obtained, and the length of the compressed feature depends on the size of the matrix, the size of the pooling window, and the step size.

[0083] S71, performing a convolution operation on the compressed features to obtain weight parameters.

[0084] In one embodiment of the present application, starting from the edge of the compressed feature (for example, any end of the compressed feature), the convolution kernel is aligned with the corresponding area of ​​the same size in the compressed feature; each parameter in the convolution kernel is multiplied by the element of the corresponding area of ​​the compressed feature, and all products are added, and the result obtained is the new eigenvalue of the central element of the area; after completing a convolution operation, the area corresponding to the convolution kernel in the compressed feature during the next convolution operation is determined according to the preset step size and sliding direction. The preset step size can be the number of columns of the convolution kernel, or it can be other values. For example, when the preset step size is 1 and the sliding direction is to slide to the right, the area of ​​the compressed feature with the same size as the convolution kernel that participated in the previous convolution operation can be moved one column to the right to determine the area in the compressed feature that participates in this convolution operation, and the above calculation process is repeated until all areas of the compressed feature are traversed; after the above operation, a new feature data is obtained, and the feature data is the weight parameter of the convolution operation.

[0085] See also Fig. 9 , Fig. 9 1 is a functional module diagram of an action recognition device provided in an embodiment of the present application. An action recognition device 91 includes a first recognition module 901, a second recognition module 902, a fusion module 903 and a third recognition module 904. The module / unit referred to in the present application refers to a series of computer-readable instruction segments that can be executed by the processor 13 and can complete a fixed function, which is stored in the memory 12. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0086] The first recognition module 901 is used to process a first image in a surveillance video based on a pre-trained first model to obtain image features and human body regions of the first image.

[0087] The second recognition module 902 is used to determine the spatiotemporal features of the surveillance video based on a pre-trained second model, where the spatiotemporal features are used to characterize the association relationship between adjacent video frames in the surveillance video.

[0088] The fusion module 903 is used to determine fusion features according to the image features and the spatiotemporal features.

[0089] The third recognition module 904 is used to determine the action recognition result of the monitoring video according to the human body area and the fusion feature.

[0090] In some embodiments, the first recognition module 901 is also used to perform image enhancement processing on a pre-stored first sample image to obtain a second sample image; based on the second sample image, using a pre-constructed initial recognition model, obtain prediction data in the second sample image, the prediction data including a prediction detection box, a prediction category of the prediction detection box, and a prediction probability of the prediction category; according to the prediction data and the label data in the second sample image, determine the loss value of the initial recognition model; the label data includes a label detection box and a corresponding label category; use a back propagation algorithm to update the initial recognition model until the loss value meets a preset condition to obtain a trained first recognition model.

[0091] In some embodiments, the first recognition module 901 is also used to determine the position difference between the label detection box and the predicted detection box according to the position of the label detection box and the position of the predicted detection box; determine the category difference according to the predicted category, the label category and the predicted probability; determine the loss value according to the position difference and the category difference.

[0092] In some embodiments, the fusion module 903 is further used to determine corresponding weight parameters according to the image features; and determine fusion features according to the weight parameters and the spatiotemporal features.

[0093] In some embodiments, the fusion module 903 is further used to perform a pooling operation on the image features to obtain corresponding compressed features; and perform a convolution operation on the compressed features to obtain weight parameters.

[0094] In some embodiments, the fusion module 903 is further used to determine the product of the weight parameter and the spatiotemporal feature as the fusion feature.

[0095] In some embodiments, the third recognition module 904 is further used to determine corresponding regional features from the fusion features according to the position of the human body region in the first image; and determine the action recognition result of the surveillance video according to the regional features.

[0096] See also Fig.10 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 100 includes a memory 12 and a processor 13. The memory 12 is used to store computer-readable instructions, and the processor 13 is used to execute the computer-readable instructions stored in the memory to implement an action recognition method described in any of the above embodiments.

[0097] In an embodiment of the present application, the electronic device 100 further includes a bus, a computer program stored in the memory 12 and executable on the processor 13, such as a motion recognition program.

[0098] Fig.10 Only the electronic device 100 having the memory 12 and the processor 13 is shown, and those skilled in the art can understand that Fig.10 The structure shown does not constitute a limitation on the electronic device 100 , and the electronic device 100 may include fewer or more components than shown in the figure, or combine some components, or arrange the components differently.

[0099] Combination Figure 2 The memory 12 in the electronic device 100 stores a plurality of computer-readable instructions to implement an action recognition method, and the processor 13 can execute the plurality of instructions to implement: processing a first image in a surveillance video based on a pre-trained first model to obtain image features and a human body region of the first image; determining spatiotemporal features of the surveillance video based on a pre-trained second model, wherein the spatiotemporal features are used to characterize the association relationship between adjacent video frames in the surveillance video; determining fusion features based on the image features and the spatiotemporal features; and determining an action recognition result of the surveillance video based on the human body region and the fusion features.

[0100] Specifically, the specific implementation method of the processor 13 for the above instructions can refer to Figure 2 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0101] Those skilled in the art will appreciate that the schematic diagram is merely an example of the electronic device 100 and does not constitute a limitation on the electronic device 100. The electronic device 100 may be a bus-type structure or a star-type structure. The electronic device 100 may also include more or less other hardware or software than shown in the diagram, or a different arrangement of components. For example, the electronic device 100 may also include input and output devices, network access devices, etc.

[0102] It should be noted that the electronic device 100 is only an example, and other existing or future electronic products that are suitable for the present application should also be included in the protection scope of the present application and included here by reference.

[0103] Among them, the memory 12 includes at least one type of readable storage medium, and the readable storage medium can be non-volatile or volatile. The readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the electronic device 100, such as a mobile hard disk of the electronic device 100. In other embodiments, the memory 12 can also be an external storage device of the electronic device 100, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the electronic device 100. The memory 12 can not only be used to store application software and various types of data installed in the electronic device 100, such as a code of a motion recognition program, etc., but also can be used to temporarily store data that has been output or is to be output.

[0104] In some embodiments, the processor 13 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor 13 is the control core (Control Unit) of the electronic device 100, and uses various interfaces and lines to connect various components of the entire electronic device 100, and executes various functions and processes data of the electronic device 100 by running or executing programs or modules stored in the memory 12 (for example, executing a motion recognition program, etc.), and calling data stored in the memory 12.

[0105] The processor 13 executes the operating system of the electronic device 100 and various installed applications. The processor 13 executes the applications to implement the steps in each of the above-mentioned motion recognition method embodiments, for example Figure 2 Steps shown.

[0106] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present application. The one or more modules / units may be a series of computer-readable instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the electronic device 100. For example, the computer program may be divided into a first recognition module 901, a second recognition module 902, a fusion module 903, and a third recognition module 904.

[0107] The above-mentioned integrated unit implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to execute a part of a motion recognition method described in each embodiment of the present application.

[0108] If the module / unit integrated in the electronic device 100 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also instruct the relevant hardware devices to complete through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented.

[0109] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory and other memory, etc.

[0110] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0111] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. Fig.10 Only one arrow is used in the figure, but it does not mean that there is only one bus or one type of bus. The bus is configured to realize the connection and communication between the memory 12 and at least one processor 13, etc.

[0112] An embodiment of the present application also provides a computer-readable storage medium (not shown), in which computer-readable storage medium is stored computer-readable instructions, and the computer-readable instructions are executed by a processor in an electronic device to implement the action recognition method described in any of the above embodiments.

[0113] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0114] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0115] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0116] In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the specification can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any specific order.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application has been described in detail with reference to the preferred embodiments, a person of ordinary skill in the art should understand that the technical solution of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present application.

Claims

1. A motion recognition method, applied to an electronic device, characterized in that: The method comprises: Processing a first image in a surveillance video based on a pre-trained first model to obtain image features and a human body region of the first image; Determining the spatiotemporal features of the surveillance video based on a pre-trained second model, wherein the spatiotemporal features are used to characterize the association relationship between adjacent video frames in the surveillance video; Determine a fusion feature according to the image feature and the spatiotemporal feature; The action recognition result of the monitoring video is determined according to the human body area and the fusion feature.

2. The motion recognition method according to claim 1, characterized in that: The method further includes training the first model, wherein the training the first model includes: Performing image enhancement processing on the pre-stored first sample image to obtain a second sample image; Based on the second sample image, using a pre-built initial recognition model, obtaining prediction data in the second sample image, the prediction data including a prediction detection box, a prediction category of the prediction detection box, and a prediction probability of the prediction category; Determining the loss value of the initial identification model according to the predicted data and the label data in the second sample image; the label data includes a label detection box and a corresponding label category; The initial recognition model is updated using a back propagation algorithm until the loss value meets a preset condition, thereby obtaining a trained first recognition model.

3. The motion recognition method according to claim 2, characterized in that: Determining the loss value of the initial model includes: Determining a position difference between the label detection frame and the predicted detection frame according to the position of the label detection frame and the position of the predicted detection frame; Determine the category difference according to the predicted category, the label category and the predicted probability; The loss value is determined according to the position difference and the category difference.

4. The motion recognition method according to claim 1, characterized in that: Determining the fusion feature according to the image feature and the spatiotemporal feature comprises: Determine corresponding weight parameters according to the image features; A fusion feature is determined according to the weight parameter and the spatiotemporal feature.

5. The motion recognition method according to claim 4, characterized in that: Determining the corresponding weight parameter according to the image feature comprises: Performing a pooling operation on the image features to obtain corresponding compressed features; A convolution operation is performed on the compressed features to obtain weight parameters.

6. The motion recognition method according to claim 4, characterized in that: The determining of the fusion feature according to the weight parameter and the spatiotemporal feature comprises: The product of the weight parameter and the spatiotemporal feature is determined as the fusion feature.

7. The motion recognition method according to claim 1, characterized in that: Determining the action recognition result of the surveillance video according to the human body region and the fusion feature includes: Determining corresponding regional features from the fused features according to the position of the human body region in the first image; The action recognition result of the surveillance video is determined according to the regional features.

8. A motion recognition device, characterized in that: The device comprises: A first recognition module is used to process a first image in the surveillance video based on a pre-trained first model to obtain image features and a human body area of ​​the first image; A second recognition module, used to determine the spatiotemporal features of the surveillance video based on a pre-trained second model, wherein the spatiotemporal features are used to characterize the association relationship between adjacent video frames in the surveillance video; A fusion module, used to determine a fusion feature according to the image feature and the spatiotemporal feature; The third recognition module is used to determine the action recognition result of the monitoring video according to the human body area and the fusion feature.

9. An electronic device, characterized in that: The electronic device comprises a processor and a memory, and the processor is used to implement the action recognition method as claimed in any one of claims 1 to 7 when executing a computer program stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the action recognition method according to any one of claims 1 to 7 is implemented.

Images