Action recognition method and device, electronic equipment and storage medium

Abstract

The invention provides an action recognition method and device, electronic equipment and a storage medium. The action recognition method comprises the following steps: recognizing human body regions in a plurality of first images based on a pre-trained first recognition model; identifying foreground regions in the plurality of first images based on a second identification model; determining a comprehensive area corresponding to the plurality of first images according to the foreground areas of the plurality of first images; wherein the comprehensive region is used for representing a region containing action information in each first image; determining a to-be-recognized area according to the comprehensive area and the human body area; wherein the to-be-recognized area is used for representing an area containing human body information and action information in each first image; and determining an action recognition result based on the image information of the to-be-recognized area. The invention relates to the technical field of artificial intelligence, and can improve the accuracy of action recognition.

Description

Technical Field

[0001] The present application relates to the field of motion recognition technology, and in particular to a motion recognition method, device, electronic device and storage medium. Background Art

[0002] At present, with the development of artificial intelligence technology, more and more organizations or individuals tend to use artificial intelligence models to assist production in order to improve productivity. Some organizations or individuals use edge computing devices to run artificial intelligence models, and use artificial intelligence models to process surveillance videos collected by cameras, so as to identify and analyze the movements of targets in surveillance videos. For example, in a production environment, artificial intelligence models can be used to analyze surveillance videos of factory areas or production lines to identify the degree of standardization of workers' movements during the production process, thereby improving the safety of the production process; for another example, in public areas, artificial intelligence models can be used to analyze the movements of targets in surveillance videos to identify conflicting movements of targets in public areas, thereby promptly identifying movements that endanger public safety.

[0003] Since the current scenarios where artificial intelligence models are used for action recognition usually involve dense targets and small individual targets, the target actions may not be accurately identified, resulting in low accuracy in identifying the actions of targets in surveillance videos. Summary of the invention

[0004] In view of the above, it is necessary to propose a motion recognition method, device, electronic device and storage medium to solve the technical problem of low accuracy of motion recognition.

[0005] The present application provides an action recognition method, which is applied to an electronic device, and the method includes: identifying human body areas in multiple first images based on a pre-trained first recognition model; identifying foreground areas in the multiple first images based on a second recognition model; determining a comprehensive area corresponding to the multiple first images based on the foreground areas of the multiple first images; wherein the comprehensive area is used to characterize an area containing action information in each first image; determining an area to be recognized based on the comprehensive area and the human body area; wherein the area to be recognized is used to characterize an area containing human body information and action information in each first image; and determining an action recognition result based on the image information of the area to be recognized.

[0006] An embodiment of the present application also provides a motion recognition device, which includes: a detection module, used to identify human body areas in multiple first images based on a pre-trained first recognition model; a first recognition module, used to identify foreground areas in the multiple first images based on a second recognition model; the first recognition module is also used to determine a comprehensive area corresponding to the multiple first images based on the foreground areas of the multiple first images; wherein the comprehensive area is used to characterize the area containing motion information in each first image; the first recognition module is also used to determine a to-be-recognized area based on the comprehensive area and the human body area; wherein the to-be-recognized area is used to characterize the area containing human body information and motion information in each first image; and a second recognition module is used to determine a motion recognition result based on image information of the to-be-recognized area.

[0007] An embodiment of the present application also provides an electronic device, which includes: a memory storing at least one instruction; and a processor executing the instruction stored in the memory to implement the action recognition method.

[0008] An embodiment of the present application also provides a computer-readable storage medium, in which at least one instruction is stored. The at least one instruction is executed by a processor in an electronic device to implement the action recognition method.

[0009] It can be seen from the above technical solutions that when the embodiment of the present application performs action recognition on the surveillance video, the human body detection frame in the video frame is first determined, and then the foreground area in the video frame is determined, so that the area containing the human body can be preliminarily screened out, thereby avoiding the misidentification of irrelevant background. Combined with the foreground area, the interference of static background or irrelevant dynamic objects can be further eliminated, thereby improving the accuracy of action recognition. And the combined use of the human body detection frame and the foreground area can achieve accurate positioning of the target human body. It is helpful to accurately identify the actions of specific people in complex monitoring scenes. The system's adaptability to complex scenes can be enhanced and the stability of recognition can be improved. In addition, by first determining the human body detection frame and the foreground area, the search range of action recognition can be narrowed, thereby reducing the amount of calculation and improving the processing speed. And the response time of action recognition can be shortened, so that the system can respond faster. Finally, the area to be identified that needs to be action recognized is jointly determined according to the foreground area and the human body detection frame, which is convenient for associating the video data with the subsequent action recognition results to form a complete monitoring data chain. Thereby improving the accuracy of action recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is an application scenario diagram of the action recognition method provided in one embodiment of the present application.

[0011] Figure 2It is a flowchart of a motion recognition method provided by an embodiment of the present application.

[0012] Figure 3 is a schematic diagram of a first image provided by an embodiment of the present application.

[0013] Figure 4 This is a flowchart of a method for training a first recognition model provided in one embodiment of the present application.

[0014] Figure 5 It is a flowchart of a method for determining a comprehensive area provided by an embodiment of the present application.

[0015] Figure 6 This is a flowchart of a method for determining an area to be identified provided in an embodiment of the present application.

[0016] Figure 7 It is a functional module diagram of a motion recognition device provided in one embodiment of the present application.

[0017] Figure 8 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to more clearly understand the purpose, features and advantages of the present application, the present application is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other. In the following description, many specific details are set forth to facilitate a full understanding of the present application, and the embodiments described are only a part of the embodiments of the present application, rather than all of the embodiments.

[0019] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by those skilled in the art to which the present application belongs. The terms used herein in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0021] The embodiment of the present application provides a motion recognition method, which can be applied to one or more electronic devices. The electronic device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a programmable gate array (Field-Programmable Gate Array, FPGA), a digital processor (Digital Signal Processor, DSP), an embedded device, etc.

[0022] An electronic device can be any electronic product that can interact with customers, such as personal computers, tablet computers, smart phones, personal digital assistants (PDAs), game consoles, Internet Protocol Television (IPTV), smart wearable devices, etc.

[0023] The electronic device may also include a network device and / or a client device, wherein the network device includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud consisting of a large number of hosts or network servers based on cloud computing.

[0024] The networks where electronic devices are located include but are not limited to the Internet, wide area network, metropolitan area network, local area network, virtual private network (VPN), etc.

[0025] like Figure 1 As shown, an action recognition method provided by the present application can be applied to an electronic device 100, and the electronic device 100 is communicatively connected to a camera device 200, and the electronic device 100 is used to receive a surveillance video sent by the camera device 200. Among them, the camera device 200 is used to shoot a target (for example, a pedestrian, a vehicle, etc.) within an imaging range to obtain a surveillance video including the target. A first recognition model for target recognition and a second recognition model for foreground recognition are running in the electronic device 100. After the electronic device 100 receives the surveillance video input by the camera device 200, the first recognition model and the second recognition model are used to process multiple video frames in the surveillance video, respectively, to obtain an action recognition result of the target in the surveillance video.

[0026] Exemplarily, when the surveillance video captured by the camera device 200 includes multiple people, the first recognition model in the electronic device 100 receives multiple video frames in the surveillance video and identifies the human body area where the people in the multiple video frames are located. For example, the multiple targets (e.g., people) in the surveillance video can be multiple people with conflicting behaviors and physical contact; the multiple targets can also be multiple players competing for basketball in a basketball court; the multiple targets can also be multiple pedestrians walking on the street. The present application does not limit the specific surveillance scene of the surveillance video. The second recognition model in the electronic device 100 also identifies the foreground area (e.g., the area belonging to the person in the video frame) and the background area (e.g., the area not belonging to the person in the video frame) in the multiple video frames. The electronic device 100 also determines the comprehensive area based on the foreground area in the multiple video frames, characterizes the area where the person may appear in the surveillance video based on the comprehensive area, and determines the area to be identified in the video frame that needs to be action recognized based on the comprehensive area and the human body area. For example, the area to be identified may be an area in the video frame where a human body exists and where a physical conflict may occur, or an area in the video frame where a human body may exist and where a scramble may occur, or an area in the video frame where a human body may exist and where a walking action may occur. In this way, before performing action recognition on the surveillance video, the area to be identified that needs to be action recognized can be determined based on the human body area and the foreground area in the video frame, which can increase the probability that the area to be identified includes human information, thereby improving the accuracy of action recognition. It can also narrow the scope of action recognition, thereby improving the efficiency of action recognition.

[0027] like Figure 2 , is a flow chart of a method for motion recognition provided in an embodiment of the present application. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted. A method for motion recognition provided in an embodiment of the present application includes the following steps.

[0028] S20: Identify human body regions in a plurality of first images based on a pre-trained first recognition model.

[0029] In one embodiment of the present application, when identifying human body movements in a surveillance video, in order to improve the accuracy of movement recognition, the human body region in the video frame of the surveillance video may be first identified. The first image may be any video frame in the surveillance video, and the first recognition model may be a recognition model with a target detection function. Exemplarily, the first recognition model may be the V3 version of the YOLO model, the V4 version of the YOLO model, the V8S version of the YOLO model, or a convolutional neural network model, etc., which is not limited in the present application.

[0030] Among them, the YOLO (You Only Look Once) model is an artificial intelligence model with target detection function. Exemplarily, the specific steps of using the YOLO model to identify the human body area in the first image are as follows: pre-set the parameters such as the size of the input data and the size of the output data of the first recognition model. For example, the size of the input data of the first recognition model can be 960x576, 416x416, or 448x448, and the size of the output data can be 1x6. This application does not limit the size of the input data and the size of the output data of the first recognition model. Read the surveillance video to be analyzed from the database, segment the surveillance video, and read the image to be detected by the image processing library (for example, OpenCV). Scale the image to the input size required by the model, convert the image data into a tensor, and perform normalization. The first recognition model uses an encoder to encode the input first image to obtain an output tensor, and the size and size of the output tensor depend on the configuration of the model and the size of the input image. The decoder of the first recognition model extracts the detection results from the output tensor. Specifically, the predicted bounding box information of each grid is extracted from the output tensor, including the center point coordinates, width, height and confidence of the bounding box. Among them, the encoder and the decoder both include multiple neurons, each neuron corresponds to an activation function, and multiple activation functions are used to transmit information related to the image data in the encoder and the decoder. Exemplarily, the activation function can be a LEAKYRELU function or a SILU function, which is not limited in the present application. According to the category probability information in the output tensor, the category to which each bounding box belongs is determined. The extracted bounding boxes are subjected to non-maximum suppression processing, and the bounding boxes are screened by comparing the confidence and overlap rate of the bounding boxes to eliminate overlapping bounding boxes and retain the best detection results. According to the bounding box information and the category probability, the final detection box position and category are determined. Draw a detection box on the first image, and mark the category and confidence information. As Figure 3 The figure shows a schematic diagram of a human body region 310 corresponding to a target person in the first image 300. In this way, the region containing the target (eg, person) in the video frame can be obtained, thereby narrowing the scope of action recognition and improving the accuracy and efficiency of action recognition.

[0031] In one embodiment of the present application, when the pixel area occupied by the target in the surveillance video accounts for a small proportion of the area in the video frame, the accuracy of determining the human body area based on the overlap rate is low. In order to improve the accuracy of identifying the human body area in the first image based on the first recognition model, the loss value of the first recognition model can be determined based on the position and area of ​​the detection frame of the human body area during the training of the first recognition model, thereby improving the performance of the first recognition model. For details, please refer to the method for training the first recognition model. Figure 4 The corresponding detailed description.

[0032] S21, identifying foreground areas in the plurality of first images based on a second recognition model.

[0033] In one embodiment of the present application, when performing action recognition on a surveillance video, the foreground area of ​​multiple video frames in the surveillance video is first identified, which can improve the accuracy of action recognition. Specifically, identifying the foreground area in the first image helps to separate the moving object (i.e., the foreground) from the static background in the video. By eliminating the interference of background information, the subsequent action recognition process can focus on analyzing the motion information in the foreground, thereby improving the accuracy of action recognition. And identifying the foreground area can accurately delineate the area containing the action, which can ensure that the second recognition model more accurately identifies and analyzes the action of the target in the surveillance video. In complex surveillance scenes, the identification of the foreground area helps to avoid misidentification of irrelevant areas, and can also improve the efficiency of action recognition. By first identifying the foreground area, action recognition can be performed within a smaller image range, which helps to reduce the amount of calculation and improve the processing speed of action recognition. In addition, surveillance videos usually contain complex backgrounds and various moving objects. By first identifying the foreground area, the system can better adapt to these complex scenes and improve the stability and reliability of recognition.

[0034] Exemplarily, the second recognition model may be a Visual Background Extractor (VIBE) model. VIBE is an artificial intelligence algorithm model with background modeling and foreground detection. Processing the first image using the VIBE algorithm includes: using the first first image in the surveillance video to initialize the VIBE model. For each pixel in the first image, a pixel in its neighborhood (e.g., 8 adjacent pixels) is randomly selected as the sample value of the VIBE model of the pixel. For each pixel in the subsequent multiple first images, the similarity between the sample value and the pixel value of each pixel in the VIBE model is calculated. If the similarity between the pixel and multiple pixels in the VIBE model is less than a preset similarity threshold, the pixel is classified as background; otherwise, the pixel is classified as foreground.

[0035] S22, determining a comprehensive area corresponding to the multiple first images according to the foreground areas of the multiple first images; wherein the comprehensive area is used to represent an area containing action information in each first image.

[0036] In one embodiment of the present application, in order to analyze the image information of multiple video frames in the surveillance video during the process of action recognition to improve the accuracy of action recognition, when analyzing the foreground area in a surveillance video, the foreground areas in multiple first images in the surveillance video can be fused into a comprehensive area, and then the image information in the first image involved in action recognition is determined based on the comprehensive area, thereby improving the accuracy of action recognition. The comprehensive area is used to characterize the area containing action information in each first image.

[0037] In one embodiment of the present application, the surveillance video can be preprocessed, continuous video frames can be extracted from the surveillance video, and the foreground area can be separated from the video frame by using methods such as background difference method, mixed Gaussian model or deep learning. In this way, the foreground (moving target) and the background can be distinguished, so as to accurately identify pedestrians, vehicles and other targets in the surveillance scene. Then the foreground area in multiple continuous video frames is spliced. Among them, before the video splicing, the video frame can also be corrected to avoid various distortions in the video frame caused by the manufacturing, installation, process and other reasons of the camera device. The picture can be visually stretched and adjusted by fisheye correction and other technologies to eliminate the distortion. Perspective transformation: Due to the different heights, distances and angles of different cameras, the captured pictures are not on the same projection plane. Therefore, it is necessary to perform perspective transformation on the image and adjust it to a consistent viewing angle for seamless splicing. For multi-source video data with overlapping areas, the splicing algorithm can be used for seamless real-time splicing to eliminate the overlapping areas and form a wide-angle, large-field-of-view video image. Crop the stitched images, remove the redundant parts, and keep the final video image.

[0038] In some embodiments, the fused video frames can also be post-processed and optimized. Since different cameras may have color differences, the spliced ​​video needs to be color corrected to ensure the overall color consistency. The frame rate and resolution of the video can also be adjusted to meet different application scenarios and display requirements. For example, in banks, jewelry stores, parks and other places, through monitoring video analysis technology, suspicious persons can be discovered and circled in time to prevent crime and ensure public safety; in places with large traffic such as stations and shopping malls, through regional wandering video analysis gateway and other technologies, the flow of personnel can be analyzed, and management personnel can be warned in advance to guide the flow of people and avoid congestion and safety hazards; in smart parks, through real-time access of cameras to the analysis platform video analysis gateway and other technologies, blind spots of human control can be monitored, and personnel in the area can be monitored and analyzed in real time to improve the safety management level of the park.

[0039] In one embodiment of the present application, for a specific method of determining the integrated area corresponding to the plurality of first images according to the foreground areas of the plurality of first images, see Figure 5The corresponding detailed description.

[0040] S23, determining a region to be identified according to the comprehensive region and the human body region; wherein the region to be identified is used to represent a region in each first image that contains human body information and action information.

[0041] In one embodiment of the present application, when performing action recognition on a surveillance video, a human body detection frame in a video frame of the surveillance video is first identified to determine the human body area in the video frame, and then the foreground area in the video frame is determined. Finally, the area to be identified that needs to be action recognized is determined based on the foreground area and the human body area, wherein the area to be identified is used to characterize the area containing human body information and action information in each first image. In this way, the accuracy of action recognition can be improved. By determining the human body detection frame, the area containing the human body can be preliminarily screened out to avoid misidentification of irrelevant backgrounds. Combined with the foreground area, the interference of static backgrounds or irrelevant dynamic objects can be further eliminated, thereby improving the accuracy of action recognition. The combined use of the human body detection frame and the foreground area can achieve accurate positioning of the target human body. This helps to accurately identify the actions of specific people in complex surveillance scenes. The efficiency of action recognition can also be improved. By first determining the human body detection frame and the foreground area, the search range of action recognition can be narrowed, thereby reducing the amount of calculation and improving the processing speed. In surveillance videos, real-time performance is very important. This approach can shorten the response time of action recognition and enable the system to respond faster. In practical applications, monitoring scenes are often complex and changeable. By combining the human detection frame and foreground area, the adaptability of the action recognition method to complex scenes can be enhanced and the stability of the recognition results can be improved. By first determining the human detection frame and foreground area, the impact of factors such as lighting changes and camera shake on action recognition can be reduced to a certain extent, and the anti-interference ability of action recognition can be improved. By continuously performing action recognition on the area to be recognized, in-depth analysis of human behavior can be achieved, such as behavior trajectory tracking and abnormal behavior detection.

[0042] In one embodiment of the present application, for a specific method of determining the area to be identified based on the comprehensive area and the human body area, see Figure 6 The corresponding detailed description.

[0043] S24, determining an action recognition result based on the image information of the area to be recognized.

[0044] In one embodiment of the present application, the image information in the area to be identified can be used to characterize the area where people and foreground exist in the video frame of the surveillance video, so the action recognition result can be determined based on the image information in the area to be identified. Specifically, the image information in the area to be identified can be input into the third recognition model to obtain the action recognition result output by the third recognition model. Among them, the third recognition model can be any artificial intelligence model with action recognition function. For example, the third recognition model can be a YOWO model (You Only Watch Once) or a SlowFast model (SlowFast model), which is not limited in this application.

[0045] Exemplarily, the SlowFast model includes two network branches, namely a slow network and a fast network, which are used to process the static and dynamic content in the area to be identified, respectively. Specifically, the slow network is used to process high-resolution video frames to extract static environmental information in the video. The fast network is used to process low-resolution video frames to extract dynamic motion information in the video. The slow network samples video frames in the surveillance video at a lower frame rate and uses a convolution operation with more layers to extract spatial features in the video frames. The fast network samples video frames based on a smaller time span (e.g., 1 millisecond) so that more video frames can be collected in each sampling period (e.g., 1 second). In the SlowFast model, the features output by the slow network and the fast network can be fused, so that the SlowFast model can simultaneously utilize the global information of the slow network and the local information of the fast network to further improve the accuracy of video understanding. This ensures that the SlowFast model can simultaneously focus on the static and dynamic content of the video, thereby achieving more accurate action recognition.

[0046] It can be seen from the above technical solutions that when the embodiment of the present application performs action recognition on the surveillance video, the human body detection frame in the video frame is first determined, and then the foreground area in the video frame is determined, so that the area containing the human body can be preliminarily screened out, thereby avoiding the misidentification of irrelevant background. Combined with the foreground area, the interference of static background or irrelevant dynamic objects can be further eliminated, thereby improving the accuracy of action recognition. And the combined use of the human body detection frame and the foreground area can achieve accurate positioning of the target human body. It is helpful to accurately identify the actions of specific people in complex monitoring scenes. The system's adaptability to complex scenes can be enhanced and the stability of recognition can be improved. In addition, by first determining the human body detection frame and the foreground area, the search range of action recognition can be narrowed, thereby reducing the amount of calculation and improving the processing speed. And the response time of action recognition can be shortened, so that the system can respond faster. Finally, the area to be identified that needs to be action recognized is jointly determined according to the foreground area and the human body detection frame, which is convenient for associating the video data with the subsequent action recognition results to form a complete monitoring data chain. Thereby improving the accuracy of action recognition.

[0047] like Figure 4 , is a flow chart of a method for determining a preset duration provided by an embodiment of the present application. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted. The method for determining a preset duration provided by an embodiment of the present application includes the following steps.

[0048] S30, performing image enhancement processing on the pre-stored first sample image to obtain a second sample image.

[0049] In one embodiment of the present application, in order to improve the quality of the image input to the first recognition model, the pre-stored first sample image may be enhanced to obtain a second sample image. The second sample image has a higher resolution than the first sample image. The image enhancement process includes interpolation, super-resolution reconstruction, image transformation, etc., which are not limited in the present application.

[0050] Exemplarily, processing the first image according to the interpolation algorithm includes: obtaining new pixel values ​​by calculation or prediction based on the continuity or correlation of the image pixel values ​​in space, and improving the resolution of the first image by inserting new pixel values ​​between the pixels of the first image. For example, the interpolation algorithm can be a nearest neighbor interpolation algorithm, and the value of the new pixel directly adopts the value of the known pixel closest to it. The bilinear interpolation algorithm linearly interpolates the values ​​of the upper and lower and left and right pixels of the target pixel and takes the weighted average. The bicubic interpolation algorithm considers the grayscale values ​​of the 16 pixels around the target pixel and performs interpolation calculations through a cubic polynomial. The spline interpolation algorithm performs interpolation calculations by constructing a piecewise polynomial function so that the interpolation curve smoothly transitions at the known pixel points. Through the processing of the interpolation algorithm, the first image with a lower resolution can be enlarged to obtain a second image with a higher resolution, and it can ensure that the clarity and image information of the first image are maintained in the second image.

[0051] In one embodiment of the present application, when improving the resolution of a first image, the first image may be first transformed to improve the quality of the first image; then an interpolation algorithm or a super-resolution reconstruction algorithm may be used to enlarge the first image and restore its details to obtain a high-resolution second image.

[0052] S31: Based on the second sample image, using a pre-built initial recognition model, obtain a predicted detection box in the second sample image.

[0053] In one embodiment of the present application, the pre-built initial recognition model may be a convolutional neural network model, a YOLO model, etc., and the present application does not limit this. Exemplarily, when the initial recognition model is a YOLO model, using the pre-built initial recognition model, obtaining the predicted detection frame in the second sample image includes: dividing the second image into multiple grids. For example, the second image can be divided into multiple 7x7 grids, and the second image can also be divided into multiple grids of different sizes (for example, 13x13, 26x26, 52x52) to identify targets of different sizes in the second image. Each grid is responsible for predicting B bounding boxes. These bounding boxes contain the location information (x, y, w, h) and confidence of the target. Among them, x and y are the coordinates of the center point of the bounding box, and w and h are the width and height of the bounding box. The confidence is used to characterize the probability of containing the target in the bounding box and the accuracy of the bounding box. The processed bounding box and the corresponding category probability are output as the final detection result, and displayed in the image in the form of a rectangular box, while marking the category and confidence of the target.

[0054] S32, determining the loss value of the initial recognition model according to the predicted detection box and the label detection box in the second sample image, and updating the initial recognition model using a back propagation algorithm until the loss value meets a preset condition, thereby obtaining a trained first recognition model.

[0055] In one embodiment of the present application, the loss value of the initial recognition model is determined according to the predicted detection frame and the label detection frame in the second sample image, and the model performance can be evaluated. Among them, the loss value can characterize the gap between the predicted result output by the first recognition model and the actual label. When the loss value is small, it means that the gap between the predicted detection frame and the label detection frame is small, and the overlap between the predicted detection frame and the label detection frame is high, that is, the result predicted by the initial recognition model is more accurate. According to the change of the loss value, the performance of the initial recognition model at different training stages can be judged, and the optimization strategy can be adjusted accordingly (for example, adjusting the learning rate, batch size, etc.). During the model training process, the loss value is used as the input of the gradient descent algorithm to guide the update direction of the model parameters. By continuously reducing the loss value through the back propagation algorithm, the model can gradually learn the ability to extract effective features from the input image and make accurate predictions. Calculating the loss value can ensure that the parameters in the model pay more attention to the features related to target recognition in the image during the iterative update process, thereby improving the generalization ability and robustness of the model. During the training process, the first recognition model can dynamically adjust the attention to targets of different scales according to the change of the loss value, thereby improving the robustness of action recognition in complex scenes.

[0056] In one embodiment of the present application, determining the loss value of the initial recognition model includes: determining the difference between the label detection frame and the predicted detection frame according to the first parameter of the label detection frame and the second parameter of the predicted detection frame; wherein the first parameter includes the first center point coordinates, the first width and the first height of the label detection frame, and the second parameter includes the second center point coordinates, the second width and the second height of the predicted detection frame; determining the loss value according to the difference and the area of ​​the label detection frame. Specifically, the calculation method of the loss value of the initial recognition model satisfies the following relationship: ; Among them, Loss represents the loss value of the initial recognition model, exp represents the natural exponential function, and cx a Represents the horizontal coordinate of the first center point, cy a Represents the ordinate of the first center point, w a / 2 represents the first width, h a / 2 represents the first width, cx b Represents the horizontal coordinate of the second center point, cy b Represents the ordinate of the second center point, w b / 2 represents the second width, h b / 2 represents the second width, T represents the transposed vector, and S represents the mean area of ​​all label detection boxes in the first image.

[0057] like Figure 5 , is a flow chart of a method for training a first recognition model provided in an embodiment of the present application. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted. The method for training a first recognition model provided in an embodiment of the present application includes the following steps.

[0058] S40: Determine a ratio of an area of ​​a foreground region of each first image in the plurality of first images to an area of ​​the first image.

[0059] In one embodiment of the present application, in order to improve the confidence of the comprehensive area determined based on multiple foreground areas, the ratio of the foreground area of ​​each first image to the area of ​​the first image can be determined first. The larger the ratio, the higher the proportion of the area occupied by the foreground area in the image, the more image information the foreground area contains, and the more reliable the action recognition result is when the action recognition is performed based on the image information in the foreground area.

[0060] Exemplarily, a first number of pixels occupied by a foreground area in a first image may be determined, and a second number of all pixels in the first image may be determined, and a ratio of the first number to the second number may be determined as a ratio of an area of ​​the foreground area to an area of ​​the first image.

[0061] S41, determining a candidate image from the plurality of first images according to the time when each first image is acquired.

[0062] In one embodiment of the present application, in order to identify the action of a target in a surveillance video, multiple first images adjacent in the time dimension may be obtained as candidate images, and then a comprehensive area may be determined based on the foreground area in the multiple candidate images, so as to ensure that the image information in the comprehensive area has a correlation in the time dimension, thereby improving the accuracy of subsequent action recognition. Specifically, the candidate image may be determined from the multiple first images based on the time at which each first image is obtained. For example, if the time difference between each two adjacent video frames is 1 millisecond, a video frame with a time interval of 5 milliseconds may be determined as a candidate image, and a video frame with a time interval of 10 milliseconds may also be determined as a candidate image, and the present application does not limit this.

[0063] S42, when the ratio corresponding to the candidate image meets a preset condition, merging the foreground areas of the plurality of first images to obtain a comprehensive area.

[0064] In one embodiment of the present application, in order to ensure that the integrated area generated based on the foreground area of ​​the candidate image can contain more image information in the first image, it is possible to first determine whether the ratio corresponding to the candidate image meets the preset conditions. When the ratio corresponding to the candidate image meets the preset conditions, it indicates that the integrated area generated based on the fusion of the foreground area of ​​the candidate image contains more image information in the first image.

[0065] In one embodiment of the present application, the preset condition includes: the number of candidate images corresponding to the candidate images whose ratio is greater than the first threshold reaches the second threshold. Exemplarily, the first threshold may be 0.5, and the second threshold may be 6. When the number of candidate images whose ratio is greater than 0.5 is greater than or equal to 6, the foreground areas in the candidate images may be merged to obtain a comprehensive area.

[0066] like Figure 6 , is a flow chart of a method for determining a comprehensive area provided in an embodiment of the present application. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted. The method for determining a comprehensive area provided in an embodiment of the present application includes the following steps.

[0067] S50, determining candidate regions in the plurality of first images based on an intersection of the comprehensive region and the human body region.

[0068] In one embodiment of the present application, in order to improve the accuracy of action recognition of a target (e.g., a pedestrian) in a surveillance video, a candidate region in the first image may be determined based on the intersection of the comprehensive region and the human region in any first image. In this way, the candidate region includes image information of the human region in the first image and image information of the foreground region, and action recognition is subsequently performed based on the image information of the candidate region, which can improve the accuracy of action recognition.

[0069] S51, obtaining a region to be identified based on the candidate region.

[0070] In one embodiment of the present application, after obtaining a candidate area including image information of a human body area and image information of a foreground area in a first image, in order to further improve the accuracy of action recognition, an area to be recognized can also be obtained based on the candidate area. Specifically, the area of ​​the candidate area can be expanded according to a preset expansion ratio to obtain the area to be recognized, thereby ensuring that the area to be recognized includes more image information, thereby utilizing more image information for action recognition; the area of ​​the candidate area can also be reduced according to a preset reduction ratio to obtain the area to be recognized, thereby reducing the amount of image information in the area to be recognized, thereby improving the efficiency of action recognition, and removing redundant image information contained in the area to be recognized, thereby improving the accuracy of action recognition. Exemplarily, when the preset expansion ratio is 20%, it can be determined that the distance from the boundary to the center of the area to be recognized is 1.2 times the distance from the boundary to the center of the candidate area.

[0071] See also Figure 7 , Figure 7 6 is a functional module diagram of a motion recognition device provided in an embodiment of the present application. A motion recognition device 61 includes a detection module 610, a first recognition module 611, and a second recognition module 612. The module / unit referred to in the present application refers to a series of computer-readable instruction segments that can be executed by the processor 13 and can complete fixed functions, which are stored in the memory 12. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0072] The detection module 610 is used to identify human body regions in a plurality of first images based on a pre-trained first recognition model.

[0073] The first recognition module 611 is used to recognize the foreground area in the multiple first images based on the second recognition model.

[0074] The first recognition module 611 is further used to determine a comprehensive area corresponding to the multiple first images according to the foreground areas of the multiple first images; wherein the comprehensive area is used to represent an area containing action information in each first image.

[0075] The first recognition module 611 is further used to determine a region to be recognized based on the comprehensive region and the human body region; wherein the region to be recognized is used to represent a region in each first image that contains human body information and action information.

[0076] The second recognition module 612 is used to determine the action recognition result based on the image information of the area to be recognized.

[0077] In some embodiments, the detection module 610 is also used to perform image enhancement processing on a pre-stored first sample image to obtain a second sample image; based on the second sample image, using a pre-built initial recognition model, obtain a predicted detection box in the second sample image; according to the predicted detection box and the label detection box in the second sample image, determine the loss value of the initial recognition model, and use a back propagation algorithm to update the initial recognition model until the loss value meets a preset condition to obtain a trained first recognition model.

[0078] In some embodiments, the detection module 610 determines the loss value of the initial identification model including: determining the difference between the label detection frame and the predicted detection frame based on the first parameter of the label detection frame and the second parameter of the predicted detection frame; wherein the first parameter includes the first center point coordinates, the first width and the first height of the label detection frame, and the second parameter includes the second center point coordinates, the second width and the second height of the predicted detection frame; and determining the loss value based on the difference and the area of ​​the label detection frame.

[0079] In some embodiments, the first recognition module 611 determines the integrated area corresponding to the multiple first images based on the foreground area of ​​the multiple first images, including: determining the ratio of the foreground area of ​​each first image in the multiple first images to the area of ​​the first image; determining candidate images from the multiple first images based on the time when each first image is acquired; and merging the foreground areas of the multiple first images to obtain the integrated area when the ratio corresponding to the candidate images meets a preset condition.

[0080] In some embodiments, the first recognition module 611 determines the area to be recognized based on the comprehensive area and the human body area, including: determining candidate areas in the multiple first images based on the intersection of the comprehensive area and the human body area; and obtaining the area to be recognized based on the candidate areas.

[0081] See also Figure 8, is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 100 includes a memory 12 and a processor 13. The memory 12 is used to store computer-readable instructions, and the processor 13 is used to execute the computer-readable instructions stored in the memory to implement an action recognition method described in any of the above embodiments.

[0082] In an embodiment of the present application, the electronic device 100 further includes a bus, a computer program stored in the memory 12 and executable on the processor 13, such as a motion recognition program.

[0083] Figure 8 Only the electronic device 100 having the memory 12 and the processor 13 is shown, and those skilled in the art can understand that Figure 8 The structure shown does not constitute a limitation on the electronic device 100 , and the electronic device 100 may include fewer or more components than shown in the figure, or combine some components, or arrange the components differently.

[0084] Combination Figure 2 The memory 12 in the electronic device 100 stores a plurality of computer-readable instructions to implement an action recognition method, and the processor 13 can execute the plurality of instructions to implement: identifying human body areas in a plurality of first images based on a pre-trained first recognition model; identifying foreground areas in the plurality of first images based on a second recognition model; determining a comprehensive area corresponding to the plurality of first images according to the foreground areas of the plurality of first images; wherein the comprehensive area is used to characterize an area containing action information in each first image; determining an area to be recognized according to the comprehensive area and the human body area; wherein the area to be recognized is used to characterize an area containing human body information and action information in each first image; and determining an action recognition result based on image information of the area to be recognized.

[0085] Specifically, the specific implementation method of the processor 13 for the above instructions can refer to Figure 2 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0086] Those skilled in the art will appreciate that the schematic diagram is merely an example of the electronic device 100 and does not constitute a limitation on the electronic device 100. The electronic device 100 may be a bus-type structure or a star-type structure. The electronic device 100 may also include more or less other hardware or software than shown in the diagram, or a different arrangement of components. For example, the electronic device 100 may also include input and output devices, network access devices, etc.

[0087] It should be noted that the electronic device 100 is only an example, and other existing or future electronic products that are suitable for the present application should also be included in the protection scope of the present application and included here by reference.

[0088] Among them, the memory 12 includes at least one type of readable storage medium, and the readable storage medium can be non-volatile or volatile. The readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the electronic device 100, such as a mobile hard disk of the electronic device 100. In other embodiments, the memory 12 can also be an external storage device of the electronic device 100, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the electronic device 100. The memory 12 can not only be used to store application software and various types of data installed in the electronic device 100, such as a code of a motion recognition program, etc., but also can be used to temporarily store data that has been output or is to be output.

[0089] In some embodiments, the processor 13 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor 13 is the control core (Control Unit) of the electronic device 100, and uses various interfaces and lines to connect various components of the entire electronic device 100, and executes various functions and processes data of the electronic device 100 by running or executing programs or modules stored in the memory 12 (for example, executing a motion recognition program, etc.), and calling data stored in the memory 12.

[0090] The processor 13 executes the operating system of the electronic device 100 and various installed applications. The processor 13 executes the applications to implement the steps in each of the above-mentioned motion recognition method embodiments, for example Figure 2 Steps shown.

[0091] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present application. The one or more modules / units may be a series of computer-readable instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the electronic device 100. For example, the computer program may be divided into a detection module 610, a first recognition module 611, and a second recognition module 612.

[0092] The above-mentioned integrated unit implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to execute a part of a motion recognition method described in each embodiment of the present application.

[0093] If the module / unit integrated in the electronic device 100 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also instruct the relevant hardware devices to complete through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented.

[0094] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory and other memory, etc.

[0095] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0096] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. Figure 8 Only one arrow is used in the figure, but it does not mean that there is only one bus or one type of bus. The bus is configured to realize the connection and communication between the memory 12 and at least one processor 13, etc.

[0097] An embodiment of the present application also provides a computer-readable storage medium (not shown), in which computer-readable storage medium is stored computer-readable instructions, and the computer-readable instructions are executed by a processor in an electronic device to implement the action recognition method described in any of the above embodiments.

[0098] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0099] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0100] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0101] In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the specification can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any specific order.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application has been described in detail with reference to the preferred embodiments, a person of ordinary skill in the art should understand that the technical solution of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present application.

Claims

1. A motion recognition method, applied to an electronic device, characterized in that: The method comprises: Identify human body regions in a plurality of first images based on a pre-trained first recognition model; identifying foreground areas in the plurality of first images based on a second recognition model; Determine, according to the foreground areas of the plurality of first images, a comprehensive area corresponding to the plurality of first images; wherein the comprehensive area is used to characterize an area containing action information in each first image; Determine a region to be identified according to the comprehensive region and the human body region; wherein the region to be identified is used to characterize a region containing human body information and action information in each first image; Based on the image information of the area to be recognized, an action recognition result is determined.

2. The motion recognition method according to claim 1, characterized in that: The method for training the first recognition model comprises: Performing image enhancement processing on the pre-stored first sample image to obtain a second sample image; Based on the second sample image, using a pre-built initial recognition model, obtaining a predicted detection box in the second sample image; According to the predicted detection box and the label detection box in the second sample image, the loss value of the initial recognition model is determined, and the initial recognition model is updated using a back propagation algorithm until the loss value meets a preset condition, thereby obtaining a trained first recognition model.

3. The motion recognition method according to claim 2, characterized in that: Determining the loss value of the initial model includes: Determine the difference between the label detection frame and the predicted detection frame according to the first parameter of the label detection frame and the second parameter of the predicted detection frame; wherein the first parameter includes the first center point coordinates, the first width and the first height of the label detection frame, and the second parameter includes the second center point coordinates, the second width and the second height of the predicted detection frame; The loss value is determined according to the difference and the area of ​​the label detection box.

4. The motion recognition method according to claim 1, characterized in that: The determining, according to the foreground areas of the plurality of first images, the integrated areas corresponding to the plurality of first images comprises: determining a ratio of a foreground region of each first image in the plurality of first images to an area of ​​the first image; determining a candidate image from the plurality of first images according to a time at which each first image was acquired; When the ratio corresponding to the candidate images meets a preset condition, the foreground areas of the multiple first images are merged to obtain a comprehensive area.

5. The motion recognition method according to claim 4, characterized in that: The preset conditions include: The number of candidate images whose ratio is greater than the first threshold reaches a second threshold.

6. The motion recognition method according to claim 1, characterized in that: Determining the area to be identified according to the comprehensive area and the human body area comprises: Determine candidate regions in the plurality of first images based on an intersection of the comprehensive region and the human body region; Based on the candidate regions, a region to be identified is obtained.

7. The motion recognition method according to claim 6, characterized in that: The obtaining of the to-be-identified region based on the candidate region comprises: The area of ​​the candidate region is expanded according to a preset expansion ratio to obtain the region to be identified.

8. A motion recognition device, characterized in that: The device comprises: A detection module, configured to identify a human body region in a plurality of first images based on a pre-trained first recognition model; A first recognition module, configured to recognize a foreground area in the plurality of first images based on a second recognition model; The first recognition module is further used to determine the integrated area corresponding to the multiple first images according to the foreground area of ​​the multiple first images; wherein the integrated area is used to represent the area containing action information in each first image; The first recognition module is further used to determine a region to be recognized based on the comprehensive region and the human body region; wherein the region to be recognized is used to represent a region in each first image that contains human body information and action information; The second recognition module is used to determine the action recognition result based on the image information of the area to be recognized.

9. An electronic device, characterized in that: The electronic device comprises a processor and a memory, and the processor is used to implement the action recognition method as claimed in any one of claims 1 to 7 when executing a computer program stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the action recognition method according to any one of claims 1 to 7 is implemented.

Images