Action recognition method and apparatus

Abstract

The application provides a motion recognition method and device, which can recognize motion information and optical flow information of a target object in an image based on the image, and determine the motion of the target object according to the motion information and the optical flow information, has high accuracy when recognizing the standing and sitting motions of students, reduces the error switching of the shooting mode caused by the false detection of the recording and broadcasting equipment, and thus can improve the use effect of the recording and broadcasting equipment and the viewing experience of the recorded video data.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an action recognition method and apparatus. Background Technology

[0002] With the continuous development of electronic technology and the increasing emphasis on education, information-based teaching through electronic devices has become a new trend. One of the most common ways of information-based teaching is to record the teacher's lectures in the classroom using recording equipment and other devices, so that students who are not in the classroom can watch the lectures through live or recorded audio and video data.

[0003] One important function that recording equipment needs to have is the ability to recognize student actions, mainly involving the actions of students standing up and sitting down in the classroom. When a student stands up to answer a question, the recording equipment should be able to recognize this action and capture a close-up video of that student. Then, when the student sits down after answering the question, the recording equipment should be able to recognize this action and stop capturing the close-up video of that student, switching to capturing video data of the entire classroom.

[0004] With existing technology, the recording equipment has a low accuracy rate in recognizing students' actions of standing up and sitting down. This makes it impossible for the recording equipment to accurately collect video data of students standing up to answer questions, reducing the effectiveness of the recording equipment and the viewing experience of the recorded video data. Summary of the Invention

[0005] This application provides an action recognition method and apparatus to solve the problem that the accuracy of recording and broadcasting equipment in recognizing students' actions of standing up and sitting down is low, thereby improving the usability of the recording and broadcasting equipment and the viewing experience of the recorded video data.

[0006] A first aspect of this application provides an action recognition method, comprising: acquiring a first image including a target object; determining action information of the target object in the first image; calculating the optical flow type of the target object in the first image; wherein the optical flow type includes: motion type, stationary type, or no optical flow type; and determining the action of the target object based on the action information and the optical flow type.

[0007] In one embodiment of the first aspect of this application, the target object is a student in a classroom; the actions of the target object include standing up and sitting down.

[0008] In one embodiment of the first aspect of this application, determining the motion information of the target object in the first image includes: inputting the first image into a machine learning model to obtain the motion information of the target object output by the machine learning model, and the feature region of the target object in the first image.

[0009] In one embodiment of the first aspect of this application, the machine learning model includes: a spatial attention module for extracting spatial correlation features between the edge of the target object and surrounding objects in the first image; a cross-stage portion module (CSP) for providing the base network Backbone of the machine learning model and extracting Backbone features from the first image; a feature pyramid network module (FPN) for fusing the Backbone features extracted by the CSP with the first deep features of the Backbone network to obtain multi-scale features; a path enhancement network module (PAN) for fusing the multi-scale features obtained by the FPN with the second deep features of the Backbone network; and a detection head module (YOLO) for learning and parsing the action information of the target object, the feature region, and confidence information in the first image based on the features fused by the FPN and PAN.

[0010] In one embodiment of the first aspect of this application, calculating the optical flow type of the target object in the first image includes: determining the optical flow length and optical flow direction of multiple feature points within a feature region of the target object in the first image; determining the optical flow type of the target object based on the optical flow length and optical flow direction of multiple feature points in the first image and the optical flow length and optical flow direction of multiple feature points in the second image; wherein the second image is an image of the frame preceding the first image.

[0011] In one embodiment of the first aspect of this application, determining the optical flow type of the target object based on the optical flow length and optical flow direction of multiple feature points in the first image and multiple feature points in the second image includes: determining, from the multiple feature points, a first feature point in the first image and the second image where the difference in optical flow length in the horizontal direction is less than the difference in optical flow length in the vertical direction, and the difference in optical flow value in the vertical direction is less than a negative number of a first preset threshold; and determining, from the multiple feature points, a feature point in the first image and the second image where the difference in optical flow length in the horizontal direction is less than the difference in optical flow length in the vertical direction, and the difference in optical flow value in the vertical direction is greater than a first preset threshold. The second feature point; from the plurality of feature points, the third feature point is determined where the difference in optical flow length in the vertical direction between the first image and the second image is less than the difference in optical flow length in the horizontal direction, and the difference in optical flow value in the horizontal direction is greater than a first preset threshold; when the sum of the number of the first feature point, the second feature point, and the third feature point is greater than the second preset threshold, the optical flow of the target object is determined to be of the motion type; when the sum of the number of the first feature point, the second feature point, and the third feature point is less than the second preset threshold, the optical flow of the target object is determined to be of the stationary type; when it is determined that the first image does not include the feature point, the optical flow of the target object is determined to be of the no-optical-flow type.

[0012] In one embodiment of the first aspect of this application, determining the action of the target object based on the action information and the optical flow type includes: when the action information of the target object in the first image indicates that the action of the target object is standing up, and the optical flow information of the target object in the first image is a motion type, and the action of the target object in the second image is sitting down, then the action of the target object in the first image is determined to be standing up; when the action information of the target object in the first image indicates that the action of the target object is standing up, and the optical flow information of the target object in the first image is not a motion type, and the action of the target object in the second image is standing up, then the action of the target object in the first image is determined to be standing up; when the action information of the target object in the first image indicates that the action of the target object is sitting down, and the optical flow information of the target object in the first image is a motion type, then the action of the target object in the first image is determined to be standing up; when the action information of the target object in the first image indicates that the action of the target object is sitting down, and the optical flow information of the target object in the first image is not a motion type, then the action of the target object in the first image is determined to be sitting down.

[0013] In one embodiment of the first aspect of this application, it further includes: using a queue to record the identification information of the target object and the action of the target object in each of several consecutive frames.

[0014] In one embodiment of the first aspect of this application, the method further includes: when it is determined that the action of the target object is to stand up, controlling the camera device to switch to a first shooting mode; and when it is determined that the action of the target object is to sit down, controlling the camera device to switch to a second shooting mode.

[0015] A second aspect of this application provides an action recognition device for performing the method provided in the first aspect of this application. The device includes: an acquisition module for acquiring a first image including a target object; an action determination module for determining action information of the target object in the first image; an optical flow determination module for calculating the optical flow type of the target object in the first image; wherein the optical flow type includes: a motion type, a stationary type, or a no-optical-flow type; and an action determination module for determining the action of the target object based on the action information and the optical flow information.

[0016] In summary, the action recognition method and apparatus provided in this application can identify the action information and optical flow information of target objects in images based on images, and determine the action of the target object based on the action information and optical flow information. It has high accuracy in recognizing students' standing and sitting actions, reduces the error switching of shooting modes caused by false detection of recording equipment, and thus improves the use effect of recording equipment and the viewing experience of recorded video data. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram illustrating the application scenario of this application;

[0019] Figure 2 A flowchart illustrating an embodiment of the action recognition method provided in this application;

[0020] Figure 3 A schematic diagram of the structure of an embodiment of the machine learning model provided in this application;

[0021] Figure 4 A schematic diagram of the structure of an embodiment of the spatial attention network provided in this application;

[0022] Figure 5 A schematic diagram of an embodiment of the optical flow type provided in this application;

[0023] Figure 6 A schematic diagram of the first and second images provided in this application;

[0024] Figure 7 This application provides a schematic diagram of the optical flow within the special diagnostic area. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] Figure 1 This diagram illustrates the application scenario of this application. With the continuous development of electronic technology and increasing emphasis on education, information-based teaching via electronic devices has become a new trend, enabling students in different regions to enjoy high-quality educational resources. One of the most common methods of information-based teaching is recording the teacher's lectures in the classroom using recording equipment, allowing students not in the classroom to view the recorded audio and video content. For example, in… Figure 1 In the scenario depicted, teacher A is lecturing to multiple students B in a classroom. The recording equipment 10 can record both teacher A's lecture and student B's interactions. This application is applicable to typical classroom environments, such as classrooms with a length b of 7 meters, a width a of 9 meters, and a height of 3.8 meters.

[0028] In some embodiments, such as Figure 1One important function that the recording equipment 10 in the scenario needs to have is the recognition of students' actions, mainly involving the actions of students standing up and sitting down in the classroom. When a student stands up to answer a question, the recording equipment 10 can recognize that the student's action is standing up and can collect close-up video data of that student. Then, when the student sits down after answering the question, the recording equipment 10 can also recognize that the student's action is sitting down, and can stop collecting close-up video data of the student and switch to collecting video data of the entire classroom.

[0029] In some embodiments, the recording device 10 typically employs image recognition-based methods to identify student actions in the classroom. For example, the recording device 10 uses machine learning models and other methods to identify student actions in images captured in real time within the classroom. However, existing single-method image-based action recognition suffers from low accuracy in recognizing student actions such as standing up and sitting down. Figure 1 When applied in the recording and broadcasting scenario shown, it is impossible to accurately collect video data of students standing up to answer questions, which reduces the effectiveness of the recording and broadcasting equipment and the viewing experience of the video data recorded by the equipment.

[0030] Therefore, embodiments of this application provide an action recognition method and apparatus, which can be applied to, for example... Figure 1 In the scenario shown, during recording, the recording device 10 identifies the motion information and optical flow information of the target object in the image based on the image, and determines the motion of the target object based on the motion information and optical flow information. This improves the accuracy of the recording device in recognizing students' standing and sitting actions, enhances the usability of the recording device, and improves the viewing experience of the recorded video data. The technical solution of this application will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0031] Figure 2 A flowchart illustrating an embodiment of the action recognition method provided in this application is shown below. Figure 2 The method shown can be applied to, for example, Figure 1 The scenario shown depicts a recording device that records a teacher's lecture in a classroom, and this recording is performed by the recording device 10 set up in the scenario. Specifically, as... Figure 2 The action recognition methods shown include:

[0032] S101: Obtain the first image including the target object.

[0033] Specifically, in this embodiment, the recording and broadcasting equipment, which is the main body of the execution, can take each consecutive frame of the video data collected in real time as the first image during the recording process, and identify the action of the target object in the first image. The target object can be a teacher or student in the classroom. In this embodiment, the target object is a student as an example. The actions that a student can perform in the classroom include: standing up and sitting down.

[0034] In some embodiments, the recording device may also use preset frames from the acquired video data at intervals as the first image, or the recording device may also start recognizing the action of the target object in the currently acquired first image according to received external instructions or internal instructions of the application running in the recording device.

[0035] It is understood that when the target object is a student in the classroom, the first image of the classroom captured by the recording device may include multiple students. In the subsequent embodiments of this application, the processing method of one target object (i.e., one student) in the first image is used as an example for illustration. The processing method of other target objects in the image is the same, so it will not be described again.

[0036] S102: Determine the motion information of the target object in the first image.

[0037] In some embodiments, the recording device may have a built-in machine learning model. After acquiring the first image in S101, the first image is input into the machine learning model, which identifies the target object in the first image and outputs the target object's action information, as well as the specific location of the target object's feature region in the first image. The action information can be used to indicate the target object's action; for example, it can indicate whether a student is standing up or sitting down.

[0038] In some embodiments, before inputting the first image into the machine learning model, the recording device may perform regular cropping (Resize) on the first image, resize the first image according to the requirements of the machine learning model, and then input the resized first image into the machine learning model.

[0039] In some embodiments, Figure 3 This is a schematic diagram of the structure of an embodiment of the machine learning model provided in this application, as shown in... Figure 3The machine learning model shown includes the following modules: Modified Spatial Attention Module (MSAM), Cross Stage Partial (CSP), Feature Pyramid Network (FPN), Path Aggregation Network (PAN), Convolution (Conv), and YOLO detection head module.

[0040] Specifically, when the first image is... Figure 3 After the input image shown is input into the machine learning model, in step S1, the first image is processed by a spatial attention network to extract the spatial correlation features between the edge of the target object and surrounding objects in the first image. The spatial attention module is specifically designed for images captured by recording equipment, where students are typically seated or have a table in front of them. This allows the subsequent model to better integrate the features of the student's environment to identify the student's behavior. For example, assuming the student is sitting in a chair with the chair back behind them in the first image, the subsequent model can determine that the student's action is more likely to be sitting down and less likely to be standing up, based on the positional relationship between the chair and the student.

[0041] In some embodiments, Figure 4 This is a schematic diagram of the structure of an embodiment of the spatial attention network provided in this application. The left side represents the network input, and the right side represents the network output. CBL-N stands for Convolution+Batchnorm+LeakyReLU, where N represents the output channel dimension of the convolutional layer. The three digits after CBL-N represent Kernel Size, Stride, and Pad, respectively. Eltwise refers to Eltwise (SUM), which is the sum of pixels at corresponding positions on corresponding channels of two feature maps. The embodiments provided in this application... Figure 4 The spatial attention model shown does not lose accuracy compared to the traditional SAM, but has faster inference speed and is easier to deploy.

[0042] After the spatial attention module completes the processing of the first image in S1, it continues to transmit the first image to the next network, where the CSP processes the first image in S2. The CSP can provide the network's backbone and extract the backbone features of the first image, thereby improving the learning ability of the machine learning model while reducing the amount of parameter calculation for the first image in the network.

[0043] Subsequently, FPN and PAN can jointly extract and process feature information of the first image at multiple scales. Specifically, FPN can be used in S3 to fuse the backbone features of the first image with the first deep features of the network to enhance the expressive power of the feature map and obtain multi-scale features. Then, PAN can be used in S4 to fuse the multi-scale features obtained by FPN with the second deep features of the backbone network. The layer containing the second deep features is deeper than the layer containing the first deep features, thereby further enhancing the expressive power of the feature map. In some embodiments, the FPN and PAN provided in this application perform depth compression on the required scales, greatly reducing the number of model parameters. Figure 3 The parameter m is the width factor, which can take values ​​of 1, 0.75, 0.5, 0.25, etc. The specific value can be selected according to the specific computing power of the recording and broadcasting equipment's chip.

[0044] Finally, the feature information fused from FPN and PAN is learned and parsed in S6 by three YOLO detection head modules labeled ①-③ to extract the action information label of the target object in the first image, the coordinates of the top-left and bottom-right vertices of the rectangular feature region of the target object in the first image ((point_left, point_top) and (point_right, point_bottom)), and the confidence score of the action information label. The action information label can correspond to at least two values, used to indicate whether the student in the first image is standing or sitting.

[0045] In some embodiments, the feature information after the fusion of FPN and PAN can be further processed by the convolution module in S5, and then learned and parsed by the YOLO detection head module.

[0046] In some embodiments, when the first image includes multiple target objects, the machine learning network will output the action information, feature regions, etc. of each target object. At this time, a queue viou_tracker can be used to store the information of all target objects in the first image, which is beneficial for subsequent tracking processing.

[0047] S103: Calculate the optical flow information of the target object in the first image.

[0048] In this embodiment, the execution order of S103 and S102 is not limited; they can be executed sequentially or simultaneously.

[0049] In some embodiments, after the recording and broadcasting device, as the execution subject, acquires the first image in S101, it can first perform regular cropping (Resize) processing on the first image to accelerate the subsequent optical flow extraction speed. Subsequently, the first image is converted to grayscale, and FAST feature point extraction is performed on the grayscale image. The feature points can also be called corner points, and the acquired corner points are denoted as pre_points.

[0050] In some embodiments, since calculating optical flow information requires reference to images preceding it in time sequence, if the first image is the first frame acquired by the recording device, then the optical flow information of that image is not extracted. However, if the first image is not the first frame acquired by the recording device, subsequent optical flow calculations can continue. For example, optical flow can be extracted based on the acquired corner points pre_points using an LK pyramid method to obtain the current corner points cur_points. This application does not limit the specific definition and calculation method of optical flow; reference can be made to existing technologies. Specifically, after obtaining the optical flow, this application classifies the optical flow information according to its direction and length, dividing it into motion type, stationary type, and no optical flow type. The specific definition of the above-mentioned optical flow information proposed in this application will be explained below with reference to the accompanying drawings.

[0051] Figure 5 This is a schematic diagram of an embodiment of the optical flow type provided in this application, wherein, as Figure 5 This paper demonstrates how, based on the characteristics of student movement in a classroom setting, the optical flow changes reflected in the image are divided into upper, lower, left, and right directions. Specifically, the movement of students when standing up or sitting down exhibits the following characteristics: movement to the left followed by a "stationary" position; movement to the right followed by a "stationary" position; movement upwards followed by a "stationary" position; and movement downwards followed by a "stationary" position. Therefore, considering that students' movements during standing up, sitting down, moving left, or moving right are accompanied not only by the corresponding movement trend but also by other forms of movement, this application does not directly determine the student's movement based on the length and direction of the optical flow in the first image. Instead, it further combines the changes in optical flow in the first and second images to define the type of optical flow change in the image, denoted as optical flow information. Subsequently, the optical flow information, combined with motion information, can be used to jointly determine the student's movement.

[0052] Specifically, such as Figure 5In the scenario shown, optical flow types can be defined based on changes in optical flow within the feature regions of a target object in the image: moving optical flow, stationary optical flow, and no optical flow. A threshold `MOVEMENTFLOW` can be given; if an optical flow length (including four types: upward, downward, leftward, and rightward) is greater than `MOVEMENTFLOW`, this optical flow is classified as moving optical flow. Similarly, a threshold `STATICFLOW` (called the stationary optical flow length) can be given; if an optical flow length is less than this threshold, this optical flow is classified as stationary optical flow. The number of moving optical flows is called motion sensitivity, denoted as `SENSITIVITY`. For the target object in the first image, if no corner points are extracted within the feature region (the detection box of the optical flow) of the target object, the target object is said to have no optical flow, also known as no corner points, etc.

[0053] Based on such Figure 5 The definition of the optical flow type shown in S103 is that, after determining the optical flow length and direction of multiple feature points (corner points) within the feature region (detection box) of the target object in the first image, the recording and broadcasting device can determine whether the optical flow type of the target object in the first image is a motion type, a stationary type, or a no-optical-flow type based on the optical flow length and direction of these multiple feature points, combined with the optical flow length and direction of multiple feature points in the second image of the previous frame of the first image.

[0054] In some embodiments, to determine the optical flow type corresponding to the target object in the first image in S103, float type variables upNum, downNum, leftrightNum, and stopNum can be defined to determine the optical flow type based on the optical flow direction and optical flow length in the preceding and following frames. For example, Figure 6 The first image and the second image provided in this application are schematic diagrams, wherein the first image T2 is a frame currently being processed by the recording and broadcasting equipment. The optical flow direction and optical flow length of each corner point within the feature region S of the target object (student) in the first image T2 can be calculated using optical flow calculation. For example... Figure 7 This application provides a schematic diagram of the optical flow within the special diagnostic area, wherein... Figure 6 Within the feature region S shown, the direction and length of the optical flow at each corner point are identified by vectors. Subsequently, as follows... Figure 7 The optical flow at each corner point in the feature region shown is compared with the optical flow at each corner point in the feature region of the second image T1. Since the second image T1 is before the first image T2, the recording device already has the optical flow data in the second image T1.

[0055] In some embodiments, for multiple feature points within a feature region S, when it is determined that the difference in optical flow length in the horizontal direction between the first image T2 and the second image T1 is less than the difference in optical flow length in the vertical direction, and the difference in optical flow value in the vertical direction is less than a negative number of the first preset threshold MOVEMENTFLOW, the feature point is recorded as the first feature point, and the count of upNum is incremented by 1. The above process can be expressed by the formula: |cur_points[j].y - pre_points[j].y| > |cur_points[j].x - pre_points[j].x| and |cur_points[j].y - pre_points[j].y| < - MOVEMENTFLOW, then: upNum = upNum + 1.0;

[0056] When it is determined that the difference in optical flow length in the horizontal direction between the first image T2 and the second image T1 is less than the difference in optical flow length in the vertical direction, and the difference in optical flow value in the vertical direction is greater than the first preset threshold MOVEMENTFLOW, the feature point is recorded as the second feature point, and the count of downNum is incremented by 1. The above process can be expressed by the formula |cur_points[j].y-pre_points[j].y|>|cur_points[j].x-pre_points[j].x| and cur_points[j].y-pre_points[j].y>MOVEMENTFLOW, then: downNum=downNum+1.0.

[0057] When it is determined that the difference in optical flow length in the vertical direction between the first image T2 and the second image T1 is less than the difference in optical flow length in the horizontal direction, and the difference in optical flow value in the horizontal direction is greater than the first preset threshold MOVEMENTFLOW, the feature point is recorded as the third feature point, and the count of leftrightNum is incremented by 1. The above process can be expressed by the formula |cur_points[j].y-pre_points[j].y|<|cur_points[j].x-pre_points[j].x| and cur_points[j].x-pre_points[j].x)>MOVEMENTFLOW, then: leftrightNum=leftrightNum+1.0.

[0058] Finally, when the sum of the counts of the first feature point (upNum), the second feature point (downNum), and the third feature point (leftrightNum) within the feature region S, totalNum, is greater than the second preset threshold SENSITIVITY, the optical flow type corresponding to the target object in the first image is determined to be motion type. Correspondingly, when totalNum is not greater than the second preset threshold SENSITIVITY, the optical flow type corresponding to the target object in the first image is determined to be stationary type. Furthermore, when no FAST feature points are detected within the feature region, the target object in the first image is determined to have no optical flow.

[0059] S104. Based on the motion information obtained in S102 and the optical flow information obtained in S103, the motion of the target object in the first image is finally determined by combining the two.

[0060] In some embodiments, the following judgment can be made in S104 of this application based on the motion information and optical flow information of the target object:

[0061] a. When the motion information of the target object in the first image indicates that the action of the target object is to stand up, and the optical flow information of the target object in the first image is the motion type, and the action of the target object in the second image is to sit down, then the action of the target object in the first image is determined to be to stand up.

[0062] b. When the motion information of the target image in the first image indicates that the action of the target object is to stand up, and the optical flow information of the target object in the first image is not a motion type (it can be a static type or a no-optical-flow type), and the action of the target object in the second image is to stand up, then the action of the target object in the first image is determined to be to stand up.

[0063] c. If the motion information of the target object in the first image indicates that the action of the target object is to sit down, and the optical flow information of the target object in the first image is a motion type, then the action of the target object in the first image is determined to be to stand up.

[0064] d. If the motion information of the target object in the first image indicates that the action of the target object is to sit down, and the optical flow information of the target object in the first image is not a motion type, then the action of the target object in the first image is determined to be sitting down.

[0065] In some embodiments, to record the actions of target objects in consecutive frame images, the recording device can maintain a double-ended queue of length M, ActionTrackqueue, to record the action information label of the target object obtained by the machine learning model, as well as the identification information such as the target object's ID. To track each target object, two states can be defined for each object tracked by the queue: clsFlow and has_flow. The value of clsFlow corresponds to the following: 1 indicates that motion flow was detected at a certain point in the past, and 0 indicates that no motion flow was detected from the appearance of the target ID until the previous frame. Therefore, since the student's state can also be tracked and maintained through the queue in this embodiment, student behavior recognition can be achieved from the time the student stands up until they sit down. This avoids relying on a single result in a single frame image to reflect the actual action, and can recognize the student's standing action even when the student is not moving while answering a question. This overcomes the adverse effects of the length of time the student answers the question on the behavior recognition effect, further improving the user experience of the recording device in practical applications.

[0066] When using the above queue to determine the action of a target object, the following steps are included: Step 1: First, calculate the totalNum of each target object and compare it with SENSITIVITY. If totalNum is greater than SENSITIVITY, then the current clsFlow = 0 and has_flow = 1 for that target; otherwise, clsFlow = 1 and has_flow = 0. Step 2: Maintain a double-ended queue of length M, ActionTrackqueue, to obtain the action information label of the target object and the identification information such as the target object's ID from the machine learning model. Step 3: Initialize ActionTrackqueue: If ActionTrackqueue.Size is less than M, compare the last target in the queue with the target in the current frame image. Compare the has_flow state of targets with the same ID. If the has_flow of the last ID in ActionTrackqueue is 1, it means that the target was previously detected to stand up. Then, update the has_flow state of the target with the corresponding ID in the current frame to 1. That is, if a target with a certain ID was previously detected to stand up, and the next frame detects a standup, regardless of the current optical flow state of the ID, it should be in motion optical flow state. Step 4: When ActionTrackqueue.Size equals M, when a new target enters the frame, the target that entered the queue last needs to be popped. At the same time, the has_flow state of all targets entering the new frame is updated, following the same principle as initialization. Step 5: Determine whether the target object's action is standing up or sitting down according to the above ad method.

[0067] In some embodiments, after the recording device determines the action of the target object student in the first image through the above S101-S104, if the action of the student is determined to be standing up, the camera device is controlled to switch to the first shooting mode to collect close-up video data of the student; if the action of the student is determined to be sitting down, the camera device is controlled to switch to the second shooting mode to collect video data of the entire classroom.

[0068] In summary, the action recognition method provided in this application can be applied to, for example, Figure 1 In the scenario shown, when the recording equipment is recording, it identifies the motion information and optical flow information of the target object in the image based on the image, and determines the motion of the target object based on the motion information and optical flow information. It has high accuracy in recognizing the actions of students standing up and sitting down, reducing the wrong switching of shooting modes caused by false detection of the recording equipment, thereby improving the use effect of the recording equipment and the viewing experience of the recorded video data.

[0069] In the foregoing embodiments, the action recognition method provided by the embodiments of this application has been described. To implement the functions of the methods provided by the embodiments of this application, the recording and broadcasting device, as the execution subject, may include hardware structures and / or software modules, implementing the above functions in the form of hardware structures, software modules, or a combination of hardware structures and software modules. Whether a particular function is executed in the form of hardware structures, software modules, or a combination of hardware structures and software modules depends on the specific application and design constraints of the technical solution.

[0070] In some embodiments, this application provides an action recognition device, comprising: an acquisition module for acquiring a first image including a target object; a behavior determination module for determining action information of the target object in the first image; an optical flow determination module for calculating the optical flow type of the target object in the first image; wherein the optical flow type includes: a motion type, a stationary type, or a no-optical-flow type; and an action determination module for determining the action of the target object based on the action information and the optical flow information. The action recognition device can be a recording and broadcasting device, and the specific execution method and principle of the action recognition device can be referred to the action recognition method in the foregoing embodiments of this application, and will not be repeated here.

[0071] It should be noted that the division of components and modules in the action recognition device described above in this embodiment is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. For example, the processing module and the driving module can be the same module, and these modules can all be implemented in software through processing elements; they can all be implemented in hardware; or some modules can be implemented in software through processing elements, and some modules can be implemented in hardware. A separate processing element can be established, or it can be integrated into a chip in the device. Alternatively, it can be stored as program code in the device's memory, and called and executed by a processing element of the device. The implementation of other modules is similar. Furthermore, these modules can be fully or partially integrated together, or they can be implemented independently. The main control component described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through the integrated logic circuit in the hardware of the main control component or through software instructions.

[0072] For example, these components / modules can be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together to implement a system-on-a-chip (SOC).

[0073] In the above embodiments, all or part of the method steps executed by the action recognition device can be implemented by software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented in whole or in part as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the process or function described in the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0074] This application also provides an electronic device, including: a processor and a memory; wherein, the memory stores a computer program, and when the processor executes the computer program, the processor can be used to execute any of the action recognition methods in the foregoing embodiments of this application.

[0075] This application also provides a computer-readable storage medium storing a computer program, which, when executed, can be used to perform any of the action recognition methods described in the foregoing embodiments of this application.

[0076] This application also provides a chip for executing instructions, the chip being used to perform the action recognition method as described in any of the foregoing embodiments of this application.

[0077] This application also provides a program product, which includes a computer program stored in a storage medium. At least one processor can read the computer program from the storage medium. When the at least one processor executes the computer program, it can implement the action recognition method as described in any of the foregoing embodiments of this application.

[0078] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method of action recognition, characterized by, The method comprises: acquiring a first image including a target object; inputting the first image into a machine learning model to obtain action information of the target object output by the machine learning model and a feature region of the target object in the first image; the machine learning model comprises: a spatial attention module configured to extract spatial correlation features of edges of the target object and surrounding objects in the first image; a cross-stage partial module CSP configured to provide a backbone network Backbone of the machine learning model and extract Backbone features of the first image; a feature pyramid network module FPN configured to fuse the Backbone features extracted by the CSP with first deep features of the Backbone network to obtain multi-scale features; a path augmentation network module PAN configured to fuse the multi-scale features obtained by the FPN with second deep features of the Backbone network; a detection head module Yolo configured to learn and analyze the action information of the target object, the feature region and confidence information of the target object in the first image according to the fused features of the FPN and the PAN; calculating an optical flow type of the target object in the first image; wherein the optical flow type comprises: a motion type, a still type or no optical flow type; determining the action of the target object according to the action information and the optical flow type.

2. The method of claim 1, wherein: the target object is a student in a classroom; the action of the target object includes standing up and sitting down.

3. The method of claim 1, wherein, The calculation of the optical flow type of the target object in the first image comprises: determining the optical flow length and the optical flow direction of a plurality of feature points in the feature region of the target object in the first image; determining the optical flow type of the target object according to the optical flow length and the optical flow direction of the plurality of feature points in the first image and the optical flow length and the optical flow direction of the plurality of feature points in a second image; wherein the second image is an image of a previous frame of the first image.

4. The method of claim 3, wherein, The determination of the optical flow type of the target object according to the optical flow length and the optical flow direction of the plurality of feature points in the first image and the optical flow length and the optical flow direction of the plurality of feature points in the second image comprises: determining a first feature point from the plurality of feature points, which has a difference between the horizontal optical flow length in the first image and the second image that is less than a difference between the vertical optical flow length, and a difference between the vertical optical flow values that is less than a negative of a first preset threshold value; determining a second feature point from the plurality of feature points, which has a difference between the horizontal optical flow length in the first image and the second image that is less than a difference between the vertical optical flow length, and a difference between the vertical optical flow values that is greater than the first preset threshold value; determining a third feature point from the plurality of feature points, which has a difference between the vertical optical flow length in the first image and the second image that is less than a difference between the horizontal optical flow length, and a difference between the horizontal optical flow values that is greater than the first preset threshold value; When the sum of the number of the first feature points, the second feature points and the third feature points is greater than a second preset threshold, it is determined that the optical flow of the target object is of a motion type; when the sum of the number of the first feature points, the second feature points and the third feature points is less than the second preset threshold, it is determined that the optical flow of the target object is of a static type; when it is determined that the first image does not include the feature points, it is determined that the optical flow of the target object is of a no optical flow type.

5. The method of claim 4, wherein, The action of the target object is determined according to the action information and the optical flow type, including: When the action information of the target object in the first image indicates that the action of the target object is standing up, and the optical flow information of the target object in the first image is of a motion type, and the action of the target object in the second image is sitting down, it is determined that the action of the target object in the first image is standing up; When the action information of the target object in the first image indicates that the action of the target object is standing up, and the optical flow information of the target object in the first image is not of a motion type, and the action of the target object in the second image is standing up, it is determined that the action of the target object in the first image is standing up; When the action information of the target object in the first image indicates that the action of the target object is sitting down, and the optical flow information of the target object in the first image is of a motion type, it is determined that the action of the target object in the first image is standing up; When the action information of the target object in the first image indicates that the action of the target object is sitting down, and the optical flow information of the target object in the first image is not of a motion type, it is determined that the action of the target object in the first image is sitting down.

6. The method of claim 5, wherein, Further comprising: Using a queue to record the identification information of the target object and the action of the target object in each of the continuous multiple frames of images.

7. The method according to any one of claims 1 to 6, characterized in that, Further comprising: When it is determined that the action of the target object is standing up, controlling the camera to switch to a first shooting mode; When it is determined that the action of the target object is sitting down, controlling the camera to switch to a second shooting mode.

8. An action recognition apparatus, characterized by comprising: Comprising: An acquisition module is configured to acquire a first image including a target object; An action determination module is configured to determine action information of the target object in the first image; An optical flow determination module is configured to calculate an optical flow type of the target object in the first image; wherein the optical flow type includes a motion type, a static type or a no optical flow type; An action determination module is configured to determine the action of the target object according to the action information and the optical flow information; The action determination module is specifically configured to input the first image into a machine learning model to obtain the action information of the target object output by the machine learning model and a feature region of the target object in the first image; the machine learning model includes: A spatial attention module is configured to extract spatial correlation features of edges of the target object and surrounding objects in the first image; a cross-stage partial module (CSP) configured to provide a backbone network of the machine learning model and extract backbone features of the first image; a feature pyramid network module (FPN) configured to fuse the backbone features extracted by the CSP with first deep layer features of the backbone network to obtain multi-scale features; a path aggregation network module (PAN) configured to fuse the multi-scale features obtained by the FPN with second deep layer features of the backbone network; a detection head module (Yolo) configured to learn and parse action information of a target object, the feature region and confidence information in the first image based on the fused features of the FPN and the PAN.

Images