A behavior recognition method, device, apparatus and storage medium

Abstract

This application discloses a behavior recognition method, apparatus, device, and storage medium, relating to the field of computer vision. The method includes: acquiring video data captured by a camera, the video data comprising multiple frames; using a behavior detection model to identify each target object within the current frame image to obtain the target object's target location and behavior recognition result; assigning a target ID to the target object within the current frame image based on the target location, with the same target ID for the same target object in consecutive frame images; obtaining the behavior recognition result of the target object in consecutive frame images based on the target ID of the target object in the current frame image, and obtaining the final behavior recognition result of the target object in the current frame image based on the behavior recognition result of the target object in consecutive frame images, wherein the consecutive frame images include the current frame image. This method can solve the technical problem of accurately and efficiently recognizing student behavior in teaching recording scenarios.

Description

Technical Field

[0001] This application relates to the field of computer vision, and more particularly to a behavior recognition method, apparatus, device, and storage medium. Background Technology

[0002] With social progress and development, and the continuous improvement of people's living standards, education has gradually become an important social issue. Educational equity is the foundation for building a harmonious society, and the sharing of high-quality teaching resources is a crucial means to achieve it. Against this backdrop, teaching recording and broadcasting systems have emerged and been entrusted with the important task of recording and disseminating high-quality teaching resources. These systems can integrate and synchronously record teaching videos and audios captured on-site, as well as image signals from electronic devices used during the teaching process, generating standardized streaming media files for live broadcasting, storage, post-editing, and on-demand viewing, thereby achieving the goal of sharing teaching resources.

[0003] Teaching recording systems typically use classrooms as the recording environment and need to monitor classroom interactions between teachers and students. For example, when a student stands up to answer a question, the system needs to provide a close-up shot. This requires video-based behavior recognition technology to identify the student's action and provide a close-up shot for tracking. Traditional behavior recognition technologies include two-stream methods, momentum detection, skeletonization, and 3D spatiotemporal modeling. However, two-stream methods are computationally expensive and difficult to deploy in real-time. Momentum detection ignores target attributes (such as students in the recording environment), leading to false positives. Skeletonization requires classifying the skeletons of each student in the classroom, which is time-consuming, and the annotation of key skeleton points is costly and burdensome. 3D spatiotemporal modeling is based on 3D convolution, but current Neural Network Processing Units (NPUs) often do not support 3D convolution inference, making front-end deployment difficult. In summary, in the context of recorded teaching sessions, how to accurately and efficiently identify student behavior has become a pressing technical problem that needs to be solved. Summary of the Invention

[0004] This application provides a behavior recognition method, apparatus, device, and storage medium to solve the technical problem of difficulty in accurately and efficiently recognizing student behavior in teaching recording scenarios.

[0005] In a first aspect, embodiments of this application provide a behavior recognition method, the method comprising:

[0006] Acquire video data captured by a camera, the video data containing multiple frames of images;

[0007] The behavior detection model is used to identify each target object in the current frame image, so as to obtain the target location and behavior recognition result of the target object;

[0008] Based on the target location, a target ID is assigned to the target object in the current frame image, and the same target object in consecutive frame images has the same target ID;

[0009] Based on the target ID of the target object in the current frame image, the behavior recognition result of the target object in the continuous frame images is obtained, and based on the behavior recognition result of the target object in the continuous frame images, the final behavior recognition result of the target object in the current frame image is obtained, wherein the continuous frame images include the current frame image.

[0010] Secondly, embodiments of this application provide a behavior recognition device, the device comprising:

[0011] The video acquisition module is used to acquire video data captured by the camera, the video data including multiple frames of images;

[0012] The recognition module is used to identify each target object in the current frame image using a behavior detection model, so as to obtain the target position and behavior recognition result of the target object;

[0013] The ID allocation module is used to assign a target ID to the target object in the current frame image based on the target location, and the same target object in consecutive frame images has the same target ID;

[0014] The result acquisition module is used to obtain the behavior recognition result of the target object in the continuous frame images based on the target ID of the target object in the current frame image, and to obtain the final behavior recognition result of the target object in the current frame image based on the behavior recognition result of the target object in the continuous frame images, wherein the continuous frame images include the current frame image.

[0015] Thirdly, embodiments of this application provide a behavior recognition device, the device comprising:

[0016] One or more processors;

[0017] A memory for storing one or more programs that, when executed by one or more processors, cause the one or more processors to implement the behavior recognition method as described in the first aspect.

[0018] Fourthly, embodiments of this application provide a storage medium for storing computer-executable instructions, which, when executed by a computer processor, are used to perform the behavior recognition method as described in the first aspect.

[0019] The aforementioned behavior recognition method, apparatus, device, and storage medium acquire video data containing multiple frames captured by a camera, and use a behavior detection model to identify each target object within the current frame image to obtain the target object's target position and behavior recognition result. Then, based on the target position of each target object, a target ID is assigned to the target object within the current frame image. Based on the target ID of the target object in the current frame image, the behavior recognition result of the target object in consecutive frame images is obtained. Based on the behavior recognition result of the target object in consecutive frame images, the final behavior recognition result of the target object in the current frame image is obtained. This technical means solves the technical problem of accurately and efficiently recognizing student behavior in teaching recording scenarios. This method utilizes a behavior detection model to identify the target location and behavior detection results of objects in images, improving the accuracy and speed of student behavior recognition. After obtaining the target location of each object, a target ID is assigned to each object in the image. Furthermore, the same target object has the same target ID in consecutive image frames, facilitating the acquisition of behavior recognition results for the same object across consecutive image frames. The final behavior recognition result of the target object in the current frame is confirmed by combining the behavior recognition results from consecutive image frames. This approach considers the continuity of the same behavior across consecutive image frames, avoiding misidentification caused by relying solely on the current frame, thus improving the accuracy of behavior recognition. Moreover, this behavior recognition method is relatively simple to implement and easy to deploy in real-time on the front end. Attached Figure Description

[0020] Figure 1 This is a flowchart of a behavior recognition method provided in one embodiment of this application.

[0021] Figure 2 This is a flowchart of a behavior recognition method provided in one embodiment of this application.

[0022] Figure 3 This is a schematic diagram of the structure of a behavioral attention module provided in one embodiment of this application.

[0023] Figure 4 This is a schematic diagram of the structure of a behavioral attention module provided in one embodiment of this application.

[0024] Figure 5 This is a schematic diagram of the structure of a behavioral attention module provided in one embodiment of this application.

[0025] Figure 6This is a schematic diagram of the structure of a PSC-N Module provided in one embodiment of this application.

[0026] Figure 7 This is a schematic diagram of the structure of a PSC-N Module provided in one embodiment of this application.

[0027] Figure 8 This is a schematic diagram of the structure of a feature recognition module provided in one embodiment of this application.

[0028] Figure 9 This is a schematic diagram of the structure of a behavior detection model provided in one embodiment of this application.

[0029] Figure 10 This is a schematic diagram of the structure of a behavior detection model provided in one embodiment of this application.

[0030] Figure 11 This is a schematic diagram of target ID allocation provided in one embodiment of this application.

[0031] Figure 12 This is a schematic diagram of a prohibited area provided in one embodiment of this application.

[0032] Figure 13 This is a schematic diagram of a licensed area provided for one embodiment of this application.

[0033] Figure 14 This is a schematic diagram of a double-ended queue structure provided in one embodiment of this application.

[0034] Figure 15 This is a schematic diagram of the structure of a behavior recognition device provided in one embodiment of this application.

[0035] Figure 16 This is a schematic diagram of the structure of a behavior recognition device provided in one embodiment of this application. Detailed Implementation

[0036] The following description and accompanying drawings fully illustrate specific embodiments of this application to enable those skilled in the art to practice them. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. The scope of embodiments of this application includes the entire scope of the claims and all available equivalents of the claims. In this document, each embodiment may be referred to individually or collectively by the term "invention," which is merely for convenience and is not intended to automatically limit the scope of the application to any single invention or inventive concept if more than one invention is disclosed. Relational terms such as "first" and "second" are used herein only to distinguish one entity or operation from another, without requiring or implying any actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed. The various embodiments in this document are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the structures, products, etc., disclosed in the embodiments, since they correspond to the disclosed parts, the descriptions are relatively simple; relevant details can be found in the method section.

[0037] The behavior recognition method provided in this application embodiment can be executed by a behavior recognition device, which can be implemented by software and / or hardware. The behavior recognition device can consist of two or more physical entities, or it can consist of a single physical entity. For example, the behavior recognition device can be a computer, tablet, smartphone, server, or other device capable of directly processing data.

[0038] For example, behavior recognition devices can be deployed in a teaching recording and broadcasting system to implement behavior recognition methods in teaching recording and broadcasting scenarios. Alternatively, the computer or other device with data processing capabilities deployed in the teaching recording and broadcasting system can be used as a behavior recognition device, and behavior recognition functions can be added to implement behavior recognition methods. In this case, it is not necessary to deploy new behavior recognition devices in the teaching recording and broadcasting system.

[0039] For example, after deploying a behavior recognition device in a teaching recording system, the behavior recognition device can work collaboratively with other devices (such as cameras) within the teaching recording system to achieve teaching recording. The communication method between the behavior recognition device and other devices is not limited in this embodiment.

[0040] Figure 1A flowchart of a behavior recognition method provided in one embodiment of this application is shown below. Figure 1 As shown, the behavior recognition method includes:

[0041] Step 101: Obtain video data captured by the camera. The video data contains multiple frames of images.

[0042] In one embodiment, the camera can capture images of a specific area to obtain video data. In a teaching recording scenario, the specific area refers to the classroom where live teaching takes place, i.e., the camera is used to capture the classroom. In this case, the video data captured by the camera reflects the live teaching scene in the classroom. Optionally, there can be at least one camera. When there are multiple cameras, they are located at different positions in the classroom to capture the live teaching scene from different angles. Optionally, acquiring the video data captured by the cameras can involve acquiring the video data captured by each camera and performing subsequent processing on the video data, or manually selecting and acquiring the video data captured by one camera and performing subsequent processing on that video data. It should be noted that in this embodiment, there are no limitations on how the camera captures video data or how the behavior recognition device acquires the video data. For example, when teaching recording begins, the camera starts capturing the live teaching scene and sends the captured live teaching scene to the behavior recognition device via wired or wireless means. Alternatively, the camera and the behavior recognition device communicate through a server deployed in the teaching recording system. In this case, the behavior recognition device controls the camera to capture and acquire the captured video data through the server.

[0043] In one embodiment, the video data comprises multiple frames of images, arranged in chronological order of capture time. The specific number of frames in the video data is not limited. It can be understood that video data is essentially a continuous sequence of images, composed of multiple consecutive frames. Based on the principle of visual persistence, the human eye cannot distinguish the static image of a single frame; when multiple consecutive frames are played rapidly, the human eye perceives a smooth and continuous visual effect.

[0044] Step 102: Use the behavior detection model to identify each target object in the current frame image to obtain the target location and behavior recognition results of the target objects.

[0045] In one embodiment, a behavior detection model is used to identify target objects in each frame of an image to obtain the target location and behavior recognition results of each target object in each frame. Here, the target object refers to the object being located and identified. The type of target object that the behavior detection model can identify varies depending on the application scenario. In one embodiment, the target object is a human, meaning the behavior detection model can identify humans in the video data. Optionally, in a teaching recording scenario, the target object is a student. In this case, when capturing video data, the area where the students are located in the classroom can be filmed to obtain video data containing the target object. It should be noted that in a teaching recording scenario, there may be situations where the teacher enters the area where the students are located. In this case, the video data also includes the teacher. Both the teacher and students are human; therefore, the behavior detection model will also identify the teacher as a target object and output the corresponding recognition result. It is understood that during teaching, the teacher spends a longer time in the podium area and a shorter time in the student area. Therefore, even if the teacher in the video data is identified by the behavior detection model, it will not affect the final effect of the teaching recording.

[0046] Understandably, after an image frame passes through the behavior detection model, the behavior detection model outputs the target location and behavior recognition result of the target object contained in the image. Optionally, if there is no target object in the current frame image, the behavior detection model will not output the corresponding target location and behavior recognition result; in this case, the output result of the behavior detection model can be considered empty. In one embodiment, when the behavior detection model performs recognition, it sequentially recognizes each frame image. At this time, the image currently recognized by the behavior detection model is recorded as the current frame image, and the final behavior recognition result of the target object in the current frame image is subsequently obtained.

[0047] The behavior detection model is a pre-built and trained neural network model, and this step is the application process of the behavior detection model. The model structure and parameters of the behavior detection model can be set according to actual conditions. After inputting the current frame image into the behavior detection model, the behavior detection model can identify the recognized target object, its corresponding target position, and the behavior recognition result in the current frame image. The target position refers to the position of the target object in the corresponding image. In one embodiment, the behavior detection model identifies the recognized target object in the image using a bounding box. At this time, the identified target object is located within the bounding box. Correspondingly, the target position refers to the position of the rectangular area containing the bounding box in the image. The target position can be represented in the form of coordinates. For example, a rectangular coordinate system can be established with a certain pixel point in the image (such as the pixel point in the upper left corner) as the origin of the coordinate system, and the target position of the corresponding target object can be obtained through the coordinates of the rectangular area on the coordinate system. The behavior recognition result refers to the result obtained after recognizing the behavior of the target object. For example, the recognizable behaviors are pre-defined. For ease of description, the pre-defined behaviors are referred to as preset behaviors, and there are multiple preset behaviors. When a behavior detection model identifies a target object, it finds the possible behaviors the target object might currently exhibit among preset behaviors and obtains the behavior recognition result. The content of the preset behaviors can vary depending on the application scenario. For example, in a teaching recording scenario, preset behaviors include standing up, sitting down, raising a hand, etc. In this case, the behavior detection model's identification of the target object's behavior based on the preset behaviors can also be understood as classifying the target object's behaviors to determine the category to which the behavior belongs. Each preset behavior can be considered a category. Optionally, when identifying behaviors, the behavior detection model can determine the confidence level of the target object's current behavior belonging to each preset behavior. The higher the confidence level, the greater the probability that the target object's behavior belongs to a preset behavior, and vice versa. Then, the behavior detection model uses the preset behavior with the highest confidence level as the behavior recognition result of the target object and outputs the corresponding confidence level. Alternatively, the behavior detection model can use multiple preset behaviors with the highest confidence levels as the behavior recognition result of the target object and output the corresponding confidence levels. In one embodiment, the behavior detection model uses the preset behavior with the highest confidence level as the behavior recognition result of the target object as an example.

[0048] In one embodiment, when the behavior detection model identifies the target location and behavior recognition result of a target object, it first extracts the first feature of the target object in the image. For example, the first feature includes limb association features and spatial association features. The limb association features represent the association features between the target limbs of the target object. The target limbs are pre-defined limbs that need to be identified by the behavior detection model, such as arms, shoulders, hips, and legs. The association features between the target limbs may include the relative positions between the target limbs. The behavior of the target object can be inferred through the limb association features. The spatial association features represent the association features between the target limbs of the target object and spatial objects. The spatial objects are objects that have a spatial relationship with the target object. The spatial relationship can be determined according to the application scenario. Generally speaking, objects that should exist near the target object are considered to have a spatial relationship with the target object. For example, in a teaching recording scenario, students generally sit on chairs and use tables. Therefore, chairs and tables are located near students and can be considered to have a spatial relationship with students. Spatial association features can include the relative position between the target limb and spatial objects. For example, in a teaching recording scenario, when the target object is a student, spatial association features include the association between the student's arm and the table, or the association between the student's hip and the chair. These spatial association features can be used to infer the target object's behavior and target location. After obtaining the first feature of the target object, the behavior detection model performs deep learning on the first feature to uncover richer second features. Then, the behavior detection model identifies the second feature, that is, it identifies the feature content of the second feature, obtaining the feature recognition result. This feature recognition result describes the feature content contained in the second feature. Finally, the behavior detection model analyzes the feature recognition result to obtain the target object's target location and behavior recognition result.

[0049] Generally, in teaching recording scenarios, when a camera captures students, it captures all students within the shooting area. In this case, the video data contains multiple target objects (i.e., all students within the shooting area). After the current frame image is input into the behavior detection model, the model identifies each target object within the current frame image to obtain the target location and behavior recognition results for each object. It can be understood that when a target object appears repeatedly in multiple frames, that target object has a corresponding target location and behavior recognition result in each frame.

[0050] Step 103: Assign a target ID to the target object in the current frame image based on the target location. The same target object in consecutive frame images has the same target ID.

[0051] A target ID is a unique identifier assigned to a target object. Each target object corresponds to a single target ID, which allows for the differentiation of target objects. The specific representation of the target ID is not limited in this embodiment. For example, each target object appearing in the current frame image is assigned a corresponding target ID, so that after the target IDs are assigned, two target objects can be distinguished by their respective target IDs.

[0052] In one embodiment, a target ID is assigned to the target object based on its target position (i.e., the target object's position in the corresponding image) in the current frame image. Optionally, the same target object may appear in multiple consecutive frames. Therefore, in one embodiment, the same target ID is assigned to the same target object in consecutive frames, where consecutive frames refer to multiple consecutive frames. For the current frame image, the last frame in the corresponding consecutive frames is the current frame image. It is understood that when a target object performs an action, its action in consecutive frames should be continuous. Correspondingly, its target position in consecutive frames should be relatively close. Therefore, the same target ID can be assigned to the same target object based on its target position in consecutive frames, so as to more accurately identify the target object's action by combining consecutive frames and avoid confusing the target object. Furthermore, different target objects have different target positions in a single frame image. Therefore, different target objects can be distinguished based on their target positions, and different target IDs can be assigned to different target objects.

[0053] In one embodiment, target IDs are assigned to each target object in the current frame image based on the target positions of each target object in the previous frame image and the target positions of each target object in the current frame image. Optionally, based on the target position of a target object in the current frame image, it is determined whether a target object exists at that target position in the previous frame image. If so, the target ID of the target object at that target position in the previous frame image is obtained and assigned to the target object at that target position in the current frame image. If not, it is determined that the target object is a newly appearing object, and a new ID is assigned to the target object. For the first frame image, each target object appearing therein is assigned a new ID. The rules for assigning target IDs to target objects can be set according to actual conditions. Alternatively, the existence of the same target object in consecutive frame images can be determined by calculating the IOU (Intersection over Union) value, and the same ID can be assigned to the same target object. Here, the IOU value is the result obtained by dividing the overlapping part of two regions by the set of the two regions. For example, the target location of the target object in the current frame image is obtained. Then, the Interchange of Union (IOU) value is calculated based on this target location and the target locations of each target object in the previous frame image. At this point, there is an IOU value between the target location in the current frame image and each target location in the previous frame image. It can be understood that the higher the IOU value, the higher the probability that the two target objects belong to the same target object. Accordingly, the largest IOU value is selected from all IOU values, and it is determined whether the largest IOU value is greater than a preset threshold. If it is greater than the preset threshold, the target object contained in the target location corresponding to the largest IOU value in the previous frame image and the target object contained in the target location corresponding to the largest IOU value in the current frame image can be considered the same target object. Therefore, the target ID of the target object in the previous frame image is obtained and used as the target ID of the target object in the current frame image. If it is less than the preset threshold, the target object contained in the target location corresponding to the largest IOU value in the current frame image can be considered a newly appearing target object, and a new target ID is assigned to it. At this time, when the target position of the same target object in two consecutive frames of images jitters or changes slightly, the same target object in two consecutive frames of images can still be accurately identified by the IOU value.

[0054] In one embodiment, if a target object in the previous frame is not identified in the current frame, it is considered that the target object has temporarily disappeared for one frame. The number of frames the target object disappeared from is updated, and the number of frames disappeared is compared with a preset target number. If the number of frames disappeared reaches the target number, it means that the target object has disappeared for a period of time. Therefore, it can be considered that the target object has disappeared from the video data. At this time, the target object and its corresponding ID are not added to the current frame. If the number of frames disappeared does not reach the target number, it means that the target object disappeared for a very short time, possibly due to factors such as missed identification. When a target object disappears, its location in the current frame is identified based on its position in the previous frame, and a target ID is assigned to it. Subsequent frames are processed by repeatedly checking for disappearances and updating the disappearance count accordingly. If the target object is detected again in subsequent frames, it's assumed it was missed for some reason. In this case, the disappearance count is reset to zero, and the target object and its ID are retained in subsequent frames. This ensures accurate identification and correct ID assignment for target objects in consecutive frames, preventing missed detections. Even if a target object is missed in a frame, it's retained to avoid impacting the final behavior recognition result. Furthermore, removing a target object after it's determined to have disappeared further ensures accuracy and prevents a missing target object from remaining in frames for an extended period.

[0055] Step 104: Based on the target ID of the target object in the current frame image, obtain the behavior recognition result of the target object in the continuous frame images, and based on the behavior recognition result of the target object in the continuous frame images, obtain the final behavior recognition result of the target object in the current frame image. The continuous frame images include the current frame image.

[0056] For example, since the target ID is a unique identifier for each target object, the behavior recognition result of the target object in each frame of a series of frames can be obtained based on the target ID in the current frame image. The series of frames includes the current frame image. In one embodiment, the series of frames consists of the current frame image and a predetermined number of frames preceding it, and the number of frames in the series can be set according to actual needs. Because the time interval between adjacent frames in video data is very small, the differences in the target object's actions in the series of frames are small, meaning the probability of the target object having the same behavior recognition result in the series of frames is high. Therefore, the final behavior recognition result of the target object in the current frame image can be determined based on the behavior recognition result of the target object in the series of frames. In one embodiment, determining the final behavior recognition result of the target object in the current frame image based on the behavior recognition results of the target object in consecutive frame images can be achieved by: counting the behavior recognition results of the target object in other frame images (excluding the current frame image) in the consecutive frame images, and finding the behavior recognition results that are the same as the behavior recognition result in the current frame image. This yields the number of times the behavior recognition result in the current frame image appears in the consecutive frame images. When the number of occurrences exceeds a preset threshold, the behavior recognition result in the current frame image is considered correct, and this behavior recognition result is taken as the final behavior recognition result for the current frame image. Optionally, if the number of occurrences does not exceed the preset threshold, it can be considered that the target object's action has changed. In this case, the preset default behavior recognition result is taken as the final behavior recognition result for the current frame image. The default behavior recognition result can be set according to the application scenario. For example, in a teaching recording scenario, the default behavior recognition result is "sit down." If the number of times the behavior recognition result in the current frame image appears in the consecutive frame images is less than the threshold, then "sit down" is taken as the final behavior recognition result for the current frame image.

[0057] It is understandable that for the first few frames of video data (less than the number of consecutive frames), only the target location, behavior recognition result, and target ID of the first few frames are obtained. Since the number of consecutive frames has not been reached, the final behavior recognition result of the first few frames will not be output. That is, from the user's perspective, the first few frames of video data do not have the final behavior recognition result.

[0058] Optionally, after obtaining the final behavior recognition result of the target object, subsequent control can be carried out based on the final behavior recognition result. For example, when the target object stands up, the camera can be controlled to track and capture the target object.

[0059] The above-described technical approach, which involves acquiring video data containing multiple frames captured by a camera and using a behavior detection model to identify each target object within the current frame to obtain the target location and behavior recognition results of the target objects, then assigning a target ID to each target object within the current frame based on its target location, obtaining the behavior recognition results of the target object in consecutive frames based on the target ID of the target object in the current frame, and finally obtaining the final behavior recognition result of the target object in the current frame based on the behavior recognition results of the target object in consecutive frames, solves the technical problem of accurately recognizing student behavior in teaching recording scenarios. This method utilizes a behavior detection model to identify the target location and behavior detection results of objects in images, improving the accuracy and speed of student behavior recognition. After obtaining the target location of each object, a target ID is assigned to each object in the image. Furthermore, the same target object has the same target ID in consecutive image frames, facilitating the acquisition of behavior recognition results for the same object across consecutive image frames. The final behavior recognition result of the target object in the current frame is confirmed by combining the behavior recognition results from consecutive frames, avoiding misidentification that can occur when relying solely on the current frame, thus improving the accuracy of behavior recognition. Moreover, this behavior recognition method is relatively simple to implement and easy to deploy in real-time on the front end.

[0060] Figure 2 This is a flowchart illustrating a behavior recognition method according to one embodiment of this application. This behavior recognition method is a specific embodiment based on the one described above. Figure 2 As shown, the behavior recognition method includes:

[0061] Step 201: Obtain video data captured by the camera. The video data contains multiple frames of images.

[0062] Step 202: Use the behavior detection model to identify each target object in the current frame image to obtain the target location and behavior recognition results of the target objects.

[0063] In one embodiment, the behavior detection model includes a behavior attention module, a local spatial convolution module, a feature recognition module, and an object detection module. Accordingly, step 202 includes steps 2021-2024:

[0064] Step 2021: Use the behavior attention module to extract the first feature of each target object in the current frame image. The first feature includes limb association feature and spatial association feature. The limb association feature is used to reflect the association feature between each target limb in the target object. The spatial association feature is used to reflect the association feature between the target limb and the spatial object. The spatial object is an object that has a spatial relationship with the target object.

[0065] The behavior attention module is the core module of the behavior detection model. It extracts the first features of each target object in an image. These first features include limb association features and spatial association features. For example, limb association features reflect the association between different target limbs within the target object. The target limbs are pre-defined limbs. The behavior attention module can identify the target limbs in the image to obtain the association features between them. Generally, when a target object performs an action, the relative positions between target limbs change. For example, if the target object's arm, shoulder, leg, and hip are designated as target limbs, the relative position between the arm and shoulder when the target object raises its hand differs from when it doesn't raise its hand. In this case, when the behavior attention model identifies the target object, it can obtain limb association features representing the arm and shoulder and their relative positions, thus assisting in behavior recognition. For example, spatial association features reflect the association between target limbs and spatial objects. Spatial objects are objects that have a spatial relationship with the target object. Furthermore, the spatial relationship can be determined based on the application scenario; generally, objects that should exist near the target object are considered objects with a spatial relationship to it. It is understandable that when a target object performs a certain action, the relative position between its target limb and a spatial object will change. For example, if the target limb is the buttocks and the spatial object is a chair, the positional relationship between the buttocks and the chair will be different when the target object stands or sits. At this time, when the behavior attention module identifies the target object, it can obtain limb association features that represent the buttocks and the chair and the relative positional relationship between the two, so as to assist in behavior recognition and target location determination through spatial association features.

[0066] The structure of the behavioral attention module can be configured according to actual needs. In one embodiment, the behavioral attention module includes a content-aware attention submodule and at least one convolutional submodule. The convolutional submodule is used to extract features from the input data, and the content-aware attention submodule is used to obtain features related to limb association features and features related to spatial association features based on the extracted features. Alternatively, the behavioral attention module includes a spatial attention submodule and at least one convolutional submodule. The convolutional submodule is used to extract features from the input data, and the spatial attention submodule is used to obtain features related to spatial association features and features related to limb association features based on the extracted features. Or, the behavioral attention module includes a content-aware attention submodule, a spatial attention submodule, and at least one convolutional submodule. The convolutional submodule is used to extract features from the input data, and the content-aware attention submodule is used to obtain features related to limb association features based on the extracted features. The spatial attention submodule is used to obtain features related to spatial association features based on the extracted features.

[0067] The convolutional submodule is used to convolve the input data (such as the current frame image or a feature map obtained based on the current frame image) to extract the required features from the data, and works with other submodules in the behavioral attention module to obtain the first feature. The convolutional submodule can be represented as a CBL-N submodule, where CBL consists of Convolution (for convolution) + Batchnorm (to ensure the input of each neural network layer has the same distribution) + LeakyReLU (representing a linear unit function with leakage correction). That is, the convolutional submodule includes convolutional layers, Batchnorm layers, and LeakyReLU layers. N represents the number of output channels of the convolutional submodule. The value of N can be chosen according to the actual situation. In addition to the number of channels, the parameters involved in the convolutional submodule include the kernel size, stride, and padding pixels. The specific values ​​of each parameter can be set according to the actual situation. The number of convolutional submodules used in the behavioral attention module and the connection relationships between the convolutional submodules can be set according to the actual situation.

[0068] The Contextual Attention Module (CAM) primarily learns content related to limb association features of various target objects in an image. In some cases, it can also learn content related to spatial association features. The network structure of the CAM can be set according to the actual situation. In one embodiment, the CAM consists of residual blocks and a sigmoid function, i.e., a sigmoid function is attached after the residual block. The residual block can be represented as ResBlock-N, where ResBlock represents the residual block, and its specific structure can be set according to the actual situation. N represents the number of output channels of the residual block. Optionally, the value of N in ResBlock-N and CBL-N is the same. In addition to the number of channels, the parameters involved in ResBlock-N also include the kernel size, stride, and padding pixels. The specific values ​​of each parameter can be set according to the actual situation.

[0069] The Spatial Attention Module (SAM) primarily learns spatial association features related to various objects in an image. In some cases, it can also learn features related to limbs. The network structure of the SAM can be customized according to specific requirements. In one embodiment, the SAM consists of a CBL-N and a Sigmoid function, i.e., a Sigmoid function is appended to the CBL-N. The CBL-N used in the SAM can have the same structure as the CBL-N used in the convolutional submodule.

[0070] For example, when the behavior attention module includes a convolutional submodule, a content-aware attention submodule, and a spatial attention submodule, the convolutional submodule is used to convolve the data input to itself to achieve feature extraction. Then, the content-aware attention submodule can extract content related to limb-related features (i.e., features) based on the features extracted by the convolutional submodule. Similarly, the spatial attention submodule can extract content related to spatially related features (i.e., features) based on the features extracted by the convolutional submodule. For instance... Figure 3 This is a schematic diagram of the structure of a behavioral attention module provided in one embodiment of this application, with reference to... Figure 3 The Action Contextual Spatial Attention Module (ACSAM) consists of two convolutional submodules ( Figure 3 The module is represented as CBL-N, consisting of a content-aware attention submodule and a spatial attention submodule. The three numbers following CBL-N represent the kernel size, stride, and padding pixels, respectively. For example, CBL-N,3,2,1 indicates a kernel size of 3 (i.e., the kernel's length and width are both 3, making it a square kernel), a stride of 2, and a padding pixel of 1. CBL-N,K,1,3 indicates a kernel size of K (the specific value can be set according to actual conditions), a stride of 1, and a padding pixel of 3. The three numbers following ResBlock-N also represent the kernel size, stride, and padding pixels, respectively. For example, ResBlock-N,3,1,1 indicates a kernel size of 3, a stride of 1, and a padding pixel of 1. Figure 3 In the code, ResBlock-N,3,1,1 and the Sigmoid function constitute the content-aware attention submodule, and CBL-N,3,1,1 and the Sigmoid function constitute the spatial attention submodule. In one embodiment, to increase the receptive field and improve the multimodal behavioral feature fusion efficiency, K in CBL-N,K,1,3 is set to 5 or 7. It is understood that... Figure 3 In this context, ACSAM-N represents the behavioral attention module, where N represents the number of channels for the first feature output by ACSAM. N in ACSAM-N, CBL-N, and ResBlock-N can have the same value. It should be noted that... Figure 3 The specific working process of the behavioral attention module shown in the embodiment is not limited.

[0071] Building upon this, to further reduce the size of the behavior attention module, its structure can be simplified. This simplification can be achieved by including only a convolutional submodule and a content-aware attention submodule within the behavior attention module. This means the content-aware attention submodule also has the ability to learn spatial correlation features, allowing the extraction of the first features of each target object in the image through the convolutional and content-aware attention submodules. For example, Figure 4 This is a schematic diagram of the structure of a behavioral attention module provided in one embodiment of this application. Figure 4 The behavioral attention module in the algorithm comprises two convolutional submodules (CBL-N,3,2,1 and CBL-N,K,1,3, respectively) and a content-aware attention submodule (SAM). The structure of SAM is similar to... Figure 3 The structure of SAM is the same as in the previous example and will not be described again here. In this case, SAM can obtain features related to limb association features and features related to spatial association features based on the features extracted by the convolutional submodules. Alternatively, a simplified approach is to set only convolutional and spatial attention submodules in the behavior attention module. The first features of each target object in the image are extracted by the convolutional and spatial attention submodules, meaning that the spatial attention submodule also has the ability to learn limb association features. For example, Figure 5 This is a schematic diagram of the structure of a behavioral attention module provided in one embodiment of this application. Figure 5 The behavioral attention module in the algorithm includes two convolutional submodules (CBL-N,3,2,1 and CBL-N,k,1,3) and a spatial attention submodule (CAM). The results of CAM are related to... Figure 3 The structure of CAM is the same as in the previous example, and will not be described again. Here, CAM can obtain features related to spatial association and features related to limb association based on the features extracted by the convolutional submodules. This is understandable. Figure 4 and Figure 5 The behavioral attention submodule shown has a simple structure, faster processing speed, and its processing results can meet user requirements. Figure 3 The structure of the behavior attention module shown is relatively complex, and the processing accuracy is higher. In practical applications, users can choose the structure of the behavior attention module according to their actual needs. At the same time, users can modify the number of sub-modules, connection relationships and parameters in the behavior attention module according to their actual needs.

[0072] In one embodiment, after the current frame image is input into the behavior attention module, the behavior attention module can extract the first feature of each target object in the image.

[0073] Understandably, the number of behavior attention modules in a behavior detection model can be set according to actual needs. Furthermore, the more behavior attention modules there are, the larger the behavior detection model becomes, and the longer the processing time for obtaining the first feature. Optionally, when there are multiple behavior attention modules, they can be connected in series, and different behavior attention modules can have different model parameters.

[0074] Step 2022: Use the local spatial convolution module to perform deep learning on the first feature to output a second feature with multiple channels of different numbers.

[0075] For example, the local spatial convolution module is used to perform deep learning on the first feature. Deep learning refers to learning the inherent patterns and representation levels of the first feature, thereby uncovering a second feature with richer feature content. In this embodiment, multiple second features are obtained based on the first feature, and each second feature has a different number of channels, which can be set according to actual needs.

[0076] In one embodiment, the Local Spatial Convolution Module (PSC Module) consists of multiple PSC Modules, and each PSC Module can be represented by a PSC-N Module, where N represents the number of output channels of the PSC Module. Each PSC-N Module is used for deep learning and outputs a second feature. In one embodiment, the structure of the PSC Module can be set according to actual conditions. For example, the PSC Module can be improved from the CSP (Cross Stage Partial Module), where CSP integrates gradient changes from beginning to end into the feature map (corresponding to the first feature in this embodiment), which can reduce the amount of computation while ensuring accuracy. Unlike the CSP module, the first network layer of the branch in the PSC Module is replaced by a 1x1 convolutional layer, which helps to shorten the recognition time in the NPU, where the behavior detection model can be deployed. It is understood that the number of parameters of the convolution kernel of the 1x1 convolutional layer is limited, and the impact of using a 1x1 convolutional layer PSC Module on the computational complexity of the behavior detection model is small. For example, Figure 6 This is a schematic diagram of the structure of a PSC-N Module provided in one embodiment of this application. Figure 7 This is a schematic diagram of the structure of a PSC-N Module provided in one embodiment of this application. Figure 6 and Figure 7 Different structures of the PSC-N Module are shown. Figure 6 and Figure 7In this context, Concate represents a stitching layer, and CBL-N represents a convolutional submodule with N output channels, consisting of Convolution+Batchnorm+LeakyReLu. The three numbers following CBL-N represent the kernel size, stride, and padding pixels, respectively. Figure 6 and Figure 7 In CBL-N / 2, it indicates that the number of output channels is N or 2. For the structure and parameters of the PSC-N Module, please refer to [reference needed]. Figure 6 and Figure 7 The embodiments are not described further. In one embodiment, each PSC-N Module is followed by Maxpooling, where Maxpooling represents a maximum pooling layer to perform maximum pooling on the output of the PSC-N Module.

[0077] Optionally, multiple PSC-N Modules are connected in series, with each PSC-N Module connected sequentially in ascending order of the number of output channels. It is understood that the more PSC-N Modules there are, the higher the accuracy of the behavior recognition result, but the improvement in accuracy becomes smaller and less significant. However, in front-end deployment, increasing the number of PSC-N Modules significantly increases the runtime, leading to slower inference speed, excessive latency, and poor real-time performance during behavior recognition, which is detrimental to front-end deployment. Therefore, in one embodiment, three PSC-N Modules are set, with N in the PSC-N Modules being 64, 128, and 256 respectively. In one embodiment, the output of PSC-256 Module is directly used as the second feature. For PSC-64 Module and PSC-128 Module, CBL-N / 2,1,1,0 ( Figure 6 and Figure 7 (The dashed box indicates that the output features are used as the second features.) Optionally, the larger the number of channels in the PSC-N Module, the larger the size of the target object detected by that sub-module during deep learning. In this case, the second features output by the three PSC-N Modules correspond to small-sized, medium-sized, and large-sized target objects, respectively. It can be understood that the size used to classify small, medium, and large-sized target objects can be set according to the actual situation.

[0078] Step 2023: Use the feature recognition module to identify the second feature and output the feature recognition result. Each second feature corresponds to one feature recognition result.

[0079] The feature recognition module is used to identify the second feature and output the feature recognition result, with each second feature corresponding to one feature recognition result. The feature recognition result refers to the result obtained after recognizing the second feature, representing the content of that second feature. In one embodiment, after recognizing the second feature, the feature recognition module obtains feature recognition results corresponding to target objects of different sizes. For example, after inputting the second features of large, medium, and small target objects into the feature recognition module, feature recognition results corresponding to large, medium, and small sizes can be obtained, respectively.

[0080] In one embodiment, when processing each second feature, the feature recognition module can combine features of target objects of different sizes using the `concate` function within the feature recognition module to enrich the number of features processed. The specific structure of the feature recognition module can be set according to actual conditions; for example, Figure 8 This is a schematic diagram of the structure of a feature recognition module provided in one embodiment of this application. (Reference) Figure 8 The feature recognition module consists of 5 CBL-N layers (N values ​​are 512, 255, 512, 256, and 128 respectively), 2 CBLU-N layers, 2 Concatenated layers, and 3 Conv layers. CBL-N has the same meaning as the previously mentioned CBL-N and will not be repeated here. CBLU-N represents a structure with N output channels composed of CBL+Upsample, where Upsample represents upsampling, and the specific rules for upsampling are not limited. Conv represents convolution, and the specific structure and parameter implementation of its convolutional layers are not limited in any specific embodiment. (Reference) Figure 8 When there are three second features, the second feature with the largest number of channels is input into the first CBL-512,3,1,1 in the first row. The second feature with the second largest number of output channels is input into the Concate in the middle row to combine with the features processed in the previous layer. The second feature with the smallest number of output channels is input into the Concate in the bottom row to combine with the features processed in the previous layer. The three Conv output the feature recognition results corresponding to each second feature, and from top to bottom, they are the feature recognition results corresponding to large, medium and small target objects, respectively.

[0081] Step 2024: The object detection module detects the target position and behavior recognition results of each target object based on the feature recognition results. Each feature recognition result corresponds to an object detection module, and the target objects detected by each object detection module have different sizes.

[0082] The object detection module is used to detect the target location and behavior recognition results of each target object based on feature recognition results. Optionally, the object detection module includes three branches, which are used to identify large, medium, and small target objects, respectively. For example, in a teaching recording scenario, when students are the target objects, the age range of students is relatively large, such as high school students, middle school students, and primary school students, each covering different age ranges. Furthermore, the height of students varies across different age ranges; for example, high school students are generally taller than middle school and primary school students. Therefore, in this embodiment, height is used as the standard, with tall students identified as large-sized target objects, medium-sized students as medium-sized target objects, and short students as small-sized target objects, resulting in three types. Different branches in the object detection module identify different types of students, thereby increasing the detection range of the object detection module and enhancing the application scope of the behavior recognition model.

[0083] The three branches in the object detection module can adopt the same structure. In one embodiment, each branch in the object detection module uses a YOLO detection head. The feature recognition results are input into the object detection module, and the YOLO detection head can draw a detection box containing the target object in the current frame image based on the feature recognition results. This detection box is typically a rectangle. The position of the rectangular area containing the detection box in the image is recorded as the target position, and the behavior recognition result is marked on the edge of the detection box, that is, the category to which the behavior emitted by the target object belongs (i.e., a pre-defined behavior) and the confidence level of belonging to that category are identified. The larger the size of the target object, the larger the detection box of the target object in the image after processing by the object detection module. It is understood that, in addition to the YOLO detection head, the object detection module can also use other networks or structures capable of recognizing feature recognition results; this embodiment does not limit this.

[0084] In one embodiment, Figure 9 This is a schematic diagram of the structure of a behavior detection model provided in one embodiment of this application, with reference to... Figure 9 The behavior detection model includes two behavior attention modules, with 32 and 64 channels respectively. Figure 9 The two behavioral attention modules are denoted as ACSAM-32 and ACSAM-64, respectively. Both modules have the same structure and can employ... Figures 3-5 The structure described in any of the above. The local spatial convolution module includes three PSC-N modules with channel numbers of 64, 128, and 256, corresponding to... Figure 9 The PSC-64Module, PSC-128Module, and PSC-256Module all have the same structure as the PSC-N Module and can be used... Figures 6-7 Any of the structures described herein, when employing Figure 6 In the structure, CBL-N / 2,1,1,0 in PSC-64Module and PSC-128Module Figure 6 and Figure 7 (Shown by dashed box) The output of the second feature is used as the output of the PSC-256Module. The structure of the feature recognition module in the behavior detection model is similar to... Figure 8 The structures shown are the same and will not be described again here. The object detection module uses YOLO detectors, and the three YOLO detectors output the target location and behavior recognition results of large, medium and small target objects respectively from top to bottom.

[0085] In one embodiment, to further improve the behavior detection model and its inference speed after deployment, a lightweight behavior detection model can be employed. Figure 10 This is a schematic diagram of the structure of a behavior detection model provided in one embodiment of this application. Figure 10 A lightweight behavior detection model is shown, which can replace the behavior detection module. Figure 9 The behavior detection model shown is... Figure 10 Yes Figure 9 The feature recognition module was streamlined by removing CBL-512, 3, 1, 1 and adjusting the parameters of other CBL-N modules. This effectively reduced the parameter size of the behavior detection model, thus meeting the needs of real-time front-end deployment. It should be noted that when the types of behavior recognition results (i.e., behavior classifications) are relatively few, the behavior detection model does not need to have an excessive number of channels. Even with a low number of channels in the feature recognition module, the behavior detection model can still accurately obtain the target location and behavior recognition results of the target object, and reducing the number of channels can improve the recognition speed of the behavior detection model. Figure 10 In this context, the value of m is set to any one of 1, 2, or 4, and the value of n is set to any one of 1 or 2. If m = 2 and n = 1, then CBL-128m is equivalent to CBL-256, and CBLU-32n is equivalent to CBLU-32. It can be understood that the values ​​of m and n can be set according to the hardware resources of the behavior recognition device.

[0086] After obtaining the target location and behavior recognition results of the target object in step 202, the subsequent steps are executed.

[0087] Step 203: Assign target IDs to target objects in the current frame image based on the target position of each target object. The same target object in consecutive frame images has the same target ID.

[0088] In one embodiment, step 203 may specifically include steps 2031-2032:

[0089] Step 2031: Based on the target location, calculate the IOU value between the target object in the current frame image and each target object in the previous frame image. Each target object in the previous frame image has a target ID.

[0090] Each target object in the previous frame has been assigned a corresponding target ID. For example, a target object is selected in the current frame, and the Interchange of Union (IOU) value between this target object and each target object in the previous frame is calculated. In this case, each target object in the current frame corresponds to an IOU value with each target object in the previous frame. In one embodiment, the IOU value can be determined based on the target position of the target object in the current frame and the target position of the target object in the previous frame, i.e., by calculating the ratio of the intersection to the union of the regions corresponding to the target positions. Optionally, for the first frame, the target IDs of each target object can be directly assigned without referring to the previous frame or calculating IOU values.

[0091] Step 2032: Determine the target ID of the target object in the current frame image based on the IOU value.

[0092] The IOU value reflects the similarity between a target object in the current frame and a target object in the previous frame. A higher IOU value indicates greater similarity between the two target objects. Based on this similarity, the target object most similar to the target object in the current frame can be selected from the previous frame, and the target ID of the target object in the current frame can be determined based on the target ID of the selected target object. In one embodiment, step 2032 may include steps 20321-20322:

[0093] Step 20321: Select the maximum IOU value from all IOU values ​​corresponding to the target object in the current frame image.

[0094] The higher the IOU value, the greater the similarity between the two target objects. The highest IOU value corresponds to the highest similarity between the target object in the previous frame and the target object in the current frame, meaning they may be the same target object in different frames. Therefore, the highest IOU value is selected from all IOU values.

[0095] Step 20322: If the maximum IOU value is greater than the first threshold, obtain the target ID of the target object corresponding to the maximum IOU value in the previous frame image, and use the obtained target ID as the target ID of the target object in the current frame image.

[0096] The first threshold can be understood as the minimum Interchange of Union (IOU) value at which two target objects in two frames belong to the same target object. If the maximum IOU value is greater than the first threshold, it means that the two target objects corresponding to the maximum IOU value are the same target object. Therefore, the target ID of the target object in the previous frame corresponding to the maximum IOU value is used as the target ID of the target object in the current frame. For example, Figure 11 This is a schematic diagram of target ID allocation provided in one embodiment of this application, with reference to... Figure 11 The left side shows the target object in the (i-1)th frame (i.e., the previous frame), and the right side shows the target objects in the i-th frame (i.e., the current frame). For the target object in the top left corner of the i-th frame, after calculating its IOU value with each target object in the (i-1)th frame, the target object in the top left corner of the (i-1)th frame has the largest IOU value, which is greater than the first threshold. In this case, the target ID 001 of the target object in the (i-1)th frame corresponding to the largest IOU value is taken as the target ID of the target object in the i-th frame. It can be understood that the value of the first threshold can be set according to actual needs.

[0097] In one embodiment, the target object in the current frame image can also be a newly appearing object that is different from all the target objects in the previous frame image. In this case, step 2032 further includes step 20323:

[0098] Step 20323: If the maximum IOU value is less than the second threshold, then the target object in the current frame image is determined to be the first target object to appear, and a new target ID is assigned to the target object. The second threshold is less than or equal to the first threshold.

[0099] The second threshold can be understood as the maximum threshold at which two target objects in two frames do not belong to the same target object. Generally, the second threshold is less than or equal to the first threshold. If the maximum IOU value is less than the second threshold, it means that the target object in the previous frame corresponding to the maximum IOU value is not the same target object as the target object in the current frame. That is, each target object in the previous frame is not the same target object as the target object in the current frame. In this case, the target object in the current frame is determined to be the first target object to appear, and a new target ID is assigned to it. It is understood that the specific value of the second threshold can be set according to actual needs. Optionally, the comparison order of the first and second thresholds is not limited in the embodiment.

[0100] For example, refer to Figure 11In the i-th frame image (current frame image), for the target object with target ID 008, its corresponding maximum IOU value is less than the second threshold. Therefore, the target object is determined to be a newly appearing target object, and a new target ID is assigned to the target object, that is, 008 is assigned as the target ID.

[0101] In one embodiment, there is also a possibility that the target object may be missed due to the target object moving too fast or the behavior detection model being ineffective. In this case, the target object identified in the previous frame may temporarily disappear in the current frame. To avoid this situation affecting subsequent behavior recognition, steps 2033-2036 are included after step 2032:

[0102] Step 2033: Based on the target IDs of each target object in the previous frame image and the target IDs of each target object in the current frame image, find the target objects that do not appear in the current frame image.

[0103] For example, after assigning a target ID to each target object in the current frame image according to step 2032, the target IDs of each target object in the previous frame image are compared with the target IDs of each target object in the current frame image. If the target ID of a certain target object in the previous frame image does not appear in the current frame image, it is determined that the target object corresponding to that target ID in the previous frame image does not appear in the current frame image. By comparing each target ID in the two frames one by one, the target objects that do not appear in the current frame image can be determined.

[0104] Step 2034: Update the number of frames that have not appeared where the target object has not appeared.

[0105] After identifying a target object that is not present in the current frame, it is assumed that the target object will not appear in a new frame (the current frame). Therefore, the number of frames in which the target object has not appeared is incremented by 1. At this point, the number of frames in which the target object has not appeared reflects the number of frames since the target object disappeared. It can be understood that when there are multiple target objects that have not appeared, each target object corresponds to one number of frames in which it has not appeared.

[0106] Step 2035: If the number of frames does not reach the target number, add the missing target object and its corresponding target ID to the current frame image.

[0107] When a target object does not appear in the current frame, there are two possibilities: one is that the target object has disappeared from the current frame (i.e., moved out of the shooting area corresponding to the current frame); the other is that the target object has temporarily disappeared from the current frame for some reason or has not been identified, meaning the missing target object has not actually moved out of the shooting area corresponding to the current frame. To avoid the temporary disappearance of the target object affecting the final behavior recognition result, in one embodiment, after updating the number of missing frames for the target object, it is determined whether the number of missing frames has reached the target number of frames. The target number of frames can be set according to the actual situation, and it is the minimum number of frames required to determine that the target object has disappeared in the video data. If the target number of frames is not reached, the target object may be temporarily missing. Therefore, the missing target object and its corresponding target ID are added to the current frame image. Specifically, based on the target position of the missing target object in the previous frame image, a bounding box is selected at the same target position in the current frame image to serve as the detection bounding box for the missing target object. The target ID of the missing target object in the previous frame image is then used as the target ID of the missing target object in the current frame image for further detection. At this point, in subsequent processing, the final behavior recognition result of the missing target object will be obtained in the current frame image.

[0108] For example, refer to Figure 11 The target object with target ID 005 in the (i-1)th frame (previous frame) does not appear in the i-th frame (current frame). Therefore, based on the target position of the target object with target ID 005, the target object is retained at the same target position in the i-th frame, and the target ID of the target object is set to 005.

[0109] It is understandable that after retaining the target ID of the target object that did not appear in the current frame image, for the next frame image, it will also determine whether the target ID of the target object contained therein includes the target ID of the target object that did not appear. If it does not include it, the number of frames that did not appear is incremented by 1. If it does include it, it is considered that the target object has been detected again and the target object has not disappeared. Therefore, the number of frames that did not appear is reset to zero.

[0110] Step 2036: If the target frame number is not reached, then abandon the process of adding the missing target object and its corresponding target ID to the current frame image.

[0111] For example, when the target number of missing frames is reached, it can be assumed that the missing target object has disappeared from the video data. Therefore, the missing target object and its ID are no longer added to the current frame. In this case, the final behavior recognition result of the missing target object will not be obtained from the current frame during subsequent processing. Thus, when a target object disappears from the video data, its corresponding final behavior recognition result will continue for a period of time (the duration corresponding to the target frame number) until it is confirmed that the target object has disappeared, at which point the final behavior recognition result of that target object will no longer be output.

[0112] After assigning a target ID to the target object in step 203, proceed with the subsequent steps.

[0113] Step 204: Determine whether the target object is located within a permitted area based on its target location. If it is located within a permitted area, proceed to step 205; otherwise, abandon the processing of the target object.

[0114] In one embodiment, the captured video data is divided into permitted and prohibited areas. Permitted areas are those where the target object is allowed to exist; when the target object is within a permitted area, its behavior is recognized. Prohibited areas are those where the target object is not allowed to exist; when the target object is within a prohibited area, its behavior is not recognized. The number and location of permitted and prohibited areas can be set according to actual circumstances. For example, in a teaching recording scenario, the target object is set as a student. Since both students and teachers are human, when a teacher appears in the image, the behavior detection model will also obtain the teacher's target location and behavior recognition results. In this case, it is necessary to filter out the teacher's target location and behavior recognition results. Because teachers and students move in different areas in the classroom, the area where students sit is set as a permitted area, while the corridor area (where students do not sit) and the teacher's podium are set as prohibited areas. In a teaching scenario, the teacher is generally located at the podium or in the corridor area; therefore, the permitted and prohibited areas are used to filter out teachers, preventing invalid target objects from interfering with the behavior recognition results. Optionally, the permitted and prohibited areas can be adjusted according to the specific shape and size of the classroom, task settings, desk placement, etc. In one embodiment, both the permitted and prohibited areas are convex polygons.

[0115] For example, after obtaining the target location of the target object, it is determined whether the target object is located within the permitted area based on the target location. In one embodiment, the target location of the target object is a rectangular area location, which is the location of the detection box (i.e., the rectangle) identifying the target object. Determining whether the target object is located within the permitted area can be: judging whether the rectangular area location of the target object is within the permitted area; if it is, then the target object is determined to be within the permitted area; otherwise, the target object is determined not to be within the permitted area; or, calculating the intersection area between the rectangular area location of the target object and the permitted area, and determining whether the target object is located within the permitted area based on the intersection area. In one embodiment, taking the determination of whether the target object is located within the permitted area based on the intersection area as an example, step 204 may include steps 2041-2044:

[0116] Step 2041: Obtain the location of the first area of ​​the pre-defined permitted area and the location of the second area of ​​the prohibited area.

[0117] The first region position represents the location of the permitted region in the image, and the second region position represents the location of the prohibited region in the image. Optionally, the first and second region positions can be determined manually, or machine learning can be used to identify the first region position of the permitted region and the second region position of the prohibited region in each frame of the image.

[0118] Step 2042: If none of the corner points of the target object are within the second region, then determine the intersection of the target location and the first region.

[0119] The corner point position refers to the location of the corner point within the target position. For a rectangular area, the corner point can also be understood as a vertex. The intersection area position refers to the location of the area where the target position and the first area position intersect. The intersection area is determined based on the first area position and the target position.

[0120] For example, if none of the corner points of the rectangular region containing the target object are located within the second region, it indicates that the target object is not within the prohibited region. If at least one corner point is located within the second region, it indicates that the target object is within the prohibited region. In one embodiment, the detection box corresponding to the rectangular region is a non-tilted rectangle in each frame image. Therefore, when determining whether the target object is located within the prohibited region, it can be determined by judging whether one of the two corner points (the top two or the bottom two) is located within the second region. If one corner point is located within the second region, it is determined that the target object is within the prohibited region, and the final behavior recognition of the target object within the prohibited region is abandoned. For example, Figure 12This is a schematic diagram of a prohibited area provided for one embodiment of this application. Figure 12 In the process, the lower right corner of the rectangular region Z1 containing the target object is located within the second region Z2 of the prohibited region. At this point, it is determined that the target object is located within the prohibited region, and the target objects located within the prohibited region are filtered.

[0121] Next, for target objects that are not in the prohibited area, calculate the intersection area between the rectangular area position of the target object and the first area position.

[0122] Step 2043: Calculate the area ratio of the intersection region location to the first region location.

[0123] For the permitted region, the area ratio of the intersection region location to the first region location is first calculated. In one embodiment, this area ratio is recorded as the pseudo IOU value, where the formula for calculating the pseudo IOU value is as follows:

[0124]

[0125] For example, Figure 13 A schematic diagram of a licensed area provided for one embodiment of this application, in Figure 13 In the diagram, S1 represents the area of ​​the first region, the solid rectangle represents the rectangular region of the target object, and S0 represents the area of ​​the intersection region between the rectangular region and the first region. The area ratio, or pseudo-IOU value, is calculated using the following formula:

[0126]

[0127] Step 2044: If the area ratio is greater than or equal to the ratio threshold, then the target object is determined to be located in the permitted area.

[0128] The ratio threshold represents the minimum critical value of the area ratio required for a target object to be located within a permitted area. For example, given a ratio threshold μ, if the area ratio is greater than or equal to the ratio threshold μ, the target object is determined to be within a permitted area, so that only the behavior of target objects within the permitted area is identified in subsequent processes. Otherwise, the target object is determined to be outside the permitted area. It is understood that the specific value of the ratio threshold can be set according to actual needs.

[0129] For example, if the target object is located in a permitted area, then step 205 is performed.

[0130] Step 205: Based on the target ID of the target object in the current frame image, obtain the behavior recognition result of the target object in the continuous frame images, and based on the behavior recognition result of the target object in the continuous frame images, obtain the final behavior recognition result of the target object in the current frame image. The continuous frame images include the current frame image.

[0131] For example, if the target object is located within the permitted area, it means that it is necessary to determine the final behavior recognition result of the target object. Therefore, it is necessary to perform the operation of obtaining the behavior recognition result of the target object in consecutive frame images based on the target ID of the target object in the current frame image.

[0132] In one embodiment, obtaining the behavior recognition result of the target object in consecutive frame images based on the target ID of the target object in the current frame image includes steps 2051-2052:

[0133] Step 2051: Write the target ID and behavior recognition result of each target object in the current frame image to the double-ended queue, and remove the target ID and behavior recognition result of each target object in the first frame image from the double-ended queue. The double-ended queue contains consecutive frame images.

[0134] A double-ended queue (deque) allows enqueue and dequeue operations at both ends. The logical structure of elements in a deque is linear. The two ends of the deque are called the front and back ends, respectively, and insertion and deletion operations can be performed at both ends. For example, Figure 14 This is a schematic diagram of a double-ended queue structure provided in one embodiment of this application, with reference to... Figure 14 The system continuously updates the deque by storing the target IDs and behavior recognition results of each object in the current frame as a set of data at the back end of a double-ended queue. Simultaneously, it removes the target IDs and behavior recognition results of the first frame from the front end of the deque, thus updating the queue to ensure it contains the target IDs and behavior recognition results of the latest frame. The number of elements in the deque can be adjusted as needed. Figure 14 The presence of N (N is a positive integer) frames in the double-ended queue is merely an illustrative description, and the number of elements in the double-ended queue is not limited in the embodiments.

[0135] Since the target ID and behavior recognition results corresponding to each frame are written sequentially into the deque, the frames contained in the deque are consecutive frames and include the current frame. Optionally, the elements contained in the deque will not be processed until the deque is full.

[0136] Step 2052: Based on the target ID of the target object in the current frame image, obtain the recognition results of each behavior of the target object in the double-ended queue.

[0137] For example, since the target ID is a unique identifier that distinguishes target objects, based on the target ID of the target object in the current frame image, target objects with the same target ID can be identified in the double-ended queue (i.e., consecutive frame images). Then, the behavior recognition result of the target object is obtained in the consecutive frame images. That is, the behavior recognition results corresponding to the same target ID are searched in each image contained in the double-ended queue. At this time, multiple behavior recognition results are found, and the number of behavior recognition results is less than or equal to the number of consecutive frame images.

[0138] After finding the behavior recognition results corresponding to the same target ID, the final behavior recognition result of the target object in the current frame image can be obtained based on the behavior recognition results. At this time, the final behavior recognition result of the target object in the current frame image is obtained based on the behavior recognition results of the target object in consecutive frame images. This includes: if the number of times the behavior recognition result of the target object in the current frame image appears in consecutive frame images exceeds the number threshold, then the behavior recognition result is taken as the final behavior recognition result of the target object in the current frame image.

[0139] For a target object in the current frame image, after determining the behavior recognition results for each of its corresponding actions in consecutive frames, identify the behavior recognition results that are identical to those in the current frame image and count the number of times these identical behavior recognition results occur. If the number of occurrences reaches (greater than or equal to) a preset threshold, it indicates that the target object has been identified multiple times in consecutive frames as having the behavior recognition result corresponding to the current frame image. Therefore, the behavior recognition result of the current frame image is taken as the final behavior recognition result for the target object in the current frame image. If the number of occurrences does not reach the preset threshold, it indicates that the target object has been identified a relatively small number of times in consecutive frames as having the behavior recognition result corresponding to the current frame image. In this case, it is determined that there is no corresponding final behavior recognition result for the target object in the current frame image, or the final behavior recognition result for the target object in the current frame image is determined as the default behavior recognition result. After processing each target object in the current frame image in the above manner, the final behavior recognition result for each target object can be determined. Optionally, after processing the current frame image, the next frame image can be used as the current frame image and the above processing can continue to be performed, so as to determine the final behavior recognition result of the target object in each frame image in real time during the acquisition of video data.

[0140] For example, if a target object in the current frame is identified as "standing," and the number of times this target object is identified as "standing" in consecutive frames (i.e., the number of times the behavior recognition result of "standing" appears) is greater than or equal to a threshold, then "standing" is taken as the final behavior recognition result for the target object in the current frame. It should be noted that determining the final behavior recognition result of a target object based solely on a single frame may lead to errors. Since each action of the target object lasts for a certain period of time (i.e., it is continuous), the final behavior recognition result of the target object in the current frame can be determined by examining the behavior recognition results of the target object in consecutive frames, thereby improving the accuracy of the behavior recognition structure.

[0141] In one embodiment, after determining the final behavior recognition result of the target object in the current frame image, the final behavior recognition result of the target object in the current frame image, the target position of the target object in the current frame image, and the confidence level are packaged into a data packet and then processed. For example, after obtaining the data packet, it is sent to other background applications. After receiving the data packet, the other background applications perform corresponding processing according to task requirements. For example, if the other background applications, after receiving the data packet, identify the final behavior recognition result of a target object in the current frame image as "standing up", they will segment a local image containing the target object from the current frame image based on the target position of the target object and send the local image of the target object to the large screen for display. Alternatively, they can control the camera to track and capture the target object based on the target position of the target object until the other applications determine the final behavior recognition result of the target object as "sitting down" based on the received data packet, and then control the camera to resume global shooting.

[0142] The above-described method addresses the technical challenge of accurately identifying student behavior in teaching recording scenarios by acquiring video data captured by a camera, using a behavior detection model to identify the target location and behavior of each object within the current frame of the video data, assigning target IDs to each object based on its target location (the same target object has the same target ID in consecutive frames), and obtaining the behavior recognition result of the target object in consecutive frames based on its target ID located within the permitted area in the current frame. This method then yields the final behavior recognition result of the target object in the current frame based on the behavior recognition results in consecutive frames. Specifically, the behavior detection model extracts a first feature containing limb-related and spatial-related features, and uses this first feature to identify the target location and behavior detection result of the target object in the image, improving the accuracy of student behavior recognition. Furthermore, the structure of the behavior detection model can be simplified according to actual needs, ensuring processing speed and meeting the requirements of real-time front-end deployment. By combining the target ID of the target object in the previous frame to assign a target ID to the target object in the current frame, rapid target ID assignment can be achieved. Furthermore, target objects that have not appeared in the current frame and whose absence count has not reached a threshold, along with their target IDs, are retained, avoiding the impact of missed target object identification. By setting permitted and prohibited regions, other objects in the image can be filtered out to prevent obtaining the final behavior recognition result for other objects. Moreover, by acquiring the behavior recognition results of the target object in consecutive image frames and confirming the final behavior recognition result based on these consecutive frame images, the accuracy of target object behavior recognition is improved, avoiding reliance on a single image. This results in a behavior recognition method that is robust, accurate, and easy to deploy.

[0143] Figure 15 The following is a schematic diagram of the structure of a behavior recognition device provided in one embodiment of this application. Please refer to 15. The device includes: a video acquisition module 301, a recognition module 302, an ID allocation module 303, and a result acquisition module 304.

[0144] The video acquisition module 301 is used to acquire video data captured by a camera, which includes multiple frames of images; the recognition module 302 is used to identify each target object in the current frame image using a behavior detection model to obtain the target position and behavior recognition result of the target object; the ID allocation module 303 is used to assign a target ID to the target object in the current frame image based on the target position, and the same target object in consecutive frame images has the same target ID; the result acquisition module 304 is used to obtain the behavior recognition result of the target object in consecutive frame images based on the target ID of the target object in the current frame image, and obtain the final behavior recognition result of the target object in the current frame image based on the behavior recognition result of the target object in consecutive frame images, where consecutive frame images include the current frame image.

[0145] Based on the above embodiments, the behavior detection model includes a behavior attention module, a local spatial convolution module, a feature recognition module, and an object detection module. Correspondingly, the recognition module 302 includes a first feature extraction submodule, used to extract first features of each target object in the current frame image using the behavior attention module. The first features include limb association features and spatial association features. The limb association features reflect the association features between each target limb in the target object, and the spatial association features reflect the association features between the target limb and a spatial object, where the spatial object is an object that has a spatial relationship with the target object. A second feature learning submodule is used to perform deep learning on the first features using the local spatial convolution module to output multiple second features with different numbers of channels. A second feature recognition submodule is used to recognize the second features using the feature recognition module and output the feature recognition results, with each second feature corresponding to one feature recognition result. A detection submodule is used by the object detection module to detect the target position and behavior recognition results of each target object based on the feature recognition results. Each feature recognition result corresponds to one object detection module, and the target objects detected by each object detection module have different sizes.

[0146] Based on the above embodiments, the behavioral attention module includes a content-aware attention submodule and at least one convolutional submodule. The convolutional submodule is used to extract features from the input data, and the content-aware attention submodule is used to obtain features related to limb association features and features related to spatial association features based on the extracted features; or, the behavioral attention module includes a spatial attention submodule and at least one convolutional submodule. The convolutional submodule is used to extract features from the input data, and the spatial attention submodule is used to obtain features related to spatial association features and features related to limb association features based on the extracted features; or, the behavioral attention module includes a content-aware attention submodule, a spatial attention submodule, and at least one convolutional submodule. The convolutional submodule is used to extract features from the input data, and the content-aware attention submodule is used to obtain features related to limb association features based on the extracted features, and the spatial attention submodule is used to obtain features related to spatial association features based on the extracted features.

[0147] Based on the above embodiments, the ID allocation module 303 includes: an IOU value calculation submodule, used to calculate the IOU value between the target object in the current frame image and each target object in the previous frame image according to the target position, wherein each target object in the previous frame image has a target ID; and an ID determination submodule, used to determine the target ID of the target object in the current frame image according to the IOU value.

[0148] Based on the above embodiments, the ID determination submodule includes an IOU value selection unit, which is used to select the maximum IOU value from all IOU values ​​corresponding to the target object in the current frame image; and an ID acquisition unit, which is used to acquire the target ID of the target object corresponding to the maximum IOU value in the previous frame image if the maximum IOU value is greater than a first threshold, and use the acquired target ID as the target ID of the target object in the current frame image.

[0149] Based on the above embodiments, the ID determination submodule further includes an ID assignment unit, which is used to determine that the target object in the current frame image is the first target object to appear if the maximum IOU value is less than the second threshold, and assign a new target ID to the target object, wherein the second threshold is less than or equal to the first threshold.

[0150] Based on the above embodiments, the ID determination submodule includes a target object lookup unit, used to determine the target ID of the target object in the current frame image according to the IOU value, and then look up the target objects that do not appear in the current frame image according to the target IDs of each target object in the previous frame image and the target IDs of each target object in the current frame image; an update unit, used to update the number of non-appearance frames for which no target object appears; an ID retention unit, used to add the non-appearance target object and its corresponding target ID to the current frame image if the number of non-appearance frames does not reach the target number of frames; and an ID abandonment unit, used to abandon adding the non-appearance target object and its corresponding target ID to the current frame image if the number of non-appearance frames reaches the target number of frames.

[0151] Based on the above embodiments, it further includes: a region determination module, used to determine whether the target object is located within a permitted region based on the target position of the target object before obtaining the behavior recognition result of the target object in the continuous frame images according to the target ID of the target object in the current frame image; and an action execution module, used to execute the operation of obtaining the behavior recognition result of the target object in the continuous frame images according to the target ID of the target object in the current frame image if the target object is located within a permitted region.

[0152] Based on the above embodiments, the target location of the target object is the rectangular area containing the target object; the area determination module further includes a location acquisition submodule, used to acquire the first area location of the pre-set permitted area and the second area location of the prohibited area; an intersection area location determination submodule, used to determine the intersection area location of the target location and the first area location if none of the corner points of the target location of the target object are within the second area location; an area ratio calculation submodule, used to calculate the area ratio between the intersection area location and the first area location; and a permitted area determination submodule, used to determine that the target object is located within the permitted area if the area ratio is greater than or equal to a ratio threshold.

[0153] Based on the above embodiments, the behavior recognition result acquisition module 304 includes a double-ended queue writing submodule, which is used to write the target ID and behavior recognition result of each target object in the current frame image into the double-ended queue, and remove the target ID and behavior recognition result of each target object in the first frame image from the double-ended queue. The double-ended queue contains consecutive frame images. The result acquisition submodule is used to acquire the behavior recognition results of each target object in the double-ended queue according to the target ID of the target object in the current frame image.

[0154] Based on the above embodiments, the behavior recognition result acquisition module 304 includes a final behavior recognition result determination submodule, which is used to take the behavior recognition result as the final behavior recognition result of the target object in the current frame image if the number of times the behavior recognition result of the target object appears in consecutive frame images exceeds a number threshold.

[0155] The behavior recognition device provided above can be used to execute the behavior recognition method provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0156] It is worth noting that in the embodiments of the above-mentioned behavior recognition device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of this application.

[0157] Figure 16 This is a schematic diagram of the structure of a behavior recognition device provided in one embodiment of this application, as shown below. Figure 16 As shown, the behavior recognition device includes a processor 400, a memory 401, an input device 402, and an output device 403; the number of processors 400 in the behavior recognition device can be one or more. Figure 16 Taking a processor 400 as an example; in a behavior recognition device, the processor 400, memory 401, input device 402, and output device 403 can be connected via a bus or other means. Figure 16 Taking the example of a connection between China and Israel via a bus.

[0158] The memory 401, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the behavior recognition method in the embodiments of this application (e.g., the video acquisition module 301, recognition module 302, ID allocation module 303, and result acquisition module 304 in the behavior recognition device). The processor 400 executes various functional applications and data processing of the behavior recognition device by running the software programs, instructions, and modules stored in the memory 401, thereby realizing the aforementioned behavior recognition method.

[0159] The memory 401 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on the use of the behavior recognition device. Furthermore, the memory 401 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 401 may further include memory remotely located relative to the processor 400, which can be connected to the behavior recognition device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0160] Input device 402 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the behavior recognition device. Output device 403 may include a display device such as a screen. In addition, the behavior recognition device may also include a communication device to enable communication with external devices (such as cameras).

[0161] The aforementioned behavior recognition device includes a corresponding behavior recognition apparatus, which can be used to execute any behavior recognition method and has corresponding functions and beneficial effects.

[0162] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and equipment can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0163] Furthermore, embodiments of this application also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform relevant operations in the behavior recognition method provided in any embodiment of this application, and have corresponding functions and beneficial effects.

[0164] Those skilled in the art will understand that embodiments of this application may be provided as methods, systems, or computer program products.

[0165] Therefore, this application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce implementations of the flowchart... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0166] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory. Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0167] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0168] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0169] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application for those skilled in the art.

Claims

1. A behavior recognition method, characterized by, include: Acquire video data captured by a camera, the video data containing multiple frames of images; The behavior detection model is used to identify each target object in the current frame image, so as to obtain the target location and behavior recognition result of the target object; Based on the target location, a target ID is assigned to the target object in the current frame image, and the same target object in consecutive frame images has the same target ID; Based on the target ID of the target object in the current frame image, obtain the behavior recognition result of the target object in the continuous frame images, and based on the behavior recognition result of the target object in the continuous frame images, obtain the final behavior recognition result of the target object in the current frame image, wherein the continuous frame images include the current frame image; The behavior detection model includes a behavior attention module, a local spatial convolution module, a feature recognition module, and an object detection module. The step of using a behavior detection model to identify each target object within the current frame image, and obtaining the target location and behavior recognition results of the target objects, includes: The behavior attention module is used to extract the first feature of each target object in the current frame image. The first feature includes limb association feature and spatial association feature. The limb association feature is used to reflect the association feature between each target limb in the target object. The spatial association feature is used to reflect the association feature between the target limb and a spatial object. The spatial object is an object that has a spatial relationship with the target object. The first feature is subjected to deep learning using the local spatial convolution module to output multiple second features with different numbers of channels; The feature recognition module is used to identify the second feature and output the feature recognition result. Each second feature corresponds to one feature recognition result. The object detection module detects the target position and behavior recognition result of each target object based on the feature recognition result. Each feature recognition result corresponds to one object detection module, and the target object detected by each object detection module has a different size.

2. The behavior recognition method of claim 1, wherein, The behavior attention module includes a content-aware attention submodule and at least one convolutional submodule. The convolutional submodule is used to extract features from the input data, and the content-aware attention submodule is used to obtain features related to the limb association features and features related to the spatial association features based on the extracted features. Alternatively, the behavioral attention module includes a spatial attention submodule and at least one convolutional submodule. The convolutional submodule is used to extract features from the input data, and the spatial attention submodule is used to obtain features related to the spatial association features and features related to the limb association features based on the extracted features. Or, the behavior attention module comprises a content-aware attention submodule, a spatial attention submodule, and at least one convolutional submodule, the convolutional submodule being configured to perform feature extraction on input data, the content-aware attention submodule being configured to obtain features related to the limb-associated features based on the extracted features, and the spatial attention submodule being configured to obtain features related to the spatial-associated features based on the extracted features.

3. The behavior recognition method of claim 1, wherein, The assigning of the target ID to the target object in the current frame image based on the target position comprises: According to the target position, calculating an IOU value between the target object in the current frame image and each target object in a previous frame image, each target object in the previous frame image having a target ID; According to the IOU value, determining the target ID of the target object in the current frame image.

4. The behavior recognition method of claim 3, wherein, The determining of the target ID of the target object in the current frame image according to the IOU value comprises: selecting a maximum IOU value from all the IOU values corresponding to the target object in the current frame image; if the maximum IOU value is greater than a first threshold value, obtaining the target ID of the target object corresponding to the maximum IOU value in the previous frame image, and taking the obtained target ID as the target ID of the target object in the current frame image.

5. The behavior recognition method of claim 4, wherein, Further comprising: if the maximum IOU value is less than a second threshold value, determining that the target object in the current frame image is a first-time appearing target object, and assigning a new target ID to the target object, the second threshold value being less than or equal to the first threshold value.

6. The behavior recognition method of claim 3, wherein, After the determining of the target ID of the target object in the current frame image according to the IOU value, comprising: According to the target ID of each target object in the previous frame image and the target ID of each target object in the current frame image, finding a target object that does not appear in the current frame image; updating the number of non-appearing frames of the non-appearing target object; if the number of non-appearing frames does not reach a target frame number, adding the non-appearing target object and the corresponding target ID in the current frame image; if the number of non-appearing frames reaches the target frame number, giving up adding the non-appearing target object and the corresponding target ID in the current frame image.

7. The behavior recognition method of claim 1, wherein, Before the obtaining of the behavior recognition result of the target object in the continuous frame images according to the target ID of the target object in the current frame image, comprising: determining whether the target object is located in a permitted area according to the target position of the target object; if located in the permitted area, performing the operation of obtaining the behavior recognition result of the target object in the continuous frame images according to the target ID of the target object in the current frame image.

8. The behavior recognition method of claim 7, wherein, The target position of the target object is a rectangular area position identifying the target object. The determining of whether the target object is located in the permitted area according to the target position of the target object comprises: obtaining a first area position of a permitted area and a second area position of a prohibited area that are preset; If each corner position in the target position of the target object is not in the second region position, a region position of an intersection of the target position and the first region position is determined; An area ratio of the intersection region position and the first region position is calculated; If the area ratio is greater than or equal to a ratio threshold, it is determined that the target object is located in the permitted region.

9. The behavior recognition method of claim 1, wherein, The obtaining of the behavior recognition result of the target object in the continuous frame image according to the target ID of the target object in the current frame image comprises: The target ID and the behavior recognition result of each target object in the current frame image are written into a double-ended queue, and the target ID and the behavior recognition result of each target object in the first frame image in the double-ended queue are removed, the frame images contained in the double-ended queue being continuous frame images; The behavior recognition result of the target object in the double-ended queue is obtained according to the target ID of the target object in the current frame image.

10. The behavior recognition method of claim 1 or 9, wherein, The final behavior recognition result of the target object in the current frame image is obtained according to the behavior recognition result of the target object in the continuous frame image. If the behavior recognition result of the target object in the current frame image appears in the continuous frame image more than a threshold number of times, the behavior recognition result is taken as the final behavior recognition result of the target object in the current frame image.

11. A behavior recognition apparatus, characterized by comprising: The device comprises: A video acquisition module is configured to acquire video data captured by a camera, the video data comprising a plurality of frame images; An identification module is configured to identify each target object in a current frame image by using a behavior detection model to obtain a target position and a behavior recognition result of the target object; An ID allocation module is configured to allocate a target ID to the target object in the current frame image based on the target position, the same target object in continuous frame images having the same target ID; A result acquisition module is configured to obtain a behavior recognition result of the target object in continuous frame images according to a target ID of the target object in the current frame image, and obtain a final behavior recognition result of the target object in the current frame image according to the behavior recognition result of the target object in the continuous frame images, the continuous frame images comprising the current frame image; The behavior detection model comprises a behavior attention module, a local spatial convolution module, a feature recognition module, and an object detection module; The identification module comprises: A first feature extraction submodule is configured to extract a first feature of each target object in a current frame image by using the behavior attention module, the first feature comprising a limb correlation feature and a spatial correlation feature, the limb correlation feature being used to reflect a correlation feature between target limbs in the target object, and the spatial correlation feature being used to reflect a correlation feature between the target limbs and a spatial object, the spatial object being an object having a spatial relationship with the target object; A second feature learning submodule is configured to perform deep learning on the first feature by using the local spatial convolution module to output a plurality of second features having different channel numbers. a second feature recognition submodule, configured to recognize the second features by using the feature recognition module, and output feature recognition results, each second feature corresponding to a feature recognition result; a detection submodule, configured to detect target positions and behavior recognition results of the target objects according to the feature recognition results by using the object detection module, each feature recognition result corresponding to one object detection module, and each object detection module detecting target objects of different sizes.

12. A behavior recognition device, comprising: comprising: one or more processors; a memory for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the behavior recognition method according to any one of claims 1-10.

13. A storage medium storing computer-executable instructions, wherein: The computer executable instructions, when executed by a computer processor, are used to perform the behavior recognition method according to any one of claims 1-10.

Images