Classroom action recognition method, computer equipment and readable storage medium

Abstract

The embodiment of the invention relates to the technical field of education, and discloses a classroom action recognition method, computer equipment and a readable storage medium. The method comprises the following steps: extracting multiple frames of classroom images from a classroom video for target detection processing to obtain at least one initial video clip, merging the at least one initial video clip meeting an action continuity condition into a candidate video clip, performing action recognition processing on the target video clip, and obtaining a candidate video clip; and sending the target video clip and the preset cue word to the target video large model, so that the target video large model outputs the target description information. According to the embodiment of the invention, the initial video clips belonging to the same effective action can be merged into one candidate video clip, so that the situation that the continuous action is split into a plurality of frames of classroom images for independent recognition, so that misjudgment is easy to occur can be avoided, and the target video large model has higher and better understanding ability for the video clips with the continuous action; and video clips containing effective actions can be screened out more reliably and accurately.

Description

Technical Field

[0001] This application relates to the field of educational technology, specifically to a method for recognizing classroom actions, a computer device, and a readable storage medium. Background Technology

[0002] Related technologies can capture classroom images and then analyze them using image models to determine student actions, such as raising a hand, standing, scratching a head, or resting on a desk, to reflect student behavior and help teachers manage students more effectively. However, different actions may have some similarities; for example, raising a hand and raising a cup to drink water are somewhat similar. When analyzing single-frame classroom images, the image models provided by these technologies are prone to misidentifying actions, such as misidentifying drinking water as raising a hand. Therefore, the accuracy and effectiveness of using these image models to identify student actions in the classroom are not high. Summary of the Invention

[0003] One objective of this application is to provide a method, computer device, and readable storage medium for recognizing classroom actions, in order to solve the technical problem of low accuracy in action screening in related technologies.

[0004] In a first aspect, embodiments of this application provide a method for recognizing classroom actions, comprising: acquiring a classroom video; extracting multiple frames of classroom images from the classroom video for target detection processing to obtain at least one initial video segment, wherein the initial video segment is a video segment corresponding to a person with a valid action being detected; merging at least one initial video segment that satisfies the action continuity condition into a candidate video segment; performing action recognition processing on the target video segment based on a preset video classification model to obtain action recognition information, wherein the action recognition information includes a first type of recognition information and a second type of recognition information, wherein the first type of recognition information is used to indicate that the preset video classification model has identified that the target video segment contains a valid action, and the second type of recognition information is used to indicate that the preset video classification model has not identified that the target video segment contains a valid action, wherein the target video segment is one of the candidate video segments in the at least one candidate video segment; in response to the action recognition information being the second type of recognition information, sending the target video segment and a preset prompt word to a target video model, so that the target video model responds to the preset prompt word and outputs target description information of the target video segment, wherein the target description information is used to describe whether the target video segment contains a valid action.

[0005] In the initial screening stage, single-frame images are used as the screening granularity (video clips are used as the screening granularity in the secondary screening stage below). Initial video clips with valid actions are generated. Images that do not contain valid actions or whose actions are obviously invalid will not participate in subsequent processing, which helps to improve the accuracy of action recognition.

[0006] The embodiments of this application can merge initial video segments belonging to the same valid action into a single candidate video segment. This avoids the situation where continuous actions are split into multiple frames of classroom images for independent recognition, which can easily lead to misjudgment. Furthermore, compared to the way the target video big model understands single-frame classroom images, the target video big model has a stronger and better understanding of video segments with continuous actions, and can more reliably and accurately select video segments containing valid actions.

[0007] This application embodiment, after performing target detection processing on multiple frames of classroom images in the initial screening stage, can filter out images of student figures who do not perform effective actions, thereby reducing the number of images that need to be screened again in subsequent steps and improving overall processing efficiency. However, since effective actions are represented by a continuous sequence of images, using only image frame-based detection methods can easily reduce the accuracy of action recognition. In this application embodiment, each candidate video segment is screened from the perspective of video clips to obtain at least one target video segment. This allows video clips that do not contain effective actions or whose actions are obviously invalid to be eliminated, and the obtained target video clips are more likely to contain effective actions.

[0008] This application embodiment uses a large target video model to perform in-depth analysis and understanding of the target video segment in order to determine whether the target video segment contains the target action, thereby improving the accuracy of action recognition.

[0009] Optionally, the step of performing action recognition processing on the target video segment based on a preset video classification model to obtain action recognition information includes: performing action recognition processing on the target video segment based on a preset video classification model to obtain a probability set, the probability set including multiple recognition probability values, the recognition probability values ​​being the probability that the action contained in the target video segment belongs to the corresponding action tag, and generating action recognition information based on the probability set.

[0010] Optionally, generating action recognition information based on the probability set includes: determining a first maximum probability value of the first probability set, wherein the first maximum probability value is the recognition probability value with the largest value in the first probability set; generating a first type of recognition information in response to the first maximum probability value being greater than or equal to a preset probability threshold, wherein the action contained in the target video segment belongs to the action tag corresponding to the first maximum probability value; and generating a second type of recognition information in response to the first maximum probability value being less than the preset probability threshold.

[0011] Optionally, before generating the second type of recognition information, the recognition method further includes: in response to the first maximum probability value being less than a preset probability threshold, determining classroom scene description information corresponding to the target video segment, determining a target probability threshold corresponding to the first maximum probability value based on the classroom scene description information, and generating action recognition information based on the first maximum probability value and the target probability threshold.

[0012] Optionally, determining the classroom scene description information corresponding to the target video segment includes: determining the timestamp of the target video segment; determining a target time period based on the timestamp of the target video segment, wherein the target time period includes the timestamp of the target video segment; obtaining reference video segments, wherein the reference video segments are classified video segments with timestamps within the target time period, wherein the classified video segments are candidate video segments that have undergone action recognition processing by the preset video classification model and output a second probability set, wherein the second probability set includes multiple recognition probability values, wherein the recognition probability values ​​are the probability that the action contained in the candidate video segment belongs to the corresponding action tag; determining the second maximum probability value in the second probability set of each reference video segment and the action tag corresponding to the second maximum probability value as candidate tags, wherein the second maximum probability value is the recognition probability value with the largest value in the second probability set; and generating classroom scene description information corresponding to the target video segment based on all candidate tags within the target time period.

[0013] Optionally, the classroom scene description information includes single group action information, mixed group action information, and no group action information. The step of generating classroom scene description information corresponding to the target video segment based on all candidate tags within the target time period includes: classifying all candidate tags within the target time period according to action type to obtain the number of targets belonging to each action type; generating no group action information in response to all target numbers being less than a preset number threshold; determining the number of targets greater than or equal to the preset number threshold in response to at least one target number being greater than or equal to the preset number threshold; generating single group action information in response to the number being 1; and generating mixed group action information in response to the number being greater than 1.

[0014] Optionally, the classroom scene description information includes single group action information, mixed group action information, and no group action information. Determining the target probability threshold corresponding to the first maximum probability value based on the classroom scene description information includes: in response to the classroom scene description information being single group action information, and the group action represented by the single group action information belonging to the action tag corresponding to the first maximum probability value, updating the preset probability threshold to the target probability threshold, where the target probability threshold is less than or equal to the preset probability threshold; or, in response to the classroom scene description information being single group action information, and the group action represented by the single group action information not belonging to the action tag corresponding to the first maximum probability value, maintaining the preset probability threshold corresponding to the first maximum probability value unchanged; or, in response to the classroom scene description information being mixed group action information, maintaining the preset probability threshold corresponding to the first maximum probability value unchanged; or, in response to the classroom scene description information being no group action information, maintaining the preset probability threshold corresponding to the first maximum probability value unchanged.

[0015] Optionally, generating action recognition information based on the first maximum probability value and the target probability threshold includes: generating a first type of recognition information in response to the first maximum probability value being greater than or equal to the target probability threshold, wherein the action contained in the target video segment belongs to the action tag corresponding to the first maximum probability value; and generating a second type of recognition information in response to the first maximum probability value being less than the target probability threshold.

[0016] Optionally, the step of extracting multiple frames of classroom images from the classroom video for target detection processing to obtain at least one initial video segment includes: extracting multiple frames of classroom images from the classroom video, performing target detection processing on the classroom images to obtain at least one target region image, wherein the target region image is a person region image in the classroom image corresponding to a person with effective actions, determining the image acquisition time of the target region image, and determining an initial video segment of a specified duration based on the image acquisition time of the target region image, wherein the initial video segment includes the target region image.

[0017] The effective action corresponding to the target area image is a continuous action. In this embodiment, the initial video segment with a time range is determined based on the image acquisition time of the target area image. This allows the initial video segment to cover the presentation process of the effective action as much as possible. This helps the target video large model to more accurately and reliably select the final video segment containing the effective action, thereby improving the accuracy of action recognition.

[0018] Optionally, the conditional elements of the action continuity condition include video time overlap and character consistency. Merging at least one of the initial video segments that satisfy the action continuity condition into a candidate video segment includes: determining at least two initial video segments with video time overlap as video segments to be screened, wherein the classroom image in the video segments to be screened contains a character area image corresponding to a character with valid actions, and merging at least two video segments to be screened with consistent characters into a candidate video segment.

[0019] This application embodiment, based on the two key conditions of video temporal overlap and person consistency, can quickly and effectively merge at least two video segments to be screened with consistent persons into candidate video segments. These candidate video segments comprehensively cover the entire process of the same person performing an effective action, which helps the subsequent target video model to more comprehensively and effectively understand the candidate video segments, thereby accurately and reliably selecting the final video segment containing the effective action. Furthermore, by merging at least two initial video segments belonging to the same effective action, this application embodiment reduces the number of video segments requiring processing, reduces the time and resource consumption of overlap calculations, and improves action recognition efficiency.

[0020] Optionally, merging at least two video segments to be screened with identical characters into a candidate video segment includes: performing target detection processing on the classroom image in each video segment to be screened to obtain a character detection box corresponding to the character region image; the character detection box corresponding to one of the at least two video segments to be screened is the target detection box; the character detection boxes corresponding to the remaining video segments to be screened in the at least two video segments to be screened are all reference detection boxes; calculating the similarity between the target detection box and each of the reference detection boxes to obtain multiple similarity values; finding the maximum similarity value among the multiple similarity values; if the maximum similarity value is greater than a preset similarity threshold, merging the video segment to be screened with the target detection box corresponding to the maximum similarity value and the video segment to be screened with the reference detection box corresponding to the maximum similarity value to obtain a candidate video segment.

[0021] Optionally, the recognition method further includes: determining candidate video segments whose recognition probability values ​​fall within a preset probability range as difficult samples, obtaining a video training set including difficult samples, and fine-tuning a preset large video model based on the video training set to obtain a target large video model.

[0022] Candidate video segments whose probability values ​​fall within a preset probability range are easy to obtain, and these candidate video segments are considered hard samples. Training a large video model with these hard samples can improve the convergence rate of the large video model, thereby improving the training efficiency of the large video model.

[0023] In a second aspect, embodiments of this application provide a computer device including a memory and a processor. The memory is connected to the processor, and the processor is configured to execute one or more computer programs stored in the memory. When the processor executes the one or more computer programs, it enables the computer device to implement the aforementioned classroom action recognition method.

[0024] In a third aspect, embodiments of this application provide a computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, the computer program including program instructions, which, when executed by a processor, cause the processor to perform the aforementioned classroom action recognition method. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A schematic diagram of the structure of a classroom action recognition system provided in an embodiment of this application;

[0027] Figure 2 A flowchart illustrating a classroom action recognition method provided in an embodiment of this application;

[0028] Figure 3 A schematic diagram of a classroom scene provided for an embodiment of this application;

[0029] Figure 4 A schematic diagram of the structure of a classroom action recognition device provided in an embodiment of this application;

[0030] Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0032] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. Moreover, the terms "first," "second," and "third" used in this application do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.

[0033] Identifying student actions in the classroom offers numerous benefits. It helps teachers understand student engagement promptly, allowing for timely adjustments to teaching strategies and ultimately improving teaching effectiveness. For example, teachers can track student actions such as raising hands, scratching heads, or resting their heads on desks, using the results to determine if students understand the lesson content and require further explanation or questioning. Furthermore, this approach helps teachers focus on students who need more attention, encouraging their participation in class. For instance, teachers can notice students who rarely or never raise their hands and take steps to encourage them to participate more actively, thus boosting their confidence, engagement, and overall learning motivation.

[0034] Related technologies enable the identification of student actions in the classroom using action recognition methods based on deep learning algorithms, computer vision algorithms, or traditional image processing algorithms. Deep learning-based action recognition methods utilize deep neural network models to classify student actions, learning the characteristics of different actions through training on large datasets. Computer vision-based action recognition uses image processing algorithms to extract the position and pose information of student key points, and then uses machine learning or deep learning algorithms for classification, monitoring student actions in real time. Traditional image processing-based action recognition methods use edge detection, morphological operations, and other methods to extract student actions.

[0035] All of the above methods require detecting human body regions in classroom images and then classifying the actions within those regions. This approach has the following problems: 1. Action is a continuous process; a single frame of a classroom image cannot describe the entire action process. For example, standing up and raising a hand are continuous actions, and relying on a single frame for action recognition results in poor performance. 2. The accuracy of action recognition is highly dependent on machine learning or deep learning models, exhibiting poor robustness.

[0036] This application embodiment can merge multiple actions belonging to the same valid action, which helps the target video large model to reliably and accurately identify students' actions in the classroom. Furthermore, this application embodiment not only performs initial screening of classroom images but also performs secondary screening of the video segments after merging classroom images, thus improving the accuracy of action recognition.

[0037] The following embodiments of this application provide a classroom action recognition system. Please refer to them together. Figure 1 and Figure 2 The classroom action recognition system 100 includes a camera 11 and a computer device 12.

[0038] Camera 11 is installed in classroom 13 to capture video of the classroom interior. The number of cameras 11 can be one, two, or more. Two or more cameras are positioned at different locations within the classroom to provide a full-view view of the interior.

[0039] Computer device 12 is communicatively connected to camera 11 and is used to analyze classroom videos using the classroom action recognition methods provided in the various embodiments below, in order to filter out final video segments containing valid actions.

[0040] It is understood that in some embodiments, the computer device 12 is installed in the classroom, and the computer device 12 is capable of real-time analysis of the classroom video. In some embodiments, the computer device 12 is installed in the classroom, and after the computer device 12 obtains the target video segment through real-time analysis of the classroom video, the computer device 12 uploads the target video segment to a cloud server, and the cloud server analyzes the target video segment to generate teaching feedback information. In some embodiments, the computer device 12 is a cloud server, and the camera 11 directly uploads the classroom video to the cloud server, the cloud server analyzes the classroom video in real-time, filters out the target video segment, and generates teaching feedback information based on the target video segment.

[0041] As another aspect of the embodiments of this application, the following embodiments of this application provide a method for recognizing classroom actions. Please refer to... Figure 3 The method for recognizing classroom actions includes the following steps S31 to S35.

[0042] This embodiment of the application executes step S31 to acquire classroom video. Classroom video is a video of students attending class in a classroom. There are various ways to acquire classroom video. In some embodiments, this embodiment acquires classroom video directly transmitted from a camera. In some embodiments, the classroom video is stored on a memory card; when the memory card is inserted into a computer device, this embodiment reads the classroom video stored on the memory card. In some embodiments, the classroom video is pre-saved on a cloud server; this embodiment requests the cloud server to return the classroom video. In some embodiments, the classroom video is pre-saved locally on the computer device; this embodiment directly retrieves the classroom video from the local storage.

[0043] After obtaining the classroom video in step S31, this embodiment of the application executes step S32, extracting multiple frames of classroom images from the classroom video for target detection processing to obtain at least one initial video segment.

[0044] This application embodiment extracts multiple frames of classroom images from a classroom video according to a preset frame extraction frequency. The classroom images are images captured by a camera within the classroom. The preset frame extraction frequency can be customized by the designer according to business needs, for example, a preset frame extraction frequency of 1 frame / second. The classroom images include images of each student's body area. This application embodiment can determine the students' actions by analyzing these body area images.

[0045] In some embodiments, the present application embodiments can perform target detection processing on each frame of classroom image based on traditional image detection algorithms. In some embodiments, the present application embodiments can perform target detection processing on each frame of classroom image based on a preset target detection model, wherein the preset target detection model includes any model such as the YOLO series model or the R-CNN series model.

[0046] The initial video clip consists of video clips corresponding to individuals exhibiting valid actions. Valid actions are those specified by the designer, while invalid actions are those that do not fall under the category of valid actions. Valid actions include raising a hand, standing, etc. Invalid actions include touching one's head, touching one's face, or drinking water, etc.

[0047] In the initial screening stage, single-frame images are used as the screening granularity (video clips are used as the screening granularity in the secondary screening stage below). Initial video clips with valid actions are generated. Images that do not contain valid actions or whose actions are obviously invalid will not participate in subsequent processing, which helps to improve the accuracy of action recognition.

[0048] Extracting multiple classroom images from a classroom video and performing target detection processing to obtain at least one initial video segment includes the following steps: extracting multiple classroom images from the classroom video, performing target detection processing on the classroom images to obtain at least one target region image, the target region image being the image of the person in the classroom image corresponding to a person with effective actions, determining the image acquisition time of the target region image, and determining an initial video segment of a specified duration based on the image acquisition time of the target region image, the initial video segment containing the target region image.

[0049] The process of performing target detection processing on classroom images to obtain at least one target region image includes the following steps: performing target detection processing on classroom images based on a preset target detection model to obtain a person detection box, and determining the image region selected by the person detection box as the target region image.

[0050] The embodiments of this application can train a preset target detection model. The steps of training the preset target detection model include: obtaining an action training set, which includes multiple image training samples of different actions, wherein the image training samples are sample images of students performing different types of actions; and training the preset detection model based on the action training set to obtain the preset target detection model.

[0051] For example, the preset target detection model is a YOLOv8 model, and this embodiment can train and generate a YOLOv8 model using the YOLOv8 algorithm. When training the YOLOv8 model, this embodiment extracts multiple frames of image training samples from a pre-training video, where the pre-training video is a video taken in a classroom during YOLOv8 model training. Next, this embodiment annotates each frame of image training samples with actions, obtaining image training samples with different action category labels. For example, actions include raising a hand, writing, standing, and drinking water. The action category labels for each image training sample are "raising hand," "writing," "standing," and "drinking water," respectively. All image training samples with action category labels form an action training set. Then, this embodiment uses the action training set to train the preset detection model to obtain the preset target detection model.

[0052] The YOLOv8 model, through the features and weights learned during training, can perform object detection processing on each frame of classroom image, thereby obtaining the bounding box (bbox) for each student, and extracting the target action features of the image region corresponding to the bounding box. It then determines whether the target action features match the action features of a valid action. If they match, the image region corresponding to the bounding box is determined to be the target region image; otherwise, it is determined that the image region corresponding to the bounding box is not the target region image.

[0053] It is understandable that a classroom image includes a character region image corresponding to one person or character region images corresponding to two or more people. Therefore, the number of target region images obtained from a single classroom image frame can be one or more. It is also understandable that the actions in the target region images corresponding to different people can be the same or different. For example, students A, B, C, and D all appear in the same classroom image frame. Students A and B both raise their hands, student C stands, and student D drinks water. After the above processing, student A corresponds to the first target region image, student B corresponds to the second target region image, and student C corresponds to the third target region image. Since student D's action is not a valid action, the character region image corresponding to student D will not be used as the fourth target region image.

[0054] The image acquisition time is the time when the classroom image containing the target area is captured. In this embodiment, the time when the classroom image containing the target area is captured is determined and used as the image acquisition time for the target area image.

[0055] The specified duration is customized by the designer according to design requirements, such as specifying a duration of 2 seconds or 3 seconds.

[0056] Determining the initial video segment of a specified duration based on the image acquisition time of the target area image includes the following steps: determining the start and end times of the initial video segment, and extracting a video segment from the classroom video that falls between the start and end times as the initial video segment.

[0057] There are many ways to determine the start and end times of an initial video segment based on the image acquisition time.

[0058] In some embodiments, the start time of the initial video segment is the image acquisition time of the target area image, and the end time is the time obtained by adding a specified duration to the start time.

[0059] In some embodiments, the specified duration is the sum of the first duration and the second duration, the start time of the initial video segment is the time obtained by subtracting the first duration from the image acquisition time of the target area image, and the end time is the time obtained by adding the second duration to the image acquisition time of the target area image.

[0060] The effective action corresponding to the target area image is a continuous action. In this embodiment, the initial video segment with a time range is determined based on the image acquisition time of the target area image. This allows the initial video segment to cover the presentation process of the effective action as much as possible. This helps the target video large model to more accurately and reliably select the final video segment containing the effective action, thereby improving the accuracy of action recognition.

[0061] After obtaining the initial video segments in step S32, this embodiment of the application executes step S33, merging at least one initial video segment that satisfies the condition of continuous action into a candidate video segment. This embodiment of the application can merge initial video segments belonging to the same valid action into a single candidate video segment. This avoids the situation where continuous actions are split into multiple frames of classroom images for independent identification, which can easily lead to misjudgment. Furthermore, compared to the way the target video large model understands single-frame classroom images, the target video large model has a stronger and better understanding of video segments with continuous actions, and can more reliably and accurately filter out video segments containing valid actions.

[0062] The action continuity condition setting is used to merge initial video clips involving valid actions of the same person. Designers can define the conditional elements of the action continuity condition according to business requirements.

[0063] In some embodiments, the condition element for the action continuity condition is that the effective actions of each initial video segment conform to a preset motion trajectory. For example, student A performs a hand-raising action, obtaining a first initial video segment at time t1 and a second initial video segment at time t2. The effective actions of the first and second initial video segments conform to the hand-raising motion trajectory. Therefore, the first and second initial video segments satisfy the action continuity condition. Thus, in this embodiment, the first and second initial video segments are merged into a candidate video segment.

[0064] In some embodiments, the elements of the action continuity condition include video temporal overlap and character consistency. Merging at least one initial video segment that satisfies the action continuity condition into a candidate video segment includes: determining that at least two initial video segments with overlapping video temporalities are both selected as video segments to be screened; the classroom image in the video segments to be screened contains a character region image corresponding to a character with valid actions; and merging at least two video segments to be screened with consistent characters into a candidate video segment. The overlap here can be partial or complete.

[0065] For example, the video duration of initial video segment K1 is from 5s to 7s, that of initial video segment K2 is from 6s to 8s, that of initial video segment K3 is from 6s to 8s, that of initial video segment K4 is from 9s to 11s, and that of initial video segment K5 is from 10s to 12s. The video durations of initial video segments K1, K2, and K3 overlap; therefore, all three are considered as video segments to be screened. The video duration of initial video segment K4 overlaps with that of initial video segment K5; therefore, both initial video segments K4 and K5 are considered as another video segment to be screened.

[0066] Merging at least two video clips with identical characters into a candidate video clip includes the following steps: Performing object detection processing on the classroom image in each video clip to obtain a character detection box corresponding to the character region image; using the character detection box corresponding to one of the at least two video clips as the target detection box; and using the character detection boxes corresponding to the remaining video clips as reference detection boxes; calculating the similarity between the target detection box and each reference detection box to obtain multiple similarity values; finding the maximum similarity value among these values; if the maximum similarity value is greater than a preset similarity threshold, merging the video clip with the target detection box corresponding to the maximum similarity value with the video clip with the reference detection box corresponding to the maximum similarity value to obtain a candidate video clip; if the maximum similarity value is less than or equal to the preset similarity threshold, then determining that the characters in at least two video clips are inconsistent.

[0067] Similarity values ​​are represented using the Intersection over Union (IoU) ratio. The preset similarity threshold is customized by the designer based on business needs, for example, a preset similarity threshold of 0.5. Calculating the similarity between the target detection box and each reference detection box to obtain multiple similarity values ​​includes the following steps: Calculating the IoU ratio between the target detection box and each reference detection box using the IoU algorithm.

[0068] For example, initial video segments K1, K2, and K3 are all used as video segments to be screened. In this embodiment, the classroom images in the initial video segments K1, K2, and K3 are processed using the YOLOv8 model to obtain a person detection box BBOX1 corresponding to initial video segment K1, a person detection box BBOX2 corresponding to initial video segment K2, and a person detection box BBOX3 corresponding to initial video segment K3, respectively. In this embodiment, one of the person detection boxes BBOX1, BBOX2, and BBOX3 is selected as the target detection box; for example, BBOX1 is the target detection box, and BBOX2 and BBOX3 are both reference detection boxes.

[0069] In this embodiment, the similarity between person detection boxes BBOX1 and BBOX2 is calculated to obtain a similarity value X1, which is 0.95. The similarity between person detection boxes BBOX1 and BBOX3 is also calculated to obtain a similarity value X2, which is 0.2. Since the similarity value X1 is greater than the similarity value X2, and the similarity value X1 is greater than a preset similarity threshold, this embodiment merges the initial video segment K1 and the initial video segment K2 into a candidate video segment.

[0070] This application embodiment, based on the two key conditions of overlapping video time and identical individuals, can quickly and effectively merge at least two video segments to be screened with identical individuals into candidate video segments. These candidate video segments comprehensively cover the entire process of the same individual performing an effective action, which helps the subsequent target video model to more comprehensively and effectively understand the candidate video segments, thereby accurately and reliably selecting the final video segment containing the effective action. Furthermore, by merging at least two initial video segments belonging to the same effective action, this application embodiment reduces the number of video segments requiring processing, reduces the time and resource consumption for overlap calculations, and improves action recognition efficiency.

[0071] It is understood that the embodiments of this application can further process the video segments to be screened by combining time-series analysis methods, motion trajectory methods and other methods, so as to obtain reliable and accurate candidate video segments.

[0072] After obtaining candidate video segments in step S33, this embodiment of the application executes step S34, which performs action recognition processing on the target video segment based on a preset video classification model to obtain action recognition information. The target video segment is one of the candidate video segments in at least one candidate video segment.

[0073] The preset video classification model is a TSN (Temporal Segment Networks) model, a CNN-LSTM model, a 3D CNN model, or another video classification model. This application embodiment obtains a video sample set, which includes multiple video samples labeled with action categories. The TSN model is trained and generated based on this video sample set. The TSN model can analyze and understand videos. During training, the TSN model learns the features of different action categories, thereby accurately classifying candidate video segments.

[0074] Action recognition information includes a first type of recognition information and a second type of recognition information. The first type of recognition information is used to indicate that the preset video classification model has identified that the target video segment contains a valid action, and the second type of recognition information is used to indicate that the preset video classification model has not identified that the target video segment contains a valid action.

[0075] This application embodiment, after performing target detection processing on multiple frames of classroom images in the initial screening stage, can filter out images of student figures who do not perform effective actions, thereby reducing the number of images that need to be screened again in subsequent steps and improving overall processing efficiency. However, since effective actions are represented by a continuous sequence of images, using only image frame-based detection methods can easily reduce the accuracy of action recognition. In this application embodiment, from the perspective of video segments, a preset video classification model is used to perform secondary screening processing on candidate video segments. This can eliminate video segments that do not contain effective actions or whose actions are obviously invalid, and the resulting candidate video segments can contain effective actions with a higher probability.

[0076] The action recognition process for a target video segment based on a preset video classification model to obtain action recognition information includes the following steps: inputting the target video segment into the preset video classification model for action recognition processing to obtain a first probability set, which includes multiple recognition probability values. The recognition probability value is the probability that the action contained in the target video segment belongs to the corresponding action label. Based on the first probability set, action recognition information is generated.

[0077] For example, a preset video classification model is configured with N action labels. In some embodiments, the N action labels include a hand-raising label, a standing label, and an invalid action label. Invalid action labels include labels such as drinking water, touching the face, or touching the head. In this embodiment, the target video clip is input into the preset video classification model, and the preset video classification model outputs the recognition probability values ​​of the actions contained in the target video clip belonging to the N action labels respectively. For example, when the target video clip is input into the preset video classification model, the preset video classification model outputs: the recognition probability value of the action contained in the target video clip belonging to the hand-raising label is 0.8, the recognition probability value of the action contained in the target video clip belonging to the standing label is 0.1, and the recognition probability value of the action contained in the target video clip H1 belonging to the invalid action label is 0.1. Therefore, the first probability set Q1 = {0.8, 0.1, 0.1}.

[0078] Generating action recognition information based on a first probability set includes the following steps: determining a first maximum probability value of the first probability set, wherein the first maximum probability value is the recognition probability value with the largest value in the first probability set; generating first type of recognition information in response to the first maximum probability value being greater than or equal to a preset probability threshold, wherein the action contained in the target video segment belongs to the action tag corresponding to the first maximum probability value; and generating second type of recognition information in response to the first maximum probability value being less than the preset probability threshold.

[0079] The preset probability threshold is customized by the designer based on engineering experience. For example, the preset probability threshold is 0.5, 0.6, 0.7, 0.8, or any value between 0.5 and 0.8. It is understood that the preset probability thresholds corresponding to different action labels may be the same or different.

[0080] If the first maximum probability value is greater than or equal to the preset probability threshold, it means that the action contained in the target video segment can belong to the action label corresponding to the first maximum probability value with high confidence. As mentioned above, the preset probability threshold is 0.7, and the recognition probability value of the action contained in the target video segment H1 belonging to the hand-raising label is 0.8. Therefore, the action contained in the target video segment H1 belongs to the hand-raising label.

[0081] If the first maximum probability value is less than a preset probability threshold, it indicates that the preset video classification model cannot confidently identify the action type of the action contained in the target video segment. For example, when a target video segment is input into a preset video classification model, the preset video classification model outputs a first probability set Q2 = {0.5, 0.3, 0.2}. Since the first maximum probability value of 0.5 is less than the preset probability threshold of 0.7, the preset video classification model cannot confidently identify the action type of the action contained in the target video segment. Therefore, this embodiment generates a second type of recognition information.

[0082] In developing this application, the inventors observed numerous classroom videos and conducted experiments, discovering that, taking raising hands as an example, at the same time point, if multiple students simultaneously raise their hands during the detection and identification of valid actions, the probability of a student raising their hand in the candidate video segment is higher. Considering that the accuracy of the preset video classification model is not 100%, if the preset probability threshold for the hand-raising action remains at 0.7, the preset video classification model is prone to incorrectly rejecting individual target video segments containing the hand-raising action.

[0083] In some embodiments, before generating the second type of recognition information, the recognition method further includes: in response to a first maximum probability value being less than a preset probability threshold, determining classroom scene description information corresponding to the target video segment, determining a target probability threshold corresponding to the first maximum probability value based on the classroom scene description information, and generating action recognition information based on the first maximum probability value and the target probability threshold.

[0084] This application provides a dynamic threshold adjustment mechanism. In this mechanism, when the preset video classification model cannot accurately identify the action type of the action contained in the target video segment, classroom scene description information is generated. The preset probability threshold of the action corresponding to the first maximum probability value is dynamically adjusted using the classroom scene description information, and then the action recognition information is re-determined. In this way, the classroom scene description information can be used as supervision information to reliably generate action recognition information, so as to accurately and reliably determine whether it is necessary to call the target video large model to further identify the candidate video segment.

[0085] Classroom scene description information is used to describe the actions that occur in the classroom within the target time period. For example, 6 students raise their hands or stand up in turn in the classroom. For example, 6 students raise their hands in turn in the classroom, and 7 students stand up in turn in the classroom.

[0086] Determining the classroom scene description information corresponding to the target video segment includes the following steps: determining the timestamp of the target video segment; determining the target time period based on the timestamp of the target video segment, the target time period containing the timestamp of the target video segment; obtaining reference video segments, which are classified video segments with timestamps within the target time period; the classified video segments are candidate video segments that have undergone action recognition processing by a preset video classification model and output a second probability set; the second probability set includes multiple recognition probability values, which are the probability that the actions contained in the candidate video segment belong to the corresponding action tags; determining the second maximum probability value and the action tag corresponding to the second maximum probability value in the second probability set of each reference video segment as candidate tags; the second maximum probability value is the recognition probability value with the largest value in the second probability set; and generating classroom scene description information corresponding to the target video segment based on all candidate tags within the target time period.

[0087] In some embodiments, the timestamp of the target video segment is the center point of the target time period, and the length of the target time period is a preset duration. For example, the preset duration is 1 second.

[0088] For example, this application embodiment provides the following video clips: target video clip H1, candidate video clip H2, candidate video clip H3, candidate video clip H4, and candidate video clip H5. The preset duration is 1 second. The timestamp of target video clip H1 is at 10 seconds, the timestamp of candidate video clip H2 is at 10.2 seconds, the timestamp of candidate video clip H3 is at 10.3 seconds, the timestamp of candidate video clip H4 is at 9.4 seconds, and the timestamp of candidate video clip H5 is at 12 seconds. Target video clip H1, candidate video clip H2, candidate video clip H3, candidate video clip H4, and candidate video clip H5 are all processed by a preset video classification model for action recognition, and a second probability set is output.

[0089] In this embodiment, target video segment H1, candidate video segment H2, candidate video segment H3, and candidate video segment H4 are all used as reference video segments. The second probability sets for target video segment H1, candidate video segment H2, candidate video segment H3, candidate video segment H4, and candidate video segment H5 are as follows:

[0090] After the target video segment H1 is processed by the preset video classification model, the second probability set M1 = {0.4, 0.3, 0.3}. The action label corresponding to the second maximum probability value of the target video segment H1 in the second probability set is the "raise hand" label. Therefore, the "raise hand" label of the target video segment H1 is a candidate label.

[0091] After candidate video segment H2 is processed by the preset video classification model, the probability set M2 = {0.8, 0.1, 0.1}. The action label corresponding to the second maximum probability value of candidate video segment H2 in the second probability set is the "raise hand" label. Therefore, the "raise hand" label of candidate video segment H2 is the candidate label.

[0092] After candidate video clip H3 is processed by the preset video classification model, the second probability set M3 = {0.3, 0.7, 0}. The action label corresponding to the second maximum probability value of candidate video clip H3 in the second probability set is the standing label. Therefore, the standing label of candidate video clip H3 is the candidate label.

[0093] After candidate video segment H4 is processed by the preset video classification model, the second probability set M4 = {0.4, 0.3, 0.3}. The action label corresponding to the second maximum probability value of candidate video segment H4 in the second probability set is the "raise hand" label. Therefore, the "raise hand" label of candidate video segment H4 is a candidate label.

[0094] This application embodiment generates classroom scene description information based on the raised hand label of target video segment H1, the raised hand label of candidate video segment H2, the standing label of candidate video segment H3, and the raised hand label of candidate video segment H4.

[0095] Classroom scene description information includes information on no group actions, information on single group actions, and information on mixed group actions. Information on no group actions indicates that no group actions occur in the classroom during the target time period. For example, no effective actions performed by students occur in the classroom during the target time period, or a small number of effective actions performed by students occur in the classroom during the target time period.

[0096] Group action refers to multiple students performing the same action within the same time period in a classroom. For example, if five students raise their hands to answer a question at the same time in a classroom, then a group action of raising hands has occurred. However, if one, two, or three students raise their hands to answer a question at the same time in a classroom, since the number of students raising their hands is relatively small, this embodiment of the application considers that a group action of raising hands has not occurred in the classroom.

[0097] Single group action information indicates that group actions occurred in the classroom during the target time period. For example, multiple students raised their hands or stood up during the target time period. Mixed group action information indicates that multiple group actions occurred in the classroom during the target time period. For example, multiple students raised their hands and multiple students stood up during the target time period.

[0098] Based on all candidate tags within the target time period, generating classroom scene description information corresponding to the target video segment includes the following steps: classifying all candidate tags within the target time period according to action type, obtaining the number of targets belonging to each action type, generating no group action information in response to the total number of targets being less than a preset threshold, determining the number of targets greater than or equal to the preset threshold in response to at least one target being greater than or equal to the preset threshold, generating single group action information in response to the number being 1, and generating mixed group action information in response to the number being greater than 1.

[0099] The preset quantity threshold is customized by the designer based on engineering experience; for example, the preset quantity threshold may be 2 or 3. For instance, if the preset quantity threshold is 2, the target number of "raise hand" tags is 3, and the target number of "stand" tags is 1. Furthermore, the number of target quantities greater than or equal to the preset quantity threshold is 1. Therefore, this embodiment generates single-group action information to indicate that hand-raising actions occurred in the classroom within a target time period.

[0100] For another example, if the target number of the "raise hand" tag is 1 and the target number of the "stand" tag is 0, and both the target number of the "raise hand" tag and the target number of the "stand" tag are less than a preset threshold, this application embodiment generates no group action information to indicate that no effective actions occur in the classroom during the target time period.

[0101] For example, the target number of "raise hand" tags is 4, the target number of "stand" tags is 5, and the number of target numbers greater than or equal to a preset threshold is 2. This application embodiment generates mixed group action information to indicate that both "raise hand" and "stand" actions occur in the classroom during the target time period.

[0102] The classroom scene description information includes single group action information, mixed group action information, and no group action information. In some embodiments, determining the target probability threshold corresponding to the first maximum probability value based on the classroom scene description information includes the following steps: in response to the classroom scene description information being single group action information, and the group action represented by the single group action information belonging to the action label corresponding to the first maximum probability value, updating the preset probability threshold to the target probability threshold, wherein the target probability threshold is less than or equal to the preset probability threshold.

[0103] For example, if the group action information represents a hand-raising action, and the action label corresponding to the highest probability value is a hand-raising action, it means that although the preset video classification model cannot identify the action contained in the target video segment as a hand-raising action with high confidence, it selects the hand-raising action as the action contained in the target video segment with low confidence between the hand-raising action and the standing action. Then, combined with the feedback of the classroom scene description information, it generates single group action information, which represents the group action as a hand-raising action. This can further verify and determine that the action contained in the target video segment is a hand-raising action with high confidence. Therefore, this embodiment of the application chooses to dynamically reduce the preset probability threshold, for example, reducing the preset probability threshold from 0.7 to 0.5, that is, the target probability threshold is 0.5, so as to further judge and identify the action contained in the target video segment.

[0104] In this embodiment of the application, when a group performs the same effective action, the preset probability threshold corresponding to the effective action is lowered. This can cater to classroom scenarios where group behavior leads to high confidence in performing effective actions, and can also make up for the inherent deficiency that the accuracy of the preset video classification model does not reach 100%, which is conducive to improving the recall rate.

[0105] In some embodiments, determining the target probability threshold corresponding to the first maximum probability value based on classroom scene description information includes the following steps: in response to the classroom scene description information being single group action information, and the group action represented by the single group action information not belonging to the action label corresponding to the first maximum probability value, the preset probability threshold corresponding to the first maximum probability value is maintained unchanged.

[0106] For example, the group action information of a single group represents a hand-raising action. However, the action label corresponding to the first maximum probability value is a standing action. This indicates that the group action that occurred in the classroom during the target time period does not match the action label given by the preset video classification model corresponding to the first maximum probability value. Therefore, in this embodiment, it is not necessary to reduce the preset probability threshold and keep the preset probability threshold unchanged.

[0107] In some embodiments, determining the target probability threshold corresponding to the first maximum probability value based on classroom scene description information includes the following steps: in response to the classroom scene description information being mixed group action information, maintaining the preset probability threshold corresponding to the first maximum probability value unchanged.

[0108] For example, the group actions represented by the mixed group action information include both raising hand actions and standing actions. Although the action label corresponding to the first maximum probability value is standing action, this situation still cannot accurately and reliably determine whether a standing action actually occurred within the target time period. Therefore, in this application embodiment, it is not necessary to reduce the preset probability threshold and keep the preset probability threshold unchanged.

[0109] In some embodiments, determining the target probability threshold corresponding to the first maximum probability value based on classroom scene description information includes the following steps: in response to the classroom scene description information being no group action information, maintaining the preset probability threshold corresponding to the first maximum probability value unchanged.

[0110] Generating action recognition information based on a first maximum probability value and a target probability threshold includes the following steps: in response to the first maximum probability value being greater than or equal to the target probability threshold, generating a first type of recognition information, wherein the action contained in the target video segment belongs to the action tag corresponding to the first maximum probability value; in response to the first maximum probability value being less than the target probability threshold, generating a second type of recognition information.

[0111] In this embodiment, after processing by the dynamic threshold mechanism, the first maximum probability value is compared with the target probability threshold again. In this way, even if the preset video classification model cannot accurately identify whether the target video segment contains a valid action, it can still reliably determine whether the target video segment contains a valid action.

[0112] After obtaining the action recognition information in step S34, this embodiment of the application executes step S35. In response to the action recognition information being the second type of recognition information, the target video segment and preset prompt words are sent to the target video big model, so that the target video big model responds to the preset prompt words and outputs the target description information of the target video segment. The target description information is used to describe whether the target video segment contains a valid action.

[0113] This application embodiment retains candidate video segments that the preset video classification model cannot identify as containing valid actions, and inputs these candidate video segments into the target video large model for further identification. This approach can improve the recall rate. Furthermore, this application embodiment, while ensuring accuracy, can retain more candidate video segments as target video segments for visual understanding, thus reducing the omission rate. Simultaneously, this application embodiment uses the target video large model to perform in-depth analysis and understanding of the target video segments to determine whether the target video segments contain target actions, thereby improving the accuracy of action recognition.

[0114] The target video model is a large model obtained by fine-tuning and training the video model using prompt words. The video model can be a Pllava model, a Cogvlm2-video model, or an InternVideo model, etc.

[0115] The PLLaVA model is a method for adapting image-language pre-trained models to video understanding tasks without requiring additional parameters. The PLLaVA model employs a pooling strategy to smooth the feature distribution along the temporal dimension, reducing the dominance of extreme features and extending image-language models to video understanding. The CogVLM2-Video model introduces multiple video frames and timestamps as encoder input, utilizing automated temporal localization data construction methods to generate time-related video question-answering data. The InternVideo model is a general-purpose video foundation model that, by combining masked video modeling and video-language contrastive learning, updates its state on multiple datasets, including action recognition, video retrieval, and video question answering, demonstrating transferability across different tasks.

[0116] During the training phase, embodiments of this application identify candidate video segments whose recognition probability values ​​fall within a preset probability range as hard samples, obtain a video training set including hard samples, and fine-tune the preset large-scale video model based on the video training set to obtain the target large-scale video model. The preset probability range is a numerical range centered on the target probability threshold, with a first percentage floating upwards and a second percentage floating downwards. For example, if the target probability threshold is 0.5, and both the first and second percentages are 20%, then the preset probability range is 0.4-0.6; or, if the first percentage is 20% and the second percentage is 40%, then the preset probability range is 0.4-0.7.

[0117] Candidate video segments whose probability values ​​fall within a preset probability range are easy to obtain, and these candidate video segments are considered hard samples. Training a large video model with these hard samples can improve the convergence rate of the large video model, thereby improving the training efficiency of the large video model.

[0118] This application embodiment filters out difficult samples based on the motion analysis results of candidate video segments using a preset video classification model. These difficult samples are video segments that the preset video classification model is prone to misclassifying and require further processing and analysis. This application embodiment also manually annotates the video segments to be trained, and the annotated video segments form a video training set.

[0119] This embodiment uses the PLLava model as the base model and fine-tunes it using the aforementioned training set. During training, this embodiment can also incorporate techniques such as LoRa to further optimize and adjust the base model to suit specific application scenarios and needs.

[0120] This application embodiment obtains a prompt word template and generates preset prompt words based on the prompt word template. For example, this application embodiment directly uses the prompt word template as the preset prompt words, and the preset prompt words are as follows: Prompt: "Which action do the student do in this video. Output one behavior from [raising hand, standing, touching head, touching face, writing, drinking water]"

[0121] The target video big model responds to preset prompts and outputs target description information for the target video segment. In this embodiment, the target video segment is determined to contain a valid action based on the target description information.

[0122] For example, when the target description information is used to describe the student's action in the target video clip as a hand-raising action, this embodiment of the application determines that the target video clip contains a hand-raising action. Similarly, when the target description information is used to describe the student's action in the target video clip as a standing action, a head-touching action, a face-touching action, a writing action, or a drinking action, this embodiment of the application determines that the target video clip contains a standing action, a head-touching action, a face-touching action, a writing action, or a drinking action.

[0123] In general, the embodiments of this application achieve at least the following technical effects:

[0124] 1. Solving the problem of action continuity: Compared with related technologies that rely solely on image frame-based methods to identify actions, the embodiments of this application combine continuous actions for video classification and video understanding, which can better describe the entire action process. For example, standing and raising hand actions can be accurately captured.

[0125] 2. Improved Accuracy: This application embodiment employs video classification and a dynamic threshold mechanism for rescreening. By combining a preset video classification model with dynamically set thresholds, this application embodiment can more accurately identify and classify continuous actions, thereby filtering out video segments with invalid actions. Furthermore, this application embodiment uses a large target video model to process the filtered candidate video segments, and by visually describing the candidate video segments, the accuracy of action recognition can be further improved.

[0126] 3. Balancing High Recall and Precision: In the initial screening stage, this embodiment prioritizes high recall by setting a low probability threshold for the preset target detection model, thus retaining more initial video segments. In the secondary screening stage, this embodiment utilizes a dynamic threshold mechanism to ensure high recall, and leverages a large video model to improve action recognition precision. Therefore, this embodiment achieves both high recall and high action recognition precision.

[0127] It should be noted that in the above embodiments, there is no necessarily a certain order between the steps. Those skilled in the art can understand from the description of the embodiments of this application that the above steps may have different execution orders in different embodiments, that is, they may be executed in parallel or in turn, etc.

[0128] As another aspect of the embodiments of this application, this application provides a classroom action recognition device. The classroom action recognition device can be a software module, which includes several instructions stored in a memory. A processor can access the memory, call the instructions, and execute them to complete the classroom action recognition methods described in the various embodiments above.

[0129] In some embodiments, the classroom action recognition device can also be built from hardware components. For example, the classroom action recognition device can be built from one or more chips, which can work together to complete the classroom action recognition methods described in the various embodiments above. As another example, the classroom action recognition device can also be built from various logic devices, such as general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), microcontrollers, ARM (Acorn RISC Machine) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination of these components.

[0130] Please see Figure 4 The classroom action recognition device 400 includes a video acquisition module 41, a video initial screening module 42, a video merging module 43, a video rescreening module 44, and a visual analysis module 45.

[0131] The video acquisition module 41 is used to acquire classroom videos. The video initial screening module 42 is used to extract multiple frames of classroom images from the classroom video for target detection processing to obtain at least one initial video segment, which is the video segment corresponding to the detected person with valid actions. The video merging module 43 is used to merge at least one initial video segment that meets the action continuity condition into a candidate video segment. The video rescreening module 44 is used to perform action recognition processing on the target video segment based on a preset video classification model to obtain action recognition information. The action recognition information includes a first type of recognition information and a second type of recognition information. The first type of recognition information is used to indicate that the preset video classification model has identified that the target video segment contains valid actions, and the second type of recognition information is used to indicate that the preset video classification model has not identified that the target video segment contains valid actions. The target video segment is one of the candidate video segments among at least one candidate video segment. The visual analysis module 45 is used to send the target video segment and a preset prompt word to the target video big model in response to the action recognition information being the second type of recognition information, so that the target video big model responds to the preset prompt word and outputs the target description information of the target video segment. The target description information is used to describe the actions contained in the target video segment.

[0132] In the initial screening stage, single-frame images are used as the screening granularity (video clips are used as the screening granularity in the secondary screening stage below). Initial video clips with valid actions are generated. Images that do not contain valid actions or whose actions are obviously invalid will not participate in subsequent processing, which helps to improve the accuracy of action recognition.

[0133] The embodiments of this application can merge initial video segments belonging to the same valid action into a single candidate video segment. This avoids the situation where continuous actions are split into multiple frames of classroom images for independent recognition, which can easily lead to misjudgment. Furthermore, compared to the way the target video big model understands single-frame classroom images, the target video big model has a stronger and better understanding of video segments with continuous actions, and can more reliably and accurately select video segments containing valid actions.

[0134] This application embodiment, after performing target detection processing on multiple frames of classroom images in the initial screening stage, can filter out images of student figures who do not perform effective actions, thereby reducing the number of images that need to be screened again in subsequent steps and improving overall processing efficiency. However, since effective actions are represented by a continuous sequence of images, using only image frame-based detection methods can easily reduce the accuracy of action recognition. In this application embodiment, each candidate video segment is screened from the perspective of video clips to obtain at least one target video segment. This allows video clips that do not contain effective actions or whose actions are obviously invalid to be eliminated, and the obtained target video clips are more likely to contain effective actions.

[0135] This application embodiment uses a large target video model to perform in-depth analysis and understanding of the target video segment in order to determine whether the target video segment contains the target action, thereby improving the accuracy of action recognition.

[0136] In some embodiments, the video rescreening module 44 is specifically used to: input the target video segment into a preset video classification model for action recognition processing to obtain a first probability set, the first probability set including multiple recognition probability values, the recognition probability values ​​being the probability that the action contained in the target video segment belongs to the corresponding action tag, and generate action recognition information based on the first probability set.

[0137] In some embodiments, the video rescreening module 44 is further configured to: determine a first maximum probability value of the first probability set, wherein the first maximum probability value is the recognition probability value with the largest value in the first probability set; generate a first type of recognition information in response to the first maximum probability value being greater than or equal to a preset probability threshold, wherein the action contained in the target video segment belongs to the action tag corresponding to the first maximum probability value; and generate a second type of recognition information in response to the first maximum probability value being less than the preset probability threshold.

[0138] In some embodiments, the video rescreening module 44 is further configured to: in response to the first maximum probability value being less than a preset probability threshold, determine classroom scene description information corresponding to the target video segment, determine a target probability threshold corresponding to the first maximum probability value based on the classroom scene description information, and generate action recognition information based on the first maximum probability value and the target probability threshold.

[0139] In some embodiments, the video rescreening module 44 is further specifically configured to: determine the timestamp of the target video segment; determine a target time period based on the timestamp of the target video segment, the target time period including the timestamp of the target video segment; obtain reference video segments, the reference video segments being classified video segments with timestamps within the target time period, the classified video segments being candidate video segments that have undergone action recognition processing by the preset video classification model and output a second probability set, the second probability set including multiple recognition probability values, the recognition probability values ​​being the probability that the action contained in the candidate video segment belongs to the corresponding action tag; determine the second maximum probability value in the second probability set of each of the reference video segments and the action tag corresponding to the second maximum probability value as candidate tags, the second maximum probability value being the recognition probability value with the largest value in the second probability set; and generate classroom scene description information corresponding to the target video segment based on all candidate tags within the target time period.

[0140] In some embodiments, the classroom scene description information includes single group action information, mixed group action information, and no group action information. The video screening module 44 is further specifically used to: classify all candidate tags within the target time period according to action type to obtain the number of targets belonging to each action type; generate no group action information in response to all target numbers being less than a preset number threshold; determine the number of targets greater than or equal to the preset number threshold in response to at least one target number being greater than or equal to the preset number threshold; generate single group action information in response to the number being 1; and generate mixed group action information in response to the number being greater than 1.

[0141] In some embodiments, the classroom scene description information includes single group action information, mixed group action information, and no group action information. The video rescreening module 44 is further specifically used to: in response to the classroom scene description information being single group action information, and the group action represented by the single group action information belonging to the action label corresponding to the first maximum probability value, update the preset probability threshold to the target probability threshold, where the target probability threshold is less than or equal to the preset probability threshold; or, in response to the classroom scene description information being single group action information, and the group action represented by the single group action information not belonging to the action label corresponding to the first maximum probability value, maintain the preset probability threshold corresponding to the first maximum probability value unchanged; or, in response to the classroom scene description information being mixed group action information, maintain the preset probability threshold corresponding to the first maximum probability value unchanged; or, in response to the classroom scene description information being no group action information, maintain the preset probability threshold corresponding to the first maximum probability value unchanged.

[0142] In some embodiments, the video rescreening module 44 is further specifically used to: generate a first type of recognition information in response to a first maximum probability value being greater than or equal to a target probability threshold, wherein the action contained in the target video segment belongs to the action tag corresponding to the first maximum probability value; and generate a second type of recognition information in response to a first maximum probability value being less than the target probability threshold.

[0143] In some embodiments, the video screening module 42 is specifically used to: extract multiple frames of classroom images from the classroom video, perform target detection processing on the classroom images to obtain at least one target region image, wherein the target region image is a person region image in the classroom image corresponding to a person with effective actions, determine the image acquisition time of the target region image, and determine an initial video segment of a specified duration based on the image acquisition time of the target region image, wherein the initial video segment contains the target region image.

[0144] In some embodiments, the conditional elements of the action continuity condition include video time overlap and character consistency. The video merging module 43 is specifically used to: determine that at least two initial video segments with video time overlap are both video segments to be screened, the classroom image in the video segments to be screened contains a character area image corresponding to a character with valid action, and merge at least two video segments to be screened with consistent characters into a candidate video segment.

[0145] In some embodiments, the video merging module 43 is specifically used to: perform target detection processing on the classroom image in each video segment to be screened to obtain a person detection box corresponding to the person region image; the person detection box corresponding to one of the at least two video segments to be screened is the target detection box; the person detection boxes corresponding to the remaining video segments to be screened in the at least two video segments to be screened are all reference detection boxes; calculate the similarity between the target detection box and each reference detection box to obtain multiple similarity values; find the maximum similarity value among the multiple similarity values; if the maximum similarity value is greater than a preset similarity threshold, merge the video segment to be screened with the target detection box corresponding to the maximum similarity value and the video segment to be screened with the reference detection box corresponding to the maximum similarity value to obtain a candidate video segment.

[0146] In some embodiments, the visual analysis module 45 is specifically used to: determine candidate video segments whose recognition probability values ​​fall within a preset probability range as hard samples, obtain a video training set including hard samples, and fine-tune and train a preset large video model based on the video training set to obtain a target large video model.

[0147] It should be noted that the aforementioned classroom action recognition device can execute the classroom action recognition method provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects of the method. Technical details not described in detail in the embodiments of the classroom action recognition device can be found in the classroom action recognition method provided in the embodiments of this application.

[0148] See Figure 5 , Figure 5 This is a schematic diagram of a computer device provided in an embodiment of this application. The computer device 500 includes one or more processors 51 and a memory 52. ​​The memory 52 is connected to one or more processors 51, for example, via a bus.

[0149] Processor 51 is configured to support the computer device in performing the corresponding functions in the methods described in the above method embodiments. Processor 51 may be a central processing unit (CPU), a network processor (NP), a hardware chip, or any combination thereof. The aforementioned hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0150] Memory 52 is used to store program code, etc. Memory may include volatile memory (VM), such as random access memory (RAM); memory may also include non-volatile memory (NVM), such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); memory may also include combinations of the above types of memory.

[0151] The memory 52 can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the classroom action recognition method in the embodiments of this application. The processor executes the various functional applications and data processing of the classroom action recognition method and the classroom action recognition device by running the non-volatile software programs, instructions, and modules stored in the memory, that is, it realizes the functions of each module or unit of the classroom action recognition method and the classroom action recognition device provided in the above method embodiments.

[0152] The memory 52 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function. The data storage area may store data created based on the use of the classroom action recognition device. In some embodiments, the memory may include memory remotely located relative to the processor, which can be connected to the classroom action recognition device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0153] One or more modules are stored in memory. When executed by one or more processors, they perform the classroom action recognition method in any of the above method embodiments. For example, they perform the method steps described in the above method embodiments to realize the functions of the modules described in the above device embodiments.

[0154] This application also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the methods described in the foregoing embodiments.

[0155] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0156] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A method for recognizing classroom actions, characterized in that, include: Get the class videos; Multiple frames of classroom images are extracted from the classroom video and subjected to target detection processing to obtain at least one initial video segment, wherein the initial video segment is the video segment corresponding to the detected person with effective actions; At least one of the initial video segments that satisfies the action continuity condition is merged into a candidate video segment; Action recognition processing is performed on the target video segment based on a preset video classification model to obtain action recognition information. The action recognition information includes a first type of recognition information and a second type of recognition information. The first type of recognition information is used to indicate that the preset video classification model has identified that the target video segment contains a valid action. The second type of recognition information is used to indicate that the preset video classification model has not identified that the target video segment contains a valid action. The target video segment is one of the multiple candidate video segments. In response to the action recognition information being of the second type, the target video segment and a preset prompt word are sent to the target video big model, so that the target video big model responds to the preset prompt word and outputs the target description information of the target video segment, the target description information being used to describe the actions contained in the target video segment.

2. The identification method according to claim 1, characterized in that, The action recognition processing of the target video segment based on the preset video classification model to obtain action recognition information includes: The target video segment is input into a preset video classification model for action recognition processing to obtain a first probability set. The first probability set includes multiple recognition probability values, and the recognition probability values ​​are the probability that the action contained in the target video segment belongs to the corresponding action tag. Action recognition information is generated based on the first probability set.

3. The identification method according to claim 2, characterized in that, The generation of action recognition information based on the first probability set includes: Determine the first maximum probability value of the first probability set, wherein the first maximum probability value is the recognition probability value with the largest value in the first probability set; In response to the first maximum probability value being greater than or equal to a preset probability threshold, a first type of recognition information is generated, wherein the action contained in the target video segment belongs to the action tag corresponding to the first maximum probability value; In response to the first maximum probability value being less than a preset probability threshold, a second type of identification information is generated.

4. The identification method according to claim 3, characterized in that, Before generating the second type of identification information, the identification method further includes: In response to the first maximum probability value being less than a preset probability threshold, classroom scene description information corresponding to the target video segment is determined; Based on the classroom scenario description information, a target probability threshold corresponding to the first maximum probability value is determined; Action recognition information is generated based on the first maximum probability value and the target probability threshold.

5. The identification method according to claim 4, characterized in that, The classroom scene description information corresponding to the target video segment includes: Determine the timestamp of the target video segment; A target time period is determined based on the timestamp of the target video segment, and the target time period includes the timestamp of the target video segment. Obtain reference video segments, which are classified video segments with timestamps within the target time period. The classified video segments are candidate video segments that have undergone action recognition processing by the preset video classification model and output a second probability set. The second probability set includes multiple recognition probability values, which are the probability that the action contained in the candidate video segment belongs to the corresponding action tag. The second maximum probability value and the action label corresponding to the second maximum probability value in the second probability set of each of the reference video segments are determined as candidate labels, and the second maximum probability value is the recognition probability value with the largest value in the second probability set; Based on all candidate tags within the target time period, classroom scene description information corresponding to the target video segment is generated.

6. The identification method according to claim 5, characterized in that, The classroom scene description information includes single group action information, mixed group action information, and no group action information. The step of generating classroom scene description information corresponding to the target video segment based on all candidate tags within the target time period includes: All candidate tags within the target time period are categorized according to action type to obtain the target number of each action type. In response to the fact that the number of all targets is less than a preset threshold, information on no group action is generated. In response to at least one of the target quantities being greater than or equal to a preset quantity threshold, the number of target quantities greater than or equal to the preset quantity threshold is determined; In response to the number being 1, single group action information is generated; In response to the number being greater than 1, mixed group action information is generated.

7. The identification method according to claim 4, characterized in that, The classroom scene description information includes single-group action information, mixed-group action information, and no-group action information. Determining the target probability threshold corresponding to the first maximum probability value based on the classroom scene description information includes: In response to the classroom scene description information being single-group action information, and the group action represented by the single-group action information belonging to the action tag corresponding to the first maximum probability value, the preset probability threshold is updated to a target probability threshold, where the target probability threshold is less than or equal to the preset probability threshold; or, In response to the classroom scene description information being single-group action information, and the group action represented by the single-group action information not belonging to the action label corresponding to the first maximum probability value, the preset probability threshold corresponding to the first maximum probability value remains unchanged; or, In response to the classroom scene description information being mixed group action information, the preset probability threshold corresponding to the first maximum probability value remains unchanged; or, In response to the classroom scene description information indicating no group action information, the preset probability threshold corresponding to the first maximum probability value remains unchanged.

8. The identification method according to claim 7, characterized in that, The step of generating action recognition information based on the first maximum probability value and the target probability threshold includes: In response to the first maximum probability value being greater than or equal to the target probability threshold, a first type of recognition information is generated, wherein the action contained in the target video segment belongs to the action tag corresponding to the first maximum probability value; In response to the first maximum probability value being less than the target probability threshold, a second type of identification information is generated.

9. The identification method according to any one of claims 1 to 8, characterized in that, The step of extracting multiple frames of classroom images from the classroom video and performing target detection processing to obtain at least one initial video segment includes: Extract multiple frames of classroom images from the classroom video; The classroom image is subjected to target detection processing to obtain at least one target region image, wherein the target region image is the person region image in the classroom image corresponding to a person with effective action; Determine the image acquisition time of the target area image; An initial video segment of a specified duration is determined based on the image acquisition time of the target area image, and the initial video segment contains the target area image.

10. The identification method according to claim 9, characterized in that, The action continuity condition includes two elements: overlapping video time and identical characters. Merging at least one of the initial video segments that satisfy the action continuity condition into a candidate video segment includes: At least two initial video segments with overlapping video times are identified as video segments to be screened, and the classroom images in the video segments to be screened include character area images corresponding to characters with effective actions; Merge at least two video segments with identical characters into candidate video segments.

11. The identification method according to claim 10, characterized in that, The step of merging at least two video clips of the same person into a candidate video clip includes: Perform target detection processing on the classroom image in each of the video segments to be screened to obtain a person detection box corresponding to the person region image. The person detection box corresponding to one of the at least two video segments to be screened is the target detection box, and the person detection boxes corresponding to the remaining video segments to be screened in at least two video segments to be screened are all reference detection boxes. Calculate the similarity between the target detection box and each of the reference detection boxes to obtain multiple similarity values; Find the maximum similarity value among the multiple similarity values; If the maximum similarity value is greater than a preset similarity threshold, the video segment to be screened in the target detection box corresponding to the maximum similarity value is merged with the video segment to be screened in the reference detection box corresponding to the maximum similarity value to obtain a candidate video segment.

12. The identification method according to any one of claims 2 to 8, characterized in that, Also includes: Candidate video segments whose recognition probability values ​​fall within a preset probability range are identified as hard samples; Obtain a video training set, which includes hard samples; The target video model is obtained by fine-tuning the preset video model based on the video training set.

13. A computer device, characterized in that, The device includes a memory and a processor, the memory being connected to the processor, the processor being configured to execute one or more computer programs stored in the memory, the processor causing the computer device to implement the classroom action recognition method as described in any one of claims 1-12 when executing the one or more computer programs.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the classroom action recognition method as described in any one of claims 1-12.

Images