Target behavior detection method, system, equipment and medium

Abstract

The invention relates to a target behavior detection method, system and device and a medium. The method comprises the steps of obtaining a video picture; inputting the video picture into an automatic matting model, performing matting processing on a portrait in the video picture, and splicing the portrait after matting with a preset simplified background to obtain a composite picture corresponding to the video picture; and inputting a synthesized image obtained by splicing into a target detection model to obtain a detection result of the character behavior, and identifying a target behavior of the person based on a detection result. According to the invention, the accuracy and practicability of behavior recognition in a complex or diversified scene are improved.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to a method, system, device and medium for detecting target behavior. Background Art

[0002] With the rise of deep learning, computer vision technology has been continuously developed, especially object detection technology represented by target detection and instance segmentation, which has good implementation solutions in various industries. However, in specific scenarios such as experimental operations or industrial process operations, this traditional target detection method has low detection accuracy due to complex backgrounds. Therefore, it is necessary to provide a behavior detection method, system, device and medium. Summary of the invention

[0003] In view of the above-mentioned shortcomings of the prior art, an object of the present invention is to provide a method, system, device and medium for detecting target behavior, which improves the problem of low accuracy of human behavior recognition in complex backgrounds or diverse scenes in the prior art.

[0004] To achieve the above-mentioned purpose and other related purposes, the present invention provides a method for detecting a target behavior, comprising: acquiring a video screen; inputting the video screen into an automatic cutout model, cutting out a portrait in the video screen, and splicing the cutout portrait with a preset simplified background to obtain a composite image corresponding to the video screen; inputting the spliced ​​composite image into a target detection model to obtain a detection result of the person's behavior; and identifying the person's target behavior based on the detection result.

[0005] In one embodiment of the present invention, the obtaining of video images includes: extracting a corresponding single video image from one or more surveillance videos at a preset time point; wherein, when extracting the video image, presetting one of the surveillance videos as the main surveillance.

[0006] In one embodiment of the present invention, when a single video frame is extracted from a surveillance video, the target behavior of a person is identified based on the detection result, including: determining whether the detection result is valid: if invalid, reacquiring the next video frame and identifying it; if valid, using the detection result as the target behavior of the person.

[0007] In one embodiment of the present invention, when a single video frame is extracted from each of multiple surveillance videos, the target behavior of a person is identified based on the detection results, including: for each extracted video frame: selecting the detection result corresponding to the composite image in the main channel monitoring from the detection results corresponding to each video frame; judging whether the detection result corresponding to the composite image in the main channel monitoring is valid: if invalid, determining the target behavior of the person based on the detection results of the remaining composite images; if valid, using the detection result of the composite image in the main channel monitoring as the target behavior of the person.

[0008] In one embodiment of the present invention, determining the target behavior of a person based on the detection results of the remaining composite images includes: judging whether the detection results of the remaining composite images are consistent: if they are consistent, taking the detection results of the remaining composite images as the target behavior of the person; otherwise, re-extracting a single video frame from each of the surveillance videos and performing detection.

[0009] In one embodiment of the present invention, the acquisition of video images includes: extracting corresponding multiple video images from one or more surveillance videos at preset time intervals within a preset time length; wherein, when extracting the video images, one of the surveillance videos is pre-set as the main surveillance.

[0010] In one embodiment of the present invention, when multiple video frames are extracted from a surveillance video, the target behavior of a person is identified based on the detection results, including: counting the detection results corresponding to each extracted video frame; determining whether the total amount of video frames with consistent detection results is greater than or equal to a preset total frame threshold: if so, taking the consistent detection results as the target behavior of the person; otherwise, re-extracting multiple video frames from the surveillance video and performing detection.

[0011] In one embodiment of the present invention, when multiple video frames are correspondingly extracted from each of multiple surveillance videos, the target behavior of a person is identified based on the detection results, including: for each video frame extracted at the same time; from the detection results of each video frame, selecting the detection result corresponding to the composite image in the main channel monitoring: judging whether the detection result corresponding to the composite image in the main channel monitoring is valid: if invalid, determining the initial target behavior of the person based on the detection results of the remaining composite images; if valid, taking the detection result of the composite image in the main channel monitoring as the initial target behavior of the person; counting the detection results corresponding to all video frames extracted within the time length; judging whether the total amount of consistent initial target behaviors is greater than a preset behavior total amount threshold: if so, taking the initial target behavior as the target behavior of the person; otherwise, re-extracting multiple video frames from each of the surveillance videos and performing detection.

[0012] In one embodiment of the present invention, the method of determining the initial target behavior of a person based on the detection results of the remaining composite images includes: judging whether the detection results of the remaining composite images are consistent: if they are consistent, then using the detection results of the remaining composite images as the initial target behavior of the person; otherwise, re-extracting multiple video frames from each of the surveillance videos and performing detection.

[0013] In one embodiment of the present invention, a target behavior detection system is also provided, and the system includes: a data acquisition module, which is used to acquire video images; an automatic cutout module, which is used to input the video images into an automatic cutout model, cut out the portrait in the video images, and splice the cutout portrait with a preset simplified background to obtain a composite image corresponding to the video images; a behavior detection module, which is used to input the spliced ​​composite image into a target detection model to obtain a detection result of the person's behavior; and a behavior determination module, which is used to identify the target behavior of the person based on the detection result.

[0014] In one embodiment of the present invention, an electronic device is also provided, including: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the electronic device implements any of the above-mentioned methods for detecting the target behavior.

[0015] In one embodiment of the present invention, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor of a computer, the computer executes any of the above-mentioned methods for detecting target behaviors.

[0016] As described above, the target behavior detection method, system, device and medium of the present invention have the following beneficial effects: by using an automatic cutout model to cut out the video screen, and obtaining a synthetic image corresponding to the video screen after replacing it with a simplified background, the interference factors caused by the background can be effectively reduced, greatly improving the accuracy of subsequent target detection model recognition. The synthetic image is input into the target detection model to make the detection results of the person's behavior more accurate, and the target behavior of the person is determined by comprehensively analyzing the detection results to reduce misjudgment, greatly improving the accuracy and practicality of behavior recognition in complex or diverse scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of a flow chart of a method for detecting target behavior provided by an embodiment of the present invention;

[0018] Figure 2 Shown is a structural block diagram of a target behavior detection system provided by an embodiment of the present invention;

[0019] Figure 3Shown is a structural schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0021] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0022] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0023] The inventor found that although target detection and instance segmentation and other target detection methods are currently applied in many fields, there are still many deficiencies in relying solely on this target detection method in complex and changeable environments such as experimental operations or industrial process operations. For example, in teaching scenarios, teachers often cannot take into account the operation of each student, so they cannot stop some dangerous operations or students from doing behaviors unrelated to the classroom in time, so it is necessary to monitor and remind students of their wrong behaviors. And because of the complex environment, a simple target detection method cannot accurately judge students' intentions, so students cannot be warned in time when they do behaviors unrelated to the classroom. In the examination scenario, traditional manual proctoring and cameras installed in the classroom cannot accurately monitor the behavior of each student, so once cheating occurs, it will affect the fairness of the examination.

[0024] Furthermore, the inventors also found that the common method for judging the behavior of characters in the above-mentioned scenes currently mainly only uses the target detection method, performs target detection annotation on the character behavior data, and trains the target detection model. Due to the single data source and direct detection, the speed is relatively slow and the accuracy is reduced, which cannot meet the real-time scenario.

[0025] In order to improve the above problems, the present invention provides a method for detecting target behavior, which uses an automatic cutout model to cut out the video screen, and obtains a synthetic image corresponding to the video screen after replacing it with a simplified background, which can effectively reduce the interference factors caused by the background and greatly improve the accuracy of subsequent target detection model recognition. The synthetic image is input into the target detection model to make the detection result of the character's behavior more accurate, and the target behavior of the character is determined by comprehensively analyzing the detection results to reduce misjudgment, greatly improving the accuracy and practicality of behavior recognition in complex or diverse scenarios.

[0026] See also Figure 1 , the target behavior detection method includes the following steps:

[0027] S1. Obtain video images.

[0028] For ease of description, the present invention uses the scene of a student classroom as an example for relevant explanation, but it can be understood that the present invention can be applied not only to student classrooms, but also to scenes such as industrial process operations, without limitation here. In a classroom or industrial environment, the video picture is obtained by extracting the corresponding video frame from the surveillance video. To ensure the recognition accuracy, a camera can be set on each desk or device to capture and shoot the image of the student or operator in the corresponding position in real time and analyze it. Among them, the video picture can be a picture extracted from the surveillance video, or it can be a comprehensive analysis of multiple pictures, without limitation here.

[0029] Specifically, in one embodiment of the present invention, the obtaining of video images includes: extracting a corresponding single video image from one or more surveillance videos at a preset time point; wherein, when extracting the video image, presetting one of the surveillance videos as the main surveillance.

[0030] When a video screen is extracted from the surveillance video each time, multiple time points can be pre-set. A video screen can be extracted from the surveillance video of a single shooting angle at each time point, or a video screen can be extracted from multiple surveillance videos of different shooting angles. For example, a video screen is extracted once per second. It can be understood that when there are multiple surveillance videos, the video screens of each surveillance video must be extracted synchronously to ensure that the video screens of different shooting angles can accurately correspond to the scene at the same time point. In order to improve the accuracy and speed of subsequent recognition, one of the surveillance videos will be pre-designated as the main monitoring. Specifically, when there is one surveillance video, the surveillance video is used as the main monitoring. When there are multiple surveillance videos, the one with the most complete shooting information can be selected as the main monitoring based on the shooting angle. For example, there are three surveillance videos, which are installed at different positions of the desk to shoot students from three angles: front view, side view and top view. To ensure that the most comprehensive field of view can be obtained, the front view video is used as the main monitoring.

[0031] In another embodiment of the present invention, the acquisition of video images includes: extracting corresponding multiple video images from one or more surveillance videos at preset time intervals within a preset time length; wherein, when extracting the video images, one of the surveillance videos is pre-set as the main surveillance.

[0032] When extracting multiple video images from a surveillance video, a time length can be preset, and when extracting video images at a preset time interval within the time length, multiple video images can be extracted sequentially from a surveillance video of a single shooting angle. It is also possible to extract a video image at the same sampling point of each video surveillance from multiple surveillance videos of different shooting angles, and there is a time interval between adjacent sampling points. For example, within 3 seconds, a video image is extracted every 1 second. In addition, one frame can be arbitrarily extracted from two adjacent frames of the surveillance video as a video image. It is understandable that when there are multiple surveillance videos, the video images of each surveillance video must be extracted synchronously to ensure that the video images of different shooting angles can accurately correspond to the scene of the same sampling point. In order to improve the accuracy and efficiency of subsequent recognition, one of the surveillance videos will be pre-designated as the main surveillance. Specifically, when there is one surveillance video, the video is used as the main surveillance; when there are multiple surveillance videos, the one with the most comprehensive information can be selected as the main surveillance based on the shooting angle. For example, if there are three surveillance videos, the cameras are installed in different positions of the desks, and students are filmed from three angles: front view, side view, and top view. To ensure the most comprehensive field of view, the front view video can be used as the main surveillance channel.

[0033] S2. Input the video screen into the automatic cutout model, cut out the portrait in the video screen, and splice the cutout portrait with the preset simplified background to obtain a composite image corresponding to the video screen.

[0034] The video image obtained above is input into the automatic cutout model, and the video image is analyzed by image recognition technology to identify the portrait area therein, and the portrait area is separated from the video image. The cutout portrait area is spliced ​​with the preset simplified background to form a composite image of the original video image. Among them, the automatic cutout model is a MODnet model without Trimap, and the simplified background is used to remove interference in the original background, so that the character image is clearer and more visible, and improve the accuracy and efficiency of subsequent character behavior recognition. It includes but is not limited to a solid color background, a gradient color background, or a background with a simple pattern, etc. Those skilled in the art can make adaptive choices based on actual image processing needs, and are not limited here. Furthermore, in order to make the character image more prominent and simplify visual complexity, in one embodiment of the present invention, the simplified background is a white background.

[0035] In one embodiment of the present invention, for each video screen: the video screen is input into the automatic cutout model, a portrait in the video screen is cutout, and the cutout portrait is spliced ​​with a preset simplified background to obtain a composite image, including:

[0036] Inputting a video screen into the automatic cutout model, extracting a first resolution feature generated by a first module of the automatic cutout model, and a second resolution feature generated by a second module of the automatic cutout model; wherein the first resolution is smaller than the second resolution;

[0037] Based on the first resolution feature and the second resolution feature, identifying a foreground portrait from the video picture, and separating the foreground portrait from the video picture;

[0038] The foreground portrait is spliced ​​with a preset simplified background to obtain a composite image.

[0039] When the background in the video screen needs to be replaced, each video screen is input into the automatic cutout model. First, the first module composed of the first several layers in the automatic cutout model extracts the low-resolution features (i.e., the first resolution features) in the screen according to the low-resolution supervision signal, and the features are used to predict the overall outline of the portrait and generate a rough foreground mask. Then, the second module composed of the latter several layers in the model extracts the high-resolution features (i.e., the second resolution features) in the screen according to the high-resolution supervision signal, which are used to predict the edge details, so as to more accurately distinguish the transition area between the portrait area and the background area. The first resolution features and the second resolution features are combined, and the approximate position and shape of the portrait are determined by the first resolution features, and the specific details and edge features of the portrait are determined by the second resolution features, so that the foreground portrait can be identified from the video screen, and the portrait can be separated from the video screen by the cutout technology. The cutout portrait is spliced ​​with the preset simplified background (such as a white background) to ensure that the position of the portrait on the new background is consistent with the original background, and to ensure a smooth transition of the edge area, so as to obtain a composite image of the original video screen.

[0040] S3. Input the spliced ​​composite image into the target detection model to obtain the detection result of the person's behavior.

[0041] The composite image obtained by the above splicing is input into the target detection model. The main branch of the model includes multiple interconnected layers of convolutional neural networks, which capture multi-scale features in the composite image through pooling and non-pooling methods to obtain feature maps of multiple different scales. While the main branch captures multi-scale features, the auxiliary branch of the model will further process the feature map extracted by the main branch through other additional convolutional layers to help enhance the feature extraction ability of the main branch. All feature maps obtained by the main branch and the auxiliary branch are input into the prediction head of the target detection model, and each feature map is parsed to obtain the detection result of the character behavior in the input composite image. Among them, the target detection model is a YOLO series model. Preferably, in order to further improve the detection accuracy and speed, the target detection model is a lightweight target detection network based on YOLOv9.

[0042] It should be noted that for classroom scenes, the detection results can be students' classroom behaviors such as "sleeping", "playing with mobile phones", "talking", "no abnormality", etc. It is understandable that for different scenes, the corresponding detection results are different, and those skilled in the art can adaptively set the corresponding detection results based on the actual scene needs, which is not limited here.

[0043] S4. Identify the target behavior of the person based on the detection results.

[0044] Since video surveillance can be one channel or multiple channels, the extracted video images can be multiple different images from multiple time points, or a single image at a single time point, so corresponding analysis will be performed according to different situations. Specifically, in the case of multiple channels and multiple images, the detection results of each viewpoint and time point will be combined to determine the target behavior of the person. In the case of a single channel and multiple images, the detection results of each time point will be combined to determine the target behavior of the person. In the case of multiple channels and single images, the detection results of each viewpoint will be combined to determine the target behavior of the person. In the case of a single channel and multiple images, the target behavior of the person will be determined directly based on the detection results.

[0045] Specifically, in one embodiment of the present invention, when a single video frame is extracted from a surveillance video, the target behavior of a person is identified based on the detection result, including:

[0046] Determine whether the test result is valid:

[0047] If invalid, re-acquire the next video frame and perform recognition;

[0048] If valid, the detection result is used as the target behavior of the character.

[0049] When there is only one surveillance video, and a video frame is extracted from the surveillance video at a preset time point each time for analysis to determine the target behavior of the person, the judgment will be made directly based on the detection result generated by the target detection model. Specifically, if the confidence of the target detection model for all candidate detection results is lower than the preset threshold, it means that the current video frame does not recognize the behavior of the person, and the detection result can be considered invalid at this time. At this time, the next video frame will continue to be obtained from the surveillance video, and a composite image of the video frame will be obtained by the aforementioned cutout method, and the new composite image will be input into the target detection model, and a new detection result will be regenerated and analyzed again. On the contrary, if the confidence of a candidate detection result in the target detection model is the highest and higher than the preset threshold, the detection result is considered valid, and the detection result is used as the target behavior of the person, and the next video frame will continue to be obtained from the surveillance video for analysis.

[0050] Exemplarily, a video screen is extracted from the surveillance video every 1 second. For the video screen extracted in the first second, the composite image obtained after the cutout process is input into the target detection model, and the confidence of "speaking" is the highest and higher than the preset threshold, then "speaking" is used as the target behavior of the student. Then for the video screen extracted in the second second, the above process is repeated, and the confidence of all candidate detection results (including but not limited to "playing with mobile phone", "speaking", "no abnormality", etc.) is lower than the preset threshold, then the current video screen is no longer analyzed, and the third second video screen is selected for analysis. Through the above process, each screen is analyzed frame by frame, which can more accurately capture the changes in student behavior and issue early warning information in time.

[0051] In another embodiment of the present invention, when a single video frame is correspondingly extracted from each of the multiple surveillance videos, the target behavior of the person is identified based on the detection result, including:

[0052] For each extracted video frame:

[0053] From the detection results corresponding to the various video images, select the detection result corresponding to the composite image in the main road monitoring;

[0054] Determine whether the detection result corresponding to the synthetic image in the main road monitoring is valid:

[0055] If invalid, determine the target behavior of the character based on the detection results of the remaining composite images;

[0056] If valid, the detection result of the synthetic image in the main road monitoring is used as the target behavior of the person.

[0057] When there are multiple channels of surveillance video (such as front view, side view and top view), and according to the preset time point, a video screen is extracted from each channel of the multi-channel surveillance for analysis each time to determine the target behavior of the person. Each extracted video screen is cut out by the automatic cutout model to obtain a composite image corresponding to each video screen. Each composite image is input into the target detection model to obtain the detection result of each composite image; wherein each detection result corresponds to a video screen. In the case of multi-channel surveillance, since one channel of surveillance has been pre-set as the main channel surveillance, the detection result corresponding to the composite image in the main channel surveillance is selected. Based on the confidence of the detection result, the detection result of the composite image in the main channel surveillance is judged to determine whether it is valid. If the confidence of each candidate detection result output in the target detection model is lower than the preset threshold, the detection result is invalid, and it is necessary to analyze according to the detection results of the remaining composite images. On the contrary, if the confidence of a candidate detection result in the is the highest and higher than the preset threshold, the detection result is considered valid, and the detection result is used as the target behavior of the person, and a video screen is extracted from each channel of the multi-channel surveillance and analyzed. Through this method of jointly analyzing multiple monitoring channels and giving priority to the detection results of the main monitoring channel, it is possible to capture the multiple characteristics of a person and avoid problems such as inaccurate or even failed recognition caused by occlusion or monitoring failure in a single monitoring channel, thereby improving the robustness of the model and the accuracy of recognition.

[0058] In one embodiment of the present invention, determining the target behavior of the person based on the detection results of the remaining composite images includes:

[0059] Determine whether the detection results of the remaining composite images are consistent:

[0060] If they are consistent, the detection results of the remaining composite images are used as the target behavior of the person;

[0061] Otherwise, a single video frame is re-extracted from each of the surveillance videos and detected.

[0062] When the detection result of the main channel is invalid, the first step is to determine whether the detection results of the remaining composite images are consistent. If they are consistent, it means that the detection results of the remaining perspectives are the same. At this time, the consistent detection results will be used as the target behavior of the person. After the judgment is completed, the corresponding single video screen will continue to be re-extracted from each channel of each surveillance video and analyzed again. For example, when there is occlusion in the main channel monitoring, the detection result of the main channel is invalid, and the remaining detection results are all "playing with mobile phones", then it is considered that the student's target behavior at this moment is "playing with mobile phones". Furthermore, if the detection results of the remaining composite images are inconsistent, it means that the behaviors recognized from different perspectives are different (such as the detection result of the top view is "sleeping" and the detection result of the side view is "talking"), and the target behavior of the person cannot be determined at this time, so the corresponding single video screen needs to be re-extracted from each channel of each surveillance video and analyzed again.

[0063] In yet another embodiment of the present invention, when a plurality of video frames are extracted from a surveillance video, the target behavior of a person is identified based on the detection result, including:

[0064] Count the detection results corresponding to each extracted video screen;

[0065] Determine whether the total amount of video images with consistent detection results is greater than or equal to the preset total amount threshold:

[0066] If so, the consistent detection result is used as the target behavior of the character;

[0067] Otherwise, multiple video frames are re-extracted from the surveillance video and detected.

[0068] In order to improve the processing accuracy and reduce the possibility of single-frame misjudgment, the present invention can also use multiple video images for comprehensive analysis to determine the target behavior of the person. Specifically, when there is only one monitoring video, and within a preset time length, multiple video images are extracted for comprehensive analysis at preset time intervals, and the synthetic image corresponding to each video image is input into the target detection model to obtain the detection result corresponding to each video image. The detection results of all video images extracted within the current time length are counted, and it is determined whether the total amount of images with consistent detection results is greater than or equal to the total amount of images threshold: if so, the detection result is used as the target behavior of the person; otherwise, the target behavior of the person cannot be determined. After the above judgment and execution are completed, multiple video images are extracted from the monitoring video again and analyzed again.

[0069] In another embodiment of the present invention, when a plurality of video frames are correspondingly extracted from each of the plurality of surveillance videos, the target behavior of the person is identified based on the detection result, including:

[0070] For each video frame extracted at the same time:

[0071] From the detection results of each video screen, select the detection result corresponding to the composite image in the main channel monitoring:

[0072] Determine whether the detection result corresponding to the synthetic image in the main road monitoring is valid:

[0073] If invalid, the initial target behavior of the character is determined based on the detection results of the remaining composite images;

[0074] If valid, the detection result of the synthetic image in the main road monitoring is used as the initial target behavior of the person;

[0075] Counting the detection results corresponding to all video images extracted within the time length;

[0076] Determine whether the total amount of initial target behavior is greater than the preset behavior total amount threshold:

[0077] If so, the initial target behavior is used as the character's target behavior;

[0078] Otherwise, multiple video frames are re-extracted from each of the surveillance videos and detected.

[0079] In order to improve the accuracy of behavior recognition and fully capture the dynamics of behavior, the present invention still uses multi-channel monitoring and multi-screen for comprehensive analysis. When there are multiple monitoring videos, and within a preset time length, multiple video screens are extracted for comprehensive analysis in each monitoring according to a preset time interval. After each extracted video screen is obtained by the automatic cutout model to obtain a synthetic image, it is input into the target detection model to obtain the detection result of the video screen. Since one monitoring channel has been pre-designated as the main monitoring channel, the target behavior of the person will be judged based on the detection result of the main monitoring channel. Specifically, the candidate detection results corresponding to the synthetic image in the main monitoring channel are first counted. If the confidence of each detection result is lower than the preset threshold, it means that the detection result corresponding to the synthetic image in the main monitoring channel is invalid. At this time, the detection results of the remaining synthetic images need to be comprehensively considered to determine the initial target behavior of the person. On the contrary, if the confidence of one of the candidate detection results is the highest and higher than the preset threshold, it means that the detection result is valid and it is used as the initial target behavior of the person. The detection results corresponding to all video frames extracted within the time length are counted, and it is determined whether the number of consistent initial target behaviors is greater than the preset total behavior threshold: if the consistent number is greater than the total threshold, the initial target behavior is used as the target behavior of the person, and multiple video frames are re-extracted from each of the surveillance videos and detected; otherwise, multiple video frames are continued to be re-extracted from each of the surveillance videos and detected.

[0080] In one embodiment of the present invention, the determining of the initial target behavior of the person based on the detection results of the remaining composite images includes:

[0081] Determine whether the detection results of the remaining composite images are consistent:

[0082] If they are consistent, the detection results of the remaining composite images are used as the initial target behavior of the person;

[0083] Otherwise, multiple video frames are re-extracted from each of the surveillance videos and detected.

[0084] When the detection result of the main monitoring is invalid, the detection results of the composite images generated by the remaining monitoring videos will be analyzed. These composite images come from monitoring videos from different perspectives and represent the shooting images from different angles at the same time point. Compare the detection results of the remaining composite images. If these results are consistent, the consistent detection results are used as the initial target behavior of the person. For example, if the detection results of the remaining composite images are all "sleeping", "sleeping" is used as the student's initial target behavior at this moment. On the contrary, if the detection results of these composite images from different perspectives are inconsistent, it means that there are differences in the behavior recognition results under different perspectives (such as the detection result of one perspective is "sleeping" and the other perspective is "speaking"). At this time, the student's initial target behavior cannot be determined, and it is necessary to re-acquire the video screen from the next time point for detection.

[0085] It should be noted that the target detection model is obtained by training a pre-acquired synthetic image data set. In one embodiment of the present invention, in the synthetic image data set, each video screen has multiple synthetic images, and the acquisition process of the multiple synthetic images is:

[0086] Obtain at least one video image;

[0087] The video screen is input into the automatic cutout model, the portrait in the video screen is cutout, and the cutout portrait is spliced ​​with a plurality of preset different simplified backgrounds to obtain a plurality of composite images.

[0088] In order to expand the training data, when training the target detection model, the present invention first performs a cutout process on the video screen through an automatic cutout model, and then splices the cutout portrait with multiple different simplified backgrounds to obtain multiple synthetic images. In this way, generating a variety of backgrounds can greatly increase the diversity of the data set, so that the trained target detection model can adapt to different environments and increase the generalization ability.

[0089] In the training phase of the target detection model, data collection is required first. You can use the AI ​​terminal student assessment device, which uses three cameras to record three-view data, namely front view, side view and top view. The device contains an AI development module based on RK3588, which can integrate AI, IoT and edge computing. The module uses an eight-core 64-bit processor, ARM Mali-G610 MP4 GPU and integrated AI acceleration NPU, and supports 8K, 32GB RAM video encoding and decoding, and has multiple video input and output interfaces. Use this device to record and collect videos of some students that are not related to learning and cheating in exams, such as looking around, sleeping, playing with lighters, spinning pens, reading notes, playing with mobile phones, etc. After collecting three videos, extract the corresponding behavior action key frames, perform target detection and annotation, and then use the automatic cutout model to cut out the portrait part and replace the background for data augmentation.

[0090] It should be further explained that, in order to solve the problem of information loss, the present invention also adopts a compilable gradient information method when training the target detection model, improves the gradient calculation method, and designs a new network structure - a generalized efficient layer aggregation network to output the detection results. In the actual application stage of the target detection model, if it is continuously judged that several frames are performing dangerous or cheating operations, a reminder is output on the display interface to help monitor and analyze student behavior.

[0091] It is understandable that the detection method described in the present invention can be used for both real-time behavior detection and offline detection. When real-time detection of video is required, considering factors such as device computing power, the video images extracted at each time point can be analyzed in sequence according to the preset time point, or after analyzing a video image, the next video image closest to the time point of the current detection result can be selected for analysis, without limitation. For example, if the sampling time point is 0.01 seconds, the video image obtained at 10:05 passes through the target detection model, and the time when the detection result is obtained is 10:07. At this time, the video image at 10:07 will be selected for analysis again, rather than the video image at 10:06.

[0092] See also Figure 2, the target behavior detection system 100 includes: a data acquisition module 110, an automatic cutout module 120, a behavior detection module 130 and a behavior determination module 140. The data acquisition module 110 is used to acquire the video screen. The automatic cutout module 120 is used to input the video screen into the automatic cutout model, cut out the portrait in the video screen, and splice the cutout portrait with the preset simplified background to obtain a composite image corresponding to the video screen. The behavior detection module 130 is used to input the spliced ​​composite image into the target detection model to obtain the detection result of the character behavior. The behavior determination module 140 is used to identify the target behavior of the character based on the detection result.

[0093] For the specific definition of the target behavior detection system, please refer to the definition of the target behavior detection method mentioned above, which will not be repeated here. Each module in the above-mentioned target behavior detection system can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware format, or can be stored in the memory in the computer device in software format, so that the processor can call the operations corresponding to the above modules.

[0094] It should be noted that, in order to highlight the innovative part of the present invention, the present embodiment does not introduce modules that are not closely related to solving the technical problem proposed by the present invention, but this does not mean that there are no other modules in the present embodiment.

[0095] See also Figure 3 The electronic device 1 may include a memory 12, a processor 13 and a bus, and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a target behavior detection program.

[0096] Among them, the memory 12 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. The memory 12 can be an internal storage unit of the electronic device 1 in some embodiments, such as a mobile hard disk of the electronic device 1. The memory 12 can also be an external storage device of the electronic device 1 in other embodiments, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1. Further, the memory 12 can also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 12 can not only be used to store application software and various types of data installed in the electronic device 1, such as the code for detecting the target behavior, etc., but also can be used to temporarily store data that has been output or is to be output.

[0097] In some embodiments, the processor 13 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 13 is the control core (Control Unit) of the electronic device 1, and uses various interfaces and lines to connect the various components of the entire electronic device 1, and executes or executes programs or modules (such as target behavior detection programs, etc.) stored in the memory 12, and calls the data stored in the memory 12 to execute various functions of the electronic device 1 and process data.

[0098] The processor 13 executes the operating system and various installed applications of the electronic device 1. The processor 13 executes the applications to implement the steps in the above-mentioned target behavior detection method.

[0099] Exemplarily, the computer program may be divided into one or more modules, which are stored in the memory 12 and executed by the processor 13 to complete the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the electronic device 1. For example, the computer program may be divided into a data acquisition module 110, an automatic cutout module 120, a behavior detection module 130, and a behavior determination module 140.

[0100] The above-mentioned integrated unit implemented in the form of a software function module can be stored in a computer-readable storage medium, and the computer-readable storage medium can be non-volatile or volatile. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to perform part of the functions of the target behavior detection method described in each embodiment of the present application.

[0101] In summary, the present invention discloses a method, system, device and medium for detecting target behavior, which performs cutout processing on the video screen through an automatic cutout model, and obtains a synthetic image corresponding to the video screen after replacing it with a simplified background, which can effectively reduce the interference factors caused by the background and greatly improve the accuracy of subsequent target detection model recognition. The synthetic image is input into the target detection model to make the detection result of the character's behavior more accurate, and the target behavior of the character is determined by comprehensively analyzing the detection results to reduce misjudgment, which greatly improves the accuracy and practicality of behavior recognition in complex or diversified scenarios. For experimental teaching scenarios, this method has the advantages of high detection accuracy and wide application scenarios, and can better carry out more accurate examination, teaching and evaluation for related chemical experiments. Therefore, the present invention effectively overcomes the various shortcomings in the prior art and has a high industrial utilization value.

[0102] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A method for detecting target behavior, characterized in that: The method comprises: Get the video screen; The video screen is input into the automatic cutout model, the portrait in the video screen is cutout, and the cutout portrait is spliced ​​with the preset simplified background to obtain a composite image corresponding to the video screen; The spliced ​​composite image is input into the target detection model to obtain the detection result of the human behavior; The target behavior of the person is identified based on the detection results.

2. The method for detecting target behavior according to claim 1, characterized in that: The step of obtaining the video image includes: At a preset time point, a corresponding single video frame is extracted from one or more surveillance videos; wherein, when extracting the video frame, one of the surveillance videos is preset as the main surveillance.

3. The method for detecting target behavior according to claim 2, characterized in that: When a single video frame is extracted from a surveillance video, the target behavior of a person is identified based on the detection result, including: Determine whether the test result is valid: If invalid, re-acquire the next video frame and perform recognition; If valid, the detection result is used as the target behavior of the character.

4. The method for detecting target behavior according to claim 2, characterized in that: When a single video frame is correspondingly extracted from each of the multiple surveillance videos, the target behavior of the person is identified based on the detection result, including: For each extracted video frame: From the detection results corresponding to the various video images, select the detection result corresponding to the composite image in the main road monitoring; Determine whether the detection result corresponding to the synthetic image in the main road monitoring is valid: If invalid, determine the target behavior of the character based on the detection results of the remaining composite images; If valid, the detection result of the synthetic image in the main road monitoring is used as the target behavior of the person.

5. The method for detecting target behavior according to claim 4, characterized in that: The step of determining the target behavior of the person based on the detection results of the remaining composite images includes: Determine whether the detection results of the remaining composite images are consistent: If they are consistent, the detection results of the remaining composite images are used as the target behavior of the person; Otherwise, a single video frame is re-extracted from each of the surveillance videos and detected.

6. The method for detecting target behavior according to claim 1, characterized in that: The step of obtaining the video image includes: From one or more surveillance videos, corresponding multiple video frames are extracted within a preset time length and at preset time intervals; wherein, when extracting the video frames, one of the surveillance videos is pre-set as the main surveillance.

7. The method for detecting target behavior according to claim 6, characterized in that: When multiple video frames are extracted from a surveillance video, the target behavior of the person is identified based on the detection result, including: Count the detection results corresponding to each extracted video screen; Determine whether the total amount of video images with consistent detection results is greater than or equal to a preset total amount threshold: if so, take the consistent detection results as the target behavior of the person; Otherwise, multiple video frames are re-extracted from the surveillance video and detected.

8. The method for detecting target behavior according to claim 6, characterized in that: When a plurality of video frames are correspondingly extracted from each of the plurality of surveillance videos, the target behavior of the person is identified based on the detection result, including: For each video frame extracted at the same time: From the detection results of each video screen, select the detection result corresponding to the composite image in the main channel monitoring: Determine whether the detection result corresponding to the synthetic image in the main road monitoring is valid: If invalid, the initial target behavior of the character is determined based on the detection results of the remaining composite images; If valid, the detection result of the synthetic image in the main road monitoring is used as the initial target behavior of the person; Counting the detection results corresponding to all video images extracted within the time length; Determine whether the total amount of initial target behavior is greater than the preset behavior total amount threshold: If so, the initial target behavior is used as the character's target behavior; Otherwise, multiple video frames are re-extracted from each of the surveillance videos and detected.

9. The method for detecting target behavior according to claim 8, characterized in that: The determining of the initial target behavior of the person based on the detection results of the remaining composite images includes: Determine whether the detection results of the remaining composite images are consistent: If they are consistent, the detection results of the remaining composite images are used as the initial target behavior of the person; Otherwise, multiple video frames are re-extracted from each of the surveillance videos and detected.

10. A target behavior detection system, characterized in that: The system comprises: A data acquisition module, used to acquire video images; The automatic cutout module is used to input the video screen into the automatic cutout model, cut out the portrait in the video screen, and splice the cutout portrait with the preset simplified background to obtain a composite image corresponding to the video screen; The behavior detection module is used to input the spliced ​​synthetic image into the target detection model to obtain the detection result of the person's behavior; The behavior determination module is used to identify the target behavior of the person based on the detection results.

11. An electronic device, characterized in that: The electronic device comprises: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the target behavior detection method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor of a computer, the computer is caused to execute the method for detecting a target behavior as described in any one of claims 1 to 9.

Images