Feature and action based video behavior detection artificial intelligence method and robot

Abstract

The application discloses a feature and action based video behavior detection artificial intelligence method and robot, which comprises a typical feature selection step, a typical feature detection step, a to-be-detected frame marking step, a to-be-detected video segment cutting step, a preset type of behavior action detection step and a to-be-detected video marking step. The method, system and robot can obtain typical feature categories of behavior actions by selecting typical features first, then detecting the typical features, and then detecting the preset type of behavior actions in those video segments with the typical features, so that the detection of many video segments without the typical features is reduced, and the detection speed and efficiency are greatly improved. The method is also applicable to real-time video acquisition, and the preset type of behavior action detection is not needed when there is no typical feature, so that the calculation cost is greatly saved, the detection speed is improved, and the real-time detection effect is achieved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an artificial intelligence method and robot for video behavior detection based on features and actions. Background Technology

[0002] Object detection projects have undergone years of development, resulting in a mature modeling system. This mainly includes two types: two-stage object detection algorithms, which first acquire candidate regions and then classify and correct their locations. These algorithms are highly accurate but slow. One-stage object detection algorithms, on the other hand, directly output the target's location and position information via a neural network. Representative examples of two-stage object detection algorithms include R-CNN, SPPNet, and Mask R-CNN; representative examples of one-stage object detection algorithms include YOLO, SSD, FPN, and EfficientDet.

[0003] Action detection, similar to object detection in images, requires knowing not only whether an action occurs in a video, but also when that action occurs. Action detection is significantly more difficult than action recognition, so there are currently no highly effective methods. It requires processing long, unsegmented videos, which often contain significant interference, with the target action typically occupying only a small portion of the video. Depending on whether the video is read in as a single block or sequentially, it is categorized as online or offline. Offline action detection reads in an entire video block at once and then locates the time of action within that block. Online action detection continuously reads in new frames, aiming to detect actions as early as possible during frame reading (before the action has finished). Real-time performance is also crucial for online action detection, which limits computational complexity (given current computing power). Frame-by-frame detection independently determines the type of action in each frame of the video sequence, utilizing methods like CNNs, but only spatial information. The sliding window method involves setting a fixed sliding window size, sliding the window across the video sequence, and then classifying the resulting video segments using action recognition.

[0004] In the process of realizing this invention, the inventors discovered that the prior art has at least the following problems: Under the prior art, if the behavior detection in video is done by sliding window, it is too slow and time-consuming. If the frame-by-frame detection method is used, it can only detect static content and is inaccurate for dynamic action detection.

[0005] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0006] Therefore, it is necessary to provide feature- and action-based artificial intelligence methods and robots for video behavior detection to address the shortcomings or deficiencies of existing technologies. These methods can assist manual video behavior detection, improve the efficiency and effectiveness of video behavior detection, and ensure that preset types of behaviors can be detected without omission.

[0007] In a first aspect, embodiments of the present invention provide an artificial intelligence method, the method comprising:

[0008] Typical feature selection steps: Obtain K video clips (K is a natural number greater than 1) for each preset type of behavior action; use each video clip as input, and calculate each object type in each video clip for each preset type of behavior action through a preset object detection deep learning model; add each object type in each video clip to the typical feature candidate set; count the number of each object type in the typical feature candidate set; if the number of each object type / K is greater than or equal to a preset ratio, then each object type is selected as a typical feature.

[0009] Typical feature detection steps: Acquire each frame of the video to be detected. Use each frame as input and calculate it through a typical feature detection deep learning model. The output is used as the label of the typical features of each frame. If the label of the typical features of each frame is not empty, extract the image within the marked range from each frame according to the marked range in the label of the typical features of each frame. Use the image within the marked range as input and calculate it through a person recognition image deep learning model. The output is used as the information of the person in the label of the typical features of each frame.

[0010] The steps for marking the frames to be detected are as follows: Obtain the labels of each frame image and its typical features in the video to be detected; if the labels of the typical features of each frame image are not empty, determine the maximum duration of the corresponding preset type of behavior based on the names of the typical features in the labels, and use this as the preset duration to be detected; if multiple labels of the typical features are not empty, determine multiple maximum durations of the corresponding preset type of behavior based on the names of the multiple typical features in the multiple labels, and take the largest of the multiple maximum durations as the preset duration to be detected; if the labels of the typical features of each frame image are not empty, mark each frame image as a frame to be detected, mark all frames before each frame image with a duration equal to / 2 of the preset duration to be detected as frames to be detected, and mark all frames after each frame image with a duration equal to / 2 of the preset duration to be detected as frames to be detected; some frames will be repeatedly marked as frames to be detected, while some frames will not be marked as frames to be detected.

[0011] The steps for extracting video segments to be detected are as follows: extract the continuous frames marked as frames to be detected but not marked as extracted from the video to be detected as a video segment to be detected, and mark the extracted frames as extracted.

[0012] The preset type of behavior and action detection steps are as follows: The video segment to be detected is used as input, and the output calculated by the preset type of behavior and action detection deep learning model is used as the label of the preset type of behavior and action in the video segment to be detected; Video segments within the marked range are used as input, and the output calculated by the person video recognition deep learning model is used as the information of the person in the preset type of behavior and action labels of the video segment; The labels of preset type of behavior and action in all the video segments to be detected are used as the labels of preset type of behavior and action in the video segment to be detected.

[0013] The step of tagging the video to be detected is as follows: Replace the corresponding video segment in the video to be detected with the video segment labeled with each preset type of behavior action, so as to obtain the video to be detected after being tagged with the preset type of behavior action.

[0014] Preferably, the method further includes:

[0015] Video acquisition steps: Acquire the video to be tested;

[0016] Steps for setting preset behavior actions: Obtain the preset behavior actions set by the user;

[0017] Steps for obtaining videos of preset types of behavioral actions: Obtain video samples of each preset type of behavioral action;

[0018] Steps for obtaining typical characteristics of preset behavior types: Obtain the typical characteristics of preset behavior types set by the user;

[0019] The steps for obtaining image samples of typical features of a preset type of behavior are as follows: Obtain image samples of each of the aforementioned typical features.

[0020] Preferably, the method further includes:

[0021] Steps to establish a video behavior knowledge graph: Establish multiple pre-defined types of behavior action entities, multiple typical feature entities, multiple person entities, multiple regulatory department entities, and multiple regulatory personnel entities;

[0022] Certain individuals, such as fugitives, need to be identified during video surveillance and sent to regulatory authorities.

[0023] The steps for defining relationships in a video behavior knowledge graph are as follows: A person entity points to a behavior action entity of a preset type through a relationship; a person entity points to a typical characteristic entity through a relationship; a person entity points to a regulatory department entity through a relationship; a regulatory personnel entity points to a regulatory department entity through a relationship.

[0024] The steps for generating static relationships in a video behavior knowledge graph are as follows: Based on the typical characteristics of user-defined preset types of behavior actions, establish relationships between typical characteristic entities and preset type behavior action entities.

[0025] Preferably, the method further includes:

[0026] The typical feature detection model construction steps are as follows: take the image sample of each typical feature as input, take the label of each typical feature as the expected output, train and test the deep learning model to obtain the typical feature detection deep learning model.

[0027] The steps for building a behavior and action detection model of preset types are as follows: take video samples of each preset type of behavior and action as input, take the label of each preset type of behavior and action as the expected output, train and test the deep learning model, and obtain a deep learning model for detecting behavior and action of preset types.

[0028] Steps for building a person image recognition model: Obtain a photo of each person, use the photo of each person as input, use the information of the person as the expected output, perform transfer learning on the face image recognition deep learning model, and obtain the person image recognition deep learning model.

[0029] Steps for building a person video recognition model: acquire a video of each person, use the video of each person as input, use the information of the person as the expected output, perform transfer learning on the face video recognition deep learning model, and obtain the person video recognition deep learning model.

[0030] In a second aspect, embodiments of the present invention provide an artificial intelligence system, the system comprising:

[0031] Typical Feature Selection Module: This module acquires K video clips (K is a natural number greater than 1) for each preset type of behavior action. Using each video clip as input, it calculates each object type within each video clip of each preset type of behavior action through a preset object detection deep learning model. Each object type in each video clip is added to a typical feature candidate set. The module then counts the number of each object type in the typical feature candidate set. If the number of each object type / K is greater than or equal to a preset ratio, then each object type is selected as a typical feature.

[0032] Typical Feature Detection Module: Acquires each frame of the video to be detected, uses each frame as input, and calculates the typical feature detection deep learning model. The output is used as the label of the typical feature of each frame. If the label of the typical feature of each frame is not empty, then the image within the marked range in the label of the typical feature of each frame is extracted from the frame. The image within the marked range is used as input, and calculated by the person recognition image deep learning model. The output is used as the information of the person in the label of the typical feature of each frame.

[0033] The detection frame labeling module acquires the labels of each frame image and typical features of each frame image in the video to be detected. If the labels of typical features of each frame image are not empty, the preset maximum duration of the corresponding preset type of behavior action is determined according to the name of the typical feature in the labels of the typical feature, and is used as the detection preset duration. If multiple labels of the typical feature are not empty, multiple preset maximum durations of the corresponding preset type of behavior action are determined according to the names of the multiple typical features in the multiple labels of the typical feature, and the largest of the multiple preset maximum durations is taken as the detection preset duration. If the labels of typical features of each frame image are not empty, each frame image is labeled as a detection frame, all frames before each frame image with a duration of the detection preset duration / 2 are labeled as detection frames, and all frames after each frame image with a duration of the detection preset duration / 2 are labeled as detection frames. Some frames will be repeatedly labeled as detection frames, and some frames will not be labeled as detection frames.

[0034] The video segment extraction module extracts consecutive frames marked as frames to be detected but not marked as extracted from the video to be detected as a video segment to be detected, and marks the extracted frames as extracted.

[0035] The preset type of behavior and action detection module: takes the video segment to be detected as input, and the output calculated by the preset type of behavior and action detection deep learning model is used as the label of the preset type of behavior and action in the video segment to be detected; takes the video segment within the marked range as input, and the output calculated by the person video recognition deep learning model is used as the information of the person in the label of the preset type of behavior and action in the video segment; and uses the labels of the preset type of behavior and action in all the video segments to be detected as the labels of the preset type of behavior and action in the video to be detected.

[0036] The video tagging module replaces the corresponding video segment in the video to be detected with the video segment labeled with each preset type of behavior action, thus obtaining the video to be detected after being tagged with the preset type of behavior action.

[0037] Preferably, the system further includes:

[0038] Video acquisition module: Acquires the video to be tested;

[0039] Preset type behavior action setting module: retrieves user-defined preset type behavior actions;

[0040] A video module for acquiring preset types of behavioral actions: acquires video samples of each preset type of behavioral action;

[0041] The module for obtaining typical characteristics of preset types of behavior actions: obtains the typical characteristics of preset types of behavior actions set by the user;

[0042] Image module for obtaining typical features of preset type of behavior: obtains image samples of each of the aforementioned typical features.

[0043] Preferably, the system further includes:

[0044] Establish a video behavior knowledge graph module: establish multiple preset types of behavior action entities, multiple typical feature entities, multiple person entities, multiple regulatory department entities, and multiple regulatory personnel entities;

[0045] The video behavior knowledge graph relationship definition module defines: a person entity points to a behavior action entity of a preset type through a relationship; a person entity points to a typical feature entity through a relationship; a person entity points to a regulatory department entity through a relationship; and a regulatory personnel entity points to a regulatory department entity through a relationship.

[0046] The video behavior knowledge graph static relationship generation module establishes relationships between typical feature entities and preset type behavior action entities based on the typical features of user-defined preset types of behavior actions.

[0047] Preferably, the system further includes:

[0048] Typical Feature Detection Model Construction Module: Takes image samples of each typical feature as input, and labels of each typical feature as expected output, to train and test the deep learning model to obtain a typical feature detection deep learning model;

[0049] Preset type behavior action detection model construction module: Take video samples of each preset type of behavior action as input, take the label of each preset type of behavior action as expected output, train and test the deep learning model to obtain the preset type of behavior action detection deep learning model;

[0050] The person image recognition model construction module: acquires a photo of each person, takes the photo of each person as input, takes the information of the person as the expected output, performs transfer learning on the face image recognition deep learning model, and obtains the person image recognition deep learning model.

[0051] Person video recognition model construction module: acquire the video of each person, take the video of each person as input, take the information of the person as the expected output, and perform transfer learning on the face video recognition deep learning model to obtain the person video recognition deep learning model.

[0052] Thirdly, embodiments of the present invention provide an artificial intelligence device, the device comprising modules of the system described in any one of the embodiments of the second aspect.

[0053] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of the method described in any one of the embodiments of the first aspect.

[0054] Fifthly, embodiments of the present invention provide a robot, including a memory, a processor, and an artificial intelligence robot program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any one of the methods described in the first aspect of the embodiments.

[0055] This embodiment provides a feature- and action-based video behavior detection artificial intelligence method and robot, including: a typical feature selection step; a typical feature detection step; a frame marking step; a video segment extraction step; a preset type of behavior action detection step; and a video marking step. The above method, system, and robot, by first selecting typical features to obtain the typical feature types of behavior actions, then detecting the typical features, and finally detecting preset types of behavior actions in those video segments containing typical features, reduces the detection of many video segments without typical features, thereby greatly improving detection speed and efficiency. This approach is also applicable to real-time video acquisition; when no typical features appear, there is no need to detect preset types of behavior actions, thus greatly saving computational costs, increasing detection speed, and achieving real-time detection. Attached Figure Description

[0056] Figure 1 A block diagram of an artificial intelligence system provided as an embodiment of the present invention;

[0057] Figure 2 A block diagram of an artificial intelligence system provided as an embodiment of the present invention;

[0058] Figure 3 A block diagram of an artificial intelligence system provided as an embodiment of the present invention;

[0059] Figure 4 A block diagram of an artificial intelligence system provided as an embodiment of the present invention. Detailed Implementation

[0060] The technical solutions in the embodiments of the present invention will be described in detail below with reference to the embodiments of the present invention.

[0061] Basic Embodiments of the Invention

[0062] Firstly, embodiments of the present invention provide an artificial intelligence method, the method comprising: a typical feature selection step; a typical feature detection step; a frame marking step; a video segment extraction step; a preset type of behavior / action detection step; and a video marking step. Technical effect: By first selecting typical features, the typical feature types of behavior / actions can be obtained. Then, by detecting the typical features, and then performing preset type behavior / action detection on those video segments containing typical features, the detection of many video segments without typical features is reduced, thereby greatly improving the detection speed and efficiency. This method is also applicable to real-time video acquisition. When no typical features appear, there is no need to perform preset type behavior / action detection, thereby greatly saving computational costs, improving detection speed, and achieving the effect of real-time detection.

[0063] Preferably, the method further includes: a video acquisition step; a preset type of behavior action setting step; a video acquisition step for the preset type of behavior action; a typical feature acquisition step for the preset type of behavior action; and an image acquisition step for the typical features of the preset type of behavior action. Technical effect: Allowing users to set typical features and preset type of behavior actions enables personalized detection tailored to each individual user, meeting the needs of different users. Since users may have different desired detection standards, if these standards are fixed in advance and user settings are not allowed, there will be technical defects. This application effectively overcomes this technical defect, enabling more personalized detection.

[0064] Preferably, the method further includes: establishing a video behavior knowledge graph; defining relationships within the video behavior knowledge graph; and generating static relationships within the video behavior knowledge graph. Technical benefits: The knowledge graph makes the relationships between individuals, regulatory bodies, typical characteristics, and preset types of behaviors readily apparent, improving user experience and facilitating more accurate detection using these relationships.

[0065] Preferably, the method further includes: a typical feature detection model construction step; a preset type behavior and action detection model construction step; a person image recognition model construction step; and a person video recognition model construction step. Technical effect: Training typical feature and action models using deep learning technology can overcome the shortcomings of traditional pattern matching and expert systems.

[0066] Secondly, embodiments of the present invention provide an artificial intelligence system, such as... Figure 1 As shown, the system includes: a typical feature selection module; a typical feature detection module; a frame marking module; a video segment extraction module; a behavior and action detection module of a preset type; and a video marking module.

[0067] Preferably, such as Figure 2 As shown, the system further includes: a video acquisition module; a preset type of behavior action setting module; a video module for acquiring preset type of behavior actions; a preset type of behavior action typical feature setting module; and an image module for acquiring preset type of behavior action typical features.

[0068] Preferably, such as Figure 3 As shown, the system also includes: a module for establishing a video behavior knowledge graph; a module for defining relationships in the video behavior knowledge graph; and a module for generating static relationships in the video behavior knowledge graph.

[0069] Preferably, such as Figure 4As shown, the system also includes: a typical feature detection model construction module; a preset type behavior action detection model construction module; a person image recognition model construction module; and a person video recognition model construction module.

[0070] Thirdly, embodiments of the present invention provide an artificial intelligence device, the device comprising modules of the system described in any one of the embodiments of the second aspect.

[0071] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of the method described in any one of the embodiments of the first aspect.

[0072] Fifthly, embodiments of the present invention provide a robot, including a memory, a processor, and an artificial intelligence robot program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any one of the methods described in the first aspect of the embodiments.

[0073] Preferred embodiments of the present invention

[0074] Video acquisition steps: Acquire the video to be detected; preferably, acquire the video to be detected in real time.

[0075] Preset behavior action setting steps: Obtain user-defined preset behavior actions, including violent actions. Preferably, obtain multiple user-defined preset behavior actions.

[0076] The steps for obtaining videos of preset types of behavioral actions are as follows: obtaining video samples of each preset type of behavioral action; preferably, obtaining a large number of video samples of each preset type of behavioral action; preferably, obtaining a first preset number of video samples of each preset type of behavioral action; preferably, the first preset number of different types of preset types of behavioral actions is different, with a larger first preset number for complex preset types of behavioral actions, a smaller first preset number for simple preset types of behavioral actions, and a larger first preset number for more complex preset types of behavioral actions.

[0077] The steps for obtaining typical features of preset types of behavior actions are as follows: Obtain typical features of user-defined preset types of behavior actions, including typical features of violent actions such as knives, guns, or fists. Preferably, obtain typical features of user-defined preset types of behavior actions. It can be seen that a typical feature of a certain action does not necessarily indicate that the action itself, and conversely, if it is a certain action, its typical features will definitely appear. That is, relying on typical features can easily lead to false detections, mistaking actions not belonging to a preset type for such actions. However, it will not result in false negatives, meaning that a preset type of behavior action will not be detected without its typical features. Therefore, typical features of preset types of behavior actions can be used for initial detection, thereby improving the speed and efficiency of preset type behavior action detection, saving detection time and costs. This is because typical features can be described by static images, while preset type behavior actions are continuous actions that can only be described by video. Obviously, the efficiency and speed of detecting specific images from a video to be detected is much higher than the efficiency and speed of detecting a specific video, thus reducing detection costs. This is of great significance for carbon neutrality and energy conservation. However, relying solely on typical features for detection is insufficient, as typical features cannot uniquely identify a pre-defined type of behavior or action. For example, a knife, which is typically characterized by its features, could also be a knife found on a supermarket shelf. Therefore, after initial screening using images with typical features, further confirmation is needed through action video.

[0078] The steps for obtaining image samples of typical features of a preset type of behavior are as follows: obtaining image samples of each of the typical features; preferably, obtaining a large number of image samples of each of the typical features; preferably, obtaining a second preset number of image samples of each of the typical features; preferably, the second preset number of typical features of different types is different, the second preset number of complex typical features is large, the second preset number of simple typical features is small, and the second preset number of more complex typical features is larger.

[0079] Steps to establish a video behavior knowledge graph: Establish multiple preset types of behavior action entities, multiple typical feature entities, multiple person entities, multiple regulatory department entities, and multiple regulatory personnel entities.

[0080] The steps for defining relationships in a video behavior knowledge graph are as follows: Typical feature entities point to predefined type behavior action entities through membership relationships. A typical feature entity has a membership relationship with one or more predefined type behavior action entities, and a predefined type behavior action entity has a membership relationship with one or more typical feature entities. Human entities point to predefined type behavior action entities through association relationships. A human entity has an association relationship with zero or more predefined type behavior action entities, and a predefined type behavior action entity has an association relationship with zero or more human entities. Human entities point to regulatory department entities through association relationships. A human entity has an association relationship with one regulatory department entity, and a regulatory department entity has an association relationship with multiple human entities. Regulatory personnel entities point to regulatory department entities through association relationships. A regulatory personnel entity has an association relationship with one regulatory department entity, and a regulatory department entity has an association relationship with multiple regulatory personnel entities.

[0081] The steps for generating static relationships in a video behavior knowledge graph are as follows: Based on the typical characteristics of user-defined preset types of behavior actions, establish relationships between typical characteristic entities and preset type behavior action entities.

[0082] Typical feature selection steps: First, obtain a preset object detection deep learning model capable of detecting a wide variety of objects in images, such as YOLO, SSD, FPN, and EfficientDet. Second, obtain multiple video clips for each preset type of action. Using each video clip as input, the preset object detection deep learning model calculates the object type for each video clip of each preset type of action. Then, add each object type from each video clip to a typical feature candidate set (e.g., obtain K video clips for each preset type of action, using each of the K video clips as input, and calculate the object type for each object in each video clip of each preset type of action using the preset object detection deep learning model; add the object type of each object from the 1st to the kth video clips to the typical feature candidate set). Each frame in the video clips of each preset type of action constitutes the preset type of action. The number of each object type in the candidate set of typical features is counted. If the number of each object type / k is greater than or equal to a preset ratio, then each object type is regarded as a typical feature, thus obtaining multiple typical features.

[0083] The steps for constructing a typical feature detection model are as follows: Image samples of each typical feature are used as input, and the labels of each typical feature are used as the expected output. The deep learning model is trained and tested to obtain a deep learning model for typical feature detection. In each image sample containing a typical feature, a preset marker is used to mark the range containing that typical feature. For example, if the typical feature is a knife, gun, or fist, a box is used to enclose the knife, gun, or fist in the image. The label for each typical feature includes the name of that feature (e.g., knife, gun, or fist) and the position of the marker in the image, such as the position of the box (coordinates of the four vertices). If the position of the marker in the image is empty in the label for each typical feature, it indicates that the image sample does not contain that typical feature.

[0084] The steps for building a preset type of behavior detection model are as follows: Video samples of each preset type of behavior are used as input, and the labels of each preset type of behavior are used as the expected output. The deep learning model is trained and tested to obtain a deep learning model for detecting preset types of behavior. Each preset type of behavior is marked with a preset label in the video samples, and the labeling range must cover each preset type of behavior. This range includes both spatial and temporal ranges. The temporal range extends from the start frame to the end frame of each preset type of behavior. For example, if the preset type of behavior is a violent action, then every frame in the video showing a violent action is enclosed in a bounding box. The label for each preset type of behavior includes the name of the behavior (e.g., a violent action), the start frame, the end frame, and the position of the label in each frame, such as the position of the bounding box in each frame (coordinates of four vertices). If the start frame and end frame of each preset type of behavior action are empty in the label of each preset type of behavior action, it indicates that the video sample does not contain each preset type of behavior action.

[0085] Steps for building a person image recognition model: Obtain a photo of each person, use the photo of each person as input, use the information of the person (such as the person's name, ID number, or student ID) as the expected output, perform transfer learning on the face image recognition deep learning model, and obtain the person image recognition deep learning model.

[0086] Steps for building a person video recognition model: Obtain a video of each person, use the video of each person as input, use the information of the person (such as the person's name, ID number, or student ID) as the expected output, and perform transfer learning on the face video recognition deep learning model to obtain the person video recognition deep learning model.

[0087] Typical feature detection steps: Acquire each frame of the video to be detected. Use each frame as input and calculate using a typical feature detection deep learning model. The output is used as the label of the typical features of each frame. If the label of the typical features of each frame is not empty (if empty, it indicates that the frame does not possess typical features), then extract an image within the marked range from each frame according to the marked range in the label of the typical features of each frame. Use the image within the marked range as input and calculate using a person recognition image deep learning model. The output is used as the person information in the label of the typical features of each frame (the person information can be empty, because some actions may not necessarily involve a person). If the label of the typical features of each frame is not empty, then each frame contains one or more typical features.

[0088] The step of marking the frame to be detected is as follows: Obtain the labels of each frame image and its typical features in the video to be detected. If the labels of the typical features of each frame image are not empty, determine the maximum preset duration of the corresponding preset type of behavior based on the names of the typical features in the labels, and use this as the preset duration to be detected. If multiple labels of the typical features are not empty, determine multiple maximum preset durations of the corresponding preset type of behavior based on the names of the multiple typical features in the multiple labels, and take the largest of the multiple maximum preset durations as the preset duration to be detected. If the labels of the typical features of each frame image are not empty, mark each frame image as a frame to be detected, mark all frames before each frame image with a duration equal to / 2 of the preset duration to be detected as frames to be detected, and mark all frames after each frame image with a duration equal to / 2 of the preset duration to be detected as frames to be detected. Some frames may be marked repeatedly as frames to be detected, while others may not be marked.

[0089] The steps for extracting video segments to be detected are as follows: Continuous frames marked as "to be detected" but not marked as "extracted" are extracted from the video to be detected and treated as a single video segment. Extracted frames are then marked as "extracted." Zero, one, or more video segments to be detected can be extracted from the video to be detected.

[0090] The preset type of behavior and action detection steps are as follows: The video segment to be detected is used as input, and the output calculated by the preset type of behavior and action detection deep learning model is used as the label for the preset type of behavior and action in the video segment to be detected. Video segments within the marked range are used as input, and the output calculated by the person video recognition deep learning model is used as the person information (the person information can be empty, because some actions may not involve a person) in the preset type of behavior and action labels of the video segment. If the label of the typical features of each frame is not empty, then each frame contains one or more typical features. The labels of the preset type of behavior and action in all the video segments to be detected are used as the labels for the preset type of behavior and action in the video segment to be detected.

[0091] The step of tagging the video to be detected is as follows: Replace the corresponding video segment in the video to be detected with the video segment labeled with each preset type of behavior action, so as to obtain the video to be detected after being tagged with the preset type of behavior action.

[0092] By first detecting typical features and then performing pre-defined behavior detection on video segments containing those features, the detection of many video segments lacking typical features is reduced, thus significantly improving detection speed and efficiency. This method is also applicable to real-time video acquisition; when typical features are absent, there is no need to perform pre-defined behavior detection, thereby greatly saving computational costs, increasing detection speed, and achieving real-time detection capabilities.

[0093] The steps for constructing a detection model for each typical feature are as follows: Image samples of each typical feature are used as input, and the labels of each typical feature are used as the expected output. An existing deep learning model for feature detection is trained and tested (transfer learning) to obtain the deep learning model for each typical feature. The image samples of each typical feature are marked with preset markers to indicate the range containing each typical feature. For example, if the typical feature is a knife, gun, or fist, a box is used to enclose the knife, gun, or fist in the image. The label of each typical feature includes the name of each typical feature (e.g., knife, gun, or fist) and the position of the marker for each typical feature in the image, such as the position of the box (coordinates of the four vertices). If the position of the marker for each typical feature in the image is empty in the label of each typical feature, it indicates that the image sample does not contain each typical feature.

[0094] The steps for building a behavior action detection model for each preset type are as follows: take video samples of each preset type of behavior action as input, take the label of each preset type of behavior action as the expected output, train and test the existing deep learning model for action detection (transfer learning), and obtain the deep learning model for each preset type of behavior action detection.

[0095] Each preset type of behavior action is marked with a preset tag in the video sample, and the marked area must cover each preset type of behavior action. This area includes both spatial and temporal ranges. The temporal range extends from the start frame to the end frame of each preset type of behavior action. For example, if the preset type of behavior action is violent, then every frame in the video showing a violent behavior action of the preset type of behavior action from the start frame to the end frame is enclosed in a box. The tag for each preset type of behavior action includes the name of each preset type of behavior action (e.g., violent action), the start frame, the end frame, and the position of the tag in each frame, such as the position of the box in each frame (coordinates of the four vertices). If the start frame and end frame of each preset type of behavior action are empty in the tag, it indicates that the video sample does not contain each preset type of behavior action.

[0096] Training a deep learning model for each typical feature and each preset type of behavior can make the deep learning model more targeted and accurate. Although this increases the computational cost to some extent, the cost can be reduced by targeted detection.

[0097] Steps for building a person image recognition model: Obtain a photo of each person, use the photo of each person as input, use the information of the person (such as the person's name, ID number, or student ID) as the expected output, perform transfer learning on the face image recognition deep learning model, and obtain the person image recognition deep learning model.

[0098] Steps for building a person video recognition model: Obtain a video of each person, use the video of each person as input, use the information of the person (such as the person's name, ID number, or student ID) as the expected output, and perform transfer learning on the face video recognition deep learning model to obtain the person video recognition deep learning model.

[0099] Typical feature detection steps: Acquire each frame of the video to be detected. Use each frame as input and calculate using a typical feature detection deep learning model. The output is used as the label of the typical features of each frame. If the label of the typical features of each frame is not empty (if empty, it indicates that the frame does not possess typical features), then extract an image within the marked range from each frame according to the marked range in the label of the typical features of each frame. Use the image within the marked range as input and calculate using a person recognition image deep learning model. The output is used as the person information in the label of the typical features of each frame (the person information can be empty, because some actions may not necessarily involve a person). If the label of the typical features of each frame is not empty, then each frame contains one or more typical features.

[0100] The step of marking the frame to be detected is as follows: Obtain the labels of each frame image and its typical features in the video to be detected. If the labels of the typical features of each frame image are not empty, determine the maximum preset duration of the corresponding preset type of behavior based on the names of the typical features in the labels, and use this as the preset duration to be detected. If multiple labels of the typical features are not empty, determine multiple maximum preset durations of the corresponding preset type of behavior based on the names of the multiple typical features in the multiple labels, and take the largest of the multiple maximum preset durations as the preset duration to be detected. If the labels of the typical features of each frame image are not empty, mark each frame image as a frame to be detected, mark all frames before each frame image with a duration equal to / 2 of the preset duration to be detected as frames to be detected, and mark all frames after each frame image with a duration equal to / 2 of the preset duration to be detected as frames to be detected. Some frames may be marked repeatedly as frames to be detected, while others may not be marked.

[0101] The steps for extracting video segments to be detected are as follows: Continuous frames marked as "to be detected" but not marked as "extracted" are extracted from the video to be detected and treated as a single video segment. Extracted frames are then marked as "extracted." Zero, one, or more video segments to be detected can be extracted from the video to be detected.

[0102] The steps for determining the behavior and action of the candidate preset type of the video segment to be detected are as follows: obtain each video segment to be detected, obtain the name of the typical feature in the non-empty label of the typical features of all frames in each video segment to be detected, obtain the name of multiple typical features, remove duplicates of the names of the multiple typical features, and obtain a set of names of different typical features, which is used as the set of typical features to be detected.

[0103] The steps for determining the pre-defined type of behavior action to be detected are as follows: Obtain the name of each typical feature in the set of typical features to be detected. Based on the relationship between typical feature entities and pre-defined type behavior action entities in the video behavior knowledge graph, obtain the names of one or more pre-defined type behavior actions corresponding to the name of each typical feature. Remove duplicate names from all pre-defined type behavior action names obtained from the names of all typical features in the set of typical features to be detected, resulting in a set of distinct pre-defined type behavior action names, which serves as the set of pre-defined type behavior actions to be detected.

[0104] The specific preset type of behavior action detection steps are as follows: Each video segment to be detected is used as input. The output calculated by the deep learning model for each preset type of behavior action in the set of preset type behaviors is used as the label for each preset type of behavior action in the video segment to be detected. Video segments within the labeled range are used as input. The output calculated by the deep learning model for human video recognition is used as the information of the human being in the label for each preset type of behavior action in the video segment (the human being information can be empty, because there may not be a human being in the action). If the label of the typical features of each frame image is not empty, then each frame image contains one or more typical features. The labels of each preset type of behavior action in all the video segments to be detected are used as the labels for each preset type of behavior action in the video to be detected.

[0105] This approach eliminates the need to detect all preset types of actions in a video clip; it only requires detecting preset types of actions corresponding to typical features, thereby reducing the computational load of detecting preset types of actions.

[0106] The step of tagging the video to be detected is as follows: Replace the corresponding video segment in the video to be detected with the video segment labeled with each preset type of behavior action, so as to obtain the video to be detected after being tagged with the preset type of behavior action.

[0107] By first detecting typical features and then performing pre-defined behavior detection on video segments containing those features, the detection of many video segments lacking typical features is reduced, thus significantly improving detection speed and efficiency. This method is also applicable to real-time video acquisition; when typical features are absent, there is no need to perform pre-defined behavior detection, thereby greatly saving computational costs, increasing detection speed, and achieving real-time detection capabilities.

[0108] The steps for creating a portrait of a person based on typical features are as follows: Obtain the names of typical features and the information of the person from the non-empty labels of the typical features in each frame of the video to be detected, and establish a relationship between the typical feature entities and the person entities in the video behavior knowledge graph.

[0109] Steps for creating a character profile based on preset types of behavior: Obtain the name of the preset type of behavior and the character's information from the tags of each preset type of behavior in the video segment to be detected, where the tags are not empty, and establish a relationship between the preset type of behavior entities and the character entities in the video behavior knowledge graph.

[0110] Steps for determining a person with a preset type of action: Obtain each person in the video behavior knowledge graph. If each person has a relationship connection with a preset type of action entity in the video behavior knowledge graph, then each person has a preset type of action.

[0111] Preset type action warning steps: Obtain the regulatory department with which each person has a preset type of behavior in the video behavior knowledge graph has a relationship, obtain the video behavior supervisors with which the regulatory departments have a relationship in the video behavior knowledge graph, and send the information that each person has a preset type of behavior to the video behavior supervisors.

[0112] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. An artificial intelligence method, characterized in that, The method includes: Typical feature selection steps: Obtain K video clips for each preset type of behavior, where K is a natural number greater than 1. Take each video clip as input and calculate using a preset object detection deep learning model to obtain each object type in each video clip for each preset type of behavior. Add each object type in each video clip to the typical feature candidate set. Count the number of each object type in the typical feature candidate set. If the number of each object type / K is greater than or equal to a preset ratio, then each object type is selected as a typical feature. The steps for building a typical feature detection model are as follows: take the image sample of each typical feature as input, take the label of each typical feature as the expected output, train and test the deep learning model to obtain the typical feature detection deep learning model. Typical feature detection steps: Acquire each frame of the video to be detected. Use each frame as input and calculate it through the typical feature detection deep learning model. The output is used as the label of the typical features of each frame. If the label of the typical features of each frame is not empty, then extract the image within the marked range from each frame according to the marked range in the label of the typical features of each frame. Use the image within the marked range as input and calculate it through the person recognition image deep learning model. The output is used as the information of the person in the label of the typical features of each frame. The steps for marking the frames to be detected are as follows: Obtain the labels of each frame image and its typical features in the video to be detected; if the labels of the typical features of each frame image are not empty, determine the maximum duration of the corresponding preset type of behavior based on the names of the typical features in the labels, and use this as the preset duration to be detected; if multiple labels of the typical features are not empty, determine multiple maximum durations of the corresponding preset type of behavior based on the names of the multiple typical features in the multiple labels, and take the largest of the multiple maximum durations as the preset duration to be detected; if the labels of the typical features of each frame image are not empty, mark each frame image as a frame to be detected, mark all frames before each frame image with a duration equal to / 2 of the preset duration to be detected as frames to be detected, and mark all frames after each frame image with a duration equal to / 2 of the preset duration to be detected as frames to be detected; some frames will be repeatedly marked as frames to be detected, while some frames will not be marked as frames to be detected. The steps for extracting video segments to be detected are as follows: extract the continuous frames marked as frames to be detected but not marked as extracted from the video to be detected as a video segment to be detected, and mark the extracted frames as extracted. The preset type of behavior and action detection steps are as follows: The video segment to be detected is used as input, and the output calculated by the preset type of behavior and action detection deep learning model is used as the label of the preset type of behavior and action in the video segment to be detected; Video segments within the marked range are used as input, and the output calculated by the person video recognition deep learning model is used as the information of the person in the preset type of behavior and action labels of the video segment; The labels of preset type of behavior and action in all the video segments to be detected are used as the labels of preset type of behavior and action in the video segment to be detected. The step of tagging the video to be detected is as follows: Replace the corresponding video segment in the video to be detected with the video segment labeled with each preset type of behavior action, so as to obtain the video to be detected after being tagged with the preset type of behavior action.

2. The artificial intelligence method according to claim 1, characterized in that, The method further includes: Video acquisition steps: Acquire the video to be tested; Steps for setting preset behavior actions: Obtain the preset behavior actions set by the user; Steps for obtaining videos of preset types of behavioral actions: Obtain video samples of each preset type of behavioral action; Steps for obtaining typical characteristics of preset behavior types: Obtain the typical characteristics of preset behavior types set by the user; The steps for obtaining image samples of typical features of a preset type of behavior are as follows: Obtain image samples of each of the aforementioned typical features.

3. The artificial intelligence method according to claim 1, characterized in that, The method further includes: Steps to establish a video behavior knowledge graph: Establish multiple pre-defined types of behavior action entities, multiple typical feature entities, multiple person entities, multiple regulatory department entities, and multiple regulatory personnel entities; The steps for defining relationships in a video behavior knowledge graph are as follows: A person entity points to a behavior action entity of a preset type through a relationship; a person entity points to a typical characteristic entity through a relationship; a person entity points to a regulatory department entity through a relationship; a regulatory personnel entity points to a regulatory department entity through a relationship. The steps for generating static relationships in a video behavior knowledge graph are as follows: Based on the typical characteristics of user-defined preset types of behavior actions, establish relationships between typical characteristic entities and preset type behavior action entities.

4. The artificial intelligence method according to claim 1, characterized in that, The method further includes: The steps for building a behavior and action detection model of preset types are as follows: take video samples of each preset type of behavior and action as input, take the label of each preset type of behavior and action as the expected output, train and test the deep learning model, and obtain a deep learning model for detecting behavior and action of preset types. Steps for building a person image recognition model: Obtain a photo of each person, use the photo of each person as input, use the information of the person as the expected output, perform transfer learning on the face image recognition deep learning model, and obtain the person image recognition deep learning model. Steps for building a person video recognition model: acquire a video of each person, use the video of each person as input, use the information of the person as the expected output, perform transfer learning on the face video recognition deep learning model, and obtain the person video recognition deep learning model.

5. An artificial intelligence system, characterized in that, The system includes: Typical Feature Selection Module: This module acquires K video clips for each preset type of behavior, where K is a natural number greater than 1. Each video clip is used as input, and a preset object detection deep learning model is used to calculate the object type for each video clip of each preset type of behavior. Each object type in each video clip is added to a typical feature candidate set. The number of each object type in the typical feature candidate set is counted. If the number of each object type / K is greater than or equal to a preset ratio, then each object type is selected as a typical feature. Typical Feature Detection Model Construction Module: Takes image samples of each typical feature as input, and labels of each typical feature as expected output, to train and test the deep learning model, thereby obtaining a deep learning model for typical feature detection. Typical Feature Detection Module: Acquires each frame of the video to be detected. Using each frame as input, the typical feature detection deep learning model calculates and outputs the results as the label of the typical features of each frame. If the label of the typical features of each frame is not empty, the model extracts an image within the marked range from each frame based on the marked range in the label of the typical features of each frame. Using the image within the marked range as input, the model calculates and outputs the results as the information of the person in the label of the typical features of each frame. The detection frame labeling module acquires the labels of each frame image and typical features of each frame image in the video to be detected. If the labels of typical features of each frame image are not empty, the preset maximum duration of the corresponding preset type of behavior action is determined according to the name of the typical feature in the labels of the typical feature, and is used as the detection preset duration. If multiple labels of the typical feature are not empty, multiple preset maximum durations of the corresponding preset type of behavior action are determined according to the names of the multiple typical features in the multiple labels of the typical feature, and the largest of the multiple preset maximum durations is taken as the detection preset duration. If the labels of typical features of each frame image are not empty, each frame image is labeled as a detection frame, all frames before each frame image with a duration of the detection preset duration / 2 are labeled as detection frames, and all frames after each frame image with a duration of the detection preset duration / 2 are labeled as detection frames. Some frames will be repeatedly labeled as detection frames, and some frames will not be labeled as detection frames. The video segment extraction module extracts consecutive frames marked as frames to be detected but not marked as extracted from the video to be detected as a video segment to be detected, and marks the extracted frames as extracted. The preset type of behavior and action detection module: takes the video segment to be detected as input, and the output calculated by the preset type of behavior and action detection deep learning model is used as the label of the preset type of behavior and action in the video segment to be detected; takes the video segment within the marked range as input, and the output calculated by the person video recognition deep learning model is used as the information of the person in the label of the preset type of behavior and action in the video segment; and uses the labels of the preset type of behavior and action in all the video segments to be detected as the labels of the preset type of behavior and action in the video to be detected. The video tagging module replaces the corresponding video segment in the video to be detected with the video segment labeled with each preset type of behavior action, thus obtaining the video to be detected tagged with the preset type of behavior action.

6. The artificial intelligence system according to claim 5, characterized in that, The system also includes: Video acquisition module: Acquires the video to be tested; Preset type behavior action setting module: retrieves user-defined preset type behavior actions; A video module for acquiring preset types of behavioral actions: acquires video samples of each preset type of behavioral action; The module for obtaining typical characteristics of preset types of behavior actions: obtains the typical characteristics of preset types of behavior actions set by the user; Image module for obtaining typical features of preset type of behavior: obtains image samples of each of the aforementioned typical features.

7. The artificial intelligence system according to claim 5, characterized in that, The system also includes: Establish a video behavior knowledge graph module: establish multiple preset types of behavior action entities, multiple typical feature entities, multiple person entities, multiple regulatory department entities, and multiple regulatory personnel entities; The video behavior knowledge graph relationship definition module defines: a person entity points to a behavior action entity of a preset type through a relationship; a person entity points to a typical feature entity through a relationship; a person entity points to a regulatory department entity through a relationship; and a regulatory personnel entity points to a regulatory department entity through a relationship. The video behavior knowledge graph static relationship generation module establishes relationships between typical feature entities and preset type behavior action entities based on the typical features of user-defined preset types of behavior actions.

8. The artificial intelligence system according to claim 5, characterized in that, The system also includes: Preset type behavior action detection model construction module: Take video samples of each preset type of behavior action as input, take the label of each preset type of behavior action as expected output, train and test the deep learning model to obtain the preset type of behavior action detection deep learning model; The person image recognition model construction module: acquires a photo of each person, takes the photo of each person as input, takes the information of the person as the expected output, performs transfer learning on the face image recognition deep learning model, and obtains the person image recognition deep learning model. Person video recognition model construction module: acquire the video of each person, take the video of each person as input, take the information of the person as the expected output, and perform transfer learning on the face video recognition deep learning model to obtain the person video recognition deep learning model.

9. A robot, comprising a memory, a processor, and an artificial intelligence robot program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1-4.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method according to any one of claims 1-4.

Images