Video frame-based behavior detection method and device, electronic equipment and storage medium

Abstract

The application discloses a behavior detection method and device based on video frames, electronic equipment and storage medium, which are used for quickly, accurately and flexibly detecting behaviors in video frames. Since the application can obtain a video frame set to be analyzed, the video frame set can carry a corresponding target service scene; a target judgment rule corresponding to the target service scene is determined according to the correspondence between the saved sample service scene and the sample judgment rule; and for any video frame in the video frame set, each target object and position information of each target object contained in the video frame are identified based on a pre-trained object recognition model; whether the behaviors in the video frame set are compliant is detected by judging whether each target object and the position information of each target object contained in each video frame conform to the target judgment rule, so that the behaviors in the video frames can be quickly, accurately and flexibly detected.

Description

Technical Field

[0001] This application relates to the field of behavior detection technology, and in particular to a behavior detection method, apparatus, electronic device and storage medium based on video frames. Background Technology

[0002] Currently, when detecting behavior in video frames, it is often necessary to train a dedicated model for each business scenario. For example, in the agricultural field, identifying whether there are pests or diseases on corn leaves requires a specially trained model. Training this model typically involves first constructing a training set for various scenarios. This training set must include not only positive samples (where there are corn leaves and pests / diseases, with the pests / diseases on the leaves) but also negative samples (where there are only corn leaves, only pests / diseases, and both corn leaves and pests / diseases, but not on the leaves). Constructing the training set is complex and time-consuming. Furthermore, due to the complexity of the scenarios, imbalance between positive and negative samples may occur, affecting the behavior detection results.

[0003] Therefore, how to quickly, accurately, and flexibly detect behavior in video frames is a technical problem that urgently needs to be solved. Summary of the Invention

[0004] This invention provides a video frame-based behavior detection method, apparatus, electronic device, and storage medium for quickly, accurately, and flexibly detecting behavior in video frames.

[0005] In a first aspect, this application provides a video frame-based action detection method, the method comprising:

[0006] Obtain a set of video frames to be analyzed, wherein the set of video frames carries the corresponding target business scenario;

[0007] Based on the correspondence between the saved sample business scenarios and sample judgment rules, the target judgment rule corresponding to the target business scenario is determined; and for any video frame in the video frame set, based on the pre-trained object recognition model, each target object contained in the video frame and the location information of each target object are identified.

[0008] By determining whether the target objects and their location information contained in each video frame conform to the target determination rules, the behavior in the video frame set is detected as compliant.

[0009] In one possible implementation, the step of detecting whether the behavior in the video frame set is compliant by determining whether the target objects and their location information contained in each video frame conform to the target determination rules includes:

[0010] For any video frame, based on the pre-saved correspondence between sample objects and sample judgment sub-rules, the target judgment sub-rule in the target judgment rule corresponding to the target object contained in the video frame is determined; based on the target object contained in the video frame, the position information of the target object, and the target judgment sub-rule, it is determined whether the video frame has a corresponding target sub-operation; wherein, the sample judgment sub-rule contains the position information of the sample object;

[0011] Based on whether each video frame has a corresponding target sub-operation, the compliance of the behavior in the set of video frames is detected.

[0012] In one possible implementation, the step of detecting whether the behavior in the video frame set is compliant based on whether each video frame has a corresponding target sub-operation includes:

[0013] If the target determination rule includes operation flow rules for each sub-operation, then the target video frame with the target sub-operation is determined.

[0014] Based on the sequential order of each target video frame, the behavior in the video frame set is checked for compliance by determining whether the target sub-operations in each target video frame conform to the operation flow rules.

[0015] In one possible implementation, the correspondence between the sample business scenario and the sample judgment rule is determined using the following steps:

[0016] Obtain a first set of sample video frames, which carries the corresponding sample business scenario.

[0017] For any first sample video frame in the first sample video frame set, based on the object recognition model, identify each sample target object and the sample location information of each sample target object contained in the first sample video frame;

[0018] For each sample target object, the corresponding sample target judgment sub-rule is determined and displayed based on the pre-saved correspondence between sample objects and sample judgment sub-rules;

[0019] If an input sample judgment rule is received, the correspondence between the sample business scenario and the sample judgment rule is saved.

[0020] In one possible implementation, the correspondence between the sample object and the sample judgment sub-rule is determined using the following steps:

[0021] Acquire a second sample video frame; based on the object recognition model, identify each sample object and its sample location information contained in the second sample video frame;

[0022] For each sample object, if an input sample judgment sub-rule is received, the correspondence between the sample object and the sample judgment sub-rule is saved.

[0023] In one possible implementation, the process of training the object recognition model includes:

[0024] Obtain any sample image containing an object from the sample set. The sample image corresponds to a sample category label and the sample location information of the object in the sample image. The sample category label is used to identify the category of the object contained in the sample image.

[0025] Using the original object recognition model, the object category labels and corresponding recognition location information contained in the sample image are determined;

[0026] The original object recognition model is trained based on the sample category label, the recognition category label, the sample location information, and the recognition location information.

[0027] In one possible implementation, the method further includes:

[0028] If the behavior in the set of video frames is non-compliant, the set prompt message will be output.

[0029] Secondly, this application provides a video frame-based behavior detection device, the device comprising:

[0030] The acquisition module is used to acquire a set of video frames to be analyzed, wherein the set of video frames carries its corresponding target business scenario.

[0031] The determination module is used to determine the target judgment rule corresponding to the target business scenario based on the correspondence between the saved sample business scenarios and sample judgment rules; and for any video frame in the video frame set, based on the pre-trained object recognition model, to identify each target object contained in the video frame and the location information of each target object.

[0032] The detection module is used to detect whether the behavior in the video frame set is compliant by judging whether each target object and its position information contained in each video frame conforms to the target judgment rules.

[0033] In one possible implementation, the detection module is specifically configured to, for any given video frame, determine the target judgment sub-rule in the target judgment rule corresponding to the target object contained in the video frame, based on a pre-saved correspondence between sample objects and sample judgment sub-rules; and determine whether the video frame contains a corresponding target sub-operation based on the target object contained in the video frame, the location information of the target object, and the target judgment sub-rule; wherein the sample judgment sub-rule contains the location information of the sample object.

[0034] Based on whether each video frame has a corresponding target sub-operation, the compliance of the behavior in the set of video frames is detected.

[0035] In one possible implementation, the detection module is specifically used to determine the target video frame containing the target sub-operation if the target determination rule includes the operation flow rules for each sub-operation;

[0036] Based on the sequential order of each target video frame, the behavior in the video frame set is checked for compliance by determining whether the target sub-operations in each target video frame conform to the operation flow rules.

[0037] In one possible implementation, the determining module is specifically used to obtain a first sample video frame set, the first sample video set carrying its corresponding sample service scenario;

[0038] For any first sample video frame in the first sample video frame set, based on the object recognition model, identify each sample target object and the sample location information of each sample target object contained in the first sample video frame;

[0039] For each sample target object, the corresponding sample target judgment sub-rule is determined and displayed based on the pre-saved correspondence between sample objects and sample judgment sub-rules;

[0040] If an input sample judgment rule is received, the correspondence between the sample business scenario and the sample judgment rule is saved.

[0041] In one possible implementation, the determining module is specifically used to acquire a second sample video frame; and based on the object recognition model, to identify each sample object and the sample location information of each sample object contained in the second sample video frame.

[0042] For each sample object, if an input sample judgment sub-rule is received, the correspondence between the sample object and the sample judgment sub-rule is saved.

[0043] In one possible implementation, the detection module is further configured to output a predefined prompt message if the behavior in the video frame set is non-compliant.

[0044] Thirdly, this application provides an electronic device, which includes at least a processor and a memory, wherein the processor is configured to execute a computer program stored in the memory to implement the steps of any of the above-described video frame-based behavior detection methods.

[0045] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-described video frame-based behavior detection methods.

[0046] This application can obtain a set of video frames to be analyzed, which can carry the corresponding target business scenario; based on the correspondence between the saved sample business scenarios and sample judgment rules, the target judgment rule corresponding to the target business scenario is determined; and for any video frame in the video frame set, based on a pre-trained object recognition model, each target object contained in the video frame and the location information of each target object are identified; by judging whether each target object contained in each video frame and the location information of each target object conform to the target judgment rule, the compliance of the behavior in the video frame set is detected, thereby enabling quick, accurate and flexible detection of the behavior in the video frames. Attached Figure Description

[0047] To more clearly illustrate the implementation methods in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0048] Figure 1 The diagram illustrates a video frame-based behavior detection process according to some embodiments.

[0049] Figure 2 The diagram illustrates a target determination rule construction method provided by some embodiments.

[0050] Figure 3 The diagram shows a video frame-based behavior detection device according to some embodiments;

[0051] Figure 4 A schematic diagram of an electronic device structure provided by some embodiments is shown. Detailed Implementation

[0052] To quickly, accurately, and flexibly detect behavior in video frames, embodiments of this application provide a behavior detection method, apparatus, device, and medium based on video frames.

[0053] To make the objectives and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0054] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0055] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0056] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0057] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0058] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0059] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.

[0060] Example 1:

[0061] Figure 1 The diagram illustrates a video frame-based behavior detection process according to some embodiments, which includes the following steps:

[0062] S101: Obtain the set of video frames to be analyzed, wherein the set of video frames carries the corresponding target business scenario.

[0063] The video frame-based behavior detection method provided in this application is applied to an electronic device, such as a server, PC, or mobile terminal. Optionally, the electronic device may be configured with a service platform for detecting behavior in video frames (a set of video frames), and the steps of the video frame-based behavior detection method provided in this application can be performed based on this service platform.

[0064] In one possible implementation, the electronic device can acquire a set of video frames to be analyzed based on a service platform. This application does not specifically limit the number of video frames in the video frame set; it can be flexibly set according to requirements. For example, the video frame set may contain at least one video frame (image). Optionally, the video to be analyzed can be acquired based on image acquisition devices such as cameras pre-installed at the target location, according to a set time period. The electronic device can acquire the video based on a service platform, and after acquiring the video, the electronic device can parse (frame-divide) the video into a set of video frames. For example, for the video, at set time intervals, one video frame can be extracted from the video, and the video can be processed into frames to obtain individual video frames. The set of these video frames constitutes the video frame set. The time interval can be flexibly set according to requirements; this application does not specifically limit it.

[0065] In one possible implementation, the video frame set may carry the target business scenario corresponding to the video frame set. For example, the target business scenario may be a corn growth scenario, a worker climbing an escalator scenario, a gas station scenario, etc.

[0066] S102: Based on the correspondence between the saved sample business scenarios and sample judgment rules, determine the target judgment rule corresponding to the target business scenario; and for any video frame in the video frame set, identify each target object contained in the video frame and the location information of each target object based on the pre-trained object recognition model.

[0067] In one possible implementation, to detect behavior in video frames of a target business scenario, a correspondence between sample business scenarios and sample judgment rules can be pre-stored. For ease of description, the business scenarios in the correspondence are referred to as sample business scenarios, and the judgment rules in the correspondence are referred to as sample judgment rules. The process of determining the correspondence between sample business scenarios and sample judgment rules will be described in detail in subsequent embodiments and will not be repeated here. Optionally, the (applicable) target judgment rule corresponding to the target business scenario can be determined based on the stored correspondence between sample business scenarios and sample judgment rules (for ease of description, the sample judgment rule corresponding to the target business scenario is referred to as the target judgment rule).

[0068] In one possible implementation, to detect behavior in video frames, for any video frame in the video frame set, a pre-trained object recognition model can be used to identify each target object contained in the video frame and its location information. The target objects can be flexibly set according to requirements; this application does not impose specific limitations on them. For example, target objects may include any category of objects such as safety helmets, escalators, workers, corn leaves, and wheat leaves. The training process of the object recognition model will be described in detail in subsequent embodiments and will not be repeated here.

[0069] S103: By determining whether the target objects and their location information contained in each video frame conform to the target judgment rules, the behavior in the video frame set is detected as compliant.

[0070] In one possible implementation, the compliance of behavior within a video frame set can be detected by determining whether the target objects and their location information within each video frame conform to a target judgment rule. For example, taking the target judgment rule that the location of a pest coincides with the location of a corn leaf as an example, for any video frame, a pre-trained object recognition model can be used to identify whether corn leaves and pests exist among the target objects in that video frame. If both corn leaves and pests are present, the location information of the corn leaves and pests output by the object recognition model can be used to determine whether the location of the pest coincides with the location of the corn leaf. If the location of the pest coincides with the location of the corn leaf, for example, if the pest is located on the upper surface of the corn leaf, it can be considered that the target objects and their location information within that video frame conform to the target judgment rule, and the behavior of detecting pests on corn leaves can be considered non-compliant. Optionally, when non-compliant behavior is detected in the video frame set, a pre-defined prompt message can be output. This application does not specify the exact content of the prompt message; it can be flexibly set according to needs.

[0071] Understandably, if, based on a pre-trained object recognition model, the target objects in the video frame are identified as either only having corn leaves without pests, only having pests without corn leaves, or having both corn leaves and pests, but the pests are not on the surface of the corn leaves (i.e., the location of the pests does not coincide with the location of the corn leaves), then it can be considered that the target objects and their location information in the video frame do not conform to the target judgment rules. It can be considered that the behavior of not detecting pests on corn leaves has not been detected, meaning that the behavior in the video frame set is compliant.

[0072] This application can obtain a set of video frames to be analyzed, which can carry the corresponding target business scenario; based on the correspondence between the saved sample business scenarios and sample judgment rules, the target judgment rule corresponding to the target business scenario is determined; and for any video frame in the video frame set, based on a pre-trained object recognition model, each target object contained in the video frame and the location information of each target object are identified; by judging whether each target object contained in each video frame and the location information of each target object conform to the target judgment rule, the compliance of the behavior in the video frame set is detected, thereby enabling quick, accurate and flexible detection of the behavior in the video frames.

[0073] Furthermore, compared to related technologies that require a dedicated model to identify pests and diseases on corn leaves, this application eliminates the need for complex training sets. These sets typically involve constructing a training set for various scenarios, including positive samples (where the pests and diseases are present on the corn leaves) and negative samples (where the pests and diseases are present but not on the leaves). This training set construction is complex and time-consuming, and the complexity can lead to imbalances in the positive and negative sample classifications, affecting the behavior detection results. In contrast, this application eliminates the need for a complex training set. The object recognition model only needs to identify the corn leaves and pests, as well as their location information. Based on simple and flexible judgment rules, this application achieves fast, accurate, and flexible behavior detection in video frames.

[0074] Furthermore, the judgment rules in this application can be simple textual rules that do not require complex code. Therefore, the judgment rules in this application can be configured by ordinary business personnel (business experts) who do not understand code. In contrast, when related technologies detect behavior, even if they are based on low-code platforms, they usually require programmers who understand code. The business platform in this application can be an AI zero-code business platform that can be operated by ordinary business personnel who do not understand code. Based on this, the purpose of quickly, accurately and flexibly detecting behavior in video frames can be achieved.

[0075] Example 2:

[0076] To quickly, accurately, and flexibly detect behavior in video frames, based on the above embodiments, in this embodiment, the step of detecting whether the behavior in the video frame set is compliant by determining whether the target objects and their position information contained in each video frame conform to the target judgment rules includes:

[0077] For any video frame, based on the pre-saved correspondence between sample objects and sample judgment sub-rules, the target judgment sub-rule in the target judgment rule corresponding to the target object contained in the video frame is determined; based on the target object contained in the video frame, the position information of the target object, and the target judgment sub-rule, it is determined whether the video frame has a corresponding target sub-operation; wherein, the sample judgment sub-rule contains the position information of the sample object;

[0078] Based on whether each video frame has a corresponding target sub-operation, the compliance of the behavior in the set of video frames is detected.

[0079] In one possible implementation, when detecting behavior in a video frame, for any given video frame, the target judgment sub-rule corresponding to the target object contained in that video frame can be determined based on a pre-saved correspondence between sample objects and sample judgment sub-rules. The process of determining the correspondence between sample objects and sample judgment sub-rules will be described in detail in subsequent embodiments and will not be repeated here.

[0080] This application does not specifically limit the sample judgment sub-rules corresponding to the sample objects, and can flexibly set them according to needs. Optionally, the sample judgment sub-rules may include the position information of the sample objects. For example, when a video frame contains the target object of an electrostatic discharge device, the target judgment sub-rule corresponding to the target object of the electrostatic discharge device can be determined according to the pre-saved correspondence between sample objects and sample judgment sub-rules. For example, for the target sub-operation of electrostatic discharge, the corresponding target judgment sub-rule could be that the position of the worker's hand coincides with the position of the electrostatic discharge device. When the target objects contained in the video frame include both a worker and an electrostatic discharge device, and the position of the worker's hand coincides with the position of the electrostatic discharge device, it can be considered that the target sub-operation of electrostatic discharge exists in the video frame; otherwise, it can be considered that the target sub-operation of electrostatic discharge does not exist in the video frame.

[0081] Optionally, the compliance of behavior in a video frame set can be detected based on whether a corresponding target sub-operation exists in each video frame. For example, the target judgment rule can be a global rule. Taking wearing a safety helmet as an example, the target judgment sub-rule for each video frame in the video frame set could include wearing a safety helmet (or the position of the safety helmet coinciding with the worker's head position). Based on this target judgment sub-rule, for any video frame, it can be determined whether a corresponding target sub-operation for wearing a safety helmet exists in that video frame. If the target sub-operation for wearing a safety helmet exists in every video frame, the operation (sub-operation) can be considered compliant, the behavior in the video frame set is compliant, and the detection result can be compliant. Conversely, if the target sub-operation for wearing a safety helmet does not exist in any video frame, the operation (sub-operation) can be considered non-compliant, the behavior in the video frame set is non-compliant, and the detection result can be non-compliant. Optionally, when non-compliant behavior is detected in the video frame set, a set prompt message can be output to remind the worker to wear a safety helmet correctly. This application does not specifically limit the specific content and form of the prompt message and can be flexibly set according to needs.

[0082] In one possible implementation, the target determination rule can also be a procedural rule that requires a certain order between sub-operations; for ease of description, this is called an operation flow rule. Optionally, if the target determination rule includes operation flow rules for each sub-operation, when detecting whether the behavior in the video frame set is compliant based on whether each video frame has a corresponding target sub-operation, the following steps can be taken: first, determine the target video frames with target sub-operations; determine the order of each video frame based on timestamp information, etc.; then, based on the order of each target video frame, determine whether each target sub-operation in each target video frame conforms to the operation flow rule. If it does, the behavior of the video frame set can be considered compliant and standardized; otherwise, if each target sub-operation in each target video frame does not conform to the operation flow rule, the behavior of the video frame set can be considered non-compliant and non-standard.

[0083] To facilitate understanding, the video frame-based behavior detection method provided in this application will be explained below through a specific embodiment. For example, suppose the target judgment rule includes the following operation flow rule for each sub-operation: perform the electrostatic discharge sub-operation first, then perform the oil unloading pipe connection sub-operation. If the operator does not perform electrostatic discharge and directly performs the oil unloading pipe connection sub-operation, then only target video frames containing the oil unloading pipe connection target sub-operation may be obtained from the video frame set, and no target video frames containing the electrostatic discharge target sub-operation may be obtained. It can be considered that the target sub-operations in each target video frame in the video frame set do not conform to the operation flow rule, and the behavior in that video frame set is considered non-compliant. A pre-defined prompt message can be generated to remind the operator to perform the electrostatic discharge sub-operation first, and then perform the oil unloading pipe connection sub-operation, thereby improving operational safety.

[0084] As can be seen from the above embodiments, the operation process rules in this application, based on simple textual forms, can achieve the purpose of quickly, accurately, and flexibly detecting behaviors in video frames. Compared with the prior art, which requires complex code for behavior detection, this can greatly save development costs and shorten development time.

[0085] Example 3:

[0086] If we consider the entire process of determining whether behaviors within a video frame set are compliant based on the aforementioned target-based judgment rules as a capability, see [reference needed]. Figure 2 , Figure 2 The diagram illustrates a target determination rule construction method provided by some embodiments, such as... Figure 2 As shown, the initial development of this capability generally involved three parts: object recognition, intelligent capability building, and task scenario orchestration. These three parts will be described in detail below.

[0087] Part 1: Object Recognition.

[0088] Optionally, the object recognition step can be performed based on a trained object recognition model. For example, the process of training the object recognition model includes:

[0089] Obtain any sample image containing an object from the sample set. The sample image corresponds to a sample category label and the sample location information of the object in the sample image. The sample category label is used to identify the category of the object contained in the sample image.

[0090] Using the original object recognition model, the object category labels and corresponding recognition location information contained in the sample image are determined;

[0091] The original object recognition model is trained based on the sample category label, the recognition category label, the sample location information, and the recognition location information.

[0092] In one possible implementation, a business expert can first determine the object (target) to be identified, referred to as the newly created object in the diagram for ease of description. The business expert can then upload a sample set consisting of sample images containing the object to an electronic device. When training the object recognition model, the sample set contains multiple sample images, each containing an object. Each sample image corresponds to a sample category label. Optionally, the business expert or others can label the sample images with these category labels, which identify the category of the object contained in the sample image. For example, when the object includes a safety helmet, the sample image corresponds to a sample category label containing a safety helmet; when the object includes a person, the sample image corresponds to a sample category label containing a person; when the object includes an escalator, the sample image corresponds to a sample category label containing an escalator, and so on.

[0093] In order to obtain the object's position information, the sample image also includes the object's sample position information within that sample image. This sample position information can include the coordinates of the top-left pixel of the object's bounding box within the sample image, as well as information such as the bounding box's length and width.

[0094] When training the original object recognition model, any sample image containing an object can be obtained from the sample set. This sample image corresponds to a sample category label and the object's location information within the sample image. This obtained sample image is then input into the original object recognition model, which retrieves the object's category label and corresponding location information for that sample image.

[0095] In practice, after determining the recognition category label and corresponding recognition location information of the input sample image, since the sample category label and the sample location information of the object corresponding to that sample category label in the sample image are pre-saved, the accuracy of the object recognition model's recognition result can be determined by whether the sample category label and the recognition category label are consistent, and whether the sample location information and the recognition location information are consistent. In practice, if they are inconsistent, it indicates that the object recognition model's recognition result is inaccurate, and the parameters of the object recognition model need to be adjusted to train the object recognition model.

[0096] In practice, when adjusting the parameters in the object recognition model to be trained (the original object recognition model), the gradient descent algorithm can be used to backpropagate the gradient of the parameters of the object recognition model to be trained, thereby training the object recognition model.

[0097] In one possible implementation, the above operation can be performed on each sample image in the sample set, and when the preset convergence condition is met, the object recognition model is determined to be trained successfully.

[0098] The preset convergence conditions can be such that the number of sample images correctly recognized by the original object recognition model in the sample set is greater than a set number, or the number of iterations for training the object recognition model reaches the set maximum number of iterations. These conditions can be flexibly set in practice and are not specifically limited here.

[0099] In one possible implementation, when training the original object recognition model, the sample images in the sample set can be divided into training sample images and test sample images. The original object recognition model is first trained based on the training sample images, and then the reliability of the trained object recognition model is verified based on the test sample images. Optionally, after the object recognition model is trained, the version of the object recognition model can be updated, and the updated object recognition model can be saved.

[0100] Part Two: Building Intelligent Capabilities

[0101] In one possible implementation, the process of establishing the correspondence between sample objects and sample judgment sub-rules within a single sample video frame can be termed intelligent capability construction. Optionally, the correspondence between sample objects and sample judgment sub-rules is determined during intelligent capability construction using the following steps:

[0102] Acquire a second sample video frame; based on the object recognition model, identify each sample object and its sample location information contained in the second sample video frame;

[0103] For each sample object, if an input sample judgment sub-rule is received, the correspondence between the sample object and the sample judgment sub-rule is saved.

[0104] In one possible implementation, during the development of intelligent capabilities, business experts can upload video frames (images) to the AI ​​no-code business platform, referred to as second sample video frames for ease of description. Electronic devices can obtain these second sample video frames based on the business platform and, using the object recognition module trained in the first part, identify the sample objects contained in the second sample video frames and their sample location information.

[0105] After the electronic device obtains the sample objects and their location information from the second sample video frame based on the business platform, it displays these sample objects on the screen. Business experts can view the displayed sample objects and their location information. For sample objects and their location information not recognized by the object recognition model, business experts can supplement them. For example, business experts can add (supplement) sample objects and their location information by clicking buttons such as "Supplement Objects". In one possible implementation, business experts can quickly and flexibly set corresponding sample judgment sub-rules for each sample object, i.e., design capability rules. For example, for the escalator sample object, the business expert could set a sample judgment sub-rule such as "When worker 1 is above the escalator, worker 2 must be below the escalator, and worker 2's position intersects with the escalator." When the electronic device receives the sample judgment sub-rules input by the business expert for a sample object, it can save the correspondence between the sample object and the sample judgment sub-rule.

[0106] This section also demonstrates that business experts in this application, even without coding knowledge, can quickly, accurately, and flexibly generate sample judgment sub-rules based on convenient and flexible text. Furthermore, the judgment rules and sub-rules in this application can be quickly and easily reused in other business scenarios. For example, taking wearing a safety helmet as an example, the target judgment sub-rule for wearing a safety helmet can be applied not only to the business scenario of workers climbing escalators but also to the gas station business scenario. Different business scenarios can directly reuse this target judgment sub-rule without reconfiguration, improving the reusability of AI capabilities and facilitating the accumulation of industry knowledge among business experts.

[0107] Part Three: Job Scenario Arrangement

[0108] In one possible implementation, the job scenario orchestration can integrate and encapsulate the correspondence between sample objects and sample judgment sub-rules established for a single sample video frame in the second part, forming a complete set of rule capabilities capable of detecting behavior in a set of video frames (videos). Optionally, the job scenario orchestration process may include a process of establishing the correspondence between sample business scenarios and sample judgment rules. The correspondence between sample business scenarios and sample judgment rules is determined using the following steps:

[0109] Obtain a first set of sample video frames, which carries the corresponding sample business scenario.

[0110] For any first sample video frame in the first sample video frame set, based on the object recognition model, identify each sample target object and the sample location information of each sample target object contained in the first sample video frame;

[0111] For each sample target object, the corresponding sample target judgment sub-rule is determined and displayed based on the pre-saved correspondence between sample objects and sample judgment sub-rules;

[0112] If an input sample judgment rule is received, the correspondence between the sample business scenario and the sample judgment rule is saved.

[0113] In one possible implementation, when orchestrating work scenarios, business experts can upload a specific work scenario video or a set of work scenario images to the AI ​​no-code business platform. Optionally, if a work scenario video is uploaded, the electronic device can perform frame segmentation processing on the video to obtain the video frames contained in the work scenario video, forming a first sample video frame set. If a set of work scenario images is uploaded, the electronic device can use these work scenario images as the first sample video frame set. Understandably, the first sample video set can carry its corresponding sample business scenario, such as a corn growing scene, a worker climbing an escalator scene, a gas station scene, etc.

[0114] For any given first sample video frame in the first sample video frame set, the electronic device can first identify the sample target objects and their sample location information within that first sample video frame based on an object recognition model. For each sample target object within that first sample video frame, the corresponding sample target judgment sub-rule can be determined and displayed based on the correspondence between sample objects and sample judgment sub-rules saved during the intelligent capability construction process in Part Two. That is, scene capability analysis is performed on the first sample video frame (image) to extract capability rules.

[0115] Optionally, business experts can supplement and add capability rules (sample target judgment sub-rules) that are not identified by the business platform, that is, business experts can supplement scenario capabilities. Business experts can orchestrate (set) capability rules. When orchestrating, they can orchestrate (set) the full-process rules in the above embodiments, or they can orchestrate (set) the process-oriented rules in the above embodiments, and can be flexibly set according to needs.

[0116] In one possible implementation, after the business expert has compiled the rule capabilities (sample judgment rules), they can click the "OK" or "Complete" button, etc. The business platform can receive the sample judgment rules and save the correspondence between the sample business scenarios and the sample judgment rules.

[0117] In one possible implementation, after the third part, namely the arrangement of work scenarios, is completed, business experts can upload a simulated work video to verify the arranged sample judgment rules, etc., that is, to conduct scenario capability drills.

[0118] The embodiments of this application enable visual orchestration without requiring any coding, allowing business experts without coding knowledge to participate in capability building and accelerating the entire capability development process. Furthermore, the capabilities described in this application can be reused in different business scenarios, which is beneficial for accumulating industry knowledge and further speeding up the capability building process.

[0119] Example 4:

[0120] Based on the same technical concept, this application provides a video frame-based behavior detection device. Figure 3 The diagram illustrates a behavior detection device based on video frames according to some embodiments, such as... Figure 3 As shown, the device includes:

[0121] The acquisition module 31 is used to acquire a set of video frames to be analyzed, wherein the set of video frames carries its corresponding target business scenario;

[0122] The determination module 32 is used to determine the target judgment rule corresponding to the target business scenario based on the correspondence between the saved sample business scenarios and sample judgment rules; and for any video frame in the video frame set, based on the pre-trained object recognition model, to identify each target object contained in the video frame and the location information of each target object.

[0123] The detection module 33 is used to detect whether the behavior in the video frame set is compliant by judging whether the target objects and their position information contained in each video frame conform to the target judgment rules.

[0124] In one possible implementation, the detection module 33 is specifically used to, for any video frame, determine the target judgment sub-rule in the target judgment rule corresponding to the target object contained in the video frame according to the pre-saved correspondence between sample objects and sample judgment sub-rules; and determine whether the video frame has a corresponding target sub-operation based on the target object contained in the video frame, the position information of the target object, and the target judgment sub-rule; wherein, the sample judgment sub-rule includes the position information of the sample object;

[0125] Based on whether each video frame has a corresponding target sub-operation, the compliance of the behavior in the set of video frames is detected.

[0126] In one possible implementation, the detection module 33 is specifically used to determine the target video frame containing the target sub-operation if the target determination rule includes the operation flow rules for each sub-operation;

[0127] Based on the sequential order of each target video frame, the behavior in the video frame set is checked for compliance by determining whether the target sub-operations in each target video frame conform to the operation flow rules.

[0128] In one possible implementation, the determining module 32 is specifically used to obtain a first sample video frame set, the first sample video set carrying its corresponding sample service scenario;

[0129] For any first sample video frame in the first sample video frame set, based on the object recognition model, identify each sample target object and the sample location information of each sample target object contained in the first sample video frame;

[0130] For each sample target object, the corresponding sample target judgment sub-rule is determined and displayed based on the pre-saved correspondence between sample objects and sample judgment sub-rules;

[0131] If an input sample judgment rule is received, the correspondence between the sample business scenario and the sample judgment rule is saved.

[0132] In one possible implementation, the determining module 32 is specifically used to acquire a second sample video frame; and based on the object recognition model, to identify each sample object and the sample location information of each sample object contained in the second sample video frame.

[0133] For each sample object, if an input sample judgment sub-rule is received, the correspondence between the sample object and the sample judgment sub-rule is saved.

[0134] In one possible implementation, the detection module 33 is further configured to output a predefined prompt message if the behavior in the video frame set is non-compliant.

[0135] Example 5:

[0136] Based on the same technical concept, this application provides an electronic device. Figure 4 This application provides a schematic diagram of an electronic device structure, such as... Figure 4 As shown, it includes: processor 41, communication interface 42, memory 43 and communication bus 44, wherein processor 41, communication interface 42 and memory 43 communicate with each other through communication bus 44.

[0137] The memory 43 stores a computer program, which, when executed by the processor 41, causes the processor 41 to perform the following steps:

[0138] Obtain a set of video frames to be analyzed, wherein the set of video frames carries the corresponding target business scenario;

[0139] Based on the correspondence between the saved sample business scenarios and sample judgment rules, the target judgment rule corresponding to the target business scenario is determined; and for any video frame in the video frame set, based on the pre-trained object recognition model, each target object contained in the video frame and the location information of each target object are identified.

[0140] By determining whether the target objects and their location information contained in each video frame conform to the target determination rules, the behavior in the video frame set is detected as compliant.

[0141] In one possible implementation, the processor 41 is specifically configured to, for any given video frame, determine the target judgment sub-rule in the target judgment rule corresponding to the target object contained in the video frame, based on a pre-saved correspondence between sample objects and sample judgment sub-rules; and determine whether the video frame has a corresponding target sub-operation based on the target object contained in the video frame, the position information of the target object, and the target judgment sub-rule; wherein the sample judgment sub-rule includes the position information of the sample object.

[0142] Based on whether each video frame has a corresponding target sub-operation, the compliance of the behavior in the set of video frames is detected.

[0143] In one possible implementation, processor 41 is specifically configured to determine a target video frame with a target sub-operation if the target determination rule includes operation flow rules for each sub-operation.

[0144] Based on the sequential order of each target video frame, the behavior in the video frame set is checked for compliance by determining whether the target sub-operations in each target video frame conform to the operation flow rules.

[0145] In one possible implementation, the processor 41 is specifically used to acquire a first sample video frame set, the first sample video set carrying its corresponding sample service scenario.

[0146] For any first sample video frame in the first sample video frame set, based on the object recognition model, identify each sample target object and the sample location information of each sample target object contained in the first sample video frame;

[0147] For each sample target object, the corresponding sample target judgment sub-rule is determined and displayed based on the pre-saved correspondence between sample objects and sample judgment sub-rules;

[0148] If an input sample judgment rule is received, the correspondence between the sample business scenario and the sample judgment rule is saved.

[0149] In one possible implementation, the processor 41 is specifically configured to acquire a second sample video frame; and based on the object recognition model, identify each sample object contained in the second sample video frame and the sample location information of each sample object;

[0150] For each sample object, if an input sample judgment sub-rule is received, the correspondence between the sample object and the sample judgment sub-rule is saved.

[0151] In one possible implementation, the processor 41 is specifically configured to acquire any sample image containing an object in the sample set, the sample image corresponding to a sample category label and sample location information of the object in the sample image; wherein the sample category label is used to identify the category of the object contained in the sample image;

[0152] Using the original object recognition model, the object category labels and corresponding recognition location information contained in the sample image are determined;

[0153] The original object recognition model is trained based on the sample category label, the recognition category label, the sample location information, and the recognition location information.

[0154] In one possible implementation, the processor 41 is further configured to output a predefined prompt message if the behavior in the set of video frames is non-compliant.

[0155] Since the principle behind the problem-solving by the aforementioned electronic device is similar to that of the video frame-based behavior detection method, the implementation of the aforementioned electronic device can be found in the implementation of the method, and the repetitive parts will not be repeated.

[0156] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0157] Communication interface 42 is used for communication between the above-mentioned electronic device and other devices.

[0158] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0159] The processors mentioned above can be general-purpose processors, including central processing units, network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0160] Example 6:

[0161] Based on the above embodiments, this invention provides a computer-readable storage medium storing a computer program executable by an electronic device. When the program is run on the electronic device, the electronic device performs the following steps:

[0162] Obtain a set of video frames to be analyzed, wherein the set of video frames carries the corresponding target business scenario;

[0163] Based on the correspondence between the saved sample business scenarios and sample judgment rules, the target judgment rule corresponding to the target business scenario is determined; and for any video frame in the video frame set, based on the pre-trained object recognition model, each target object contained in the video frame and the location information of each target object are identified.

[0164] By determining whether the target objects and their location information contained in each video frame conform to the target determination rules, the behavior in the video frame set is detected as compliant.

[0165] In one possible implementation, the step of detecting whether the behavior in the video frame set is compliant by determining whether the target objects and their location information contained in each video frame conform to the target determination rules includes:

[0166] For any video frame, based on the pre-saved correspondence between sample objects and sample judgment sub-rules, the target judgment sub-rule in the target judgment rule corresponding to the target object contained in the video frame is determined; based on the target object contained in the video frame, the position information of the target object, and the target judgment sub-rule, it is determined whether the video frame has a corresponding target sub-operation; wherein, the sample judgment sub-rule contains the position information of the sample object;

[0167] Based on whether each video frame has a corresponding target sub-operation, the compliance of the behavior in the set of video frames is detected.

[0168] In one possible implementation, the step of detecting whether the behavior in the video frame set is compliant based on whether each video frame has a corresponding target sub-operation includes:

[0169] If the target determination rule includes operation flow rules for each sub-operation, then the target video frame with the target sub-operation is determined.

[0170] Based on the sequential order of each target video frame, the behavior in the video frame set is checked for compliance by determining whether the target sub-operations in each target video frame conform to the operation flow rules.

[0171] In one possible implementation, the correspondence between the sample business scenario and the sample judgment rule is determined using the following steps:

[0172] Obtain a first set of sample video frames, which carries the corresponding sample business scenario.

[0173] For any first sample video frame in the first sample video frame set, based on the object recognition model, identify each sample target object and the sample location information of each sample target object contained in the first sample video frame;

[0174] For each sample target object, the corresponding sample target judgment sub-rule is determined and displayed based on the pre-saved correspondence between sample objects and sample judgment sub-rules;

[0175] If an input sample judgment rule is received, the correspondence between the sample business scenario and the sample judgment rule is saved.

[0176] In one possible implementation, the correspondence between the sample object and the sample judgment sub-rule is determined using the following steps:

[0177] Acquire a second sample video frame; based on the object recognition model, identify each sample object and its sample location information contained in the second sample video frame;

[0178] For each sample object, if an input sample judgment sub-rule is received, the correspondence between the sample object and the sample judgment sub-rule is saved.

[0179] In one possible implementation, the process of training the object recognition model includes:

[0180] Obtain any sample image containing an object from the sample set. The sample image corresponds to a sample category label and the sample location information of the object in the sample image. The sample category label is used to identify the category of the object contained in the sample image.

[0181] Using the original object recognition model, the object category labels and corresponding recognition location information contained in the sample image are determined;

[0182] The original object recognition model is trained based on the sample category label, the recognition category label, the sample location information, and the recognition location information.

[0183] In one possible implementation, the method further includes:

[0184] If the behavior in the set of video frames is non-compliant, the set prompt message will be output.

[0185] The aforementioned computer-readable storage medium can be any available medium or data storage device that can be accessed by the processor in an electronic device, including but not limited to magnetic storage such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), optical storage such as CDs, DVDs, BDs, HVDs, etc., and semiconductor storage such as ROMs, EPROMs, EEPROMs, non-volatile memory (NAND flash), solid-state drives (SSDs), etc.

[0186] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0187] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0188] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0189] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0190] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A behavior detection method based on video frames, characterized in that, The method is applied to an electronic device, and the method includes: Obtain a set of video frames to be analyzed, wherein the set of video frames carries the corresponding target business scenario; Based on the correspondence between the saved sample business scenarios and sample judgment rules, the target judgment rule corresponding to the target business scenario is determined; and for any video frame in the video frame set, based on the pre-trained object recognition model, each target object contained in the video frame and the location information of each target object are identified. By determining whether the target objects and their location information contained in each video frame conform to the target judgment rules, the behavior in the video frame set is detected as compliant; wherein, the target judgment rules include: a sub-rule that the location of any target object coincides with that of any other target object.

2. The method according to claim 1, characterized in that, The step of detecting whether the behavior in the video frame set is compliant by determining whether the target objects and their location information contained in each video frame conform to the target determination rules includes: For any video frame, based on the pre-saved correspondence between sample objects and sample judgment sub-rules, the target judgment sub-rule in the target judgment rule corresponding to the target object contained in the video frame is determined; based on the target object contained in the video frame, the position information of the target object, and the target judgment sub-rule, it is determined whether the video frame has a corresponding target sub-operation; wherein, the sample judgment sub-rule contains the position information of the sample object; Based on whether each video frame has a corresponding target sub-operation, the compliance of the behavior in the set of video frames is detected.

3. The method according to claim 1, characterized in that, The step of detecting whether the behavior in the video frame set is compliant based on whether each video frame has a corresponding target sub-operation includes: If the target determination rule includes operation flow rules for each sub-operation, then the target video frame with the target sub-operation is determined. Based on the sequential order of each target video frame, the behavior in the video frame set is checked for compliance by determining whether the target sub-operations in each target video frame conform to the operation flow rules.

4. The method according to claim 1, characterized in that, The correspondence between the sample business scenarios and the sample judgment rules is determined using the following steps: Obtain a first set of sample video frames, which carries the corresponding sample business scenario. For any first sample video frame in the first sample video frame set, based on the object recognition model, identify each sample target object and the sample location information of each sample target object contained in the first sample video frame; For each sample target object, the corresponding sample target judgment sub-rule is determined and displayed based on the pre-saved correspondence between sample objects and sample judgment sub-rules; If an input sample judgment rule is received, the correspondence between the sample business scenario and the sample judgment rule is saved.

5. The method according to claim 4, characterized in that, The correspondence between the sample object and the sample judgment sub-rule is determined using the following steps: Acquire a second sample video frame; based on the object recognition model, identify each sample object and its sample location information contained in the second sample video frame; For each sample object, if an input sample judgment sub-rule is received, the correspondence between the sample object and the sample judgment sub-rule is saved.

6. The method according to any one of claims 1-5, characterized in that, The process of training the object recognition model includes: Obtain any sample image containing an object from the sample set. The sample image corresponds to a sample category label and the sample location information of the object in the sample image. The sample category label is used to identify the category of the object contained in the sample image. Using the original object recognition model, the object category labels and corresponding recognition location information contained in the sample image are determined; The original object recognition model is trained based on the sample category label, the recognition category label, the sample location information, and the recognition location information.

7. The method according to claim 1, characterized in that, The method further includes: If the behavior in the set of video frames is non-compliant, the set prompt message will be output.

8. A behavior detection device based on video frames, characterized in that, The device is used in an electronic device, and the device includes: The acquisition module is used to acquire a set of video frames to be analyzed, wherein the set of video frames carries its corresponding target business scenario. The determination module is used to determine the target judgment rule corresponding to the target business scenario based on the correspondence between the saved sample business scenarios and sample judgment rules; and for any video frame in the video frame set, based on the pre-trained object recognition model, to identify each target object contained in the video frame and the location information of each target object. The detection module is used to detect whether the behavior in the video frame set is compliant by judging whether the target objects and their position information contained in each video frame conform to the target judgment rules; wherein, the target judgment rules include: a sub-rule that the position of any target object coincides with that of any other target object.

9. An electronic device, characterized in that, The electronic device includes at least a processor and a memory, wherein the processor is used to execute a computer program stored in the memory to implement the steps of the video frame-based behavior detection method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the video frame-based behavior detection method according to any one of claims 1-7.

Images