Behavior detection method, storage medium and computer program product

Abstract

The invention discloses a behavior detection method, a storage medium and a computer program product, and relates to the technical field of computer vision, and the method comprises the steps: edge end equipment obtains continuous frame image data collected by image collection equipment, and inputs the continuous frame image data into an image detection model; performing behavior detection based on the continuous frame image data through the image detection model to obtain an initial detection result; and sending the initial detection result to the cloud equipment. And the cloud device performs behavior detection based on the initial detection result through the multi-modal large model to obtain an intermediate detection result, compares the intermediate detection result with the initial detection result to obtain a comparison result, and outputs the initial detection result and the intermediate detection result based on the comparison result. The processing efficiency and the recognition precision of behavior detection based on the image are improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a behavior detection method, a storage medium, and a computer program product. Background Art

[0002] As an important branch of artificial intelligence, computer vision technology has been widely used in the traditional security field. Computer vision mainly uses image recognition technology and video streams output by cameras to collect data according to different business scenarios, and trains models corresponding to business scenarios based on specific algorithms to meet diverse business needs. Traditional image detection models are prone to data drift when docking with new detection scenarios, that is, the performance of the model in the new detection scenario will drop significantly, affecting the recognition accuracy. In order to cope with these problems, the multimodal large model (MLLM) based on the large language model has developed rapidly in recent years. On the basis of the original powerful generalization and reasoning capabilities, the multimodal large model further introduces multimodal information processing capabilities, including image description, visual question answering, visual positioning, etc. Compared with traditional image detection models, the multimodal large model has stronger data generalization capabilities and more general image understanding capabilities. However, the resources required to deploy the multimodal large model are huge, and it takes a long time to perform behavior detection based on images, and the detection efficiency is low.

[0003] In summary, the recognition accuracy and detection efficiency of image-based behavior detection in the prior art are insufficient.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide a behavior detection method, storage medium and computer program product, aiming to solve the technical problems of insufficient recognition accuracy and detection efficiency in image-based behavior detection.

[0006] To achieve the above purpose, the present application proposes a behavior detection method, which is applied to an edge device of an image acquisition device, an image detection model is deployed on the edge device, the edge device is communicatively connected with a cloud device, and a multimodal large model is deployed on the cloud device. The behavior detection method includes:

[0007] Acquire continuous frame image data acquired by the image acquisition device, and input the continuous frame image data into the image detection model;

[0008] Performing behavior detection based on the continuous frame image data by the image detection model to obtain an initial detection result;

[0009] The initial detection result is sent to the cloud device, wherein the cloud device performs behavioral detection based on the initial detection result through the multimodal large model to obtain an intermediate detection result, compares the intermediate detection result with the initial detection result to obtain a comparison result, and outputs the comparison result based on the comparison result, the initial detection result and the intermediate detection result.

[0010] In one embodiment, the image detection model includes a plurality of functional modules;

[0011] Before the step of performing behavior detection based on the continuous frame image data by the image detection model to obtain an initial detection result, the method further includes:

[0012] Determine a behavior detection service to be executed, and split the behavior detection service into a plurality of detection sub-services, wherein the detection sub-services are services that can be executed by the functional module;

[0013] Determine the target module corresponding to each of the detection sub-services from each of the functional modules;

[0014] The step of performing behavior detection based on the continuous frame image data by the image detection model to obtain an initial detection result comprises:

[0015] Each of the target modules is called through the image detection model, and each of the target modules executes its corresponding detection sub-service based on the continuous frame image data to obtain an initial detection result.

[0016] In one embodiment, the step of splitting the behavior detection service into multiple detection sub-services includes:

[0017] Determine a detection process corresponding to the behavior detection service, and split the behavior detection service into a plurality of split sub-services according to each process node of the detection process, wherein each process node corresponds to a unique split sub-service;

[0018] Traversing each of the split sub-businesses, taking the split sub-businesses as target businesses, and detecting whether the target business is an executable business, wherein the executable business is a business that can be executed by the functional module;

[0019] If the target service is the executable service, the target service is used as a detection sub-service;

[0020] If the target business is not the executable business, the target business is split into multiple subordinate sub-businesses, each of the subordinate sub-businesses is traversed, the target business is updated through the subordinate sub-businesses, and the step of detecting whether the target business is an executable business is performed.

[0021] In one embodiment, the step of calling each of the target modules through the image detection model includes:

[0022] Determining the data dependency relationship between each of the target modules, and determining the calling order of each of the target modules based on the data dependency relationship;

[0023] Each of the target modules is called in sequence through the image detection model according to the calling order.

[0024] To achieve the above purpose, the present application proposes a behavior detection method, wherein the behavior detection method comprises a cloud device, the cloud device is communicatively connected with an edge device of an image acquisition device, an image detection model is deployed on the edge device, and a multimodal large model is deployed on the cloud device. The behavior detection method comprises:

[0025] Receiving an initial detection result sent by the edge device, wherein the edge device obtains continuous frame image data collected by the image acquisition device, and performs behavior detection based on the continuous frame image data through the image detection model to obtain an initial detection result;

[0026] Performing behavior detection based on the initial detection result through the multimodal large model to obtain an intermediate detection result;

[0027] The intermediate detection result is compared with the initial detection result to obtain a comparison result, which is based on the comparison result, the initial detection result and the intermediate detection result and outputted.

[0028] In one embodiment, the initial detection result includes behavior information corresponding to the continuous frame image data and an edge behavior detection result, wherein the behavior information includes target frame image data in which a dynamic behavior occurs in the continuous frame image data and first description information corresponding to the dynamic behavior;

[0029] The step of performing behavior detection based on the behavior information through the multimodal large model to obtain an intermediate detection result includes:

[0030] Performing behavior detection based on the target frame image data through the multimodal large model to obtain second description information and cloud-based behavior detection results, wherein the second description information is used to describe the dynamic behavior;

[0031] The second description information and the cloud behavior detection result are used as intermediate detection results.

[0032] In one embodiment, the step of comparing the intermediate detection result with the initial detection result to obtain a comparison result includes:

[0033] Comparing the second description information with the first description information to obtain a first comparison result, wherein the first comparison result indicates whether the second description information matches the first description information;

[0034] Comparing the cloud behavior detection result with the edge behavior detection result to obtain a second comparison result, wherein the second comparison result indicates whether the cloud behavior detection result matches the edge behavior detection result;

[0035] A comparison result is generated based on the first comparison result and the second comparison result.

[0036] In one embodiment, the step based on the comparison result, the initial detection result and the intermediate detection result includes:

[0037] If the comparison result shows that the initial detection result and the intermediate detection result are consistent, the initial detection result and the intermediate detection result are used as the target detection result;

[0038] If the comparison result shows that the initial detection result and the intermediate detection result are inconsistent, the cause of the abnormality is determined based on the comparison result, the initial detection result and the intermediate detection result, and the cause of the abnormality, the comparison result, the initial detection result and the intermediate detection result are used as the target detection result.

[0039] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the behavior detection method described above are implemented.

[0040] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the behavior detection method described above are implemented.

[0041] In the present application, the edge device obtains continuous frame image data collected by the image acquisition device, inputs the continuous frame image data into the image detection model; performs behavior detection based on the continuous frame image data through the image detection model to obtain an initial detection result; and sends the initial detection result to the cloud device. The cloud device performs behavior detection based on the initial detection result through the multimodal large model to obtain an intermediate detection result, compares the intermediate detection result with the initial detection result to obtain a comparison result, and outputs it based on the comparison result, the initial detection result and the intermediate detection result.

[0042] In this application, the image detection model is deployed on the edge device of the image acquisition device, and the edge device is close to the image data source, so that the image detection model can obtain and process the image data in a timely manner, reduce the image data transmission delay, and improve the real-time performance of image-based behavior detection.

[0043] Compared to using a large multimodal model to perform behavior detection based on raw image data, this application example uses the powerful generalization and image understanding capabilities of the large multimodal model to further verify and optimize the initial detection results sent by the edge, so that behavior detection can adapt to data changes in different scenarios, reduce the impact of data drift, and improve the recognition accuracy of the entire system. In addition, the large multimodal model can make full use of cloud computing resources, avoid resource limitations, and improve the efficiency and accuracy of detection by the large multimodal model.

[0044] Thus, the present application achieves the improvement of processing efficiency and recognition accuracy of image-based behavior detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 A flowchart of the first embodiment of the behavior detection method of the present application is provided;

[0048] Figure 2 A flow chart of the second embodiment of the behavior detection method of the present application;

[0049] Figure 3 A brief flowchart of a behavior detection method provided in one embodiment of the present application;

[0050] Figure 4 A specific flow chart of a behavior detection method provided in one embodiment of the present application;

[0051] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the behavior detection method in the embodiment of the present application.

[0052] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0053] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0054] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0055] The main solution of the embodiment of the present application is: the edge device obtains the continuous frame image data collected by the image acquisition device, and inputs the continuous frame image data into the image detection model; the image detection model performs behavior detection based on the continuous frame image data to obtain an initial detection result; and the initial detection result is sent to the cloud device. The cloud device performs behavior detection based on the initial detection result through the multimodal large model to obtain an intermediate detection result, compares the intermediate detection result with the initial detection result to obtain a comparison result, and outputs it based on the comparison result, the initial detection result and the intermediate detection result.

[0056] As a branch of artificial intelligence, computer vision technology has been widely used in the traditional security field. The usual practice is to use image recognition technology and the video stream output by the camera to collect data for different business scenarios and train models for corresponding business scenarios based on different algorithms to achieve different businesses.

[0057] With the continuous development of artificial intelligence technology, image detection based on traditional models can no longer meet the needs of today's increasingly diverse business scenarios and rapid deployment. The commonly used technologies currently have the following problems. First, the ordered combination of traditional detection, classification, tracking and other convolutional networks is used to implement corresponding business scenarios. The coupling between business scenarios and models is too high. For each new business scenario, an independent set of technical implementations needs to be provided, which increases the development cost and deployment cycle. Second, traditional detection models require a lot of labeling and training, and are prone to data drift when docked with new scenarios.

[0058] In recent years, the multimodal large language model (MLLM) based on the large language model has developed rapidly. On the basis of its original powerful generalization and reasoning capabilities, it has further introduced multimodal information processing capabilities, including image description, visual question answering, visual localization, etc. [3]. The architecture of the multimodal large model includes an encoder, a connector, and a large language model. The encoder is used to encode the original image information into features, and the connector further processes the features into a form that is easy for the large language model to understand. The large language model acts as a thinking brain to integrate this information for understanding and reasoning, and generates corresponding answers. Since the training data of the large language model on which the multimodal large model relies is often in the billions, which is more than the training data of the traditional model, it has stronger data generalization ability than the traditional model and has more general image understanding ability. However, due to the large number of model parameters, the deployment of the multimodal large model occupies a huge amount of resources, and the reasoning time for a single image is also in the seconds level, which cannot achieve real-time reasoning like the traditional model. Therefore, using only the multimodal large model cannot meet the requirements of most current business scenarios.

[0059] In summary, the recognition accuracy and detection efficiency of image-based behavior detection in the prior art are insufficient.

[0060] The present application provides a solution by deploying an image detection model on an edge device of an image acquisition device, where the edge device is close to the image data source, so that the image detection model can obtain and process image data in a timely manner, reduce image data transmission delays, and improve the real-time performance of image-based behavior detection.

[0061] Moreover, compared to using a large multimodal model to perform behavior detection based on raw image data, this application example uses the powerful generalization and image understanding capabilities of the large multimodal model to further verify and optimize the initial detection results sent by the edge, so that behavior detection can adapt to data changes in different scenarios, reduce the impact of data drift, and improve the recognition accuracy of the entire system. In addition, the large multimodal model can make full use of cloud computing resources, avoid resource limitations, and improve the efficiency and accuracy of detection by the large multimodal model.

[0062] Thus, the present application achieves the improvement of processing efficiency and recognition accuracy of image-based behavior detection.

[0063] It should be noted that the executor of this embodiment is the edge device of the image acquisition device. The edge device refers to a device close to the data source, which can perform certain data processing tasks without sending all data to the cloud for processing. The edge device can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device that can achieve the above functions, etc., which is not limited here. The following describes this embodiment with the edge device as the executor. In this embodiment, an image detection model is deployed on the edge device, the edge device is communicatively connected to the cloud device, and a multimodal large model is deployed on the cloud device.

[0064] Based on this, the present application embodiment provides a behavior detection method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the behavior detection method of the present application.

[0065] In this embodiment, the behavior detection method includes steps S10 to S30:

[0066] Step S10, acquiring continuous frame image data acquired by the image acquisition device, and inputting the continuous frame image data into the image detection model;

[0067] It should be noted that image acquisition devices are able to capture static images or continuous video streams and convert them into digital signals for subsequent processing and analysis. Image acquisition devices usually include core components such as optical lenses and image sensors, as well as possible auxiliary components such as light sources and image acquisition cards.

[0068] Acquiring a series of temporally continuous image data from the image acquisition device and inputting these data into the image detection model can achieve real-time data acquisition and input, providing a basis for subsequent behavior detection. It can be understood that continuous frame images refer to a series of temporally continuous images, which are used to capture changes in dynamic scenes, such as the movement of objects, human behavior, etc. By analyzing continuous frame images, information such as the movement trajectory and speed of the object can be extracted.

[0069] Step S20, performing behavior detection based on the continuous frame image data by the image detection model to obtain an initial detection result;

[0070] It should be noted that behavior detection refers to the process of identifying specific behaviors from video or image sequences, which usually involves analyzing the motion information in the image sequence to determine whether the target behavior exists. Behavior detection can be applied to various scenarios, such as video surveillance, human-computer interaction, motion analysis, etc. By detecting behaviors, functions such as anomaly detection and behavior recognition can be achieved.

[0071] An image detection model is a machine learning model used to identify specific objects, scenes, or behaviors in an image. The model extracts features from the input image and determines whether the target object or behavior exists in the image based on these features. The image detection model can be a convolutional neural network (CNN), a region proposal network (R-CNN) series, or a YOLO (You Only LookOnce) series, and there is no restriction here.

[0072] The image detection model processes and analyzes the input continuous frame image data, identifies the target behavior, and outputs the initial detection result. The initial detection result refers to the preliminary result obtained by the image detection model after performing behavior detection on the continuous frame image data, which may include information such as the existence, location, and confidence of the target behavior. The initial detection result can be used for subsequent further processing and analysis. This step realizes the preliminary processing and analysis of the image data and provides a basis for subsequent further processing.

[0073] Step S30, sending the initial detection result to the cloud device, wherein the cloud device performs behavior detection based on the initial detection result through the multimodal large model to obtain an intermediate detection result, compares the intermediate detection result with the initial detection result to obtain a comparison result, and outputs it based on the comparison result, the initial detection result and the intermediate detection result.

[0074] Cloud devices refer to computing devices deployed on cloud servers, which are used to process data from edge devices. Cloud devices have powerful computing and storage capabilities and can handle large-scale data and complex computing tasks. They usually collaborate with edge devices to achieve real-time data processing and analysis.

[0075] In this embodiment, the edge device is connected to the cloud device in communication, and the edge device sends the initial detection result to the cloud device. The cloud device further processes and analyzes the initial detection result through the multimodal large model to obtain an intermediate detection result, compares the intermediate detection result with the initial detection result, obtains a comparison result, and finally outputs the final target detection result based on the comparison result, the initial detection result and the intermediate detection result.

[0076] It should be noted that the intermediate detection result refers to the result obtained after the cloud device further processes and analyzes the initial detection result through a multimodal large model. The intermediate detection result may contain more detailed behavioral characteristics, classification information, or more accurate location information. The target detection result refers to the final output behavior detection result, which is the result obtained based on the comprehensive analysis of the initial detection result and the intermediate detection result. It may include detailed information such as the existence, type, location, and confidence of the behavior, which is not limited here.

[0077] Multimodal large models refer to deep learning models that can process multiple types of data (such as text, images, audio, etc.). Multimodal large models usually have a large number of parameters and complex network structures, can capture multiple features in the data, can fuse information from different modalities, and perform cross-modal reasoning and understanding. For example, they can combine image and text information to better understand the image content, or combine audio and video information to recognize speech and actions.

[0078] In a feasible implementation, the image detection model includes a plurality of functional modules;

[0079] Before the step S20: performing behavior detection based on the continuous frame image data by the image detection model to obtain an initial detection result, the step further includes:

[0080] Step S01, determining a behavior detection service to be executed, and splitting the behavior detection service into a plurality of detection sub-services, wherein the detection sub-services are services that can be executed by the functional module;

[0081] At present, the corresponding business scenarios are realized by using an ordered combination of traditional detection, classification, tracking and other convolutional networks. In this way, the coupling between business scenarios and models is too high. For each new business scenario, a set of independent technical implementations needs to be provided, which increases the development cost and deployment cycle. In order to solve the problem that business scenarios and models cannot be decoupled, this embodiment proposes a modular image detection model, which realizes the modular access of models and capabilities in most business scenarios. According to the needs of business scenarios, the corresponding models or capabilities can be imported from the developed model library and capability library, and the modules can be connected in order to obtain the initial detection results.

[0082] In this embodiment, the image detection model includes multiple functional modules. A functional module refers to a relatively independent model component designed to achieve specific functions in the target behavior detection method. Different functional modules can achieve different functional goals. For example, it can be a feature extraction module, an image tracking module, etc.

[0083] The edge device determines the behavior detection business to be executed, and splits the behavior detection business into multiple detection sub-businesses. By splitting the behavior detection business to be executed into multiple detection sub-businesses, it is more convenient to use the functional modules in the image detection model for processing. Each detection sub-business corresponds to one or more functional modules in the model, and these modules can work together to complete the entire behavior detection task.

[0084] It should be noted that the behavior detection business to be executed refers to the specific tasks or scenarios that require behavior detection, which may involve different application scenarios, such as video surveillance, human-computer interaction, motion analysis, etc. The behavior detection business to be executed usually has clear goals and requirements, such as detecting specific types of behaviors (such as running, jumping, waving, etc.), identifying behaviors in specific scenarios (such as indoors, outdoors, at night, etc.), and real-time requirements, etc. Business requirements determine the design and functional configuration of the image detection model. Detection sub-business refers to splitting the behavior detection business to be executed into smaller and more specific tasks or sub-tasks, which can be independently executed by the functional modules in the image detection model.

[0085] By splitting the overall business into multiple sub-businesses, it is easier to configure and adjust according to actual needs, improving the flexibility of edge devices. Different detection sub-businesses may require different functional modules and computing resources. The split sub-businesses can use these resources more effectively and avoid resource waste. The split sub-businesses are easier to manage and maintain, which helps reduce the complexity and maintenance costs of edge devices.

[0086] Step S02, determining the target modules corresponding to the respective detection sub-services from the respective functional modules;

[0087] The target module refers to the functional module or component in the image detection model used to perform a specific detection sub-business. According to the split detection sub-business, select the appropriate target module from the functional modules in the image detection model.

[0088] It is understandable that by selecting the most appropriate target module to process each detection sub-service, the output results can be ensured to be more accurate and reliable. Different target modules may have different performance and efficiency. Through reasonable selection, the performance of the entire edge device can be optimized, and the detection speed and efficiency can be improved. As the application scenarios and requirements change, it may be necessary to add new detection sub-services or modify existing services. Through modular design, these changes can be more easily implemented, enhancing the scalability of the image detection model.

[0089] It can be understood that the process of determining the target modules corresponding to each detection sub-business from each functional module can be: first, a detailed demand analysis is performed on the detection sub-business, which includes clarifying the business requirements of each detection sub-business, which may specifically include business objectives, input data, output requirements, and business logic that may be involved. For example, one detection sub-business may be to identify pedestrians in a video and track their motion trajectories, while another detection sub-business may be to identify a specific type of vehicle and extract its features; then a matching strategy is determined, that is, based on the business requirements of the detection sub-business, the matching relationship between the functional modules and the detection sub-business is determined, which functional modules can meet the requirements of which detection sub-businesses, and how to combine multiple functional modules when necessary to achieve complex functions. For example, for the detection sub-business of pedestrian tracking, it may be necessary to combine an image detection module (for identifying pedestrians) and an image tracking module (for tracking the motion trajectories of pedestrians); then, according to the matching strategy, a matching process is performed to associate each detection sub-business with the corresponding functional module, and configure necessary parameters and interfaces. For example, for the detection sub-business of vehicle identification, it may be necessary to configure an image detection module to identify a specific type of vehicle, and set output parameters to include the type, location, and confidence of the vehicle.

[0090] The step S20: the step of performing behavior detection based on the continuous frame image data by the image detection model to obtain an initial detection result, comprises:

[0091] Step S201, calling each of the target modules through the image detection model, and obtaining an initial detection result by executing a corresponding detection sub-service based on the continuous frame image data through each of the target modules.

[0092] The image detection model calls each target module to perform the corresponding detection sub-service according to the mapping relationship between the detection sub-service and the target module. The target module processes the input continuous frame image data and outputs the initial detection result. The initial detection will serve as the basis for subsequent processing and analysis to generate the final target detection result. By automatically calling the target module to perform the detection sub-service, manual intervention and error rate can be greatly reduced, and the degree of automation of the edge device can be improved. It can be understood that when calling each of the target modules through the image detection model, some modules may obtain too many computing resources, while other modules are insufficient. In a feasible implementation, the edge device can monitor the resource usage of each target module, such as memory, disk I / O (input / output), etc., and then set the resource usage priority for each module according to the urgency and importance of the calling order of the detection subtask, where the priority is proportional to the amount of resources, ensuring that high-priority tasks can obtain sufficient resources while avoiding low-priority tasks from occupying resources for a long time; when a module is insufficient in resources, some resources can be recovered from other low-priority modules, or allocated from the reserved resources of the edge device.

[0093] In a feasible implementation manner, the step S01: splitting the behavior detection service into multiple detection sub-services includes:

[0094] Step S011, determining the detection process corresponding to the behavior detection service, and splitting the behavior detection service into multiple split sub-services according to each process node of the detection process, wherein each process node corresponds to a unique split sub-service;

[0095] The detection process refers to a series of ordered steps or stages that need to be followed when performing behavioral detection business. These steps usually cover the entire process from data input, preprocessing, feature extraction, model reasoning to result output. Each node (or step) in the detection process undertakes a specific task and outputs intermediate results or final detection results. The connections between these nodes constitute the logical framework of the entire detection process. Split sub-businesses refer to smaller and more specific business units obtained by splitting the behavioral detection business according to the various process nodes of the detection process. Each split sub-business corresponds to a node in the detection process and undertakes the tasks assigned to the node.

[0096] By determining the detection process according to the specific needs of the behavior detection business, and splitting the business into multiple sub-businesses according to the process nodes, it facilitates subsequent module matching and execution.

[0097] Step S012, traversing each of the split sub-businesses, taking the split sub-businesses as target businesses, and detecting whether the target business is an executable business, wherein the executable business is a business that can be executed by the functional module;

[0098] An executable business refers to a business that can be directly executed by a functional module. Each split sub-business is traversed and the split sub-business is taken as the target business. It is determined whether the target is an executable business that can be directly executed by a functional module, thereby identifying which split sub-businesses can be executed directly and which need to be further split.

[0099] Step S013: if the target service is the executable service, the target service is used as a detection sub-service;

[0100] By identifying the business that can be directly executed as a detection sub-business, the subsequent execution process is simplified and the efficiency is improved.

[0101] Step S014, if the target business is not the executable business, split the target business into multiple subordinate sub-businesses, traverse each of the subordinate sub-businesses, update the target business through the subordinate sub-businesses, and execute the step of detecting whether the target business is an executable business.

[0102] For target businesses that cannot be directly executed, they are further split into lower-level sub-businesses, and the target business is updated to the split lower-level sub-businesses, and then the detection steps are repeated. By continuously splitting, complex businesses are decomposed into simpler, directly executable parts, which improves the flexibility and scalability of edge devices.

[0103] In one feasible implementation, the step S201: calling each of the target modules through the image detection model, includes:

[0104] Step S2011, determining the data dependency relationship between the target modules, and determining the calling order of the target modules based on the data dependency relationship;

[0105] Data dependency refers to the output of one module being used as the input of another module, or the execution of one module requiring the intermediate result provided by another module. Based on data dependency, the edge device can determine the calling order of each target module, that is, determine which target modules need to be executed first in order to provide the necessary input data for subsequent modules. It can be understood that by determining a reasonable calling order, unnecessary waiting and data transmission overhead can be avoided, thereby improving the execution efficiency of the entire image detection task. Clear dependencies and calling order help reduce conflicts and error propagation between modules, and enhance the stability and reliability of edge devices.

[0106] The specific steps of determining the calling order of each target module based on the data dependency relationship may be: determining the data interaction information between all target modules, including the input, output, and functional description of the target modules; constructing a dependency graph: constructing a dependency graph based on the data interaction information between the target modules, in which each target module is a node and each edge represents a data dependency relationship; performing topological sorting based on the dependency graph to obtain a linear calling order, and topological sorting is a graph theory algorithm that can find a linear sequence in a directed acyclic graph (DAG) such that for each directed edge (u, v) in the graph, vertex u appears before vertex v in the sequence. This sequence is the calling order of the target modules.

[0107] Step S2012: Call each of the target modules in sequence according to the calling order through the image detection model.

[0108] The image detection model calls each target module in sequence. After receiving the input data, each module performs its specific function and outputs the processing result to the next module. This process continues until all modules have been called, and finally the image detection result is obtained.

[0109] By calling the modules in sequence, we can ensure that each module receives the correct input data and produces the expected output results. Orderly module calls can avoid unnecessary repeated calculations and data transmission, thereby improving the processing efficiency of the entire image detection task.

[0110] In this embodiment, the image detection model is deployed on the edge device of the image acquisition device, and the edge device is close to the image data source, so that the image detection model can obtain and process the image data in a timely manner, reduce the image data transmission delay, and improve the real-time performance of image-based behavior detection.

[0111] Moreover, compared with using a multimodal large model to perform behavior detection based on raw image data, this embodiment uses the powerful generalization and image understanding capabilities of the multimodal large model to further verify and optimize the initial detection results sent by the edge, so that behavior detection can adapt to data changes in different scenarios, reduce the impact of data drift, and improve the recognition accuracy of the entire edge device. In addition, the multimodal large model can make full use of cloud computing resources, avoid resource limitations, and improve the efficiency and accuracy of multimodal large models for detection.

[0112] Thus, this embodiment achieves improved processing efficiency and recognition accuracy of image-based behavior detection.

[0113] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction and will not be repeated later.

[0114] It should be noted that the execution subject of this embodiment is a cloud device, which can be a computing service device with data processing, network communication and program running functions, such as a server, or an electronic device that can realize the above functions, etc., without limitation. The following describes this embodiment with the cloud device as the execution subject. In this embodiment, the edge device is communicatively connected with the cloud device, a multimodal large model is deployed on the cloud device, and an image detection model is deployed on the edge device.

[0115] Based on this, refer to Figure 2 , Figure 2 This is a flow chart of the second embodiment of the behavior detection method of the present application. In this embodiment, the behavior detection method includes steps A10 to A30:

[0116] Step A10, receiving the initial detection result sent by the edge device, wherein the edge device obtains the continuous frame image data collected by the image acquisition device, and performs behavior detection based on the continuous frame image data through the image detection model to obtain the initial detection result;

[0117] The cloud device receives the initial detection results from the edge device. The edge device is usually deployed close to the image acquisition device for real-time processing of image data. The edge device uses the image detection model to perform behavior detection on the continuous frame image data obtained from the image acquisition device, preliminarily identifies possible behavior patterns or events, and sends these initial detection results to the central server or cloud device for processing.

[0118] Step A20, performing behavior detection based on the initial detection result using the multimodal large model to obtain an intermediate detection result;

[0119] The cloud device uses the multimodal big model to perform further behavioral detection on the initial detection results sent by the edge device. The multimodal big model can integrate information from different modalities (such as images, videos, audio, text, etc.) to perform more comprehensive behavioral analysis. By combining the initial detection results and possible other modal information, the multimodal big model can generate more accurate intermediate detection results.

[0120] Step A30, comparing the intermediate detection result with the initial detection result to obtain a comparison result, and outputting it based on the comparison result, the initial detection result and the intermediate detection result.

[0121] The cloud device compares the intermediate test results with the initial test results to evaluate the differences and consistency between the two. Through the comparison, the cloud device can identify possible errors or inconsistencies and generate the final test result based on this information and the initial and intermediate test results. The final test result may be a more accurate test result that combines the advantages of both, or it may be a more comprehensive test result that includes the differences and uncertainties of the two.

[0122] By comparing the initial test results with the intermediate test results, the cloud device can identify and correct possible errors and improve the reliability of the final test results. The final test results provide a more comprehensive information basis for subsequent decision-making, making the decision-making process more scientific and reasonable.

[0123] In a feasible implementation manner, the initial detection result includes behavior information corresponding to the continuous frame image data and an edge behavior detection result, and the behavior information includes target frame image data in which a dynamic behavior appears in the continuous frame image data and first description information corresponding to the dynamic behavior;

[0124] The step A20: performing behavior detection based on the behavior information by the multimodal large model to obtain an intermediate detection result includes:

[0125] Step A201, performing behavior detection based on the target frame image data through the multimodal large model to obtain second description information and cloud behavior detection results, wherein the second description information is used to describe the dynamic behavior;

[0126] It should be noted that the initial detection results include not only the edge behavior detection results obtained by the edge device through the image detection model, but also the behavior information corresponding to the continuous frame image data. The behavior information is further divided into two parts: one is the target frame image data where dynamic behavior occurs, which are the frames containing key dynamic behaviors selected from the continuous frame images; the other is the first description information corresponding to the dynamic behavior, which is the preliminary text description or classification label of the dynamic behavior.

[0127] The behavior information corresponding to the continuous frame image data refers to the data or information related to a specific behavior extracted from a series of continuous images (i.e., video frames). The behavior information is usually used to describe, identify or analyze dynamic behaviors in images, such as human movements, object motion trajectories, etc.

[0128] The edge behavior detection result refers to the behavior recognition result obtained after preliminary processing and analysis of continuous frame image data on the edge computing device. Edge computing is a distributed computing architecture that pushes computing and data storage from the centralized cloud to the edge of the network, that is, close to the source of data generation.

[0129] The target frame image data with dynamic behavior in the continuous frame image data refers to the frames containing specific dynamic behavior selected from a series of continuous images. These frames are usually used for further behavior analysis or recognition. Dynamic behavior refers to the behavior that occurs in the image with temporal continuity and spatial variability, such as people walking, vehicles driving, etc.

[0130] It is understandable that the multimodal large model receives target frame image data (i.e., frames containing dynamic behaviors) from the edge device and performs in-depth behavior detection on it. Unlike the preliminary detection on the edge device, the multimodal large model in the cloud may have more powerful computing power and richer training data, and can more accurately identify and analyze dynamic behaviors. The results of the detection include the second description information and the cloud behavior detection results. The second description information is a more detailed and accurate description of the dynamic behavior, which may include detailed information such as the category of the behavior, the time and place of occurrence. The cloud behavior detection result is a comprehensive judgment of the entire dynamic behavior, which may be a classification label, a confidence score, or a more complex representation.

[0131] The multimodal large model uses more powerful computing power and richer training data to more accurately identify and analyze dynamic behaviors and improve the accuracy of detection results. The second description information provides a more detailed and accurate description of dynamic behaviors, so that subsequent analysis and decision-making can be based on richer information.

[0132] Step A202: Use the second description information and the cloud behavior detection result as intermediate detection results.

[0133] The cloud device integrates the second description information and the cloud behavior detection result as an intermediate detection result, and the intermediate detection result will be used for subsequent comparison and generation of the final detection result.

[0134] It is understandable that the intermediate detection results integrate information from different sources (edge ​​and cloud), so that subsequent analysis and decision-making can be based on more comprehensive and accurate information. Through the collaborative work of the above two steps, cloud devices can achieve more accurate and comprehensive detection of dynamic behaviors in continuous frame image data, providing strong support for various application scenarios. At the same time, this processing method also reflects the combination of edge computing and cloud computing, making full use of the advantages of both and improving the performance and efficiency of the entire cloud device.

[0135] In one feasible implementation, the step A30: comparing the intermediate detection result with the initial detection result to obtain a comparison result, comprises:

[0136] Step A301, comparing the second description information with the first description information to obtain a first comparison result, wherein the first comparison result indicates whether the second description information matches the first description information;

[0137] The remote device compares the second description information in the intermediate detection result (derived by the multimodal large model based on the target frame image data) with the first description information in the initial detection result (preliminarily obtained by the edge device based on the continuous frame image data) to obtain a first comparison result. The first comparison result is an indicator of whether the second description information matches the first description information. If the two are highly similar, it indicates that the initial recognition of the edge device and the further recognition of the multimodal large model are consistent in description information; if there are differences, further analysis of the cause may be required. The comparison can evaluate the consistency of the second description information with the first description information, thereby evaluating the accuracy of the initial recognition of the edge device and the effectiveness of the further recognition of the multimodal large model.

[0138] It is understandable that the specific method of comparison is not limited in this embodiment. The comparison may involve methods such as text similarity calculation and semantic analysis to quantify the differences between the description information. For example, indicators such as cosine similarity and Jaccard similarity coefficient can be used to measure the similarity between two description information.

[0139] Step A302, comparing the cloud behavior detection result with the edge behavior detection result to obtain a second comparison result, wherein the second comparison result indicates whether the cloud behavior detection result matches the edge behavior detection result;

[0140] The cloud device compares the cloud behavior detection result in the intermediate detection result with the edge behavior detection result in the initial detection result to obtain a second comparison result. The second comparison result is an indicator of whether the cloud behavior detection result matches the edge behavior detection result. If the two are consistent, it indicates that the behavior recognition of the multimodal large model has been verified in terms of accuracy; if they are inconsistent, it may be necessary to further adjust the model parameters or optimize the algorithm.

[0141] It is understandable that the specific method of comparison is not limited in this embodiment, and the comparison may involve methods such as comparison of classification labels and comparison of confidence scores. For example, if the classification labels of the cloud behavior detection results and the edge behavior detection results are the same and the confidence scores are high, it means that the cloud behavior detection results and the edge behavior detection results are consistent in behavior recognition; if there are differences, further analysis of the reasons may be required, such as the introduction of new information sources or algorithm improvements in the multimodal large model.

[0142] Step A303: Generate a comparison result based on the first comparison result and the second comparison result.

[0143] The cloud device combines the first comparison result and the second comparison result to generate a final comparison result. The comparison result may be a comprehensive indicator, such as a similarity score, consistency percentage, etc., which is used to quantify the difference and consistency between the initial detection result and the intermediate detection result, and is not limited here.

[0144] By comparing the initial detection results with the intermediate detection results, the cloud device can find the differences and consistency between the two, thereby evaluating the accuracy of behavior recognition. This helps to promptly discover and correct possible errors and improve the accuracy of the final detection results. The comparison results provide additional verification of the behavior recognition process, enhance the reliability of the cloud device for behavior recognition, and help maintain the stability and robustness of the cloud device in complex or uncertain scenarios.

[0145] In one feasible implementation, the step A30: the step based on the comparison result, the initial detection result and the intermediate detection result, comprises:

[0146] Step A304: if the comparison result shows that the initial detection result and the intermediate detection result are consistent, the initial detection result and the intermediate detection result are used as the target detection result;

[0147] The cloud device first determines whether the comparison result indicates that the initial detection result and the intermediate detection result are consistent. If the first comparison result is that the second description information matches the first description information, and the second comparison result is that the cloud behavior detection result matches the edge behavior detection result, the cloud device considers that the results of the image detection model and the multimodal large model in behavior recognition are consistent, and uses the target frame image data, the cloud behavior detection result, and the edge behavior detection result as the target detection result;

[0148] Step A305, if the comparison result is that the initial detection result and the intermediate detection result are inconsistent, determine the cause of the abnormality based on the comparison result, the initial detection result and the intermediate detection result, and use the cause of the abnormality, the comparison result, the initial detection result and the intermediate detection result as the target detection result.

[0149] If the first comparison result is that the second description information does not match the first description information, and / or the second comparison result is that the cloud behavior detection result does not match the edge behavior detection result, the cloud device believes that the results of the image detection model and the multimodal large model in behavior recognition are inconsistent, and determines the cause of the abnormality based on the comparison result, the initial detection result and the intermediate detection result, and uses the cause of the abnormality, the comparison result, the target frame image data, the cloud behavior detection result and the edge behavior detection result as the target detection result.

[0150] By deeply analyzing the causes of inconsistencies, the cloud device can identify and correct possible errors, thereby improving the accuracy of the final detection results and providing valuable information for subsequent optimization of the cloud device.

[0151] In this embodiment, the image detection model is deployed on the edge device of the image acquisition device, and the edge device is close to the image data source, so that the image detection model can obtain and process the image data in a timely manner, reduce the image data transmission delay, and improve the real-time performance of image-based behavior detection.

[0152] Moreover, compared with using a multimodal large model to perform behavior detection based on raw image data, this embodiment uses the powerful generalization and image understanding capabilities of the multimodal large model to further verify and optimize the initial detection results sent by the edge, so that behavior detection can adapt to data changes in different scenarios, reduce the impact of data drift, and improve the recognition accuracy of the entire cloud device. In addition, the multimodal large model can make full use of the computing resources of the cloud, avoid resource limitations, and improve the efficiency and accuracy of the multimodal large model for detection.

[0153] Thus, this embodiment achieves improved processing efficiency and recognition accuracy of image-based behavior detection.

[0154] For example, in order to help understand the implementation process of the behavior detection method obtained by combining this embodiment with the above-mentioned embodiment 1, this embodiment is further described with specific examples:

[0155] This embodiment provides a target behavior detection method for "end-edge-cloud" deployment, which can combine traditional detection modes and multimodal large models. First, preliminary results are obtained by deploying in a traditional way at the edge, and the algorithm architecture is optimized to decouple the model and business scenarios to achieve rapid deployment of various business scenarios. Secondly, the multimodal large model is accessed on the cloud, and the image understanding ability of the large model is used to verify the preliminary results, avoiding the data drift problem of traditional models and improving the accuracy of business recognition.

[0156] Specifically, in response to the problem that business scenarios and models cannot be decoupled, this embodiment proposes modular access to models and capabilities in most business scenarios. According to the needs of the business scenarios, the corresponding models or capabilities are imported from the developed model library and capability library, and the modules are connected in order to obtain the output results. In response to the problem of data drift after the traditional model is connected to the new scenario, this embodiment proposes a method that combines the traditional image detection method with the multimodal large model, and utilizes the characteristics of the multimodal large model such as strong data generalization ability and general recognition ability to improve the final recognition accuracy. In response to the problem that the multimodal large model consumes a lot of resources in the field of image monitoring and cannot achieve real-time detection, this embodiment proposes a method that uses the traditional image monitoring method as a pre-detection module, uses the traditional image detection method to first obtain preliminary recognition results, and then relies on the multimodal large model for verification.

[0157] This embodiment provides a target behavior detection method deployed in "end-edge-cloud", which aims to decouple business scenarios and models and use multimodal large models to improve the final business recognition accuracy. Figure 3 , the target behavior detection method specifically includes:

[0158] The image detection model deployed on the edge is used to analyze video data in real time. After analyzing and processing the video frames, it generates preliminary analysis results and sends them to the cloud. The preliminary analysis results include the following: discrete frame data of the video segment where the business scenario occurs, which is used to verify the authenticity of the initial analysis results (that is, the target frame image data); prompt information, which describes the prompt words of the corresponding business scenario, which is used for the cloud-based multimodal large model to perform data verification on the business scenario (that is, the first description information); edge model detection results, which are used to repeatedly verify the accuracy of the analysis results with the large model (that is, the edge behavior detection results). The multimodal large model processing module deployed in the cloud is used to analyze the preliminary analysis results sent from the edge, including extracting key information based on the images contained in the results, comparing them with the prompt information in the preliminary analysis results, and judging whether the image content is the event in the prompt information based on whether the comparison results are consistent. Finally, the information extracted from the image is compared with the edge detection results of the initial analysis results, and the final analysis results are output. The final analysis results include the following: discrete frame data of the video segment where the business scene occurs, which is used to verify the authenticity of the final analysis results (that is, the target frame image data); multimodal large model recognition results (that is, cloud-based behavior detection results); (3) edge model detection results (that is, edge behavior detection results).

[0159] In the target behavior detection method, the image detection model deployed at the edge is used to analyze video data in real time and give preliminary recognition results. It specifically includes a set of model libraries and capability libraries (i.e. functional modules).

[0160] The model library is used to perform corresponding image processing operations according to the needs of business scenarios, including: image detection component library, which detects relevant target objects according to business scenario requirements, inputs the image to be detected, and outputs the label, edge box and confidence of the target object. Image feature extraction component library, which inputs the image of the object to be identified, uses the feature extraction model to extract the feature information of the object to be identified, and outputs the feature vector. At the same time, multiple feature vectors can also be generated into a corresponding feature library for searching the feature vector to be detected. Vector retrieval component library, first builds a feature library of the object to be identified, inputs the feature vector of the object to be retrieved, uses the retrieval algorithm to match the features, and outputs the label corresponding to the feature with the highest similarity in the feature library. Image tracking component library, which inputs the edge box of the object in the previous frame and the current target image, and outputs the edge box and confidence of the tracked object in the target image. Pedestrian skeleton point extraction component library, which inputs the pedestrian edge box and the target image, and outputs the corresponding skeleton key point coordinates.

[0161] The capability library is used to execute corresponding data processing capabilities according to the needs of business scenarios, including: video encoding and decoding component library, which reads the real-time stream or offline video of the camera according to the requirements of the business scenario, slices the video frame and outputs it, or merges and encodes discrete pictures into videos. ROI (Region of Interest) Region of Interest Selection Component Library, hereinafter referred to as ROI area, sets one or more ROI areas in the image and filters the edge boxes that are not in the area. Post-processing component library, which matches the object labels, edge boxes, confidence and other results output from the model library according to the rules formulated by the business scenario. Specifically, it first specifies the detection subject, and then matches the subject and the target object. The matching rule is determined by the relative distance threshold between the subject and the target object in the image. Generally speaking, the subject is a person, and the subject contains the data related to the subject output from the model component library, including detection, tracking, skeleton key points, face recognition, human body attributes and other information supported by the model capability analysis. The target object is the object associated with the subject in the business scenario, including object labels, edge boxes, confidence, vector retrieval results and other information. The specific information of the above subject and associated objects depends on whether it is needed in the business scenario. The conditional trigger component library formulates trigger conditions according to business rules for the data matched by the post-processing component. The triggering method uses condition counting, and the trigger condition threshold can be adjusted according to the business scenario. First, determine whether the current image frame meets the matching conditions output by the post-processing component. If it does, the condition counting is performed. When the count exceeds the threshold, the corresponding business scenario is triggered. The condition counting is reset after the trigger. If the condition counting does not exceed the threshold and the post-processing component matching conditions are not met for multiple consecutive times, the condition counting is reset. The result sending component library uses tools such as message middleware to send the results to the target device. The information sent is the preliminary analysis result of the image detection model. The multimodal access component library selects the specified multimodal large model and enters multiple prompt words. It understands the image information in the preliminary analysis results, determines whether the image information is consistent with the prompt words, and outputs the final analysis results.

[0162] In this embodiment, the image detection model is deployed at the edge and provides real-time detection capabilities for business scenarios. First, the required business capabilities are split according to different business scenarios, and then split into specific executable functions according to the business capabilities. Then, the corresponding components are extracted from the model library and capability library, and multiple components are combined in an orderly manner to obtain a business scenario monitoring service that meets the deployment conditions.

[0163] In order to describe the role and execution method of the image detection model in detail, in one of the embodiments, it is required to monitor the behavior of people taking pictures of the screen with their mobile phones in the office scene and issue an alarm. The following steps are used to split and implement the functions. For details, please refer to Figure 4 :

[0164] First, according to the given business scenario requirements, the business process can be split into video decoding, screen capture behavior detection, and result sending. Video decoding and result sending directly call the video encoding and decoding component library and result sending component library in the capability library. Screen capture behavior detection has no component library directly called, so continue to split:

[0165] Step 1: Call the image detection component library to detect the edge boxes, confidence levels, and labels of three types of objects in the image: pedestrians, mobile phones, and screens. Call the ROI region component library to filter the edge boxes and retain the detection results within the region of interest.

[0166] Step 2: Call the pedestrian skeleton point extraction component library for the pedestrian detection result to obtain the object skeleton key point information;

[0167] Step 3: Match the person holding the phone through the post-processing component library, input the person and phone detection information, match the phone edge frame coordinates with the person's wrist key point coordinates. The matching rule is the pixel distance between the edge frame and the wrist key point. If the pixel distance is less than the preset value, it is determined that the person is holding the phone, and the people who do not meet the rule are filtered out;

[0168] Step 4, for the screen detection results, the usual method uses image detection to determine the screen state, but it is unavoidable that the lock screen interface is regularly replaced, resulting in detection failure. In view of this, in order to identify the current state of the screen, this embodiment is divided into four states: bright screen, lock screen, black screen, and scan QR code. For the actual screen shooting behavior, it is necessary to filter the lock screen, black screen and scan code. The method is to collect pictures of four different screen states in advance, and generate a screen feature library through the image feature extraction component. The screen feature library is loaded into the memory space when the screen shooting behavior detection service is started for vector retrieval. The screen image detected by the image is passed into the image feature extraction component library to generate the corresponding features, and the feature vector is compared with the generated screen feature library in combination with the vector retrieval component library, and the screen state corresponding to the top1 (first) result is output as the current screen recognition result. The advantage of this approach is that the lock screen interface results can be updated at any time by importing a new screen feature library without retraining the detection model.

[0169] Step 5: Match the person object holding the mobile phone and close to the screen through the post-processing component library, input the person information and screen information that meet the requirements of holding the mobile phone, match the hand key points with the screen edge frame, and the matching rule is the pixel distance between the edge frame and the wrist key point. If the pixel distance is less than the preset value, it is determined that the person holding the mobile phone is close to the screen, and filter out the people who do not meet the rule;

[0170] Step six, finally, match the people who take pictures of the screen through the post-processing component library again, input the information of the person who holds the mobile phone and is close to the screen, and match the key points of the person's eyes with the vectors connecting the edge frame of the mobile phone and the edge frame of the screen. The matching rule is that the angle between the two vectors is less than the set threshold. The person who meets this condition is detected to have taken pictures of the screen with the mobile phone.

[0171] Step seven, based on the results obtained from the above model library component analysis, call the conditional trigger component library and set the trigger count. When the event of a person taking a picture of the screen with a mobile phone is detected and the trigger count exceeds the count, the business scenario event is triggered and the mobile phone screen taking event information is generated, including the event image, model library detection results, etc.

[0172] Step 8: Call the result sending component and use kafka (an open source stream processing platform) or Http (Hypertext Transfer Protocol) to request to send the mobile phone screen capture event information to the cloud.

[0173] It should be noted that the above embodiment only lists one business scenario. Other scenarios that meet the image detection model include people staying without reception, leaving without locking the screen, valet operation, people entering without work clothes, and items left behind. Use appropriate splitting methods for different business scenarios, decouple business scenarios and models by calling model library and capability library components, and quickly implement business scenario deployment with simple deployment operations.

[0174] The multimodal large model recognition module performs recognition. For the above-mentioned real-time detection business scenarios, because the sample data is limited when the detection model is trained, the data drift that occurs in actual use will still cause different degrees of misidentification of business scenarios. In the above-mentioned screen shooting behavior detection embodiment, the mobile phone object is a small object in the image and has a low resolution, so it is difficult to recognize. For example, it is easy to misidentify similar objects such as ID cards, water cups, and rectangular foods as mobile phones. Although the recognition accuracy can be effectively improved by increasing the types and number of positive and negative samples of training data, the manpower consumption cost of model annotation and training is huge. The multimodal large model has powerful image understanding and data generalization capabilities. The data drift problem of the traditional model can be effectively curbed by the multimodal large model. The method is to send the image results to be recognized and the business scenario prompt words output by the edge image detection model to the multimodal large model, and use its visual dialogue ability, visual positioning and other capabilities to gradually verify the detection results, and filter out the results with consistent image content and business scenario prompt words and output them.

[0175] Exemplarily, the prompt words in the mobile phone screen shooting embodiment can be: Role: Personnel Behavior Analyst. Background: The user wants to know the behavior of the person in the video image to determine whether the behavior has security risks and data leakage risks. The user may have the action of shooting and recording the screen with a mobile phone. Skills: Object detection and recognition ability, good at identifying objects in the image and locating the position; image analysis ability, good at analyzing the correlation of image objects, and reasoning whether the events occurring in the image are consistent with the prompt words. Prompt words: Behavior: The person uses a mobile phone to shoot the display screen; Necessary elements: People, mobile phones, monitors; Actions: Raise both hands and take pictures. Workflow: 1. Identify objects in the image, 2. Analyze whether the events occurring in the image are consistent with the behavior described by the prompt words. 3. If it is consistent with the prompt words, output the result. If it is inconsistent, explain the reason for the inconsistency and what the person in this image may be doing. Output format requirements: Behavior event: behavior in the prompt words; Detected objects: objects identified as included in the necessary elements of the prompt words; Recognition results: output consistent or inconsistent according to the output of the workflow; Reason: When the result output is inconsistent, explain the reason for the inconsistency.

[0176] In this embodiment, the multimodal large model verification steps are divided into the following steps:

[0177] Step 1: Use the visual positioning capability of the multimodal large model to locate the labels in the preliminary analysis results. If the label is not located, filter out the label object. For example, if the mobile phone identified by the target detection model in the mobile phone screen shooting behavior is not located in the multimodal large model, the mobile phone screen shooting event triggered this time is considered to be an accidental touch, and the triggering reason is the misidentification of the target detection model.

[0178] Step 2: Utilize the visual question answering capability of the multimodal large model to analyze the discrete frame data of the video segment captured in the preliminary recognition result, and send multiple video frames to the multimodal large model. This embodiment uses VILA1.5. Input prompt words are the business scenario prompt words in the preliminary recognition result, and obtain the recognition result that is consistent with the prompt words after the large model analyzes the video frame.

[0179] Step three, combining the results obtained by the large model in the first two steps, the event is judged to have occurred if the two verification results and the preliminary recognition results are consistent. If there is any inconsistency, the multimodal large model will analyze the cause of the inconsistency in detail and write it into the final judgment result.

[0180] Secondly, for complex business scenarios, the use of traditional models sometimes cannot achieve the required detection capabilities. In this case, the multimodal large model can be used as a substitute for the traditional model for further identification. For example, in a certain embodiment, the business scenario requires identification of whether a person is sleeping in the office area. Due to the scarcity of data on samples of sleeping people in the target scenario, it is impossible to meet the model training requirements, and the identification method using skeleton point judgment is difficult and misjudgment is serious. According to the characteristics of sleeping people, the business scenario can be divided into detecting people who have not moved for a long time, and those who are lying or lying in this type of people. The image detection model is used to detect people who have not moved for a long time. This type of event is input into the multi-module large model as the initial analysis result, and prompt sentences related to sleeping actions such as lying down and lying down are added to the prompt words. The large model inference filters out the images of people who are judged to be sleeping, and outputs the results. By embedding the multimodal large model as the back-end of the implementation detection module, the difficulty of implementing complex behaviors in the traditional detection mode is simplified, and the business scenario of target behavior detection is expanded.

[0181] The usage of the above-mentioned multimodal large model module can achieve business scenario deployment with almost zero training cost, and avoid the data drift problem caused by relying solely on traditional model detection. It improves the deployment efficiency while improving the final recognition accuracy. All you need to pay attention to is changing the prompt words to adjust the recognition results of the multimodal large model.

[0182] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the behavior detection method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0183] The present application provides a behavior detection device, which can be an edge device or a cloud device. The behavior detection device includes: at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor, so that the at least one processor can execute the behavior detection method in the above-mentioned embodiment 1.

[0184] Reference below Figure 5, which shows a schematic diagram of the structure of a behavior detection device suitable for implementing the embodiment of the present application. The behavior detection device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The behavior detection device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0185] like Figure 5 As shown, the behavior detection device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the behavior detection device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the behavior detection device to communicate wirelessly or wired with other devices to exchange data. Although the behavior detection device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0186] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0187] The behavior detection device provided by the present application adopts the behavior detection method in the above embodiment, which can solve the technical problem of insufficient recognition accuracy and detection efficiency of behavior detection based on images. Compared with the prior art, the beneficial effects of the behavior detection device provided by the present application are the same as the beneficial effects of the behavior detection method provided by the above embodiment, and the other technical features in the behavior detection device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0188] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0189] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0190] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the behavior detection method in the above-mentioned embodiment.

[0191] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0192] The computer-readable storage medium may be included in the behavior detection device; or may exist independently without being assembled into the behavior detection device.

[0193] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the behavior detection device, the behavior detection device implements the behavior detection method.

[0194] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0195] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0196] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0197] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned behavior detection method, and can solve the technical problems of insufficient recognition accuracy and detection efficiency of behavior detection based on images. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the behavior detection method provided in the above-mentioned embodiment, and will not be repeated here.

[0198] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned behavior detection method when executed by a processor.

[0199] The computer program product provided by this application can solve the technical problem of insufficient recognition accuracy and detection efficiency of behavior detection based on images. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the behavior detection method provided by the above embodiment, which will not be repeated here.

[0200] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A behavior detection method, characterized in that: The behavior detection method is applied to an edge device of an image acquisition device, an image detection model is deployed on the edge device, the edge device is in communication connection with a cloud device, a multimodal large model is deployed on the cloud device, and the behavior detection method includes: Acquire continuous frame image data acquired by the image acquisition device, and input the continuous frame image data into the image detection model; Performing behavior detection based on the continuous frame image data by the image detection model to obtain an initial detection result; The initial detection result is sent to the cloud device, wherein the cloud device performs behavioral detection based on the initial detection result through the multimodal large model to obtain an intermediate detection result, compares the intermediate detection result with the initial detection result to obtain a comparison result, and outputs the comparison result based on the comparison result, the initial detection result and the intermediate detection result.

2. The behavior detection method according to claim 1, characterized in that: The image detection model includes multiple functional modules; Before the step of performing behavior detection based on the continuous frame image data by the image detection model to obtain an initial detection result, the method further includes: Determine a behavior detection service to be executed, and split the behavior detection service into a plurality of detection sub-services, wherein the detection sub-services are services that can be executed by the functional module; Determine the target module corresponding to each of the detection sub-services from each of the functional modules; The step of performing behavior detection based on the continuous frame image data by the image detection model to obtain an initial detection result comprises: Each of the target modules is called through the image detection model, and each of the target modules executes its corresponding detection sub-service based on the continuous frame image data to obtain an initial detection result.

3. The behavior detection method according to claim 2, characterized in that: The step of splitting the behavior detection service into a plurality of detection sub-services includes: Determine a detection process corresponding to the behavior detection service, and split the behavior detection service into a plurality of split sub-services according to each process node of the detection process, wherein each process node corresponds to a unique split sub-service; Traversing each of the split sub-businesses, taking the split sub-businesses as target businesses, and detecting whether the target business is an executable business, wherein the executable business is a business that can be executed by the functional module; If the target service is the executable service, the target service is used as a detection sub-service; If the target business is not the executable business, the target business is split into multiple subordinate sub-businesses, each of the subordinate sub-businesses is traversed, the target business is updated through the subordinate sub-businesses, and the step of detecting whether the target business is an executable business is performed.

4. The behavior detection method according to claim 2, characterized in that: The step of calling each of the target modules through the image detection model includes: Determining the data dependency relationship between each of the target modules, and determining the calling order of each of the target modules based on the data dependency relationship; Each of the target modules is called in sequence through the image detection model according to the calling order.

5. A behavior detection method, characterized in that: The behavior detection method comprises a cloud device, the cloud device is in communication connection with an edge device of an image acquisition device, an image detection model is deployed on the edge device, and a multimodal large model is deployed on the cloud device. The behavior detection method comprises: Receiving an initial detection result sent by the edge device, wherein the edge device obtains continuous frame image data collected by the image acquisition device, and performs behavior detection based on the continuous frame image data through the image detection model to obtain an initial detection result; Performing behavior detection based on the initial detection result through the multimodal large model to obtain an intermediate detection result; The intermediate detection result is compared with the initial detection result to obtain a comparison result, which is based on the comparison result, the initial detection result and the intermediate detection result and outputted.

6. The behavior detection method according to claim 5, characterized in that: The initial detection result includes behavior information corresponding to the continuous frame image data and an edge behavior detection result, wherein the behavior information includes target frame image data in which a dynamic behavior occurs in the continuous frame image data and first description information corresponding to the dynamic behavior; The step of performing behavior detection based on the behavior information through the multimodal large model to obtain an intermediate detection result includes: Performing behavior detection based on the target frame image data through the multimodal large model to obtain second description information and cloud-based behavior detection results, wherein the second description information is used to describe the dynamic behavior; The second description information and the cloud behavior detection result are used as intermediate detection results.

7. The behavior detection method according to claim 6, characterized in that: The step of comparing the intermediate detection result with the initial detection result to obtain a comparison result comprises: Comparing the second description information with the first description information to obtain a first comparison result, wherein the first comparison result indicates whether the second description information matches the first description information; Comparing the cloud behavior detection result with the edge behavior detection result to obtain a second comparison result, wherein the second comparison result indicates whether the cloud behavior detection result matches the edge behavior detection result; A comparison result is generated based on the first comparison result and the second comparison result.

8. The behavior detection method according to any one of claims 5 to 7, characterized in that: The steps based on the comparison result, the initial detection result and the intermediate detection result include: If the comparison result shows that the initial detection result and the intermediate detection result are consistent, the initial detection result and the intermediate detection result are used as the target detection result; If the comparison result shows that the initial detection result and the intermediate detection result are inconsistent, the cause of the abnormality is determined based on the comparison result, the initial detection result and the intermediate detection result, and the cause of the abnormality, the comparison result, the initial detection result and the intermediate detection result are used as the target detection result.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the behavior detection method according to any one of claims 1 to 8 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the behavior detection method according to any one of claims 1 to 8 are implemented.

Images