An end-to-cloud combined behavior recognition method

Abstract

The application relates to the field of artificial intelligence and computer vision technology, and discloses a behavior recognition method combining an edge and a cloud, which utilizes a multi-head lightweight model to perform single inference on a video frame at an edge end to generate preliminary perception data containing a human body and component coordinates; the preliminary perception data is time-series fused through a sliding window and an integrated confidence is calculated, according to which a discard, cache or upload state is determined; an edge device monitors CPU, memory and bandwidth resources in real time, dynamically adjusts a processing and uploading strategy based on resource states and behavior priorities, automatically triggers a load reduction mode when resources are abnormal, and when uploading is confirmed, the most representative key frames are screened and packaged and sent to the cloud end in combination with structured metadata, and the cloud end server constructs prompt text by using the structured metadata. The application balances edge computing power and cloud bandwidth, and realizes high-precision and high-reliability behavior recognition in a resource-limited environment.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer vision technology, specifically to an edge-cloud combined behavior recognition method. Background Technology

[0002] With the rapid development of IoT and AI technologies, intelligent behavior recognition is increasingly being applied in fields such as security monitoring, smart education, and industrial production. Currently, behavior recognition technology mainly manifests in two mainstream forms: edge-side solutions based on dedicated lightweight models and cloud-based solutions based on large models of a universal visual language.

[0003] The first type of approach focuses on real-time performance and low power consumption, typically using lightweight networks such as MobileNet and ShuffleNet as the backbone, combined with detection frameworks such as YOLO or SSD to run on embedded edge devices. Although techniques such as depthwise separable convolution reduce computational complexity and enable real-time inference, this model relies on a predefined closed training set, has weak generalization ability, and is difficult to handle behavior recognition tasks involving complex semantic understanding or abstract intent in open domains.

[0004] The second type of approach utilizes large visual language models (such as Qwen-VL) with strong generalization capabilities to process complex visual reasoning tasks through natural language instructions. However, directly inputting raw video frames into the large model not only faces enormous network bandwidth pressure and cloud computing costs, but also leads to unstable recognition accuracy due to hallucinations or misjudgments caused by distracted attention when processing images with complex backgrounds or subtle targets.

[0005] To balance efficiency and accuracy, a fusion scheme of multiple small models for initial screening and large model for re-examination has emerged in existing technologies. This type of scheme filters most normal frames using small human or face detection models at the edge, only invoking the large model in the cloud for re-examination under specific conditions. While this approach alleviates the computational power conflict, its core architecture still has limitations. First, existing collaborative strategies mostly employ static, fixed serial logic, lacking awareness of the runtime state of edge devices. Regardless of fluctuations in CPU temperature, memory usage, or network bandwidth of edge devices, the system executes an unchanging processing flow. Under extreme conditions of resource constraints such as device overheating and frequency throttling, memory shortages, or network congestion, this rigid strategy can easily lead to a surge in system latency or even service interruption, failing to guarantee the continuity of core business. Second, existing solutions utilize the large model in a relatively crude manner, lacking targeted prior information guidance, making it difficult to address the attention drift problem of the large model in specific scenarios. Furthermore, existing technologies lack a hierarchical resource allocation mechanism for behaviors of varying importance and complexity, resulting in low resource utilization efficiency and making it difficult to achieve the optimal balance between system robustness and recognition accuracy under limited hardware conditions. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides an edge-cloud integrated behavior recognition method. This method solves the problems of rigid computation strategies and lack of awareness of hardware resources and network status of edge devices in existing edge-cloud collaborative behavior recognition schemes, which lead to poor system stability and difficulty in ensuring service continuity under resource constraints or environmental fluctuations.

[0007] To achieve the above objectives, the first aspect of this invention provides an edge-cloud integrated behavior recognition method, which operates in an execution environment including edge devices and cloud servers. The method first performs a single forward inference on the acquired video image frames at the edge using a multi-head lightweight neural network model. It then utilizes a shared feature extraction backbone network to synchronously generate preliminary perception data including human body location, key behavioral component locations, and preliminary behavioral state probabilities, thereby achieving multi-dimensional real-time perception under low computing power conditions.

[0008] This invention further establishes a sliding window queue on the edge device to perform temporal fusion analysis on preliminary perception data from multiple consecutive frames. This analysis process calculates a comprehensive confidence score reflecting the probability of a behavior occurring by statistically analyzing the presence and interaction of various elements within the time window. Based on the comparison of this comprehensive confidence score with preset low and high thresholds, the system classifies the current event state into a discard state, a cached observation state, or a confirmed upload state. By employing a three-level state determination mechanism, transient noise and invalid data can be filtered out, retaining only potentially valuable behavioral fragments and reducing the false alarm rate.

[0009] To address the issue of limited hardware resources in edge devices, this invention introduces a resource-aware dynamic scheduling strategy. Edge devices periodically monitor hardware resource status parameters such as CPU temperature, memory usage, and available network bandwidth. The system, combined with preset behavior priority levels (including first-level, second-level, and third-level behaviors), dynamically adjusts processing logic based on real-time hardware status. When abnormal resource status is detected, the system activates corresponding load reduction modes: in memory protection mode, memory is released by pausing new event caching, reducing the sliding window length, and enabling a high compression ratio; in low bandwidth mode, structured metadata consisting of detection frames and timestamps is uploaded first, and image data is uploaded only after instructions are issued from the cloud; in temperature-controlled load reduction mode, the video capture frame rate is reduced and the temporal fusion window is extended to prioritize the detection logic of high-priority behaviors.

[0010] To reduce network transmission load and improve cloud inference efficiency, this invention performs intelligent keyframe selection upon triggering an upload. This mechanism does not upload all cached data, but rather selects the most representative keyframe images based on a weighted evaluation of normalized confidence scores and normalized behavior change scores. Subsequently, the system performs data reduction processing on the keyframes based on lossy compression or human detection bounding box region cropping, and then encapsulates the processed images with structured metadata before uploading.

[0011] In the cloud processing stage, this invention utilizes a large visual language model for refined judgment. The cloud server parses the received data packets, extracts the detection box coordinates, timestamps, and preliminary behavior labels from the structured metadata, and converts them into natural language description strings using a formatted mapping function. This description string is then concatenated with a preset general task description template to form natural language instruction text. This text serves as a prompt, guiding the large visual language model to focus its attention on specific spatiotemporal regions in the image. Combined with keyframe images, multimodal semantic analysis and reasoning are performed, ultimately outputting high-precision behavior recognition results.

[0012] To achieve the above objectives, a second aspect of the present invention also provides a computer device, comprising: a processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method described in any of the first aspects.

[0013] This invention provides an edge-cloud combined behavior recognition method. It has the following beneficial effects: 1. This invention adopts an edge-cloud collaborative architecture that combines lightweight edge screening with large-scale cloud model-based fine judgment. On the edge side, a lightweight multi-task model and a temporal confidence fusion mechanism are used to filter out a large number of invalid background frames and instantaneous noise. Key frames and structured metadata are only uploaded when the event confidence meets the threshold. This overcomes the computing power bottleneck of edge devices and reduces network bandwidth usage and cloud computing costs compared to the full video stream upload solution, achieving an organic combination of low-latency real-time perception and high-precision semantic understanding.

[0014] 2. This invention introduces a resource-aware dynamic scheduling strategy to monitor the CPU temperature, memory usage, and network bandwidth status of edge devices in real time, and dynamically adjusts the processing logic and transmission strategies of L1, L2, and L3 behaviors accordingly (such as enabling temperature-controlled load reduction, memory protection, or low-bandwidth tiered upload mode). This ensures that even under extreme conditions such as hardware overheating, insufficient memory, or network congestion, the system can still prioritize the identification and reporting of core high-risk behaviors (L1 level), avoiding system crashes or service interruptions caused by resource exhaustion.

[0015] 3. This invention utilizes structured metadata (detection box coordinates, preliminary behavior labels) generated at the edge to construct natural language prompts, guiding the attention of the cloud-based visual language large model to focus on specific spatiotemporal regions in the image. It not only utilizes the open-domain generalization ability of the large model to handle complex semantics (such as "whether homework is done seriously"), but also suppresses the hallucination phenomenon and background noise interference of the large model through edge prior information, thereby improving the recognition accuracy of long-tail scenes and subtle action interactions. Attached Figure Description

[0016] Figure 1 This is an overall flowchart of an edge-cloud combined behavior recognition method according to the present invention; Figure 2 This is a diagram illustrating the application environment and system architecture of the edge-cloud combined behavior recognition method of the present invention. Figure 3 This is a schematic diagram of the edge confidence fusion and event state determination logic of the present invention; Figure 4 This is a flowchart illustrating the logic control of the resource-aware scheduling module of the present invention. Figure 5 This is a schematic diagram of the three-level behavior recognition of the present invention. Detailed Implementation

[0017] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] See attached document Figure 2 The behavior recognition method that combines edge and cloud computing provided by this invention can run in an execution environment that includes edge devices and cloud servers.

[0019] The edge device can be a computing device with image acquisition and data processing capabilities, such as a camera or edge computing gateway integrated with a neural network processing unit (NPU). In this embodiment, the edge device is used to perform the edge-side processing steps of the behavior recognition method of the present invention.

[0020] The edge device may be configured with functional modules to perform specific steps of the behavior recognition method of the present invention. These functional modules may include: The lightweight multi-task perception module is configured to receive acquired image data and perform forward inference on the image data to generate preliminary perception data. The preliminary perception data includes: the location information of the target human body, the location information of one or more key behavioral components (such as hands, desktops, mobile phones), and the preliminary behavioral state discrimination results.

[0021] The event caching and fusion module is configured to receive preliminary sensing data from multiple consecutive time frames from the aforementioned lightweight multi-task sensing module. This module performs time-series analysis on the continuous preliminary sensing data to calculate a comprehensive confidence score, and based on the comparison result of the comprehensive confidence score with a preset threshold, executes a decision to discard data, cache data, or trigger an upload.

[0022] The resource-aware scheduling module is configured to periodically acquire one or more hardware resource status parameters of the edge device, such as CPU temperature, memory usage, and network bandwidth. Based on the acquired hardware resource status parameters, this module dynamically adjusts the execution strategy of the behavior recognition task, such as adjusting the data processing priority or the data upload method.

[0023] The keyframe selection and compression upload module is configured to select keyframe images from cached data according to preset rules in response to decisions triggering uploads and permission commands for resource scheduling. This module performs data reduction processing (e.g., lossy compression or region cropping) on ​​the selected keyframe images, encapsulates the processed image data and corresponding structured metadata into a data packet, and finally uploads it to the cloud server via a network interface.

[0024] The cloud server can be a server or server cluster with high-performance computing resources. In this embodiment, the cloud server is used to receive data uploaded by the edge device and execute the cloud-side processing steps of the behavior recognition method of the present invention.

[0025] The cloud server is configured with a large-scale model analysis module, which receives and parses data packets uploaded from edge devices. These packets may contain image data and structured metadata. This module uses a visual language model to analyze the image data and outputs the final behavior recognition result.

[0026] Edge devices establish a communication connection with cloud servers via a network for data transmission. The network can be a wired or wireless network, such as, but not limited to, Ethernet, Wi-Fi, 4G, or 5G networks.

[0027] See attached document Figure 1 This method can run in execution environments including edge devices and cloud servers.

[0028] S1. The behavior recognition method first performs a lightweight multi-task perception step on the edge device. In this step, the edge device acquires video image frames through the image acquisition unit and uses the lightweight multi-task neural network model deployed on it to perform a single forward inference on the image frames. This inference aims to simultaneously generate multi-dimensional preliminary perception data, which may specifically include the position coordinates of the target human body, the position coordinates of one or more preset key behavioral components (such as hands, desktop areas, mobile phones), and the preliminary classification confidence of the behavior state of the current frame.

[0029] S2. Next, the behavior recognition method performs edge confidence fusion and event caching steps. This step aims to establish temporal consistency in behavior judgment to avoid noise or false detections that may exist in single-frame images. Specifically, the edge device stores the temporally continuous preliminary perception data generated by the lightweight multi-task perception step into a sliding window queue. By performing temporal fusion analysis on the preliminary perception data in the sliding window queue, a comprehensive confidence score is calculated. Subsequently, the method performs a three-level decision based on the comparison result of the comprehensive confidence score with one or more preset thresholds: if the score is lower than the low threshold, it is determined as an invalid event and the relevant data is discarded; if the score is between the low threshold and the high threshold, the relevant metadata is stored in the cache and continuously observed; if the score is higher than the high threshold or the continuous observation time exceeds the preset duration, an upload decision is triggered.

[0030] S3. After triggering an upload decision or simultaneously with it, the behavior recognition method executes a resource-aware dynamic scheduling step. This step is a prerequisite for determining whether and how data can be uploaded to the cloud. Edge devices monitor their own hardware resource status in real time, including but not limited to CPU temperature, memory usage, and available network bandwidth. The behavior recognition method compares the real-time acquired resource status with preset resource thresholds and combines this with a preset complexity level of the behavior to be identified to jointly adjudicate the triggered upload decision. The adjudication result may be: allow upload, prohibit upload, or adjust the data transmission strategy based on allowing upload, such as reducing the quality of the image to be uploaded or uploading only metadata.

[0031] S4. Once the upload decision is approved, the keyframe selection and compression upload module performs the intelligent keyframe selection and compression upload step. This step aims to reduce network transmission load. The keyframe selection and compression upload module does not upload all data in the cache, but rather selects the most representative keyframe images of behavioral information from the cached data according to preset selection rules. Selection rules can be based on the overall confidence score or the degree of behavioral change. After selecting keyframes, the keyframe selection and compression upload module performs image processing on the keyframe images, such as lossy compression or region cropping based on target location information. Finally, the processed image is packaged with structured metadata containing information such as timestamps and detection box coordinates, and uploaded to the cloud server over the network.

[0032] S5. Finally, the behavior recognition method executes the cloud-based large-scale model's fine-tuning step on the cloud server. The cloud server receives and parses data packets uploaded from edge devices. The visual language large-scale model deployed on the cloud server uses the received keyframe images as visual input, and can convert the structured metadata in the data packets into text or visual cues to help the model focus its attention on key regions in the image. Based on its multimodal understanding capabilities, this visual language large-scale model performs deep semantic analysis and reasoning on behavior, and outputs the final, high-precision behavior recognition result.

[0033] In step S1, the multi-head lightweight neural network model deployed on the edge device is implemented by using a shared feature extraction backbone network and multiple parallel, task-specific prediction heads. This architecture design ensures that feature extraction computation for a single input image frame only needs to be performed once, and the resulting feature map can be reused by all subsequent prediction heads, thereby reducing the total computational load and memory access requirements when performing multi-task analysis on resource-constrained edge devices.

[0034] The shared feature extraction backbone network transforms input 2D image frame data into multi-level feature maps containing rich semantic information through a series of convolution, pooling, and non-linear activation operations. To ensure low-latency operation of the model on edge devices, this backbone network can be constructed using lightweight network architecture units. In one embodiment, the backbone network can integrate depthwise separable convolutions to replace standard convolutions, reducing the number of parameters and computational cost. In another embodiment, the backbone network can also employ architecture units with a channel shuffle mechanism to maintain information interaction between channels while reducing computational complexity.

[0035] Multiple parallel, task-specific prediction heads receive feature maps from the output of a shared backbone network and each performs independent computations to accomplish its specific task.

[0036] One of the prediction heads can be configured as a detection head. The function of this detection head is to identify and locate target objects in an image based on the input feature map. Specifically, the detection head can output a list of one or more bounding boxes. Each bounding box consists of a set of coordinate values ​​(e.g., the coordinates of the top-left vertex and its width and height) and a category label, used to identify the location of a human body in the image, as well as the locations of key behavioral components such as hands, desktops, or mobile phones.

[0037] Another prediction head can be configured as a state discriminator. This state discriminator classifies the entire image or a specific region within an image based on the input feature map. Specifically, it first aggregates the feature map into a feature vector using a global average pooling layer, then inputs this feature vector into one or more fully connected layers, ultimately outputting a one-dimensional vector containing multiple confidence scores. Each score in this one-dimensional vector corresponds to the probability of a predefined initial behavioral state (e.g., "suspected of doing homework" or "suspected of playing on a mobile phone").

[0038] Therefore, in a complete forward inference process, the input image frame first flows through a shared feature extraction backbone network to generate a feature map, which is then simultaneously fed into the detection head and the state discriminator head. After parallel processing by the two prediction heads, the multi-head lightweight neural network model synchronously outputs a set of structured preliminary perceptual data for each frame image, namely, the bounding box information of the human body and key components, and the confidence score vector of the preliminary behavioral state. This structured data will serve as the input for subsequent steps.

[0039] After completing a single forward inference pass, the multi-head lightweight neural network model generates standardized perceptual data output. This output data is not a single classification label, but a structured dataset containing spatial location information, category information, and probability values, which serves as the basic input variables for subsequent temporal fusion steps.

[0040] For any current processing frame in the video stream The perception dataset output by the lightweight multi-task perception module Specifically, it consists of three subsets: human target detection data, key component detection data, and preliminary behavioral state vectors. This perception dataset The mathematical expression is as follows: ; in, Represents the current processing frame The generated complete perception dataset; Represents the current processing frame Human target detection data in the middle; Represents the current processing frame Key component detection data; Represents the current processing frame The initial behavioral state vector.

[0041] Human target detection data This data represents the spatial location and presence confidence of the person identified as the subject in the current frame. It is defined as a vector containing five values: ; in, Represents the pixel coordinates of the top-left corner of the human body bounding box; These two coordinates represent the pixel coordinates of the lower right corner of the human body bounding box; together, they define the rectangular area of ​​the human body in the image. This represents a confidence score indicating that the region contains a human body, with a value ranging from 0 to 1. This data is used in subsequent steps to calculate the human presence rate.

[0042] Key component testing data This data characterizes the spatial distribution of objects related to a specific action in the current frame; it is defined as a set containing... One detected object instance: ; Represents the first in the set The specific format of the detected key components is as follows: ; in, This is a category index value used to indicate which category the component belongs to: "hand", "desktop area", or "mobile electronic device". and These represent the pixel coordinates of the top-left and bottom-right corners of the component's bounding box, respectively. This represents the detection confidence score for the component. These coordinate data are used for subsequent calculations to determine spatial geometric relationships, such as whether the hand is within the tabletop area.

[0043] Initial Behavioral State Vector This characterizes the preliminary probability estimate of the behavior category in the current frame by a multi-head lightweight neural network model. This preliminary behavior state vector is directly output by the classification prediction head and does not contain spatial coordinate information; it is defined as a... A probability distribution vector of dimension 1: ; in, This represents the total number of predefined categories of behaviors to be identified (e.g., categories such as "suspected of doing homework", "suspected of playing on the phone", "suspected of leaving the post"). This indicates that the current frame belongs to the first... The normalized probability value of class behavior, and satisfying The probability values ​​provided by this initial behavioral state vector serve only as an instantaneous state reference and do not directly trigger alarms. Instead, they are input as raw values ​​into the subsequent sliding window queue for time-series smoothing.

[0044] See attached document Figure 3 In step S2, the edge device performs edge-end confidence fusion and event caching on the continuous multi-frame detection results output in step S1. This step aims to calculate a comprehensive confidence score by statistically analyzing the presence and interaction of elements within a certain time window. Based on the comprehensive confidence score, the current behavior status is graded to determine whether to discard the data, cache only the metadata, or trigger an upload.

[0045] The program first maintains a length of [length] in the memory of the edge device. Sliding window queue Used to store the most recent The system detects the results of each frame. For each frame, the system extracts the confidence information of the human body detection bounding box, the desktop detection bounding box, and the hand detection bounding box. Then, the system uses a sliding window queue... In Frame data, calculate three sub-indicators: human presence rate Desktop Presence Rate and hand activity .

[0046] Human Presence Rate Defined in the sliding window queue of In the frame data, the number of frames with a human detection confidence level greater than the preset detection threshold accounts for... The proportion.

[0047] Desktop Presence Defined in the sliding window queue of In the frame data, the number of frames with a desktop detection confidence score greater than a preset detection threshold accounts for a certain percentage of the total number of frames. The proportion.

[0048] Hand activity Defined in the sliding window queue of In the frame data, the number of frames in which the hand is located near the table and the detection confidence is greater than the preset detection threshold accounts for a certain percentage of the total number of frames. The proportion.

[0049] Based on the human body presence rate Desktop Presence Rate and hand activity The system uses a weighted summation method to calculate the overall confidence score for the current time window. The calculation formula is as follows: ; in, Represents the overall confidence score; Represents the prevalence of human presence; Represents desktop presence rate; Represents hand activity level; Weighting coefficients representing the prevalence of human presence; Weighting coefficients representing desktop presence rate; Weighting coefficients representing hand activity; and All are non-negative real numbers and satisfy the normalization condition. The overall confidence score Characterized in recent The overall probability of the target action occurring within a frame time.

[0050] Calculate the overall confidence score Then, the system compares it with a preset low threshold. and high threshold The comparison is performed, and a three-tiered triage decision-making strategy is implemented. Low threshold. and high threshold Satisfying Relationships .

[0051] When the overall confidence score Less than the low threshold (Right now When the system determines that the current behavior is either non-target behavior or normal behavior, it will not perform a data saving operation and will directly discard the relevant data in the sliding window to free up memory resources.

[0052] When the overall confidence score Greater than or equal to the low threshold And less than the high threshold (Right now When the system determines the current state to be a cached observation state, the edge device enters a low-power recording mode and temporarily stores the most recent data. The frame's metadata is stored in a local cache, but no image data is uploaded at this time. The metadata includes bounding box coordinates and timestamps, but does not include raw pixel data. The preset number of buffered frames, its value is independent of .

[0053] The system determines event confirmation and triggers an upload decision when either of the following two triggering conditions is met: Condition 1: The calculated overall confidence score Greater than or equal to the high threshold (Right now ); Condition 2: The system remains in the cached observation state for a duration exceeding a preset time threshold. .

[0054] Once an upload is triggered, the system will base its actions on the cache. The frame metadata and the corresponding keyframe images are used to perform subsequent resource scheduling and uploading processes.

[0055] See attached document Figure 5 Based on the determination of event status (including discard, cache observation, and confirmed upload) in step S2, step S3 further elaborates on the resource-aware dynamic scheduling strategy. To achieve fine-grained management of edge devices under different load pressures, this embodiment pre-divides the behavioral tasks that the devices need to process into three different priority and complexity levels, namely L1 level behavior, L2 level behavior, and L3 level behavior.

[0056] Level L1 behavior (Level 1 behavior) is defined as a critical event with the highest priority. This level corresponds to the overall confidence score in step S2. satisfy The duration of a conditional or cached observation exceeds The triggered confirmation upload event. L1 level behavior represents events of high credibility or that must be reported.

[0057] Level L2 behavior (second-level behavior) is defined as an observable event with medium priority. This level corresponds to the overall confidence score in step S2. satisfy The conditions, and the duration has not yet exceeded The cached observation events. L2 level behavior represents potential events that require continuous monitoring but have not yet been definitively confirmed.

[0058] Level 3 (Level 3) behaviors are defined as background analysis tasks with the lowest priority that are computationally intensive or data-intensive. Such tasks (e.g., non-critical long-term attitude statistics or detailed interaction analysis) are independent of the immediate event (L1, L2) judgments in step S2. They have high computational overhead, but their real-time performance and priority are lower than those of Level 1 and L2 behaviors.

[0059] For L3-level behavior detection logic, cloud-based processing is the primary method, with edge devices only performing preliminary sensing. When a user actively triggers the function command, the system first assesses the current hardware resource status. If resources are sufficient (i.e., no load reduction mode is activated), the edge device will collect key segments from multiple consecutive frames and perform batch packaging and uploading. The visual language model then combines multi-frame temporal information and external knowledge for in-depth analysis to generate corresponding intent-level behavior detection content (e.g., determining "whether the user is diligently doing homework" or "what the user might do next").

[0060] Based on the above L1, L2, and L3 three-level behavior classification, edge devices monitor their main hardware status in real time, including memory usage, network bandwidth, and CPU temperature, and execute corresponding dynamic load reduction strategies.

[0061] When the edge device detects that memory usage exceeds a preset high memory threshold At this time, the system enters memory protection mode.

[0062] In memory protection mode, to release memory resources, the system will perform one or more of the following operations: clear the event cache queue that has entered cached observation state (L2 level behavior); shorten the time in step S2. ; Pause caching of new L2-level behavior events; Restore the sliding window queue defined in step S2 length The system reduces the frame rate to a preset minimum (e.g., 3 frames) to minimize memory usage. Simultaneously, it enables memory compression, storing necessary image data at a higher compression ratio than normal. Once memory usage falls below a preset minimum and remains below that level for a preset duration (e.g., 10 seconds), the system reverts to the default caching strategy.

[0063] When the edge device detects that the network bandwidth is lower than the preset low network bandwidth threshold At this time, the system enters low bandwidth mode.

[0064] In low bandwidth mode, the system implements a hierarchical data upload strategy: suspends data uploads for all L3-level behaviors; and suspends some image data uploads for L2-level behaviors.

[0065] When the edge device detects that the CPU temperature exceeds the preset CPU high temperature threshold At that time, the system enters the temperature-controlled load reduction mode.

[0066] In temperature-controlled load reduction mode, in order to prioritize the stable operation of core functions, the system will suspend all prediction calculations for L3-level behaviors, while retaining the prediction and judgment logic for L1-level and L2-level behaviors.

[0067] The resource-aware scheduling in step S3 relies on real-time and accurate monitoring of the hardware status of edge devices. This monitoring mechanism runs as a separate background process with a preset sampling period. The system polls and collects data on the device's memory status, network environment, and thermal status, and generates standardized state vectors for use by the scheduling algorithm.

[0068] For memory status monitoring, the system reads kernel statistics from the operating system to obtain the total amount of physical memory currently occupied by applications and cache. Edge devices calculate memory usage. This memory usage rate serves as the basis for determining whether to enter memory protection mode. The calculation formula is: ; in, This represents the current memory usage percentage. This represents the number of bytes of physical memory currently used by the system. This value includes the memory used for running multi-head lightweight neural network models and maintaining sliding window queues. The amount of memory consumed; This represents the total number of bytes of physical memory configured for the edge device.

[0069] For network bandwidth monitoring, the system monitors the average uplink throughput of the data transmission interface within a sliding time window to calculate the current available network bandwidth. This metric is used to determine whether a low-bandwidth mode has been triggered. The available bandwidth of this network. The calculation formula is: ; in, This represents the calculated available network bandwidth, measured in megabits per second (Mbps). Represents a time window; Represents the time window The total number of data packets transmitted internally; Representing the The size of each data packet, in megabits; Represents the time window Internal transmission The total amount of data transmitted within the time window is obtained by summing the sizes of the individual data packets.

[0070] The system will calculate the results With network low bandwidth threshold Compare them.

[0071] For CPU temperature monitoring, the system directly reads the thermal sensor values ​​from the central processing unit (CPU) core via a hardware interface. To prevent frequent switching between temperature control and normal modes due to instantaneous temperature fluctuations, the system uses an exponentially weighted moving average (EWMA) algorithm to calculate a smoothed CPU temperature value. The calculation formula is as follows: ; in, Represents the current monitoring sampling point The calculated smoothed CPU temperature value is used in conjunction with the CPU high temperature threshold. Compare; Represents the current monitoring sampling point Raw instantaneous temperature value read from the thermal sensor; This represents the previous monitoring sampling point. The calculated smoothed CPU temperature value; This represents the smoothing coefficient, and its value ranges from 0 to... <1. This coefficient determines the weight of the current measurement value in the final result.

[0072] The system integrates the data from the above three dimensions in each sampling period. Output the current device's state vector when finished. : ; This state vector The data is transmitted in real time to the resource-aware scheduling module, which then performs the necessary checks based on the aforementioned logical rules (i.e., ...). , , ), dynamically adjust the processing and uploading strategies for L1, L2 and L3 level behaviors.

[0073] See attached document Figure 4 Based on the state vector of the edge device (Include , , The document details how the resource-aware scheduling module executes dynamic scheduling strategies based on state vectors, including real-time acquisition of data and the definition of L1, L2, and L3 behaviors.

[0074] The resource-aware scheduling module obtains the latest state vector. Then, compare each of the indicators with the preset thresholds ( , , The comparison is performed to determine whether one or more load reduction modes are activated.

[0075] Under the default normal operating conditions, that is, when and and At that time, the edge device will fully execute all the functions defined in steps S1 to S2, including all prediction, caching and uploading logic for L1, L2 and L3 level behaviors.

[0076] When the resource-aware scheduling module detects the state vector Memory usage Meet the conditions When this happens, the system immediately activates memory protection mode.

[0077] In this memory protection mode, to release memory resources, the system will perform one or more of the following operations: clear the event cache queue currently occupied by L2-level behavior (cache observation state); suspend receiving new L2-level behavior into the cache; dynamically shorten the sliding window queue defined in step S2. of ; and when an upload is triggered by an L1-level action, increase the compression ratio of the image data.

[0078] When the resource-aware scheduling module detects the state vector Available network bandwidth Meet the conditions At that time, the system activates low bandwidth mode.

[0079] In this low-bandwidth mode, the system implements a tiered data upload strategy to ensure the transmission of Level 1 (critical events). Specific operations include: completely suspending data uploads for all Level 3 activities (background analysis tasks); and suspending partial image data uploads for Level 2 activities (cached observation states). Regarding the suspension of partial image data uploads for Level 2 activities, the specific implementation logic is as follows: edge devices prioritize uploading only structured metadata such as recognition detection boxes and timestamps to the cloud, without uploading image data. After receiving the metadata, the cloud server evaluates it. If the cloud determines that further verification of visual details is needed, it sends an image request command to the edge device. The edge device only uploads the image frame corresponding to the event after receiving this command. Furthermore, if the network bandwidth is detected to exceed the preset bandwidth fluctuation limit for a consecutive preset duration (e.g., 10 seconds), the system will automatically resume the normal image upload strategy.

[0080] When the resource-aware scheduling module detects the state vector Smoothing CPU temperature values Meet the conditions At that time, the system activates the temperature-controlled load reduction mode.

[0081] In this temperature-controlled load reduction mode, the system prioritizes pausing computationally intensive tasks to reduce CPU load and heat generation. Specifically, it completely suspends the prediction and computation of all L3-level behaviors (background analysis tasks). Simultaneously, the system retains the complete prediction and judgment logic for L1-level behaviors (critical events) and L2-level behaviors (cached observation states), ensuring that the core event detection function is unaffected. Furthermore, in this temperature-controlled load reduction mode, the system sends control commands to the image acquisition unit to reduce the frame rate of video acquisition and correspondingly extend the coverage time of the temporal fusion window in step S2, thereby maintaining effective capture of behavioral temporal features while reducing computation frequency.

[0082] The aforementioned load reduction mode is deactivated following an automatic recovery strategy based on time lag to prevent frequent system switching under critical conditions. Specifically: For the temperature-controlled load reduction mode, the system must monitor the CPU temperature to be below the preset lower limit and maintain it for a preset duration (e.g., 30 seconds) before automatically restoring the default frame rate and timing fusion window length. For low bandwidth mode, the network bandwidth needs to be monitored to exceed the bandwidth fluctuation limit and remain at this level for a preset time (e.g., 10 seconds) before resuming normal image upload. For memory protection mode, the default cache window length and compression strategy should be restored after the memory usage rate is detected to be less than the lower limit of memory usage and remains so for a preset time (e.g., 10 seconds).

[0083] When the system determines that an upload decision has been triggered, i.e., the L1 level behavior is confirmed, the keyframe selection and compressed upload module of this invention executes step S4, namely intelligent keyframe selection and compressed upload. This step first filters the cached event data to extract the most representative keyframes, thereby reducing the amount of data transmission while retaining the core features of the event.

[0084] The system accesses the data temporarily stored for this event in step S2, which includes... The event buffer for frame metadata. The intelligent keyframe selection algorithm is designed to extract data from this buffer. In the frame metadata, select The most representative keyframes are used for subsequent uploads.

[0085] In this definition, The preset target number of keyframes, are positive integers and satisfy ≤ conditions.

[0086] To achieve this selection, the algorithm is for the cache. Each frame in the frame metadata (where sequence index) From 1 to Calculate a comprehensive representative score This comprehensive representative score The calculation formula is obtained by weighted summation of the confidence score and behavior change score of the frame: ; in, This represents the sequence index of a frame in the event buffer, with a value range of 1. ; Representing the The overall representative score of the frame; The weighting coefficients representing the confidence score; The weighting coefficients representing the scores for behavioral change; and and All are non-negative real numbers and satisfy the normalization condition. ; Representing the Normalized confidence score of the frame; Representing the The score for changes in the normalized behavior of frames.

[0087] Normalized confidence score Based on the original confidence level of this frame Calculated. Specifically, this refers to the initial behavioral state probability generated in step S1 for this frame. The system normalizes the cache space using a normalization operation. Each frame Value mapping to Interval.

[0088] Normalized behavioral change score Changes in behavior based on the original frame Calculated. Used to quantify the Frame and the previous frame (i.e., the first frame) The differences in behavioral states between frames represent key points of behavioral change (such as the start or end of an action).

[0089] Defined as the first The center coordinates of the key detection box (taking the hand as an example) in the frame With the Center coordinates of the corresponding detection box in the frame The Euclidean distance between them. The formula for calculation is: ; in, Representing the The original behavior change score of the frame; for the first frame in the buffer (i.e. Since there is no previous frame, its Set to 0; Representing the The center coordinates of the frame key detection box, its coordinate value is ; Representing the The center coordinates of the frame key detection box, its coordinate value is The system normalizes the cache space using a normalization operation. Each frame Value mapping to Interval.

[0090] After the system has calculated all the data in the cache... Frame's overall representativeness score After that, for all Sort the values ​​in descending order and select the highest-ranking value. One frame. The metadata and original image frame corresponding to each frame are identified as keyframes and transmitted to the keyframe selection and compression upload module.

[0091] In step S4, for After selecting the keyframes, the discussion continues with... The specific implementation involves compressing the image data of keyframes and encapsulating it with structured metadata before uploading.

[0092] The keyframe selection and compression upload module selects... Each frame of the original image data undergoes a data reduction operation.

[0093] In one implementation, the data reduction operation is lossy compression. The keyframe selection and compression upload module uses the JPEG encoding standard. The original image of the frame is compressed. During JPEG compression, the keyframe selection and compression upload module uses a preset quality factor. .

[0094] This represents the quality factor of JPEG compression, and its value ranges from 0 to 100. In this embodiment, It is set to 50 to reduce the image file size while maintaining basic image recognizability.

[0095] In another implementation of data reduction, the data reduction operation is region cropping. The system first retrieves the key detection boxes (e.g., human detection boxes) generated in step S1 for the keyframe. The system is based on a human detection box. The coordinates are determined, and a preset extension margin is applied. Calculate the target cropping region The calculation formula is as follows: ; in, This represents the calculated target cropping area; Human body detection frame The coordinates of the top left corner; Human body detection frame Width and height; This represents the preset margin used to preserve the human body and surrounding area. The system then extracts only the area around the human body. Pixel data within the area is collected, and all pixel data outside the area is discarded.

[0096] After processing After generating the image data of each frame (i.e., generating compressed or cropped images), the system prepares the corresponding structured metadata for each keyframe. .

[0097] Structured metadata Includes: the detection box coordinate data of the frame generated in step S1 The original acquisition timestamp of this frame. ; and the preliminary behavior label (e.g., specifically the state probability) determined by the lightweight multi-task perception module in step S1. ).

[0098] The system will Image data after frame processing (compressed JPEG data or cropped pixel data) and its corresponding structured metadata Encapsulate and generate a data packet. The data packet Include Each image metadata pair.

[0099] Ultimately, the edge device transmits data packets through its network communication interface. Uploaded to a pre-set cloud server for further analysis, verification, or storage in the cloud.

[0100] Step S5 is primarily executed by a cloud server deployed with high-performance computing resources. The cloud server receives and parses the data packets. Extracting the contents The keyframe image sequence and its corresponding structured metadata are used to perform open-domain behavior understanding and refined judgment using a large visual language model.

[0101] The Visual Language Model (VLM) used in this embodiment is configured as a general-purpose model with multimodal understanding capabilities, and its parameter count is set to over one billion. This VLM can simultaneously receive image data and text data as joint input, and perform open-domain semantic understanding, behavior description, object localization, and complex logical reasoning on image content based on natural language instructions.

[0102] The cloud server first constructs input data pairs for model inference. Based on a pre-defined prompt template and extracted structured metadata, the system generates a natural language instruction text. This natural language instruction text is designed to guide large visual language models to focus on specific behavioral features in images or to verify preliminary behavioral labels uploaded by edge devices.

[0103] Subsequently, the cloud server will Keyframe image sequences and natural language instruction text The inputs are fed into the visual language model. The visual language model encodes multimodal features and calculates interaction attention through its internal deep neural network, outputting the final behavioral description. The mapping relationship of this reasoning process can be expressed by the formula: ; in, The final behavioral result representing the output of the visual language large model is in the form of a natural language text description that includes semantic understanding and behavioral judgment. The nonlinear inference function representing the large visual language model characterizes the computational process by which the large visual language model maps multimodal inputs to the text space. Represents the original image; Represents data packets Extracted, containing The keyframe sequence of frame image data, where For sequence index, This represents the total number of keyframes. The text represents the constructed natural language instructions.

[0104] Executing nonlinear inference functions During the process, the visual language big model utilizes its extensive pre-trained knowledge base to analyze keyframe sequences. The spatiotemporal features are extracted and combined with The semantic constraints in the image are used to identify the interaction relationships between subjects, the environmental background, and subtle motion features, thereby generating images with high-level semantic information that exclude false alarms. .

[0105] This embodiment further illustrates the implementation logic of using structured metadata to assist large models in accurate reasoning. After receiving the data packet, the cloud server not only decodes the image data but also parses the accompanying structured metadata and converts it into contextual clues in natural language form to guide the attention mechanism of the visual language large model, thereby achieving focused analysis of specific spatiotemporal regions.

[0106] The cloud server uses prompting engineering algorithms to construct natural language instruction text input to a large visual language model. The natural language instruction text is not static; rather, it is constructed by concatenating a pre-defined general task description template with a scene description field dynamically generated based on metadata. The calculation formula for this construction process is as follows: ; in, This represents the final constructed natural language instruction text used as input to the large visual language model; This represents a pre-defined general task description template, which includes the role definition and output format requirements for the visual language big model (e.g., "Please analyze the behavior of the people in the following image sequence and determine whether there are any anomalies..."). Represents a string concatenation operator; Representative to The description text corresponding to each keyframe is concatenated in sequence, where The value range is 1 to ; This represents a formatting mapping function used to convert structured data into a description string that conforms to the syntax of a natural language; Representing the The coordinate data of the detection box in the keyframe, which is generated at the edge device. Representing the The original acquisition timestamp of the keyframe; Representing the The initial behavior label generated on the edge device in the frame keyframe corresponds to the label in step S1. The highest behavioral category.

[0107] Through format mapping functions Structured metadata is converted into semantically meaningful prompt text. For example, for a given frame, Converted to text description "in area [ , , ]” Converted to "at a point in time" " Transformed into "preliminary detection as behavior" ".

[0108] The visual language big model receives information containing the above. Then, visual attention focusing based on text prompts is performed. The large-scale visual language model utilizes... The coordinate description in the input keyframe image Locate the corresponding pixel region in the middle, thereby concentrating computing resources on The subject area is defined to exclude background noise interference.

[0109] Meanwhile, the visual language big model utilizes Preliminary behavioral labels As a prior hypothesis, the visual language big data model compares the consistency between image features and this prior hypothesis during the reasoning process: if the image features support the hypothesis, it outputs a confirmation result; if the image features show subtle semantic conflicts that contradict the hypothesis (e.g., misidentifying "bending down to pick up" as "falling down" at the edge), the visual language big data model corrects the behavioral result based on its open-domain understanding ability, thus outputting the final behavioral result. .

[0110] In an alternative embodiment of the present invention, the data compression and encapsulation method in step S4 is modified. Instead of uploading the compressed original image, the edge device uploads a depth feature map extracted by the lightweight multi-task perception module (step S1) during the inference process.

[0111] Specifically, when step S2 determines that an L1-level event has been triggered, and the keyframe selection algorithm in step S4 selects the keyframe... After the keyframe, the system accesses the intermediate layers of a lightweight neural network (such as a CNN) to extract the relevant data. Feature map tensors corresponding to keyframes This feature map It is the original image. A compact data representation that has been extracted through multiple convolutions and already contains rich semantic information.

[0112] Accordingly, the data encapsulation process in step S4 is modified to: Frame feature map tensor With structured metadata (Including detection frames, timestamps, and preliminary labels) Packaged into a data package .because The data volume is much smaller than the original image. This approach further reduces the uplink bandwidth usage on edge devices.

[0113] On the cloud server (step S5), the large visual language model is adapted to receive the feature map tensor. In one implementation, the visual encoder of the large visual language model is modified to directly receive... As input, the shallow feature extraction network of the visual language large model is skipped, thereby reducing the computational load on the cloud. At this point, the inference function of the visual language large model is modified to... .

[0114] In another implementation, the cloud server is additionally configured with a decoder network corresponding to the edge S1 module. (For example, the generator part of a Generative Adversarial Network (GAN). The decoder first processes the received... Perform upsampling and reconstruction, and output the reconstructed image. ,Right now Then, the cloud will reconstruct the image. The data is input into the original visual language model in step S5 for analysis.

[0115] In another alternative embodiment of the present invention, the event-triggered logic (step S2 or step S3) introduces an additional user-defined rules module to achieve more flexible upload management.

[0116] The system is configured to allow users to define one or more sets of custom rules via configuration files or remote interfaces. Each set of rules At least include time constraints and behavioral category constraints .

[0117] Time constraints Used to define the allowed upload time range, for example Behavioral category constraints Used to define the set of allowed upload behaviors, for example or .

[0118] When the event caching and fusion module in step S2 determines that an L1-level action (or other action requiring uploading) has occurred, the system does not immediately execute the encapsulation in step S4. Instead, it first obtains the original timestamp of the event. and its behavioral categories .

[0119] The system then iterates through all user-defined rules. Perform a rule check on the event. Only when... satisfy Time constraints (Right now )and satisfy Behavioral category constraints (Right now Only when the event is considered as allowed to be uploaded will the event be deemed to be allowed.

[0120] Take a specific rule as an example: User-defined for and (Only L1 anomalous behavior at night is uploaded). When an L1 behavior is detected at 14:00, although it meets the requirements... But not satisfied The system will actively suppress the uploading of this event. If this L1 behavior is detected at 03:00, then... and If all constraints are met, the system executes step S4, the encapsulation and upload process. This method ensures that the upload strategy meets the user's specific management needs or privacy considerations.

Claims

1. A behavior recognition method combining edge and cloud computing, characterized in that, The edge-cloud combined behavior recognition method runs in an execution environment that includes edge devices and cloud servers, and includes the following steps: S1. The edge device acquires video image frames through the image acquisition unit, performs single forward inference on the video image frames using a multi-head lightweight neural network model, and synchronously generates preliminary perception data. The preliminary perception data includes human body position coordinates, key behavioral component position coordinates, and preliminary behavioral state probabilities. S2. The edge device stores the preliminary sensing data into a sliding window queue, performs time-series fusion analysis on the data in the sliding window queue to calculate a comprehensive confidence score, and determines the current event state as a discard state, a cached observation state, or a confirmed upload state based on the comparison result of the comprehensive confidence score and a preset threshold. S3. The edge device periodically monitors hardware resource status parameters, including central processing unit temperature, memory usage, and network available bandwidth. The edge device executes a resource-aware dynamic scheduling strategy based on the hardware resource status parameters and preset behavior priority division, and adjusts the processing logic and data upload strategy for the first-level behavior, the second-level behavior, and the third-level behavior. S4. When the confirmed upload status is determined, the edge device selects key frame images from the cached data according to the preset selection rules, performs data reduction processing on the key frame images, and encapsulates the processed image data and structured metadata into a data packet and uploads it to the cloud server. S5. The cloud server receives and parses the data packet, constructs a multimodal input containing natural language instruction text and the keyframe image, performs semantic analysis and reasoning using a large visual language model, and outputs the final behavior recognition result.

2. The edge-cloud combined behavior recognition method according to claim 1, characterized in that, In step S1, the multi-head lightweight neural network model includes a feature extraction backbone network, a detection prediction head, and a state discrimination prediction head; The feature extraction backbone network is used to convert the video image frames into multi-level feature maps; The detection prediction head is used to output a list of bounding boxes containing category labels and location information based on the multi-level feature map; The state discrimination prediction head is used to output the initial behavior state probability corresponding to the predefined behavior category based on the multi-level feature map.

3. The edge-cloud combined behavior recognition method according to claim 1, characterized in that, In step S2, the comprehensive confidence score is calculated as follows: The human presence rate is calculated by statistically analyzing the proportion of frames in the sliding window queue where the human detection confidence score is greater than a preset detection threshold to the total number of frames; the desktop presence rate is calculated by statistically analyzing the proportion of frames in the sliding window queue where the desktop detection confidence score is greater than a preset detection threshold to the total number of frames; and the hand activity rate is calculated by statistically analyzing the proportion of frames in the sliding window queue where the hand is located near the desktop and the detection confidence score is greater than a preset detection threshold to the total number of frames. The human presence rate, the desktop presence rate, and the hand activity rate are then weighted and summed. The specific logic for determining the current event status is as follows: If the overall confidence score is less than the low threshold, it is determined to be in the discard state and memory resources are released. If the overall confidence score is greater than or equal to the low threshold and less than the high threshold, it is determined to be a cached observation state and the metadata is temporarily stored. If the overall confidence score is greater than or equal to the high threshold, or if the duration of the cached observation state exceeds a preset time threshold, it is determined to be the confirmed upload state.

4. The edge-cloud combined behavior recognition method according to claim 1, characterized in that, In step S3, the third-level behavior is a background analysis task. For the third-level behavior, the dynamic scheduling strategy includes: When a user actively triggers a function command and the edge device does not activate the load reduction mode, the edge device collects key segments of multiple consecutive frames and performs batch packaging and uploading. The cloud server then uses the visual language big model to combine multi-frame temporal information and external knowledge to generate intent-level behavior detection content.

5. The edge-cloud combined behavior recognition method according to claim 1, characterized in that, In step S3, the dynamic scheduling strategy includes a memory protection mode. When the memory occupancy rate exceeds a preset high memory threshold, the memory protection mode is entered. In the memory protection mode, the edge device performs the following operations: pausing the caching of new second-level behavior events, reducing the length of the sliding window queue to a preset value, and enabling a memory compression mechanism to store image data at a compression ratio higher than that in the normal mode. When the memory usage rate drops below a preset lower limit and remains below that limit for a preset duration, the edge device automatically restores the default caching strategy. In step S3, the dynamic scheduling strategy includes a low bandwidth mode. When the available network bandwidth is lower than a preset low bandwidth threshold, the device enters the low bandwidth mode. In the low bandwidth mode, the edge device performs hierarchical upload logic for the second-level behavior: prioritizing the upload of structured metadata consisting of the detection box and timestamp to the cloud server. The edge device uploads the corresponding image frame only after receiving an image request instruction from the cloud server; when the available network bandwidth exceeds the preset bandwidth fluctuation limit and continues for a preset duration, the edge device automatically resumes the normal image upload strategy.

6. The edge-cloud combined behavior recognition method according to claim 1, characterized in that, In step S3, the dynamic scheduling strategy includes a temperature-controlled load reduction mode; When the temperature of the central processing unit exceeds the preset high temperature threshold, it enters the temperature control deload mode; In the temperature-controlled load reduction mode, the edge device sends a control command to the image acquisition unit to reduce the video acquisition frame rate and extend the coverage time of the sliding window queue in step S2, while retaining the prediction and judgment logic for the first-level behavior and the second-level behavior. When the temperature of the central processing unit is lower than the preset lower limit and remains below the preset lower limit for a preset duration, the edge device automatically restores the default frame rate and sliding window queue length.

7. The edge-cloud combined behavior recognition method according to claim 1, characterized in that, In step S4, the specific steps for selecting the keyframe image are as follows: A comprehensive representative score is calculated for each frame of metadata in the buffer. The comprehensive representative score is obtained by weighted summation of the normalized confidence score and the normalized behavior change score. The normalized behavior change score is calculated based on the Euclidean distance between the center coordinates of the current frame key detection box and the center coordinates of the previous frame key detection box. The edge device sorts the comprehensive representative scores in descending order and selects a preset number of frames with the highest rankings as the keyframe images.

8. The edge-cloud combined behavior recognition method according to claim 1, characterized in that, In step S4, the data reduction process includes one of the following two methods: Method 1: Using the JPEG encoding standard, lossy compression is performed on the keyframe image using a preset quality factor; Method 2: Based on the coordinates of the human detection box in the keyframe image, apply a preset extended margin to calculate the target cropping region, and extract only the pixel data within the target cropping region.

9. The edge-cloud combined behavior recognition method according to claim 1, characterized in that, In step S5, constructing a multimodal input comprising natural language instruction text and the keyframe image specifically includes: The cloud server parses the structured metadata and extracts the detection box coordinates, original collection timestamps, and preliminary behavior labels from the structured metadata. The extracted detection box coordinate data, the original acquisition timestamp, and the preliminary behavior labels are converted into natural language description strings using a formatted mapping function. The natural language description string is concatenated with a preset general task description template to generate the natural language instruction text, which is used to guide the visual language big model to focus its attention on a specific spatiotemporal region in the keyframe image.

10. A computer device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-9.

Images