Cloud-edge collaborative abnormal behavior character recognition method, device and system

Abstract

The invention relates to the technical field of computers, and provides a cloud-edge collaborative abnormal behavior character recognition method, device and system, and the method comprises the steps: edge end equipment employs a trained pedestrian detection model to cut out a video sequence containing pedestrians from a received video sequence, and uploads the video sequence containing the pedestrians to a cloud server; the cloud server cuts a video sequence containing pedestrians according to a set time length to obtain video slices, the video slices and prompt words are input into a trained multi-mode large model to obtain an identification result, the prompt words are described in a video question and answer mode, option question and answer pairs in video question and answer are constructed according to related task types, and the identification result is obtained. The options in the video questions and answers are possible behavior types. The lightweight pedestrian detection is integrated on the side end equipment to reduce the transmission, reduce the processing data volume of the cloud server, solve the problem of high demand of cloud computing resources, and improve the efficiency and response speed of the system.

Description

Technical Field

[0001] This disclosure belongs to the field of computer technology, and in particular relates to a method, device and system for identifying abnormal behavior of persons in a cloud-edge collaborative manner. Background Technology

[0002] With the rapid development of technologies such as the Internet of Things (IoT), big data, and artificial intelligence (AI), equipment monitoring systems in data centers and server rooms are becoming increasingly intelligent and automated. Traditional equipment monitoring systems typically rely on centralized cloud computing platforms to identify equipment faults through remote monitoring. While these systems can perform basic fault detection and alarms, their response speed and analytical capabilities are relatively limited. Especially when faced with equipment anomalies, they cannot quickly locate the source of the anomaly and the responsible party, resulting in low troubleshooting efficiency and long fault repair times.

[0003] In traditional deep learning methods, identifying abnormal human behavior primarily relies on small models that input single-frame or adjacent multi-frame images to locate individuals and determine behavior types. This method is inherently dependent on the formulation of behavior classification criteria and the richness of available data, and it is particularly ineffective in defining ambiguous behaviors. Multimodal large models, by fusing multiple information sources and possessing self-learning capabilities, can achieve better performance. Furthermore, traditional deep learning methods require large amounts of manually labeled data to compensate for a lack of domain knowledge, while multimodal large models already incorporate a wealth of cross-domain knowledge during the pre-training phase. They can be fine-tuned on a small amount of domain-specific data, achieving higher efficiency than traditional deep learning methods.

[0004] Although multimodal large models have significant advantages in abnormal behavior recognition, they usually require large computing resources, especially when processing video and sensor data in real time, which requires powerful computing and storage resources and has high demand for cloud computing resources. Summary of the Invention

[0005] To address the aforementioned issues, this disclosure provides a cloud-edge collaborative method, apparatus, and system for identifying abnormal behavior and individuals. By employing cloud-edge collaboration and deploying a small pedestrian detection model at the edge to filter invalid video sequences, the amount of data processed in the cloud can be reduced, thereby improving the system's efficiency and response speed.

[0006] Firstly, a cloud-edge collaborative method for identifying abnormal behavior individuals is provided, applied to edge devices with pre-deployed and trained pedestrian detection models, including: Receive the video sequence sent by the terminal acquisition device; Using a trained pedestrian detection model, video sequences containing pedestrians are cut out from video sequences; Video sequences containing pedestrians are uploaded to a cloud server so that the cloud server can use a trained multimodal large model to identify people with abnormal behavior.

[0007] Furthermore, the trained pedestrian detection model is implemented using a YOLOx network structure and deployed to edge devices after int8 quantization.

[0008] Secondly, a cloud-edge collaborative method for identifying individuals exhibiting abnormal behavior is provided, applied to a cloud server with a pre-deployed, trained multimodal large-scale model, including: Receive video sequences containing pedestrians sent by edge devices; The video sequence containing pedestrians is cut according to the set duration. For those shorter than the set duration, the actual duration is used to cut the video slices. Video slices and prompts are input into a pre-trained multimodal large model to obtain recognition results. The prompts are described using a video question-and-answer format. The question-and-answer pairs in the video question-and-answer format are constructed based on the relevant task type, and the options in the video question-and-answer format are the possible behavior types.

[0009] Furthermore, the trained multimodal large model is obtained through the following methods: By using video sequences containing pedestrians for training, the Qwen2-VL multimodal large model is fine-tuned for the task of analyzing human behavior, resulting in a well-trained multimodal large model.

[0010] Furthermore, using video sequences containing pedestrians for training, the Qwen2-VL multimodal large model was fine-tuned for a human behavior analysis task, resulting in a trained multimodal large model, including: The training video sequence containing pedestrians is cut according to a set duration, and if it is less than the set duration, it is cut according to the actual duration to obtain video slices; The video segments were annotated using two video question-and-answer methods. The first method involved describing the pedestrian behavior in the video segment in detail. The second method involved constructing option-based question-and-answer pairs based on the relevant task type, using the possible behavior types as the options. The Qwen2-VL multimodal large model was trained using LoRa fine-tuning with labeled video slices, with only the Qwen2VLModel module being fine-tuned.

[0011] Furthermore, the prompts for the first type of video Q&A are: Please briefly describe the video content, focusing on the behavior of the people in it; The prompts for the second type of video Q&A are: Based on the content of the video clip, determine whether the person in the video clip exhibits any of the following options; if so, answer with the corresponding option; if not, answer "not present". The selected actions include: A. Carrying an object other than the preset item out of the room; B. Manually opening a room equipment door that was originally closed; C. Touching or repairing equipment; D. Carrying an object other than the preset item; E. Moving an object in the room; F. Closing a room equipment door that was originally open.

[0012] Thirdly, a method for identifying abnormal behavior in a cloud-edge collaborative manner is provided, including: The edge device receives the video sequence sent by the terminal acquisition device; Edge devices use a trained pedestrian detection model to cut out video sequences containing pedestrians from video sequences; The edge device uploads video sequences containing pedestrians to the cloud server; The cloud server will cut the video sequence containing pedestrians into segments according to the set duration. If the duration is less than the set duration, the actual duration will be used to cut the video segments. The cloud server inputs video slices and prompts into a pre-trained multimodal large model to obtain recognition results. The prompts are described using a video question-and-answer format. The question-and-answer pairs in the video question-and-answer format are constructed based on the relevant task type, and the options in the video question-and-answer format are possible behavior types.

[0013] Fourthly, a cloud-edge collaborative abnormal behavior identification device is provided, comprising: a receiving module, a cutting module, and a sending module; wherein: The receiving module is used to receive video sequences sent by the terminal acquisition device; The cropping module is used to crop out video sequences containing pedestrians from a video sequence using a trained pedestrian detection model; The sending module is used to upload video sequences containing pedestrians to the cloud server, so that the cloud server can use a multimodal large model to identify people with abnormal behavior.

[0014] Fifthly, a cloud-edge collaborative abnormal behavior identification device is provided, comprising: a receiving module, a cutting module, and a recognition module, wherein: The receiving module is used to receive video sequences containing pedestrians sent by the edge device; The cutting module is used to cut video sequences containing pedestrians to a set duration, and cut them to the actual duration if the duration is less than the set duration, to obtain video slices; The recognition module is used to input video slices and prompts into a pre-trained multimodal large model to obtain recognition results. The prompts adopt a video question-and-answer method, constructing option question-and-answer pairs according to the relevant task type, and taking the possible behavior type as the option question-and-answer.

[0015] Sixthly, a cloud-edge collaborative abnormal behavior identification system is provided, comprising: edge devices and a cloud server, wherein: Edge devices are used to receive video sequences sent by terminal acquisition devices; use a trained pedestrian detection model to cut out video sequences containing pedestrians from the video sequences; and upload the video sequences containing pedestrians to the cloud server. A cloud server is used to cut video sequences containing pedestrians to a set duration, or cut them to the actual duration if the duration is less than the set duration, to obtain video slices. The video slices and prompts are then input into a trained multimodal large model to obtain recognition results. The prompts are described using a video question-and-answer method. The question-and-answer pairs in the video question-and-answer are constructed based on the relevant task type, and the options in the video question-and-answer are possible behavior types.

[0016] Compared with the prior art, this disclosure has the following advantages: (1) Introducing a multimodal large model for abnormal behavior recognition, which performs well in some scenarios compared to traditional deep learning methods, but performs poorly in handling abnormal behavior, ambiguous behavior definition and inconsistent data labeling, the accuracy of this invention is higher; (2) A cloud-edge collaborative abnormal behavior person association system was constructed. By integrating lightweight pedestrian detection into edge devices, the transmission is reduced, the amount of data processed by the cloud server is reduced, the problem of high cloud computing resource requirements is solved, and the efficiency and response speed of the system are improved.

[0017] Other features and advantages of this disclosure will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the disclosure. The objects and other advantages of this disclosure may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A schematic diagram of a cloud-edge collaborative abnormal behavior person recognition system according to an embodiment of the present disclosure is shown; Figure 2 A diagram is shown illustrating a cloud-edge collaborative method for identifying abnormal behavior individuals according to an embodiment of this disclosure. Detailed Implementation

[0020] Considering the limited computing power of edge devices, they are more suitable for handling simple business logic. Cloud computing power, on the other hand, is abundant and suitable for complex model inference. Therefore, the challenge of this invention is how to fully utilize the computing power characteristics of both the cloud and edge, rationally decompose the algorithm logic, and select a matching model. In the solution of this invention, the edge device uses a trained pedestrian detection model to cut out a video sequence containing pedestrians from a video sequence; the cloud server inputs the video slices and prompts into a trained multimodal large model to obtain the recognition result. This solution fully utilizes the computing power characteristics of both the cloud and edge, rationally decomposes the algorithm logic, and ensures the low latency and high throughput requirements of the overall business system.

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0022] In data center environments, abnormal human behavior is often complex and diverse. Traditional deep learning methods struggle to handle intricate behavioral details and dynamic changes, making them susceptible to interference and leading to false positives or false negatives. Multimodal large-scale models, however, can capture complex behavioral patterns and environmental changes, handle background interference and occlusion, and provide more robust behavior recognition capabilities. Furthermore, general object detection methods perform well in personnel detection, allowing for edge-based pedestrian screening to ensure real-time performance and low latency. This division of labor enables the system to efficiently utilize resources at both the edge and cloud, improving overall processing efficiency.

[0023] The cloud-edge collaborative abnormal behavior identification system disclosed herein consists of the following components: Figure 1 As shown, it includes: a terminal data acquisition device with a pedestrian detection module, an intelligent terminal as an edge device, and a cloud server with a human abnormal behavior recognition module, a data storage module, and a manual review module, wherein: Terminal acquisition devices are used to acquire video sequences and send the acquired video sequences to edge devices. Specifically, there can be multiple terminal acquisition devices to acquire video sequences from different locations.

[0024] The intelligent terminal is used to receive video sequences sent by the terminal acquisition device; using a trained pedestrian detection model, it cuts out video sequences containing pedestrians from the video sequences; and uploads the video sequences containing pedestrians to the cloud server. The cloud server is used to cut video sequences containing pedestrians to a set duration, or cut them to the actual duration if the duration is less than the set duration, to obtain video slices. The video slices and prompts are then input into a trained multimodal large model to obtain recognition results. The recognition results are stored in the data storage module. The recognition results are reviewed by a manual review module. The prompts are described using a video question-and-answer format. The question-and-answer pairs in the video question-and-answer format are constructed based on the relevant task type, and the options in the video question-and-answer format are possible behavior types.

[0025] Specifically, the cloud server uses video sequences containing pedestrians for training to fine-tune the Qwen2-VL multimodal large model for human behavior analysis tasks, resulting in a well-trained multimodal large model.

[0026] The cloud server's abnormal human behavior recognition module includes: a video receiving module, a preprocessing module, a Qwen2Vision Transformer PretrainedModel, a Prompt (prompt input module), a word segmenter, a Qwen2VLModel (language model), and a result output module.

[0027] The abnormal behavior recognition module's abnormal behavior recognition process during the training phase includes: receiving video sequences and prompt words; the preprocessing module trims the video sequences to obtain video slices; the Qwen2VisionTransformer PretrainedModel (inference and pretraining) encodes the video to obtain encoding vectors; the word segmenter segments the prompt words to obtain word vectors; and the word vectors and encoding vectors are input into the Qwen2VLModel to obtain the abnormal behavior person recognition result.

[0028] The process of identifying abnormal human behavior in the application phase of the trained cloud-based abnormal human behavior recognition module is as follows: The module receives a video sequence and prompt words; the preprocessing module cuts the video sequence to obtain video slices; the Qwen2Vision TransformerPretrainedModel (inference and pre-training) encodes the video to obtain encoded vectors; the word segmenter segments the prompt words to obtain word vectors; the word vectors and encoded vectors are input into the Owen2VLModel to obtain the abnormal behavior identification result.

[0029] Training phase: The wen2Vision Transformer PretrainedModel (inference and pre-training) encodes the video to obtain encoded vectors. The word segmenter segments the prompt words to obtain word vectors. The word vectors and encoded vectors are input into the Owen2VLModel to obtain the abnormal behavior person identification results. Then, the abnormal behavior person identification results and the labeled tags are compared to calculate the CrossEntropyLoss, and backpropagation is performed to adjust the model weights.

[0030] The disclosed cloud-edge collaborative abnormal behavior identification method is as follows: Figure 2 As shown, it includes the following steps: Step 201: The edge device receives the video sequence sent by the terminal acquisition device; Step 202: The edge device uses the trained pedestrian detection model to cut out the video sequence containing pedestrians from the received video sequence; Step 203: The edge device uploads the video sequence containing pedestrians to the cloud server; Step 204: The cloud server cuts the video sequence containing pedestrians according to the set duration. If the duration is less than the set duration, it is cut according to the actual duration to obtain video slices. Step 205: The cloud server inputs the video slices and prompts into the trained multimodal large model to obtain the recognition results. The prompts are described using a video question-and-answer format. The question-and-answer pairs in the video question-and-answer format are constructed based on the relevant task types, and the options in the video question-and-answer format are the possible behavior types.

[0031] The present invention will now be described in detail through a specific implementation process, which includes the following steps: (a) Perform data processing: Step 1.1 Collect video sequences from different locations. The pedestrian detection model can be used to filter video frames that do not contain pedestrians, and the frames containing pedestrians can be segmented into short videos of varying lengths. The entire short video is then divided into 10-second segments for training the multimodal large-scale model of human behavior analysis.

[0032] Model training and deployment module: (b) Training and deployment of the edge pedestrian detection model: Step 2.1 The pedestrian detection model in this invention is implemented using the YOLOx network structure.

[0033] Step 2.2 The trained pedestrian detection model is converted to int8 quantization and deployed to the edge device.

[0034] (c) Cloud-based multimodal large model training and deployment: This invention selects the Qwen2-VL multimodal large model for fine-tuning the human behavior analysis task.

[0035] Step 3.1 Label the video slice data obtained in Step 1.1 using two video question-and-answer methods.

[0036] The first approach involves a detailed description of pedestrian behavior in a video clip, with the prompt "Please briefly describe the video content, focusing on the behavior of the people in it." The second approach constructs a question-and-answer pair based on the relevant task type, using possible behavior types as answer options and requiring the user to answer with the option number. The prompt is "Based on the content of the video, determine whether the person in the video exhibits any of the following behaviors (if yes, only answer the corresponding option; if multiple behaviors exist, answer the corresponding options separated by commas; otherwise, answer G):\n A. Carrying an object other than a mobile phone out of the room; B. The door to the equipment in the room was originally closed but was manually opened; C. Touching or repairing equipment; D. Carrying an object other than a mobile phone; E. Moving objects in the room; F. The door to the equipment in the room was originally open but was closed."

[0037] This disclosure employs two video question-answering methods to enable the model to learn richer features. The first method helps the model learn detailed features related to pedestrians, while the second method aims to construct a concise question-answering approach that directly learns behavior types and outputs fewer tokens, thus improving the model's inference time. Constructing the first method improves the performance of the second task.

[0038] Step 3.2 Perform LoRa fine-tuning training on the Qwen2-VL model on the annotation task in Step 3.1, only fine-tuning the Qwen2VLModel (language model) module.

[0039] Step 3.3 Cloud Deployment: Perform only a question-and-answer task. The input is a 10-second video. The prompt is: "Based on the content of the video, please determine whether the person in the video exhibits any of the following behaviors: A. Carrying an object other than a mobile phone out of the room; B. Manually opening a door to a device in the room that was originally closed; C. Touching or repairing a device; D. Carrying an object other than a mobile phone; E. Moving an object in the room; F. Closing a door to a device in the room that was originally open." If the behavior exists, only answer the corresponding option. If multiple behaviors exist, answer the corresponding option and separate them with commas. Otherwise, answer "not present" or indicate that G is not present. Use both as model input and output the option letter.

[0040] (d) System Application: Step 4.1 By running the pedestrian detection model deployed on the edge, capture video sequences containing pedestrians and upload them to the cloud computing center.

[0041] Step 4.2 The cloud truncates the video sequence every 10 seconds, truncating any segments shorter than 10 seconds according to the actual duration. Each 10-second video segment and its accompanying prompt are fed into the finely tuned multimodal model. If the multimodal model responds with AF, it directly outputs the alarm event type and alarm video segment for manual review. If the response is G, it indicates that no potential hazards have been identified, and the system does not issue an alarm.

[0042] This invention can be applied to data center management. When there are abnormal changes in equipment, the multimodal large model technology can be used to analyze videos and effectively detect potential problems.

[0043] In summary, the solutions in this embodiment of the disclosure: 1. Introduce a multimodal large model for abnormal human behavior recognition, solving the problems of significant shortcomings, difficulties in data annotation, and poor performance of existing traditional deep learning methods in detecting abnormal behavior in complex environments. 2. Construct a cloud-edge collaborative abnormal human behavior recognition system, based on a small-model-large-model approach, providing real-time large-model inference computation, significantly reducing the demand for cloud computing resources and data transmission bandwidth, lowering network load and costs, while improving the overall efficiency of the system.

[0044] Based on the above method, this disclosure also provides a cloud-edge collaborative abnormal behavior identification device corresponding to the above method, including: a receiving module, a cutting module, and a sending module; wherein: The receiving module is used to receive video sequences sent by the terminal acquisition device; The cropping module is used to crop out video sequences containing pedestrians from a video sequence using a trained pedestrian detection model; The sending module is used to upload video sequences containing pedestrians to the cloud server, so that the cloud server can use a multimodal large model to identify people with abnormal behavior.

[0045] Based on the above method, this disclosure also provides a cloud-edge collaborative abnormal behavior identification device corresponding to the above method, including: a receiving module, a cutting module, and an identification module, wherein: The receiving module is used to receive video sequences containing pedestrians sent by the edge device; The cutting module is used to cut video sequences containing pedestrians to a set duration, and cut them to the actual duration if the duration is less than the set duration, to obtain video slices; The recognition module is used to input video slices and prompts into a pre-trained multimodal large model to obtain recognition results. The prompts adopt a video question-and-answer method, constructing option question-and-answer pairs according to the relevant task type, and taking the possible behavior type as the option question-and-answer.

[0046] Furthermore, it also includes a training module, which is used to fine-tune the Qwen2-VL multimodal large model for the task of analyzing human behavior using video sequences containing pedestrians for training, so as to obtain a trained multimodal large model.

[0047] Further, the training module is specifically used to cut the training video sequences containing pedestrians to a set duration, or cut them to the actual duration if the duration is less than the set duration, to obtain video slices; the video slices are labeled, where the labeling adopts two video question-and-answer methods. The first video question-and-answer method is to describe the specific behavior of pedestrians in the video segment; the second video question-and-answer method is to construct option question-and-answer pairs based on the relevant task type, and take the possible behavior types as option questions and answers; the labeled video slices are used to perform LoRa fine-tuning training on the Qwen2-VL multimodal large model to obtain the trained multimodal large model, where only the Qwen2VLModel module is fine-tuned.

[0048] Although the present disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A cloud-edge collaborative method for identifying individuals exhibiting abnormal behavior, characterized in that, Applications on edge devices with pre-deployed and trained pedestrian detection models include: Receive the video sequence sent by the terminal acquisition device; Using a trained pedestrian detection model, video sequences containing pedestrians are cut out from video sequences; Video sequences containing pedestrians are uploaded to a cloud server so that the cloud server can use a trained multimodal large model to identify people with abnormal behavior.

2. The method according to claim 1, characterized in that, The trained pedestrian detection model is implemented using a YOLOx network structure and deployed to edge devices after int8 quantization.

3. A cloud-edge collaborative method for identifying abnormal behavior individuals, characterized in that, Applications are performed on cloud servers where pre-trained multimodal large models have been deployed, including: Receive video sequences containing pedestrians sent by edge devices; The video sequence containing pedestrians is cut according to the set duration. For those shorter than the set duration, the actual duration is used to cut the video slices. Video slices and prompts are input into a pre-trained multimodal large model to obtain recognition results. The prompts are described using a video question-and-answer format. The question-and-answer pairs in the video question-and-answer format are constructed based on the relevant task type, and the options in the video question-and-answer format are the possible behavior types.

4. The method according to claim 3, characterized in that, A well-trained multimodal large model can be obtained through the following methods: By using video sequences containing pedestrians for training, the Qwen2-VL multimodal large model is fine-tuned for the task of analyzing human behavior, resulting in a well-trained multimodal large model.

5. The method according to claim 4, characterized in that, Using video sequences containing pedestrians for training, the Qwen2-VL multimodal large model was fine-tuned for a pedestrian behavior analysis task, resulting in a well-trained multimodal large model, including: The training video sequence containing pedestrians is cut according to a set duration, and if it is less than the set duration, it is cut according to the actual duration to obtain video slices; The video segments were annotated using two video question-and-answer methods. The first method involved describing the pedestrian behavior in the video segment in detail. The second method involved constructing option-based question-and-answer pairs based on the relevant task type, using the possible behavior types as the options. The Qwen2-VL multimodal large model was trained using LoRa fine-tuning with labeled video slices to obtain a trained multimodal large model, in which only the Qwen2VLModel module was fine-tuned.

6. The method according to claim 5, characterized in that, The prompt for the first type of video Q&A is: Please briefly describe the video content, focusing on the behavior of the people in it; The prompts for the second type of video Q&A are: based on the content of the video clip, determine whether the person in the video clip exhibits the following options; if so, answer the corresponding option. If it does not exist, then the answer is no; The selected behaviors include: A. Carrying an object other than the preset item out of the room; B. Manually opening the equipment door in the room that was originally closed. C. Touching or repairing equipment; D. Carrying objects other than the pre-set items; E. Moving objects in the room; F. Closing equipment doors that were originally open in the room.

7. A cloud-edge collaborative method for identifying abnormal behavior individuals, characterized in that, include: The edge device receives the video sequence sent by the terminal acquisition device; Edge devices use a trained pedestrian detection model to cut out video sequences containing pedestrians from video sequences; The edge device uploads video sequences containing pedestrians to the cloud server; The cloud server will cut the video sequence containing pedestrians into segments according to the set duration. If the duration is less than the set duration, the actual duration will be used to cut the video segments. The cloud server inputs video slices and prompts into a pre-trained multimodal large model to obtain recognition results. The prompts are described using a video question-and-answer format. The question-and-answer pairs in the video question-and-answer format are constructed based on the relevant task type, and the options in the video question-and-answer format are possible behavior types.

8. A cloud-edge collaborative abnormal behavior person recognition device, characterized in that, include: The module consists of a receiving module, a cutting module, and a sending module; among which: The receiving module is used to receive video sequences sent by the terminal acquisition device; The cropping module is used to crop out video sequences containing pedestrians from a video sequence using a trained pedestrian detection model; The sending module is used to upload video sequences containing pedestrians to the cloud server, so that the cloud server can use a multimodal large model to identify people with abnormal behavior.

9. A cloud-edge collaborative abnormal behavior person recognition device, characterized in that, include: The module consists of a receiving module, a cutting module, and a recognition module, wherein: The receiving module is used to receive video sequences containing pedestrians sent by the edge device; The cutting module is used to cut video sequences containing pedestrians to a set duration, and cut them to the actual duration if the duration is less than the set duration, to obtain video slices; The recognition module is used to input video slices and prompts into a pre-trained multimodal large model to obtain recognition results. The prompts adopt a video question-and-answer method, constructing option question-and-answer pairs according to the relevant task type, and taking the possible behavior type as the option question-and-answer.

10. A cloud-edge collaborative abnormal behavior person recognition system, characterized in that, include: Edge devices and cloud servers, including: Edge devices are used to receive video sequences sent by terminal acquisition devices; use a trained pedestrian detection model to cut out video sequences containing pedestrians from the video sequences; and upload the video sequences containing pedestrians to the cloud server. A cloud server is used to cut video sequences containing pedestrians to a set duration, or cut them to the actual duration if the duration is less than the set duration, to obtain video slices. The video slices and prompts are then input into a trained multimodal large model to obtain recognition results. The prompts are described using a video question-and-answer method. The question-and-answer pairs in the video question-and-answer are constructed based on the relevant task type, and the options in the video question-and-answer are possible behavior types.

Images