Multi-channel video stream management method and system

Abstract

The invention discloses a multi-channel video stream management method and system, relates to the technical field of artificial intelligence, and solves the technical problems that a multi-channel video stream intelligent management system is poor in generalization of specific tasks and large in use scene limitation. The method comprises the following steps: deploying a fine-tuning multi-modal large language model on edge side equipment to form an intelligent agent for processing multiple paths of video streams, and carrying out custom task configuration on related tasks for processing the multiple paths of video streams; through a cue word project, forming cue words for asking a question for the intelligent agent according to self-defined task configuration; key image frames of the multiple video streams are obtained, and the image frames and prompt words are used as input of an intelligent agent, so that the intelligent agent conducts reasoning on related tasks for processing the multiple video streams; and analyzing the reasoning result of the intelligent agent, forming a standard target task, and executing the target task according to the custom task configuration. According to the method, the generalization of the intelligent agent to specific tasks is enhanced, and the method can be suitable for more different scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a multi-path video stream management method and system. BACKGROUND

[0002] With the increasing demand for intelligentization in industrial monitoring and public transportation management scenarios, multi-path video stream management systems have gradually become an important application in the field of edge computing. Traditional multi-path video stream management systems are usually deployed on edge devices and are mainly used for video data acquisition, transmission and storage. In recent years, with the development of artificial intelligence technology and edge computing hardware, such systems have gradually integrated automated detection functions, such as target detection, behavior recognition and face recognition. Users can configure specific algorithm solutions for different video streams to achieve intelligent analysis and abnormal event reporting of multi-path video.

[0003] However, existing edge multi-video stream intelligent monitoring systems usually use convolutional neural networks (CNN) and attention mechanisms (such as YOLO, RCNN, etc.) for feature extraction and target detection. Such methods have limited model parameters, and actual application scenarios are prone to misidentification or missed identification. Moreover, the supported detection targets are fixed with the network weights, lacking generalization for general visual target detection.

[0004] In addition, existing edge multi-video stream management systems lack flexibility in task configuration and usually use fixed task execution modes, making it difficult to dynamically adjust computing resources or algorithm strategies according to actual needs. Although software-defined (Software-Defined, SD) technology has been widely used in network (SDN), computing, storage and other fields, its application in multi-path video stream management systems is still limited, with certain functional limitations for the use scenarios.

[0005] In the process of implementing the present application, the inventors have found that the prior art has at least the following problems:

[0006] Currently, multi-path video stream intelligent management systems have poor generalization for specific tasks and have great limitations in use scenarios. SUMMARY

[0007] The present application aims to provide a multi-path video stream management method and system to solve the technical problem of poor generalization of multi-path video stream intelligent management systems for specific tasks and great limitations in use scenarios in the prior art.

[0008] The preferred technical solutions in the many technical solutions provided by the present application can produce the many technical effects described below.

[0009] To achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0010] The application provides a multi-path video stream management method, comprising the following steps: deploying a fine-tuned multi-modal large language model on an edge device to form an intelligent agent for processing multi-path video streams and customizing task configuration for related tasks of processing the multi-path video streams; forming prompt words for asking the intelligent agent according to the customized task configuration through prompt word engineering; obtaining key image frames of the multi-path video streams, taking the image frames and the prompt words as inputs of the intelligent agent to enable the intelligent agent to infer the related tasks of processing the multi-path video streams; analyzing the inference result of the intelligent agent to form a standard target task and executing the target task according to the customized task configuration.

[0011] Optionally, the step of deploying the fine-tuned multi-modal large language model on the edge device to form the intelligent agent for processing the multi-path video streams comprises the following steps: generating N task instructions and corresponding example answers by using a target detection data set and a self-collected industrial scene data set; fine-tuning a pre-trained multi-modal large language model in an instruction fine-tuning manner according to the N task instructions and the example answers corresponding to the task instructions; obtaining a total loss function by combining a cross-entropy loss function and a contrastive learning loss function; optimizing parameters of the multi-modal large language model during fine-tuning by using the total loss function to obtain the fine-tuned multi-modal large language model; and deploying the fine-tuned multi-modal large language model on the edge device to form the intelligent agent for processing the multi-path video streams.

[0012] Optionally, the total loss function is

[0013]

[0014] wherein λ1 and λ2 are respectively a first weight coefficient and a second weight coefficient; is the cross-entropy loss function; is the contrastive loss function; y t is a one-hot encoding probability distribution of a target output text sequence, is a one-hot encoding probability distribution of a text sequence predicted by the fine-tuned multi-modal large language model; v i is an embedding vector of an image, t i is an embedding vector of a text, and sim(·) is a cosine similarity function, and τ is a temperature hyperparameter.

[0015] Optionally, the deploying the fine-tuned multi-modal large language model on the edge-side device to form an intelligent agent for processing the multi-path video stream comprises: quantizing parameters of the fine-tuned multi-modal large language model and converting the parameters into a model format supported by the edge-side device to deploy the fine-tuned multi-modal large language model on the edge-side device to form the intelligent agent for processing the multi-path video stream; wherein the parameters of the fine-tuned multi-modal large language model are quantized in an int4 quantization manner.

[0016] Optionally, the multi-modal large language model is a Qwen2.5VL model or an InternVL3 model.

[0017] Optionally, the related task comprises a task type, an event detection, and a report action.

[0018] Optionally, the forming a prompt word for asking the intelligent agent by prompt word engineering according to the custom task configuration comprises: adapting a corresponding prompt word template in advance by prompt word engineering, and automatically generating a standard prompt word according to information of the custom task configuration; the prompt word template comprises a description of a task and a requirement on an output format of the fine-tuned multi-modal large language model.

[0019] A multi-path video stream management system comprises: an edge-side deployment module configured to deploy a fine-tuned multi-modal large language model on an edge-side device to form an intelligent agent for processing a multi-path video stream; a task configuration module configured to perform custom task configuration on a related task for processing the multi-path video stream; an intelligent agent control module configured to form a prompt word for asking the intelligent agent by prompt word engineering according to the custom task configuration, and acquire a key image frame of the multi-path video stream, and use the image frame and the prompt word as an input of the intelligent agent to enable the intelligent agent to perform inference on the related task for processing the multi-path video stream; and a task execution module configured to analyze an inference result of the intelligent agent to form a standard target task, and perform the target task according to the custom task configuration.

[0020] Optionally, the system further comprises: a video stream access module configured to receive and analyze the multi-path video stream to obtain image frames of the multi-path video stream; and a channel configuration module configured to configure an IP address, a port number, and a video stream format of a current channel video stream, and match the current channel video stream to a corresponding agent of the intelligent agent.

[0021] Optionally, the intelligent agent is a proxy task processing architecture supporting a multi-channel task.

[0022] Implementing one of the technical solutions of the present application has the following advantages or beneficial effects:

[0023] The application can make the perception ability of the agent to information closer to the way of human, effectively combine text and image information when processing specific image tasks, give a clear reply to the image content, improve the model accuracy, and effectively reduce the occurrence of false positives in complex application scenarios, by deploying the fine-tuned large language model on the edge side; combined with the prompt word engineering technology, the normative interaction of the multi-modal large language model for specific tasks is realized, the large language model produces more accurate and high-quality output, the generalization for specific tasks is enhanced, and the application can be suitable for more different scenes. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings. The drawings are as follows:

[0025] Figure 1 is a flow chart of a multi-channel video stream management method according to an embodiment of the application;

[0026] Figure 2 is a flow chart of the execution of a multi-channel video stream management method according to an embodiment of the application;

[0027] Figure 3 is a schematic diagram of generating a task instruction and an example answer according to an embodiment of the application;

[0028] Figure 4 is a structural block diagram of a multi-channel video stream management system according to an embodiment of the application. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the application more clear, the various exemplary embodiments to be described below will be described with reference to the corresponding drawings, which constitute a part of the exemplary embodiments, and various exemplary embodiments that can be used to implement the application are described. Unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation described in the following exemplary embodiments does not represent all implementations consistent with the present disclosure. It should be understood that they are only examples of processes, methods and devices, etc. consistent with some aspects of the present disclosure as described in the appended claims, and other embodiments can be used, or structural and functional modifications can be made to the embodiments listed herein, without departing from the scope and spirit of the application.

[0030] In the description of the present application, it should be understood that the terms "center", "longitudinal", "transverse" and the like indicate the orientation or positional relationship shown in the drawings, which are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the elements referred to must have a particular orientation, be constructed and operated in a particular orientation. The terms "first", "second" and the like are only for the purpose of description and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. The term "a plurality of" means two or more. The terms "connected", "connected" should be interpreted broadly, for example, it can be fixed connection, detachable connection, integral connection, mechanical connection, electrical connection, communication connection, direct connection, indirect connection through intermediate medium, internal communication of two elements or interaction relationship between two elements. The term "and / or" includes any and all combinations of one or more related listed items. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0031] In order to illustrate the technical solutions described in the present application, the following will be described by specific examples, only showing the part related to the embodiment of the present application.

[0032] Example one:

[0033] As Figure 1 shown, the present application provides a multi-channel video stream management method, comprising the following steps: S1, deploying a fine-tuned multi-modal large language model on an edge device to form an intelligent agent for processing multi-channel video streams, and customizing task configuration for related tasks of processing multi-channel video streams; S2, forming prompt words for questioning the intelligent agent according to the customized task configuration through prompt word engineering; S3, acquiring key image frames of the multi-channel video stream, taking the image frames and the prompt words as inputs of the intelligent agent, so that the intelligent agent reasons on the related tasks of processing the multi-channel video stream; S4, analyzing the reasoning result of the intelligent agent to form a standard target task, and executing the target task according to the customized task configuration. Specifically, the management process is as shown in Figure 2 After forming the intelligent agent, the user can first configure the channel in the channel configuration interface to obtain the video stream corresponding to the current channel and the related information; then customize the task configuration in the task configuration interface, and send the normal prompt words generated by the prompt word engineering combined with the configuration result to the intelligent agent; the intelligent agent performs model reasoning according to the image frames of the video stream and the prompt words, and then feeds back the model output and the analysis result to the superior to execute the configured reporting action.

[0034] This embodiment deploys a fine-tuned large language model on the edge side, which enables the intelligent agent's ability to perceive information more closely like that of humans. When processing specific image tasks, it can effectively combine text and image information to give clear responses to image content, improve model accuracy, and effectively reduce the occurrence of false positives in complex application scenarios. Combined with prompt word engineering technology, it realizes standardized interaction with the multimodal large language model for specific tasks, guiding the large language model to produce more accurate and high-quality output, enhancing the generalization of specific tasks and making it applicable to more different scenarios.

[0035] Next, combine Figure 1 The specific implementation steps of the multi-channel video stream management method provided by this embodiment are described in detail:

[0036] First, execute step S1, deploy and fine-tune the multimodal large language model on the edge device, form an intelligent agent for processing multiple video streams, and customize the task configuration for related tasks of processing multiple video streams. Among them, the related tasks include task type, detection event and reporting action. It should be noted that multi-channel video streams refer to video streams from multiple different sources, usually obtained through multiple cameras or multiple transmission channels. These video streams can be transmitted and received simultaneously for various application scenarios of different edge devices. Multi-channel video streams can improve image quality and accessibility, as well as improve the accuracy and efficiency of monitoring and identification. Edge-side devices refer to the deployment of data processing and analysis tasks on edge devices close to the data source, such as local servers in factories, embedded devices at traffic intersections, AI cameras and other devices. In the field of image object detection, the intelligent agent formed by deploying a fine-tuned multimodal large language model has better detection accuracy in a variety of complex scenarios than traditional object detection algorithms such as Yolo (You Only Look Once, a real-time object detection system that predicts object bounding boxes and categories through a single neural network) and RCNN (an object detection algorithm based on convolutional neural networks) based on feature extraction methods and attention mechanisms. This can effectively reduce the occurrence of false alarms in complex application scenarios.

[0037] Further, users can customize task configurations on the system, such as task types, detection events, and reporting actions, for multi-channel video streams. The configuration of task types determines which specific operations are performed on multi-channel video streams, such as video decoding: decoding video streams from compressed format to raw image sequences for subsequent processing. Object detection: identifying and locating objects of interest (such as vehicles, pedestrians, etc.) in video frames. Object tracking: tracking detected objects across consecutive frames to obtain their motion trajectories. Behavior recognition: analyzing the behavior patterns of objects, such as whether they stop or run red lights. The configuration of detection events defines which specific events or situations need to be detected and reported, such as object appearance / disappearance: when a specific object (such as a vehicle or pedestrian) enters or leaves the monitoring area. Abnormal behavior: such as pedestrians entering restricted areas or vehicles driving in reverse. Change in target quantity: increase or decrease in the number of targets in the monitoring area. Specific condition triggering: conditions such as light changes, weather changes, etc. that may affect video quality or detection effectiveness. The configuration of reporting actions indicates which actions should be performed when specific events are detected, such as alarm: issuing an alarm signal through sound, light, or network message, etc. Data recording: recording relevant information (such as time, location, target type, etc.) of the event for subsequent analysis. Video clip saving: saving video clips before and after the triggering event for subsequent viewing and analysis. Notify relevant personnel: notify relevant personnel to handle the event through SMS, email, or instant messaging tools, etc.

[0038] Specifically, in step S1, fine-tune the multi-modal large language model on the edge device to form an intelligent agent for processing multi-channel video streams, including: S11, using the target detection dataset and the self-collected industrial scene dataset, generating N task instructions and example answers corresponding to the task instructions; the target detection dataset can use known datasets such as COCO and PascalVOC; the self-collected industrial scene dataset is an image collected by taking pictures in the industrial scene. In the case of knowing the target label and detection box in these dataset images, high-quality and large amounts of task instructions and corresponding example answers can be generated through the script of the large language model. For example, Figure 3As shown, the generated task instruction question is "Is there a person, cat, and dog in the image?", and the corresponding example answer is "Person: Yes, Cat: No, Dog: No"; the generated task instruction question is "How many people and buses are in the image?", and the corresponding example answer is "Person: 3, Bus: 1"; the generated task instruction question is "What are the coordinate boxes of the person and bus in the image?", and the corresponding example answer is "Person: {117, 117, 117, 117}, Person: {224, 224, 224, 224}, Person: {362, 362, 362, 362}, Bus: {512, 512, 512, 512}". S12, according to N task instructions and corresponding example answers, the pre-trained multi-modal large language model is fine-tuned in an instruction fine-tuning manner; Instruction fine-tuning (Instruction Tuning) is a way to adjust a large language model by providing large-scale instruction examples, which can improve the existing capabilities of a certain model in a free-shot manner, and will not lose the knowledge generalization ability of the original large model. The purpose of this embodiment is to improve the recognition and positioning ability of the model to the objects in the image. S13, combine the cross-entropy loss function and the contrastive learning loss function to obtain the total loss function; S14, optimize the parameters of the multi-modal large language model during fine-tuning using the total loss function, and obtain the fine-tuned multi-modal large language model; S15, deploy the fine-tuned multi-modal large language model on the edge side device to form an intelligent agent for processing multi-channel video streams. By fine-tuning the large language model, it can adapt to specific tasks and enhance the generalization to specific tasks. By fine-tuning the parameters of the multi-modal large language model, the perception ability of the model to the target in the image is enhanced, and the output of the model is further standardized, preparing for the subsequent extraction of the output information of the model.

[0039] Further, the total loss function is

[0040]

[0041] wherein λ1 and λ2 are the first weight coefficient and the second weight coefficient, respectively; is the cross-entropy loss function; is the contrastive loss function; y t is the one-hot encoding probability distribution of the target output text sequence, is the one-hot encoding probability distribution of the text sequence predicted by the fine-tuned multi-modal large language model; v i is the embedding vector of the image, t i is the embedding vector of the text, and sim(·) is the cosine similarity function, and τ is the temperature hyperparameter.

[0042] Specifically, step S15 includes: quantizing the parameters of the fine-tuned multi-modal large language model and converting them into a model format supported by the edge side device to deploy the fine-tuned multi-modal large language model on the edge side device to form an intelligent agent for processing multi-path video streams; wherein the parameters of the fine-tuned multi-modal large language model are quantized using an int4 quantization method. By quantizing the model parameters, the fine-tuned multi-modal large language model can be better deployed on the edge side.

[0043] As an alternative embodiment, the multi-modal large language model is a Qwen2.5-VL model or an InternVL3 model. The Qwen2.5-VL model is a visual understanding model focused on multi-modal tasks, aiming to improve the understanding of image content and combine it with natural language processing. It can process both text and image data simultaneously, achieving deep integration of the two. The InternVL3 model is a multi-modal large language model that includes 7 versions with parameter sizes ranging from 1B to 78B. It uses a native multi-modal pre-training method to train text, image, video and other modal data in a unified framework, breaking through the limitations of traditional models that need to be trained and aligned separately. Through a hybrid preference optimization algorithm and dynamic preprocessing technology, the model significantly improves multi-modal understanding and reasoning performance while maintaining strong language capabilities.

[0044] Then, step S2 is performed, and a prompt word for questioning the intelligent agent is formed according to the self-defined task configuration through prompt word engineering. Based on the strong cognitive ability of the multi-modal large language model, this embodiment combines task processing access with prompt word engineering (Prompt Engineering) technology to achieve standardized interaction of the multi-modal large language model for specific tasks, guiding the large language model to produce more accurate and high-quality outputs. On this basis, its functions can be extended to target type statistics, image content recognition, voice and video interactive dialogue, etc., and performance improvement can be directly obtained by updating the language model weight without retraining, which can be widely applied to various industries such as smart cities, industrial manufacturing, transportation hubs, public safety, etc.

[0045] Further, step S2 comprises: generating a standard prompt word in advance by prompting word engineering, adapting the corresponding prompt word template, and automatically generating a standard prompt word according to the information configured by the user-defined task. The prompt word template comprises a description of the task and a requirement for the output format of the fine-tuned multi-modal large language model. For example, the prompt word template is set as: task type: object detection; task description: detect and label all appearing vehicles and pedestrians in the video stream; detection time: pedestrian intrusion into the forbidden area; reporting action: when a pedestrian is detected to enter the preset forbidden area range; output format: JSON format, containing target type, position (coordinates), timestamp and other information. In this way, a standard prompt word can be efficiently generated according to the task information configured by the user, so as to guide the model to accurately process and output.

[0046] Next, step S3 is performed to obtain key image frames of the multi-path video stream, and the image frames are taken as inputs of the agent together with the prompt words, so that the agent performs reasoning on the related task of processing the multi-path video stream. By taking the key image frames in the video stream as inputs of the agent together with the prompt words obtained in the above steps, the agent can efficiently perform reasoning, and an accurate reasoning result can be obtained.

[0047] Finally, step S4 is performed to analyze the reasoning result of the agent, form a standard target task, and execute the target task according to the user-defined task configuration. For example, the user-defined task configuration is: task type: object detection, identify and locate vehicles and pedestrians in the video frame; detection event: pedestrian intrusion into the forbidden area and vehicle reverse; reporting action: trigger alarm and marking when detecting pedestrian intrusion into the forbidden area and vehicle reverse. When the above events are detected, alarm and marking are performed. The user is open to the configuration of task definition, and the type of task, the type of detection and the reporting action are all supported by the user to define. Through the adaptation and scheduling of the agent to the task, a complete business closed loop is realized, so as to improve the flexibility and expansibility of the system in processing complex tasks.

[0048] The embodiment is only one special case, and does not mean that the present application is in this way.

[0049] Embodiment two:

[0050] Based on the same inventive concept, for example, Figure 4As shown, the second embodiment of the application also provides a multi-channel video stream management system, comprising: an edge side deployment module (not shown in the figure) for deploying a fine-tuned multi-modal large language model on an edge side device to form an agent for processing multi-channel video streams; a task configuration module for customizing task configuration for related tasks for processing multi-channel video streams; the related tasks include task types, detection events and reporting actions. An agent control module is configured to form a prompt word for questioning the agent according to the customized task configuration through prompt word engineering, and obtain key image frames of the multi-channel video streams, and use the image frames and the prompt word as inputs of the agent to enable the agent to reason on the related tasks for processing the multi-channel video streams; the module is responsible for managing a plurality of model services running in the background and managing the state of each model, and when receiving the customized task configuration, generates a corresponding prompt word and parses the corresponding model output. A task execution module is configured to parse the reasoning result of the agent to form a standardized target task, and execute the target task according to the customized task configuration. The module is responsible for the execution of the target task, such as performing a corresponding alarm action on the parsed event, including but not limited to reporting the event to the platform, light control, visualization or calling a specific industrial interface, etc.

[0051] The multi-channel video stream management system provided by the embodiment is used to execute the multi-channel video stream management method provided by the first embodiment, and combines the idea of software-defined tasks to open the configuration of task definition to users, supports user customization from the type of task, the type of detection to the action of reporting, and realizes a complete business closed loop through the adaptation and scheduling of the agent to the task, thereby improving the flexibility and expandability of the system in processing complex tasks.

[0052] As an optional implementation, the system further comprises: a video stream access module configured to receive and parse the multi-channel video streams to obtain image frames of the multi-channel video streams; the module supports a plurality of video source formats (such as RTSP, USB, file, etc.) and performs unified encoding and buffering. A channel configuration module is configured to configure the IP address, port number and video stream format of the current channel video stream, and match the agent of the corresponding agent. Since the multi-channel video streams are video streams from multiple different sources, they are obtained through multiple transmission channels, and therefore each channel needs to be configured, which is conducive to matching the corresponding agent agent for processing.

[0053] As an alternative embodiment, the intelligent agent is a proxy task processing architecture supporting multi-channel tasks. The system structure of the embodiment can manage and control the internal resources of the intelligent agent. In order to improve the use of device computing resources and the response capability to multi-channel requests, a proxy task processing architecture supporting multi-channel and multi-task is designed, and a multi-modal model administrator class (i.e., code for managing multi-modal models) is set to be responsible for state maintenance and access control of multiple models running in the background. Users can configure multiple tasks on a single channel, each task sends an access request to the multi-modal model administrator class according to the configured request access time, the multi-modal model administrator class queries the current idle model, and converts the user's configuration into the corresponding prompt word, sends a request to the corresponding model, and after receiving the output of the model, the administrator parses the output content and returns the event format required by the user for subsequent execution of the report action. In order to stably obtain the output from the intelligent agent and accurately and effectively extract the information in the output.

[0054] The above only describes the preferred embodiments of the present application, and those skilled in the art know that various changes or equivalent replacements can be made to the features and embodiments without departing from the spirit and scope of the present application. In addition, under the guidance of the present application, the features and embodiments can be modified to adapt to specific conditions and materials without departing from the spirit and scope of the present application. Therefore, the present application is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of the present application are within the protection scope of the present application.

Claims

1. A multi-channel video stream management method, characterized in that: The following steps are involved: Deploy a fine-tuned multimodal large language model on edge devices to form an intelligent agent that processes multiple video streams and customizes task configuration related to processing the multiple video streams. By means of prompt word engineering, prompt words are formed to ask questions to the intelligent agent according to the custom task configuration; Obtaining key image frames of the multiple video streams, and using the image frames and the prompt words as inputs of an intelligent agent, so that the intelligent agent can reason about tasks related to processing the multiple video streams; Analyze the reasoning results of the agent to form a standardized target task, and execute the target task according to the custom task configuration.

2. A multi-channel video stream management method according to claim 1, characterized in that: The deployment of a fine-tuned multimodal large language model on edge devices to form an intelligent agent for processing multiple video streams includes: Using the target detection dataset and the self-collected industrial scene dataset, generate N task instructions and example answers corresponding to the task instructions; Based on the N task instructions and corresponding example answers, fine-tune the pre-trained multimodal large language model using instruction fine-tuning; Combine the cross entropy loss function and the contrastive learning loss function to obtain the total loss function; Optimizing the parameters of the multimodal large language model during fine-tuning using the total loss function to obtain the fine-tuned multimodal large language model; The fine-tuned multimodal large language model is deployed on the edge device to form an intelligent agent that processes multiple video streams.

3. A multi-channel video stream management method according to claim 2, characterized in that: The total loss function is Among them, λ1 and λ2 are the first weight coefficient and the second weight coefficient respectively; is the cross entropy loss function; is the contrast loss function; y t is the one-hot encoding probability distribution of the instance answer text sequence, The one-hot encoding probability distribution of the text sequence predicted by the fine-tuned multimodal large language model; v i is the embedding vector of the image, t i is the embedding vector of the text, sim(·) is the cosine similarity function, and τ is the hyperparameter of temperature.

4. A multi-channel video stream management method according to claim 2, characterized in that: The fine-tuning multimodal large language model is deployed on the edge device to form an intelligent agent for processing multiple video streams, including: The parameters of the fine-tuned multimodal large language model are quantized and converted into a model format supported by the edge-side device, so that the fine-tuned multimodal large language model can be deployed on the edge-side device to form an intelligent agent that processes multiple video streams; wherein, the parameters of the fine-tuned multimodal large language model are quantized using int4 quantization.

5. A multi-channel video stream management method according to claim 2, characterized in that: The multimodal large language model includes a Qwen2.5VL model and an InternVL3 model.

6. A multi-channel video stream management method according to claim 1, characterized in that: The related tasks include task type, detection event and reporting action.

7. A multi-channel video stream management method according to claim 1, characterized in that: The prompt word engineering is used to form prompt words for asking questions to the agent according to the customized task configuration, including: Through the prompt word project, the corresponding prompt word template is adapted in advance, and standardized prompt words are automatically generated according to the information configured for the custom task; the prompt word template includes a description of the task and requirements for the output format of the fine-tuned multimodal large language model.

8. A multi-channel video stream management system, characterized in that: include: The edge deployment module is used to deploy and fine-tune the multimodal large language model on edge devices to form an intelligent agent that processes multiple video streams; A task configuration module, used for performing custom task configuration on tasks related to processing the multiple video streams; an agent control module, configured to generate prompt words for questions to the agent based on the custom task configuration through prompt word engineering, obtain key image frames of the multiple video streams, and use the image frames and the prompt words as input to the agent, so that the agent can reason about tasks related to processing the multiple video streams; The task execution module is used to analyze the reasoning results of the intelligent agent, form a standardized target task, and execute the target task according to the custom task configuration.

9. A multi-channel video stream management system according to claim 8, characterized in that: The system further comprises: A video stream access module is used to receive and parse the multiple video streams to obtain image frames of the multiple video streams; The channel configuration module is used to configure the IP address, port number and video stream format of the current channel video stream and match it to the corresponding agent of the intelligent agent.

10. A multi-channel video stream management system according to claim 9, characterized in that: The intelligent agent is an agent task processing architecture that supports multi-channel tasks.

Images