Multimodal large model-based training and inference integrated video analysis method and system

Abstract

The application relates to a multi-modal large model-based training and inference integrated video analysis method and system, which solves the technical problems of high customization cost, long response period for new requirements of existing small models, and serious dependence on computing power of cascaded large model schemes, and difficulty in real-time processing of multiple video streams. A multi-task dynamic plug-in large model weight sharing inference method is adopted, a special large language model is used to convert user natural language requirements into standardized detection prompt words, and independent lightweight plug-ins that can be quickly fine-tuned through small sample data are generated for each detection task. During inference, the basic large model weight is shared among multiple tasks, and only the plug-in corresponding to the task is loaded for feature fusion and calculation. The application can achieve the positive effects of one-sentence generation of new algorithms and small-sample minute-level rapid scene optimization, and can support high-concurrency real-time analysis of hundreds of video streams with a single acceleration card, significantly improving algorithm iteration efficiency and reducing development and deployment costs.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video analysis, and in particular to a training and inference integrated video analysis method and system based on a multi-modal large model. BACKGROUND

[0002] In the field of video analysis, the current mainstream technical solution is implemented based on a small model of deep learning, and typical models include YOLO, SSD, Resnet, etc. Such a small model has the advantage of fast inference speed due to its lightweight characteristics, and is widely used in scenarios with high real-time requirements.

[0003] However, the small model technology has significant limitations: first, for each new application scenario, scene data needs to be collected, samples need to be labeled, and a dedicated model needs to be retrained, resulting in high customization costs; second, from the demand proposal to the final deployment of the model, it usually takes months to collect samples, iterate and optimize the model, and the response cycle of new demands is significantly lengthened; third, as the application scenarios continue to increase, the independent maintenance and update of each scene model leads to a linear growth trend in the overall cost, and the marginal cost continues to rise, making it difficult to adapt to large-scale, multi-scenario video analysis requirements.

[0004] With the development of large model technology, a cascading application mode of "small model + VLM (visual-language multi-modal large model)" has gradually emerged in the field of video analysis. The core advantage of this mode is that it can handle complex open-domain problems and natural language interaction requirements with the help of the powerful high-level semantic understanding ability of VLM, to some extent, expanding the application boundary of video analysis. However, this mode still has obvious performance bottlenecks: on the one hand, visual-language multi-modal large models usually contain tens of billions of parameters, which have a high dependence on hardware computing power, and it is difficult to meet the real-time requirements of video analysis in most actual application scenarios; on the other hand, due to the use of a cascading processing flow, the detection accuracy of the front-end small model directly limits the performance upper limit of the overall scheme, making it difficult to break through the inherent defects of the small model.

[0005] In summary, the existing video analysis technology still has room for improvement in terms of cost control, response efficiency, real-time performance, and detection accuracy, and there is an urgent need for an innovative technical solution that can balance low cost, fast response, high real-time performance, and high accuracy. SUMMARY

[0006] The embodiments of the present application provide a training and inference integrated video analysis method, system, computer device and computer readable storage medium based on a multi-modal large model, to at least solve the problem in the related art that a customized analysis model needs to be retrained for each new application scenario, resulting in high cost and efficiency.

[0007] In a first aspect, the embodiments of the present application provide a multi-modal large model-based training and inference integrated video analysis method, which comprises:

[0008] receiving a natural language input by a user for performing a video analysis task;

[0009] converting the natural language into a task prompt word suitable for the video analysis task by using a preset large language model;

[0010] initializing a learnable weight layer corresponding to the video analysis task in a pre-trained open-world target detection base large model, and performing preliminary inference on a real-time video stream in combination with the task prompt word to obtain a preliminary detection result;

[0011] under the condition that the weights of the open-world target detection large model are kept frozen, performing incremental fine-tuning training on the learnable weight layer based on small sample data formed by manually labeling the preliminary detection result, and outputting the learnable weight layer as a task plug-in corresponding to the video analysis task by updating the parameters of the learnable weight layer;

[0012] calling the open-world target detection base large model, dynamically loading the task plug-in and the task prompt word corresponding to the current video analysis task, and performing analysis and inference on real-time video stream data to obtain an analysis result.

[0013] In some embodiments, the open-world target detection base large model adopts a double-tower structure, including a visual encoder for extracting visual features from video stream data and a text encoder for extracting text features from the task prompt word;

[0014] The open-world target detection base large model interacts and fuses the visual features and the text features through a cross-modal fusion module to obtain fused features;

[0015] And, through a visual-text alignment module, target detection in a video is realized based on the fused features.

[0016] In some embodiments, performing incremental fine-tuning training on the learnable weight layer based on small sample data formed by manually labeling the preliminary detection result comprises:

[0017] adding the learnable weight of the initial task plug-in to the text features to generate enhanced text features;

[0018] interacting and fusing the enhanced text features and the visual features through the cross-modal fusion module to obtain fused visual features and fused text features;

[0019] calculate a matching degree of each visual region in the visual feature and each semantic unit in the text feature based on the fused visual feature and the fused text feature, and determine an alignment loss based on the matching degree;

[0020] generate a prediction box through bounding box regression based on the fused visual feature, and calculate a bounding box regression loss based on the prediction box and a real detection box in the small sample data corresponding to the prediction box;

[0021] construct a composite loss function based on the alignment loss and the bounding box regression loss, and update the weight parameters of the task plug-in through back propagation based on the composite loss function.

[0022] In some embodiments, the composite loss function is represented by the following formula:

[0023]

[0024] wherein, is the composite loss function, is the alignment loss, is the bounding box regression loss, is a cosine similarity of the i-th visual region and a positive class text, is a cosine similarity of the i-th visual region and all texts, is a temperature coefficient, and N is the number of visual regions participating in the calculation, is a prediction box coordinate, is a real box coordinate, and λ is a weight coefficient, is a loss function based on generalized intersection over union.

[0025] In some embodiments, the manual annotation of the preliminary detection result to form the small sample data comprises:

[0026] constructing an event database based on the preliminary detection result, the corresponding original video frame and the label;

[0027] manually annotating the detection result in the event database, and classifying the detection result as a correct sample or an incorrect sample, wherein:

[0028] annotated correct samples and their corresponding task prompts, original video frames and labels constitute a positive sample pair, and annotated incorrect samples and their corresponding original video frames, labels, and preset non-target prompts constitute a negative sample pair,

[0029] constructing the small sample data for the incremental fine-tuning training based on the positive sample pair and the negative sample pair.

[0030] In some embodiments, during the incremental fine-tuning training phase and the inference phase, when multiple video analysis tasks are processed in parallel, the method further comprises:

[0031] For different video analysis tasks using the same task prompt word, real-time analysis training is performed by sharing the text features extracted by the text encoder for the task prompt word;

[0032] For different video analysis tasks based on the same video stream data, real-time analysis inference is performed by sharing the visual features extracted by the visual encoder for the video stream data.

[0033] In some embodiments, the preset large language model is a video analysis prompt word dedicated large language model, which is obtained by fine-tuning a general large language model in two stages as follows:

[0034] The first stage supervised fine-tuning includes training the general large language model based on a pre-constructed instruction fine-tuning dataset, wherein the instruction fine-tuning dataset contains a plurality of mapping pairs composed of natural language descriptions in the video analysis field and their corresponding standard task prompt words;

[0035] The second stage reinforcement learning optimization includes introducing a reinforcement learning mechanism based on human feedback to continuously optimize the large language model after the first stage of fine-tuning, and obtaining the preset large language model after the generated task prompt word meets the expected result.

[0036] In a second aspect, the embodiments of the present application provide a multi-modal large model based training and inference integrated video analysis system, which comprises a preprocessing module and an execution module, wherein:

[0037] The preprocessing module is configured to receive a natural language input by a user for executing a video analysis task;

[0038] and convert the natural language into a task prompt word suitable for the video analysis task using a preset large language model;

[0039] and initialize a learnable weight layer corresponding to the video analysis task in a pre-trained open world object detection base model, and perform preliminary inference on real-time video stream combined with the task prompt word to obtain a preliminary detection result;

[0040] and based on small sample data formed by manually annotating the preliminary detection result, perform incremental fine-tuning training on the learnable weight layer, and output the learnable weight layer as a task plug-in corresponding to the video analysis task by updating the parameters of the learnable weight layer;

[0041] The execution module is configured to invoke the open-world target detection base large model, dynamically load a task plug-in and a task prompt word corresponding to a current video analysis task, analyze and infer real-time video stream data, and obtain an analysis result.

[0042] In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method in the first aspect when executing the computer program.

[0043] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the program is executable on a processor to implement the method in the first aspect.

[0044] Compared with the related art, the present application proposes a training and inference integrated video analysis method based on a multi-modal large model, solves the technical problems of high customization cost of an existing small model one-to-one, long response period of new demand, and serious dependence of algorithm power of a cascaded large model scheme, and difficulty in real-time processing of multiple video streams. A multi-task dynamic plug-in large model weight sharing inference method is adopted, a special large language model is used to convert a user natural language demand into a standardized detection prompt word, and an independent, lightweight plug-in that can be quickly fine-tuned through small sample data is generated for each detection task. During inference, the base large model weight is shared among multiple tasks, and only the plug-in corresponding to the task is loaded for feature fusion and calculation. The present application can achieve the positive effects of "one sentence generating a new algorithm" and small sample minute-level rapid scene optimization, and can support high-concurrency real-time analysis of hundreds of video streams with a single acceleration card, significantly improving algorithm iteration efficiency and reducing development and deployment cost. BRIEF DESCRIPTION OF DRAWINGS

[0045] The accompanying drawings illustrated herein are used to provide further understanding of the present application, and constitute a part of the present application. The illustrative embodiments of the present application and their description serve to explain the present application. In the drawings:

[0046] Figure 1 is a flowchart of a training and inference integrated video analysis method based on a multi-modal large model according to an embodiment of the present application;

[0047] Figure 2 is a structural block diagram of a training and inference integrated video analysis system based on a multi-modal large model according to an embodiment of the present application;

[0048] Figure 3 is an architecture diagram of a training and inference integrated video analysis system based on a multi-modal large model according to an embodiment of the present application;

[0049] Figure 4 is a schematic diagram of an internal structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. Based on the embodiments provided by the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort fall within the scope of the present application.

[0051] Obviously, the accompanying drawings in the following description are only some examples or embodiments of the present application, and for those of ordinary skill in the art, the present application can be applied to other similar scenarios without creative effort based on the accompanying drawings. In addition, it can be understood that although the efforts made in the development process can be complex and lengthy, for those of ordinary skill in the art related to the content disclosed in the present application, some design, manufacture or production changes based on the technical content disclosed in the present application are only routine technical means and should not be understood as insufficient disclosure of the content disclosed in the present application.

[0052] In the present application, the phrase "embodiments" means that the specific features, structures or characteristics described in conjunction with the embodiments can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor is it an independent or alternative embodiment to other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in the present application can be combined with other embodiments without conflict.

[0053] Unless otherwise defined, technical terms and scientific terms used in the present application shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terms "a", "an", "one", "this", and similar referents in the context of describing the application are to be construed to be inclusive, not exclusive. For example, the phrases "comprising a", "comprising one", "having a", "having one", "including a", "including one", "containing a", "containing one", "consisting of a", "consisting of one", "consisting essentially of a", "consisting essentially of one", and the like are not limited to the features listed thereafter, but include one or more of the features listed thereafter, and other features that are no less restrictive. The term "comprising", "including", "containing", and "consisting essentially of" shall be construed to encompass the elements or steps listed thereafter, as well as other non-recited elements or steps of any nature that can affect the final product or process. The terms "connected", "coupled", and "linked" are not limited to direct connections, but can include indirect connections unless otherwise specified. The term "plurality" means two or more. The term "and / or" describes the association relationship of associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that A exists alone, A and B exist together, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects. The terms "first", "second", "third", and the like are merely to distinguish similar objects, and do not represent a specific order for the objects.

[0054] The current mainstream method of video analysis is small model technology based on deep learning, such as YOLO, SSD, Resnet, etc. The advantage of small model is fast inference speed, but its disadvantage is also obvious. Since each new scene needs to collect data, label samples and retrain the model separately, the customization cost is high. From the demand to the model deployment, it usually takes months of sample collection and iterative optimization process, which significantly prolongs the response period of new demand. More importantly, as the application scenarios continue to increase, the cost of model maintenance and update increases linearly, resulting in a continuous increase in marginal cost.

[0055] With the development of large model technology, the application of large model in the field of video analysis has gradually emerged. The main mode is "small model + VLM (visual-linguistic multimodal large model) " cascade mode. The advantage of this mode is to use the powerful high-level semantic understanding ability of large model to handle complex open domain problems and natural language interaction requirements. However, this mode has obvious performance bottlenecks: first, visual language large model usually contains tens of billions of parameters, and is highly dependent on computing power, which is generally difficult to meet the real-time requirements of video analysis. More importantly, due to the use of cascade processing flow, the detection accuracy is still largely limited by the detection accuracy of the front-end small model.

[0056] Therefore, the embodiments of the present application provide a multi-modal large model training and deployment integrated video analysis method,Figure 1 is a flowchart of a multi-modal large model-based training and inference integrated video analysis method according to an embodiment of the present application, as shown in the figure, the flow includes the following steps: Figure 1

[0057] S101, receiving a natural language input by a user for performing a video analysis task;

[0058] This step is the starting point of the video analysis process, the core is to receive the video analysis demand expressed in natural language form by the user through the system interface without the user mastering professional technical terms or model instructions, and only through daily colloquial expression can the task be initiated.

[0059] Specifically, the natural language input by the user does not need to follow a specific format or technical specification. For example:

[0060] For behavior detection: "Help me detect whether the employees in the factory are wearing safety helmets" "Is there anyone smoking in the monitoring picture?";

[0061] For object recognition: "Count the number of cartons in the warehouse" "See if there is a forklift passing through the picture";

[0062] For state judgment: "Detect whether there is abnormal vibration when the equipment is running" "Judge whether there is an irrelevant person entering the area".

[0063] These natural language inputs directly reflect the actual needs of the user without involving professional information such as model parameters and feature dimensions, and the subsequent stage utilizes the advanced semantic understanding ability of the large model to handle complex open-domain problems and natural language interaction requirements.

[0064] Further, the natural language input by the user needs to be clearly directed to a specific video analysis target, which can cover three types of core information:

[0065] Target object: such as "person" "forklift" "carton" and other specific objects;

[0066] Object behavior: such as "cell phone" "wearing a safety helmet" "smoking" and other actions or states;

[0067] Scene constraint: such as "inside the factory workshop" "eastern region of the warehouse" and other space or environment restrictions (if needed).

[0068] In addition, in the present application scheme, the received natural language can support the description of various general or long-tail targets without being limited to predefined fixed categories, and the corresponding scene can adapt to diversified video analysis demand scenarios, including but not limited to: security monitoring, industrial quality inspection, and intelligent transportation, etc.

[0069] ​Further, the input carrier can be realized through a text box, voice-to-text, and the like of a system front-end interface (such as a web end or an APP end).

[0070] Step S101 serves as a starting point of the flow, connects the user with the system, converts the actual business requirement of the user into original data processable by the system, provides an input basis for subsequent steps, and ensures that the subsequent model processing can accurately match the user's intention. In addition, by supporting natural language input, the tedious process of manually configuring parameters and writing instructions for traditional small models can also be avoided.

[0071] S102, converting the natural language into a task prompt word suitable for the video analysis task by using a preset large language model;

[0072] The core logic of step S102 is to convert the user's colloquial and fuzzy natural language requirement into a task prompt word that is accurate, structured, and can be directly understood by the subsequent open world object detection model (OWOD) through a large language model optimized specially. The key of this step is to ensure that the user's requirement is accurately translated into an executable instruction of the model, and to provide accurate semantic guidance for the subsequent target detection reasoning. Specifically, the construction process of the model includes the following two stages:

[0073] First stage: supervised fine-tuning

[0074] Supervised fine-tuning is a basic step of "field adaptation" for a general large language model, and the purpose is to enable the model to master the mapping rules of natural language and task prompt words in the field of video analysis. The data set involved in fine-tuning is a high-quality labeled data set specially constructed for the video analysis scene, which is composed of mapping pairs of "natural language description-standard detection task prompt word" in this scheme.

[0075] Specifically, the natural language description simulates the user's daily input, and the "standard detection task prompt word" is an accurate expression optimized in the field. It should be noted that in this embodiment, such mapping pairs need to cover multiple scenarios and multiple target types, including general objects (such as "people" and "cars"), and also cover long-tail targets (such as "specific model of equipment" and "rare behavior state"), and need to reflect the specific features of the target (such as action, shape, and state), to ensure that the model learns the conversion logic from the fuzzy description to the accurate instruction.

[0076] During the training process of the preset large language model, the natural language description is taken as the input, and the corresponding standard prompt word is taken as the output target. By minimizing the difference between the prediction result and the standard prompt word (such as using a cross-entropy loss function), the professional terms in the field of video analysis, the target feature expression rules, and the semantic conversion logic are learned.

[0077] Second stage: reinforcement learning optimization

[0078] It can be understood that the reinforcement learning optimization is an "effect optimization" link based on supervised fine-tuning. The present scheme dynamically adjusts the model output by introducing human feedback to ensure that the generated task prompt words are more in line with actual business needs.

[0079] The model fine-tuned in the first stage may still have output bias. At this time, the prompt words generated by the model are evaluated by humans to form "feedback signals", that is, high-quality prompt words are given positive rewards and low-quality prompt words are given negative penalties. These feedback signals are used to train a "reward model" that can automatically evaluate the quality of newly generated prompt words.

[0080] Based on the reward model, the PPO (Proximal Policy Optimization) algorithm in reinforcement learning is used to adjust the strategy of the large language model: when generating prompt words, the model will dynamically optimize the output strategy according to the feedback of the reward model, and preferentially generate prompt words that are more likely to obtain high rewards (such as more accurate and more in line with target feature expressions).

[0081] Through the above two-stage fine-tuning of step S102, the pre-trained large language model can accurately understand the video analysis requirements implied in the user's natural language and generate task prompt words that not only meet the input specifications of the OWOD model but also conform to the actual detection target features, providing high-quality semantic input for subsequent target detection reasoning.

[0082] S103, in the pre-trained open world target detection base model, initialize the learnable weight layer corresponding to the video analysis task, and combine the task prompt words to perform preliminary reasoning on the real-time video stream to obtain preliminary detection results;

[0083] It should be noted that before step S103, there is also a step of training the open world target detection base model obtained, specifically:

[0084] The open world target detection base model in the present embodiment adopts a double tower structure; wherein the visual encoder is responsible for extracting visual features from the real-time video stream, and can use ViT (Visual Transformer) or CNN (Convolutional Neural Network) as the basic architecture to perform multi-scale feature extraction on video frame images, outputting visual feature vectors containing target shape, color, texture, etc. For example, in a factory monitoring scene, the visual features of "workers", "safety helmets", "forklifts", etc. can be extracted.

[0085] The text encoder is responsible for parsing the detection task prompt words to extract text semantic features; based on BERT or CLIP model architecture, the prompt words are converted into text feature vectors containing target categories and behavior descriptions, and an input sequence is constructed through CLS (sequence start) and SEP (separator) to ensure semantic integrity.

[0086] Further, the open world target detection base large model realizes deep interaction of visual features and text features through a cross-modal fusion module. Optionally, through a cross-attention mechanism, the visual features focus on the key information described in the text, and the text features focus on the corresponding target area in the vision, and finally generate features that fuse multi-modal information.

[0087] In addition, the model completes target detection based on the fused features, that is, locates and identifies objects in the image that match the text description, and the input is the visual features and text features processed by the deep cross-modal fusion module;

[0088] During the model training process, the alignment loss function is used to evaluate the matching degree of a certain area in the image and the words in the text description. At the same time, the cosine similarity between each visual area feature and the text phrase feature is calculated. If the similarity score is higher than the preset threshold, the system will consider that the area and the text phrase are successfully matched, and the final detection box coordinates are output in combination with the bounding box regression.

[0089] In addition, it should be noted that in order to train an open world target detection model that can cover all things, the data preparation needs to meet the requirements of multi-source heterogeneity, open vocabulary annotation, and coverage of long-tail distribution. Specifically, the data includes:

[0090] General scene public data set: such as COCO, ImageNet, etc., providing massive basic objects (people, vehicles, animals, etc.) images and annotations;

[0091] Vertical field specific data: such as factory, warehouse, traffic, etc. Scene-specific video / image data containing industry-specific targets (such as forklifts, safety helmets, assembly line equipment);

[0092] Self-constructed image-text alignment data: artificial annotation or through technical means to generate "image + descriptive text" pairs, for example, the image of a worker wearing a safety helmet corresponds to the text of a worker wearing a safety helmet.

[0093] Further, to break through the limitations of traditional fixed categories, rare targets or composite behaviors are allowed to be annotated; in addition, through operations such as deduplication, noise reduction, rotation / cropping, etc. Improve data quality to ensure that samples cover general scenarios and long-tail distribution, and finally form a super large-scale training library containing tens of thousands of target classes and their states, providing data support for the model to learn the characteristics of all things.

[0094] Through the above architecture design and training data, the pre-trained open world target detection base large model can process visual and text information at the same time, and realize flexible detection of various targets in the open world.

[0095] Further, step S102 uses a pre-trained open world object detection base model (OWOD) to complete preliminary detection and lay the foundation for subsequent fine-tuning optimization.

[0096] Among them, for the new video analysis task proposed by the user, a learnable weight layer is initialized for this task in the pre-trained OWOD base model; the initial parameters of the weight layer are empty or randomly assigned, and the weight layer is separated from the core weight of the base model, and the weight of the base model remains in a frozen state (does not participate in subsequent updates), and only the newly added learnable weight layer is used to adapt to new tasks.

[0097] It can be understood that the above steps avoid retraining the entire large model for each new task, reducing computational cost while preserving the general ability of the base model in all-thing recognition.

[0098] After initializing the learnable weight layer, the task prompt word and real-time video stream data of step S102 are input, the visual features of the video stream are extracted through the visual encoder, and the text features of the task prompt word are analyzed through the text encoder;

[0099] Further, the cross-modal fusion module fuses the two features, and the initialized learnable weight layer acts as an "empty plug-in" in the fusion process, but has not yet played an optimization role; the visual-text alignment module performs target matching based on the fused features and outputs the preliminary detection result.

[0100] In this embodiment, the preliminary detection result relies on the general ability of the base model to complete preliminary recognition, and is stored in the event database (including video frame images, detection boxes, labels, etc.) as "raw materials" for subsequent fine-tuning., artificial labeling forms a small sample data.

[0101] Step S103, by initializing an independent learnable weight layer combined with the base model preliminary reasoning, the first detection of the new task is completed while ensuring efficiency, which not only utilizes the general ability of the pre-trained large model, but also improves the data basis through small sample fine-tuning for subsequent.

[0102] S104, under the condition that the weight of the open world object detection large model remains frozen, based on the small sample data formed by manually labeling the preliminary detection result, the learnable weight layer is incrementally fine-tuned and trained, and the parameters of the learnable weight layer are updated and output as a task plug-in corresponding to the video analysis task;

[0103] This step optimizes the learnable weight layer through small sample data labeled by artificial marking, and updates the parameters of the learnable weight layer by freezing the base model, which not only preserves the general ability of the large model, but also quickly adapts to new tasks through small samples, so that it adapts to specific video analysis tasks, and finally forms a dedicated task plug-in.

[0104] Wherein, the preliminary detection results are manually annotated to form small sample data, which specifically includes the following sub-steps:

[0105] S1, constructing an event database based on the preliminary detection results and corresponding original video frames and labels;

[0106] S2, manually annotating the detection results in the event database to classify the detection results into correct samples or error samples, wherein: the annotated correct samples and their corresponding detection task prompt words, original video frames and labels constitute a positive sample pair, and the annotated error samples and their corresponding original video frames, labels, and preset non-target prompt words constitute a negative sample pair,

[0107] S4, based on the positive sample pair and the negative sample pair, constructing small sample data for incremental fine-tuning training.

[0108] Through the above steps S1-S4, the task data in the event database is manually annotated, that is, the results of the basic large model detection are simply classified as correct or incorrect. The training prompt word corresponding to the error sample is uniformly set to "non-target", and the training prompt word corresponding to the error sample is the inference prompt word of each task. In this embodiment, only 100 or less samples need to be annotated for a single task to form small sample data for parameter updating.

[0109] Further, after obtaining the manually annotated data, fine-tune each empty plug-in through the manually annotated data, which specifically includes the following sub-steps:

[0110] S1, feature extraction;

[0111] The video frame image data is input into the visual encoder (weight frozen) to extract visual features, and the prompt word is input into the text encoder (weight frozen) to extract text features. Specifically, the formula involved in this process is as follows:

[0112]

[0113] Wherein, Fv is the multi-scale visual feature generated by the visual encoder, I is the input image, Ft is the text feature generated by the text encoder, prompt is the prompt word, [CLS] and [SEP] represent the beginning of the sequence and the separator respectively, and are used to construct the input sequence.

[0114] S2, adding the learnable weight of the initial plug-in to the text feature to generate an enhanced text feature;

[0115] It can be understood that this step adds the newly added learnable weight layer (i.e. plug-in) consistent with the dimension of the text feature to the feature output by the text encoder to obtain the final text feature , the plug-in parameter enhances the semantic information related to the new task in the text feature, providing more accurate text guidance for subsequent cross-modal fusion;

[0116] S2, the enhanced text feature and the visual feature are interactively fused through the cross-modal fusion module to obtain the fused visual feature and the fused text feature;

[0117] The formula involved in this process is as follows:

[0118]

[0119] where P is the plug-in learnable parameter, is the fused visual feature, is the fused text feature, is the learnable parameter, and d is the dimension of the feature vector.

[0120] S3, based on the fused visual feature and the fused text feature, the matching degree of each visual region in the visual feature and each semantic unit in the text feature is calculated, and the alignment loss is determined according to the matching degree;

[0121] The formula involved in this process is as follows:

[0122]

[0123] where, is the alignment loss, si,+ is the cosine similarity between the i-th visual region and the positive class text based on the fused visual feature and the fused text feature, and si,j is the cosine similarity between the i-th visual region and all texts based on the fused visual feature and the fused text feature, is the temperature coefficient, and N is the number of visual regions involved in the calculation;

[0124] This alignment loss is used to evaluate the matching degree between image regions and text words. If the matching degree of the visual region and the corresponding semantic unit is low, the loss value will increase, and vice versa. The purpose is to constrain the model to learn the corresponding relationship between the visual feature and the text feature, and ensure that the model can correctly associate the specific region in the image with the correct text description.

[0125] S4, based on the fused visual feature, the prediction box is generated through the bounding box regression, and the bounding box regression loss is calculated according to the prediction box and the real detection box in the corresponding small sample data;

[0126] The formula involved in this process is as follows:

[0127]

[0128] This is the bounding box regression loss, where bpred is the predicted bounding box coordinates, bgt is the ground truth bounding box coordinates, and λ is the weight coefficient. It is a loss function based on the generalized intersection-union ratio (GIoU).

[0129] The bounding box regression loss is used to calculate the coordinate difference between the predicted box and the ground truth box, constraining the model to output a bounding box that is closer to the position of the ground truth target when detecting a target, thereby improving the localization accuracy of target detection.

[0130] S5 constructs a composite loss function based on alignment loss and bounding box regression loss, and updates the weight parameters of the plugin module through backpropagation based on the composite loss function.

[0131] The composite loss function can be expressed by the following formula:

[0132]

[0133] It is understandable that by using the above composite loss function and the backpropagation algorithm, only the parameters of the learnable weight layer (plugin) are adjusted to minimize the total loss. During this process, the weights of the basic large model are always frozen, and the updated learnable weight layer serves as a task plugin to achieve accurate adaptation to a specific task.

[0134] Through step S103 above, with the weights of the large open-world object detection model frozen, incremental fine-tuning training is performed on the learnable weight layer using small sample data generated from manually labeled preliminary detection results. By updating the parameters of this layer and outputting it as a task plugin for the corresponding video analysis task, it is possible to quickly and accurately adapt to a specific video analysis task without altering the core capabilities of the basic large model, significantly improving the model's object detection accuracy and adaptability for that task.

[0135] Furthermore, it should be noted that during the incremental fine-tuning training and inference phases, when processing multiple video analysis tasks in parallel, this method achieves efficient computing power reuse and real-time optimization through text feature sharing and visual feature sharing mechanisms. Specifically:

[0136] When multiple video analytics tasks use the exact same detection cue word (e.g., multiple scenes simultaneously executing "detect people holding mobile phones"), the text encoder only needs to extract features from that cue word once. The generated text features will be shared by all tasks with the same cue word. For example, when monitoring tasks in different factory areas all need to detect people holding mobile phones close to their heads, sharing the same text features avoids repeatedly parsing the cue word and reduces redundant computation.

[0137] In addition, if multiple tasks are based on the same video stream data, such as simultaneously detecting "people making phone calls", "forklifts", and "cartons" on the same monitoring screen, the visual encoder only needs to extract visual features for the video stream once, which can be reused by all related tasks. For example, when tasks 1-3 need to be performed simultaneously on the same monitoring screen in a factory, the visual features of the shared video frames do not need to be processed repeatedly, which significantly reduces the computational power consumption.

[0138] Specifically, when multiple tasks are performed simultaneously, the exemplary configuration information of each task is as follows:

[0139] Task 1 (detecting people making phone calls): prompt words = ["non-target", "people holding phones close to their heads"], and the fine-tuned output plug-in 1.

[0140] Task 2 (detecting forklifts): prompt words = ["non-target", "people holding phones close to their heads", "forklifts"], and the fine-tuned output plug-in 2.

[0141] Task 3 ("detecting cartons"): prompt words = ["non-target", "people holding phones close to their heads", "forklifts", "cartons"], and the fine-tuned output plug-in 3.

[0142] It can be understood that the above three tasks are executed in sequence, and only the prompt words have hierarchical differences; the prompt words of task 1 contain the descriptions of "non-target" and "making phone calls", task 2 adds "forklifts" based on the above, and task 3 adds "cartons". Each task generates a dedicated plug-in through fine-tuning, and shares the frozen weights of the base model, and only distinguishes tasks through prompt words and plug-ins.

[0143] In this embodiment, the feature sharing mechanism used in incremental fine-tuning reduces the repeated feature extraction process, reduces the dependence on computational power while ensuring multiple task parallelism, and can realize the reuse of the base model capability of multiple tasks, avoid repeated training, and reduce the consumption of computational power. At the same time, through incremental prompt words and plug-ins, the detection target can be quickly expanded, and the real-time performance and scalability are taken into account, solving the problem of "high customization cost and increasing marginal cost" of traditional small models, and realizing real-time and efficient inference of large models on multiple video stream tasks.

[0144] In addition, in specific deployment scenarios, the task plug-in after incremental fine-tuning is applied to the key link of actual inference scenarios, and through dynamic deployment and effect verification, it is ensured that the plug-in can effectively improve the detection performance and meet the real-time requirements.

[0145] The fine-tuned generated task plug-in is deployed to the OWOD inference service through a dynamic loading mechanism. This process does not require restarting the entire inference system, and only the weight parameters of the plug-in need to be injected into the preset interface, so that the inference service can immediately call the plug-in to process the corresponding task (for example, after adding a plug-in for detecting paper boxes based on the original detection of forklifts, the system can support this task in real time). This design in the embodiment realizes flexible expansion of tasks and avoids system interruption during traditional model updating, in line with the core architecture of "multi-task dynamic plug-in".

[0146] In addition, by comparing the detection results before and after the plug-in is updated, it is ensured that the fine-tuned plug-in can effectively correct the errors of preliminary inference and improve the detection accuracy of specific tasks. Specifically, the response speed of the inference service after the plug-in is deployed can be verified relying on acceleration hardware (such as Huawei Ascend 910B and Cambrian MLU370). For example, when processing 100 video streams in single-card concurrency, the delay of multiple tasks performed simultaneously per stream needs to be controlled within milliseconds to ensure that the needs of real-time monitoring, dynamic early warning and other scenarios are met.

[0147] S105, calling an open world object detection base model, dynamically loading a task plug-in and a task prompt word corresponding to a current video analysis task, analyzing and inferring real-time video stream data to obtain an analysis result.

[0148] The specific process includes:

[0149] 1) Data input: video stream image, plug-in, plug-in and prompt word corresponding to the task.

[0150] 2) Feature extraction stage: the video stream image is input into a visual encoder to extract visual features, and the prompt word is input into a text encoder to extract text features.

[0151] 3) Cross-modal fusion: the text features are added to the corresponding plug-in and then input into a cross-modal fusion module together with the visual features to obtain fused visual features and text features.

[0152] 4) Detection box output: the cosine similarity of each visual feature and text feature is calculated to realize the matching of regions and phrases, and then the boundary box regression is performed on the high-score region to output the detection box coordinates.

[0153] wherein, Ri is the i-th visual region feature, Tj is the j-th text phrase feature, Si,j is the cosine similarity, For the i-th visual region, the index j of the text phrase that can maximize the cosine similarity between the i-th visual region and the text phrase feature, scorei is the maximum value in the cosine similarity between the i-th visual region and all text phrase features, θ is a threshold value, and P is a text phrase that satisfies the condition The visual region features Ri and the corresponding optimal text phrase features The set that is formed For the weights of the regression layer, For the regression layer bias, For the final predicted bounding box coordinates, It is the bounding box adjustment amount.

[0154] It is understandable that this step automatically matches and loads the corresponding task plugin (such as plugin two) and task prompt words (such as ["non-target", "person holding a mobile phone and close to the head", "forklift"] for the current video analysis task (such as "detect forklift").

[0155] Furthermore, the visual encoder extracts visual features from the real-time video stream, the text encoder parses the text features of the task prompts, the text features are added with the weights of the corresponding task plugins, and then interact with the visual features through the cross-modal fusion module. Finally, the detection results are output through the visual-text alignment module.

[0156] In practical applications, the same basic large model can be quickly switched to different tasks (such as switching from "detecting forklifts" to "detecting cardboard boxes") through a dynamic loading mechanism, without the need for repeated training. At the same time, the optimization of sharing visual features of the same video and text features of the same prompt words greatly improves the efficiency of multi-task parallel inference and meets real-time requirements.

[0157] Through the steps S101 to S103 described above, the "integrated training and inference" approach in this application is an integrated architecture that deeply integrates and dynamically coordinates the two processes of model training and inference. It breaks away from the traditional model where training and inference are separated and performed sequentially. By linking "inference to generate data and training to optimize inference" in real time, it solves the problems of long model iteration cycles and high scenario adaptation costs in the traditional model.

[0158] Both training and inference are based on the same large OWOD model. During inference, the model processes real-time video streams and outputs preliminary results. During training, the model's underlying feature extraction capabilities (weight freezing) are reused, and only the newly added "plugin" parts are fine-tuned, avoiding the huge overhead of repeated calculations and full retraining. The detection results generated during inference (such as video frames and detection boxes) are automatically stored in the database. After a small amount of manual annotation, they are directly used as training data to optimize the plugins, forming an automated closed loop of "inference → data → annotation → training → optimized inference", realizing the continuous self-evolution of the model.

[0159] After the training is completed, the generated optimized plug-in can be dynamically loaded into the running inference service without interrupting or redeploying the entire model. Compared with the traditional "training-pushing separation" mode, the integrated architecture of the application scheme can realize fast response and continuous optimization. The new scene adaptation period that originally takes months is shortened to minutes, and the model accuracy can be continuously improved through an automated pipeline, significantly reducing the cost of manpower and computing power, and realizing the efficient video analysis capability of "one sentence to generate new algorithm, small sample to quickly optimize".

[0160] In another aspect, the embodiments of the application also provide a multi-modal large model-based training-pushing integrated video analysis system, Figure 2 is a structural block diagram of a multi-modal large model-based training-pushing integrated video analysis system according to an embodiment of the application, as Figure 2 shown, the system comprises a preprocessing module 20 and an execution module 21, wherein:

[0161] The preprocessing module 20 is configured to receive a natural language input by a user for performing a video analysis task; and convert the natural language into a task prompt word suitable for the video analysis task by using a preset large language model;

[0162] and initialize a learnable weight layer corresponding to the video analysis task in a pre-trained open world target detection base large model, and perform preliminary inference on a real-time video stream in combination with the task prompt word to obtain a preliminary detection result;

[0163] and under the condition that the weights of the open world target detection large model are kept frozen, based on small sample data formed by manually labeling the preliminary detection result, the learnable weight layer is incrementally fine-tuned and trained, and the parameters of the learnable weight layer are updated and output as a task plug-in corresponding to the video analysis task;

[0164] The execution module 21 is configured to, in an inference stage, call the open world target detection base large model, dynamically load the task plug-in and the task prompt word corresponding to the current video analysis task, and analyze and infer the real-time video stream data to obtain an analysis result.

[0165] In addition, Figure 3 is an architecture diagram of a multi-modal large model-based training-pushing integrated video analysis system according to an embodiment of the application, as Figure 3 shown, which shows a training-pushing integrated video analysis architecture combining a large language model (LLM) and an open world target detection large model (OWOD).

[0166] The core process of the architecture starts from Figure 3The "one-sentence generation algorithm" in the upper left corner, for example, after the user inputs the natural language requirement "help me detect the person holding the phone", the large language model will parse and optimize it into a more accurate prompt word, such as "person holding the phone close to the head", and then hand it over to the OWOD large model inference service for processing.

[0167] Further, in Figure 3 In the main process on the right, the OWOD inference service receives real-time video streams and optimized prompt words, analyzes them through visual and text encoders and cross-modal fusion modules, and outputs preliminary detection results. These results are used as real-time analysis outputs and stored in an event database.

[0168] The "small sample fast fine-tuning" section shows the manual labeling of these stored results, which can be completed with only a small amount of sample data. Subsequently, in the OWOD large model training service below, the system uses these labeled small sample data to incrementally fine-tune the base large model. This process does not update the large model itself, but generates independent, lightweight plugins, each corresponding to a specific detection task. Finally, these trained plugins are dynamically loaded back into the inference service, allowing for rapid and efficient optimization of detection accuracy in specific scenarios without changing the base model, forming a closed-loop optimization system from requirement input, model inference, data labeling, to incremental training and dynamic updating.

[0169] Through the above system, the technical problems of high customization cost of existing small models, long response period for new requirements, and serious dependence on algorithm power of cascading large model solutions, which are difficult to process multiple video streams in real time, are solved. The multi-task dynamic plugin large model weight sharing inference algorithm uses a dedicated large language model to convert user natural language requirements into standardized detection prompt words and generates independent, lightweight "plugins" that can be quickly fine-tuned with small sample data. During inference, the base large model weight is shared among multiple tasks, and the system only loads the corresponding plugin for feature fusion and calculation. This application achieves the positive effects of "one-sentence generation of new algorithms" and small sample minute-level fast scene optimization, and can support high-concurrency real-time analysis of hundreds of video streams with a single acceleration card, significantly improving algorithm iteration efficiency and reducing development and deployment costs.

[0170] In one embodiment, Figure 4 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the application, as Figure 4 shown, an electronic device, which can be a server, is provided, and the internal structure diagram of the electronic device can be as Figure 4As shown. The electronic device includes a processor, a network interface, an internal memory and a non-volatile memory connected through an internal bus, wherein the non-volatile memory stores an operating system, a computer program and a database. The processor is used to provide computing and control capabilities, the network interface is used to communicate with external terminals through network connection, the internal memory is used to provide an environment for the operation of the operating system, and the computer program is executed by the processor to implement a multi-modal large model-based training and prediction integrated video analysis method. The database is used to store data.

[0171] Those skilled in the art can understand that, Figure 4 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. Specifically, the electronic device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0172] In one embodiment, a computer program product is provided, including a computer program, characterized by being executed by a processor to implement any of the above-mentioned multi-modal large model-based training and prediction integrated video analysis methods.

[0173] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0174] The above embodiments only express several implementation ways of the present application, and the description is specific and detailed, but it should not be understood as a limitation to the patent scope of the application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A multi-modal large model-based training and inference integrated video analysis method, characterized in that, The method comprises: receiving a user input natural language for performing a video analysis task; using a preset large language model to convert the natural language into a task prompt word suitable for the video analysis task; initializing a learnable weight layer corresponding to the video analysis task in a pre-trained open-world target detection base model, and combining the task prompt word to perform preliminary inference on a real-time video stream to obtain a preliminary detection result; under the condition that the weights of the open-world target detection base model are frozen, performing incremental fine-tuning training on the learnable weight layer based on small sample data formed by manually annotating the preliminary detection result, and outputting the learnable weight layer as a task plug-in corresponding to the video analysis task by updating the parameters of the learnable weight layer; wherein the small sample data formed by manually annotating the preliminary detection result comprises: constructing an event database based on the preliminary detection result, the corresponding original video frame and label; manually annotating the detection result in the event database to classify the detection result as a correct sample or an incorrect sample, wherein: annotated correct samples and their corresponding task prompt words, original video frames and labels constitute a positive sample pair, and annotated incorrect samples and their corresponding original video frames, labels, and preset non-target prompt words constitute a negative sample pair, based on the positive sample pair and the negative sample pair, the small sample data for the incremental fine-tuning training is constructed; calling the open-world target detection base model, dynamically loading the task plug-in and the task prompt word corresponding to the current video analysis task, and performing analysis and inference on real-time video stream data to obtain an analysis result.

2. The method of claim 1, wherein, The open-world target detection base model adopts a double-tower structure, comprising: a visual encoder for extracting visual features from video stream data, and a text encoder for extracting text features from the task prompt word; The open-world target detection base model interacts and fuses the visual features and the text features through a cross-modal fusion module to obtain fusion features; and, through a visual-text alignment module, based on the fusion features, target detection in a video is realized.

3. The method of claim 2, wherein, The incremental fine-tuning training of the learnable weight layer based on the small sample data formed by manually annotating the preliminary detection result comprises: adding the learnable weight of the initial task plug-in to the text features to generate enhanced text features; interacting and fusing the enhanced text features and the visual features through the cross-modal fusion module to obtain fused visual features and fused text features; based on the fused visual features and the fused text features, calculating the matching degree of each visual region in the visual features and each semantic unit in the text features, and determining an alignment loss according to the matching degree; based on the fused visual features, generating a prediction box through a bounding box regression, and calculating a bounding box regression loss according to the prediction box and the real detection box in the small sample data corresponding to the prediction box; A composite loss function is constructed based on the alignment loss and the bounding box regression loss, and the weight parameters of the task plug-in are updated by back propagation based on the composite loss function.

4. The method of claim 3, wherein, The composite loss function is represented by the following formula: ； ； ； wherein, is the composite loss function, is the alignment loss, is the bounding box regression loss, is the cosine similarity between the i-th visual region and the positive class text, is the cosine similarity between the i-th visual region and all texts, is the temperature coefficient, N is the number of visual regions participating in the calculation, is the predicted box coordinates, is the real box coordinates, λ is the weight coefficient, is the loss function based on generalized intersection over union.

5. The method of claim 2, wherein, In the incremental fine-tuning training phase and the inference phase, when multiple video analysis tasks are processed in parallel, the method further comprises: For different video analysis tasks using the same task prompt word, real-time analysis training is performed by sharing the text features extracted by the text encoder for the task prompt word; For different video analysis tasks based on the same video stream data, real-time analysis inference is performed by sharing the visual features extracted by the visual encoder for the video stream data.

6. The method of claim 1, wherein, The pre-set large language model is a video analysis prompt word dedicated large language model, which is obtained by fine-tuning a general large language model in the following two stages: The first stage of supervised fine-tuning includes training the general large language model based on a pre-constructed instruction fine-tuning dataset, wherein the instruction fine-tuning dataset contains a plurality of mapping pairs composed of natural language descriptions in the video analysis field and their corresponding standard task prompt words; The second stage of reinforcement learning optimization includes introducing a reinforcement learning mechanism based on human feedback to continuously optimize the large language model after the first stage of fine-tuning, and obtaining the pre-set large language model after the generated task prompt word meets the expected result.

7. A multimodal large model-based training and inference integrated video analysis system, characterized in that, The system comprises a preprocessing module and an execution module, wherein: The preprocessing module is configured to receive a natural language input by a user for executing a video analysis task; and, using a pre-set large language model, the natural language is converted into a task prompt word suitable for the video analysis task; and, in a pre-trained open-world object detection base model, the learnable weight layer corresponding to the video analysis task is initialized, and the real-time video stream is preliminarily inferred in combination with the task prompt word to obtain a preliminary detection result; and, under the condition that the weights of the open-world object detection model remain frozen, the learnable weight layer is fine-tuned incrementally based on small sample data formed by manually annotating the preliminary detection result, and the parameters of the learnable weight layer are updated to be output as a task plug-in corresponding to the video analysis task; wherein the small sample data formed by manually annotating the preliminary detection result includes: based on the preliminary detection result, the corresponding original video frame and the label, an event database is constructed; The detection results in the event database are manually annotated, and the detection results are classified as correct samples or incorrect samples, wherein: The annotated correct samples and their corresponding task prompt words, original video frames and labels constitute positive sample pairs, and the annotated incorrect samples and their corresponding original video frames, labels, and pre-set non-target prompt words constitute negative sample pairs, Based on the positive sample pairs and the negative sample pairs, the small sample data used for the incremental fine-tuning training is constructed; The execution module is configured to call the open-world target detection base model, dynamically load a task plug-in and a task prompt word corresponding to a current video analysis task, analyze and infer real-time video stream data, and obtain an analysis result.

8. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method in any one of claims 1 to 6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method in any one of claims 1 to 6.

Images