Method and system for generating monitoring video user attention information based on large model

Abstract

The invention provides a method and system for generating monitoring video user attention information based on a large model, and the method comprises the steps: carrying out the fusion of a verified text description, a corresponding video path, and a prompt word, and constructing a training set; performing instruction fine tuning on the multi-modal visual language model by utilizing the training set and adopting a full-parameter fine tuning mode to obtain a fine-tuned multi-modal visual language model; deploying the fine-tuned multi-modal visual language model as a service interface through a vLLM inference engine; introducing object information into the cue word to obtain a reasoning cue word; inputting a monitoring video and a reasoning prompt word into a service interface, calling the finely-tuned multi-modal visual language model for reasoning, and generating a preliminary natural language description; performing optimization processing on the preliminary natural language description to obtain optimized natural language description; accurate understanding of the monitoring video and generation of natural language description meeting user requirements are realized.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer vision technology, and in particular to a method and system for generating user attention information in surveillance videos based on large models. Background Technology

[0002] In recent years, with the widespread use of home security cameras in daily life, ordinary users have generated a large amount of surveillance video in scenarios such as home security, childcare, and pet care. Faced with long-duration, low-information-density video playback, users often need to spend a lot of time searching to confirm key events. In actual use, users often only care about specific objects, behaviors, or events, the so-called "user-focused information." Therefore, there is an urgent need for a technical solution that can automatically understand the content of surveillance footage and generate accurate and easy-to-understand natural language descriptions based on user concerns, helping users quickly grasp what happened without having to search through recordings, thereby improving the efficiency of users obtaining key event information and meeting personalized needs.

[0003] With the rapid development of artificial intelligence technology, especially the widespread application of deep learning in computer vision and natural language processing, large-scale visual language models for multimodal tasks have gradually become a research hotspot. These models can understand image or video information and associate it with natural language, and are widely used in tasks such as video content generation and visual question answering.

[0004] While existing large-scale visual language models possess some image or video description capabilities, they still face significant limitations in surveillance scenarios. First, general-purpose models primarily rely on open-domain internet data during the pre-training phase, containing a large number of semantic patterns unrelated to surveillance (such as landscapes, art, and advertisements), lacking adaptability to the surveillance environment. Second, pre-training data is typically high-resolution, single-scene, and mostly from monocular cameras, while home surveillance footage is often low-resolution, complex, and heavily reliant on binocular cameras. This results in weak semantic representation capabilities of the models for surveillance videos, failing to capture information relevant to the user. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a method and system for generating user attention information in surveillance videos based on large models, so as to achieve accurate understanding of surveillance videos and generation of natural language descriptions that meet user needs.

[0006] In a first aspect, embodiments of the present invention provide a method for generating user attention information in surveillance videos based on a large model, the method comprising: Identify the objects that the user is interested in, and collect the raw video data of the objects in the monitoring scenario. The original video data is segmented, cleaned, and quality-filtered to obtain multiple high-quality video clips; Based on the business scenario, prompt words are set. These prompt words are used to guide the large model to focus on key information of different types of objects when generating descriptions. The multiple high-quality video clips and the prompt words are input into the visual language model to obtain a text description. The text description is validated to obtain the validated text description; The verified text description is fused with the corresponding video path and the prompt words to construct a training set; Using the training set, the multimodal visual language model is fine-tuned using a full-parameter fine-tuning method to obtain the fine-tuned multimodal visual language model. The finely tuned multimodal visual language model is deployed as a service interface through the vLLM inference engine; By incorporating object information into the prompt words, inference prompt words are obtained; The monitoring video and the inference prompts are input into the service interface, and the fine-tuned multimodal visual language model is invoked to perform inference and generate a preliminary natural language description. The initial natural language description is optimized to obtain an optimized natural language description.

[0007] Furthermore, prompt words are set according to the business scenario. These prompt words are used to guide the large model to focus on key information of different types of objects when generating descriptions, including: When the scene of the business scenario contains only a single person, the prompt words guide the large model to focus on describing the number of people, their gender, clothing characteristics, behavior, and environment. When the scene of the business scenario contains only a single pet, the prompt word guides the large model to focus on the pet's type, number, behavior status, and environmental information; When the scene of the business scenario contains multiple characters, the prompt words guide the large model to focus on describing the number of characters, group behavior characteristics, and environmental background; When both the character and the pet are present in the scene of the business scenario, the prompt word guides the large model to focus on the interactive behavior and scene features between the character and the pet; When the object of interest is not present in the scene of the business scenario, the prompt word guides the large model to provide a brief description of the overall scene.

[0008] Furthermore, the original video data is segmented, cleaned, and quality-screened to obtain multiple high-quality video clips, including: The original video data is divided according to the time length to obtain multiple segmented video clips; The multiple segmented video segments are cleaned and quality filtered to obtain multiple high-quality video segments.

[0009] Furthermore, the object information is obtained in the following ways: Acquire the surveillance video; Image frames extracted from the surveillance video are identified using a target detection algorithm to obtain the object information.

[0010] Furthermore, the finely tuned multimodal visual language model is adapted to surveillance video scenarios and has the ability to generate the user-focused information.

[0011] Secondly, embodiments of the present invention provide a system for generating user attention information in surveillance videos based on a large model, the system comprising: The acquisition module is used to acquire the objects that the user is interested in and collect the raw video data of the objects in the monitoring scenario. The processing module is used to segment, clean, and quality-screen the raw video data to obtain multiple high-quality video clips; The setting module is used to set prompt words according to the business scenario. The prompt words are used to guide the large model to focus on the key information of different types of objects when generating descriptions. The input module is used to input the multiple high-quality video clips and the prompt words into the visual language model to obtain a text description; The verification module verifies the text description to obtain a verified text description. The construction module is used to fuse the verified text description with the corresponding video path and the prompt words to construct a training set; The fine-tuning module is used to fine-tune the multimodal visual language model using the training set and a full-parameter fine-tuning method, so as to obtain the fine-tuned multimodal visual language model. The deployment module is used to deploy the fine-tuned multimodal visual language model as a service interface through the vLLM inference engine; An import module is used to import object information into the prompt words to obtain inference prompt words; The inference module is used to input the surveillance video and the inference prompts into the service interface, call the fine-tuned multimodal visual language model to perform inference, and generate a preliminary natural language description. The optimization module is used to optimize the initial natural language description to obtain an optimized natural language description.

[0012] Furthermore, the setting module is specifically used for: When the scene of the business scenario contains only a single person, the prompt words guide the large model to focus on describing the number of people, their gender, clothing characteristics, behavior, and environment. When the scene of the business scenario contains only a single pet, the prompt word guides the large model to focus on the pet's type, number, behavior status, and environmental information; When the scene of the business scenario contains multiple characters, the prompt words guide the large model to focus on describing the number of characters, group behavior characteristics, and environmental background; When both the character and the pet are present in the scene of the business scenario, the prompt word guides the large model to focus on the interactive behavior and scene features between the character and the pet; When the object of interest is not present in the scene of the business scenario, the prompt word guides the large model to provide a brief description of the overall scene.

[0013] Furthermore, the processing module is specifically used for: The original video data is divided according to the time length to obtain multiple segmented video clips; The multiple segmented video segments are cleaned and quality filtered to obtain multiple high-quality video segments.

[0014] Thirdly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the method described above.

[0015] Fourthly, embodiments of the present invention provide a computer-readable medium having processor-executable non-volatile program code that causes the processor to perform the method described above.

[0016] This invention provides a method and system for generating user-focused information from surveillance videos based on a large-scale model. The method includes: acquiring objects of user interest and collecting raw video data of these objects in a surveillance scenario; segmenting, cleaning, and quality-filtering the raw video data to obtain multiple high-quality video clips; setting prompts based on the business scenario, where the prompts guide the large-scale model to focus on key information of different types of objects when generating descriptions; inputting the multiple high-quality video clips and prompts into a visual language large-scale model to obtain text descriptions; validating the text descriptions to obtain validated text descriptions; and matching the validated text descriptions with the corresponding video paths and prompts. The process involves: merging data to construct a training set; using this training set, fine-tuning the multimodal visual language model using a full-parameter fine-tuning method to obtain a fine-tuned multimodal visual language model; deploying this fine-tuned model as a service interface via the vLLM inference engine; incorporating object information into the prompts to obtain inference prompts; inputting the surveillance video and inference prompts into the service interface, calling the fine-tuned multimodal visual language model for inference, and generating a preliminary natural language description; optimizing the preliminary natural language description to obtain an optimized natural language description; and achieving accurate understanding of the surveillance video and generation of natural language descriptions that meet user needs.

[0017] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0018] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0019] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 The flowchart illustrates the method for generating user attention information in surveillance video based on a large model, as provided in Embodiment 1 of the present invention. Figure 2 This is a schematic diagram of a system for generating user attention information for surveillance videos based on a large model, as provided in Embodiment 2 of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] To facilitate understanding of this embodiment, the embodiments of the present invention will be described in detail below.

[0023] Example 1: Figure 1 This is a flowchart of a method for generating user attention information in surveillance video based on a large model, as provided in Embodiment 1 of the present invention.

[0024] Reference Figure 1 The method includes the following steps: Step S101: Obtain the object of interest for the user and collect the original video data of the object of interest in the monitoring scenario; Step S102 involves segmenting, cleaning, and quality screening the raw video data to obtain multiple high-quality video clips. Step S103: Set prompt words according to the business scenario. The prompt words are used to guide the large model to focus on the key information of different types of objects when generating descriptions. Step S104: Input multiple high-quality video clips and prompts into the visual language model to obtain text descriptions; Step S105: Verify the text description to obtain the verified text description; Step S106: The verified text description is fused with the corresponding video path and prompt words to construct a training set; Step S107: Using the training set, the multimodal visual language model is fine-tuned using a full-parameter fine-tuning method to obtain the fine-tuned multimodal visual language model. Step S108: Deploy the fine-tuned multimodal visual language model as a service interface through the vLLM inference engine; Step S109: Incorporate object information into the prompt words to obtain inference prompt words; Step S110: Input the monitoring video and reasoning prompts into the service interface, call the fine-tuned multimodal visual language model for reasoning, and generate a preliminary natural language description; Step S111: Optimize the initial natural language description to obtain an optimized natural language description.

[0025] To address the shortcomings of existing technologies, there is an urgent need for a technical solution capable of accurately generating natural language descriptions that align with user concerns in surveillance scenarios. This application achieves accurate generation of user-focused information in surveillance video scenarios through process design and optimization.

[0026] Specifically, the process begins by collecting video data related to user-interested categories and then segmenting the videos into short time clips. Next, prompts are designed; for example, when the video subject is a person, the prompts will focus on the number of people, their gender, actions, and the surrounding environment. Then, leveraging the capabilities of a general-purpose large-scale model, the prompts and video data are input to generate corresponding text descriptions. To ensure data quality, the text descriptions are manually verified. Finally, the prompts, video data, and text descriptions are merged to generate a training set that conforms to the fine-tuning format of the large-scale model. The large-scale model can be the Qianwen open-source model.

[0027] We fine-tuned all parameters of the Qwen2.5_VL large-scale vision-language model using a full-parameter approach with the training set. Through this fine-tuning process, the model can adapt to surveillance video scenarios and learn to generate natural language descriptions that match the information users are interested in.

[0028] The finely tuned multimodal visual language model is deployed as a service interface through the vLLM inference engine, providing an efficient operating environment for the inference and generation of video data.

[0029] During the inference phase, cues are combined with object detection information, which provides auxiliary cues about objects contained in video frames. This further improves the accuracy of the large model's description of objects of interest.

[0030] The monitoring video and the designed prompts are input into the deployed service interface, which calls the large model to complete the inference and generate a preliminary natural language description, thereby capturing the information that users care about.

[0031] The generated text descriptions are post-processed to remove useless descriptions and retain the core content, making the text descriptions more concise, accurate, and in line with user needs.

[0032] This application achieves this by fine-tuning a multimodal visual language model, enabling it to accurately understand surveillance video content and generate natural language descriptions that align with user concerns. By deploying the fine-tuned multimodal visual language model using a vLLM inference engine, a high-efficiency operating environment is provided, significantly improving inference speed and service concurrency performance. Object detection information is incorporated into the design of inference prompts, effectively enhancing the accuracy of the fine-tuned multimodal visual language model's descriptions of objects of interest. Finally, the generated text results are optimized by removing unnecessary descriptions, resulting in more concise and accurate descriptions.

[0033] Furthermore, step S103 includes the following steps: Step S201: When the scene of the business scenario contains only a single person, the prompt words guide the large model to focus on describing the number of people, their gender, clothing characteristics, behavior and environment. Step S202: When the scene of the business scenario contains only a single pet, the prompt word guides the large model to focus on the pet's type, number of pets, behavioral status and environmental information; Step S203: When the scene of the business scenario contains multiple people, the prompt words guide the large model to focus on describing the number of people, group behavior characteristics and environmental background; Step S204: When both people and pets are present in the scene of a business scenario, prompts guide the large model to focus on the interactive behavior between people and pets and the scene features. Step S205: When there is no object of interest in the business scenario, the prompt word guides the large model to give a brief description of the overall scenario.

[0034] Specifically, the prompts are closely related to specific business scenarios and are used to guide the large model to focus on key information of different types of objects when generating descriptions. For example: when the scene contains only a single person, the prompts should guide the model to focus on describing the number of people, their gender, clothing characteristics, behavior, and environment; when the scene contains only a single pet, the prompts should focus on the pet's species, number, behavior, and environmental information; when the scene contains multiple people, the prompts should guide the model to focus on describing the number of people, group behavior characteristics, and environmental background; when the scene contains both people and pets, the prompts should focus on the interaction between people and pets and scene characteristics; when there is no object of interest in the scene, the prompts only need to guide the model to provide a brief description of the overall scene.

[0035] Furthermore, in complex scenarios, prompts should prioritize guiding the model to capture dynamic information, ensuring that the description highlights changes in action and the occurrence of events. For binocular camera video footage, it should be treated as a complete, unified scene, avoiding distinctions between upper and lower areas and phrases such as "upper half of the video" or "lower half of the video." To enhance the professionalism and conciseness of the description, prompts should also limit redundant or leading phrases such as "monitoring display," "in the footage," or "can be seen" generated by the model.

[0036] Furthermore, step S102 includes the following steps: Step S301: Divide the original video data according to the time length to obtain multiple segmented video segments; Step S302 involves cleaning and quality screening the multiple segmented video clips to obtain multiple high-quality video clips.

[0037] Specifically, this application first collects video data of monitoring scenes containing the main objects of user interest, including people, motor vehicles, non-motor vehicles, pets (cats and dogs), and flames. Second, the collected raw videos are segmented into approximately 4-second video clips based on their duration. Then, the segmented video clips undergo data cleaning and quality screening, including removing duplicate or highly similar videos, and eliminating low-quality video samples such as those with blurry images, decoding failures, or incomplete content. After screening and processing, over 2000 high-quality video clips are obtained. This dataset covers multiple monitoring scenarios, including both user-interested objects and background environmental information, primarily featuring dynamic footage, and includes both daytime and nighttime data, providing a rich and diverse data foundation for subsequent model fine-tuning.

[0038] Based on the cleaned video clip data and combined with the designed prompts, a general visual language model is used to generate corresponding text descriptions. To ensure that the model can fully capture the temporal features of the video content, the video frame sampling parameter is set to 2, that is, two frames are uniformly extracted per second as model input.

[0039] To ensure high-quality training data, the generated text descriptions must be manually reviewed and modified. The verification process includes two requirements: first, ensuring that the text content is relevant to the user and accurately described; and second, simplifying and optimizing the language, removing redundant elements to make the descriptions more concise and natural.

[0040] The validated text descriptions are fused with the corresponding video paths and prompts to construct JSON-formatted training data that meets the input requirements for fine-tuning the large model. Finally, this JSON file and the corresponding video data serve as the input data source for the model fine-tuning phase.

[0041] Furthermore, object information is obtained in the following ways: Step S401: Obtain surveillance video; Step S402: The image frames extracted from the surveillance video are identified using a target detection algorithm to obtain object information.

[0042] Furthermore, the finely tuned multimodal visual language model is adapted to surveillance video scenarios and has the ability to generate user-focused information.

[0043] The basic model selected in this application is the multimodal visual-language model Qwen2.5_VL_7B, which possesses both visual feature understanding and language generation capabilities. The model comprises a visual encoder, a language model, and a multimodal feature fusion module. The visual encoder extracts semantic features from the input video frames, the language model is responsible for natural language understanding and generation, and the multimodal fusion module establishes the correspondence between visual information and linguistic semantics. Qwen2.5_VL_7B offers multiple parameter scale versions; this invention selects version 7B as the basic model based on a performance and resource trade-off to achieve superior understanding and generation capabilities.

[0044] Based on this, using the constructed training set, a full-parameter fine-tuning method was employed to fine-tune the multimodal visual language model, adapting it to surveillance video scenarios and enabling it to generate user-focused information. During the fine-tuning process, this application explored two common fine-tuning strategies: full-parameter fine-tuning and LoRA fine-tuning. Comparison of the experimental results for full-parameter fine-tuning and LoRA fine-tuning revealed that full-parameter fine-tuning generates concise and accurate content, effectively focusing on key objects and their core behaviors; LoRA fine-tuning output content contains some irrelevant details (time information, license plate number, etc.), and the background and object behavior descriptions are complex, affecting the conciseness of the output content. Specific case comparisons are shown in Table 1. Therefore, full-parameter fine-tuning was selected as the core training scheme.

[0045] Table 1

[0046] The full-parameter fine-tuning process utilizes the lightweight training and inference framework Swift, with an A910E graphics card and 96GB of VRAM. Training parameters are set as follows: `train_type=full` indicates full-parameter fine-tuning; `torch_dtype=bfloat16` is used for storing and computing the model, significantly improving performance and saving VRAM; Flash Attention is used as the underlying implementation of the attention mechanism to improve training speed. The training process consists of 4 epochs, with the total number of input and output tokens limited to `max_length=15960` to ensure the model can handle multimodal information from long-term video clips. The total training time is approximately 8-9 hours.

[0047] The finely tuned multimodal visual language model Qwen2.5_VL_7B was deployed as a service interface using the vLLM inference engine. The core idea of ​​vLLM is to split the KV cache into multiple fixed-size blocks, dynamically allocate and reuse these blocks, reducing GPU memory usage. During deployment, the parameter gpu_memory_utilization=0.9 can be set, meaning vLLM can use 90% of the GPU memory during inference, ensuring the stability and efficiency of the inference process.

[0048] During the inference phase, the extracted image frames from the video first undergo an object detection algorithm to identify objects of interest within the frame. When designing prompts, this object information is incorporated into the prompts to guide the large model to focus on the relevant targets. The design of the remaining prompts remains consistent with the model training phase to ensure the unity of inference and training logic. For example, when the target detects that the object of interest in the frame is a person, the prompts will explicitly include information related to "person" to enhance the model's understanding and analysis of that target. The prompts are set as follows: #Role# You are a surveillance video analysis expert, skilled at summarizing key events from surveillance footage. You will obtain frame-by-frame images from a surveillance video recording.

[0049] Observe the image and generate a one-sentence description strictly according to the following rules.

[0050] #Description generation rules# 1) Use a subject-verb-object structure, provide a concise description in one sentence, based on clearly defined information from the image, and do not output any uncertain content; 2) Pay attention to the number of people, their clothing, their behavior, and environmental information in the picture.

[0051] #Output Requirements# 1) Outputting negative statements such as "no one appeared" is prohibited.

[0052] 2) Prohibit the output of guiding phrases such as "monitoring display," "in the video," or "can be seen"; 3) Determine if the image comes from a stereo camera. If it does, when generating the description, do not distinguish between the upper and lower regions. Please describe the entire image as a complete and unified scene. Do not output phrases such as "lower half of the video" or "upper half of the video".

[0053] The monitoring video and the designed reasoning prompts are input into the deployed service interface, which calls the fine-tuned multimodal visual language model to complete the reasoning, generate a preliminary natural language description, and achieve accurate capture of information of interest to users.

[0054] During the design phase of inference prompts, language constraints were imposed on the output content to avoid the model generating redundant and unnecessary visual guidance expressions such as "monitoring display," "in the picture," and "can be seen." However, the generation process of large language models has a certain degree of randomness, and a small number of non-compliant wordings may still occur.

[0055] To address this, the system introduces a post-processing mechanism to standardize and correct the descriptions by performing string retrieval, deletion, and replacement operations on the generated text. Specifically, this includes deleting leading phrases such as "in the picture," "displayed in the picture," or "in this video," or redundant prompts such as "no one appeared" or "no vehicle appeared." It also replaces expressions like "woman," "female," and "woman" with "female," and replaces expressions like "man," "male," and "male" with "male."

[0056] Example 2: Figure 2 This is a schematic diagram of a system for generating user attention information for surveillance videos based on a large model, as provided in Embodiment 2 of the present invention.

[0057] Reference Figure 2 The system includes: The acquisition module is used to acquire the objects that the user is interested in and collect the raw video data of the objects in the monitoring scenario. The processing module is used to segment, clean, and quality-screen the raw video data to obtain multiple high-quality video clips. The configuration module is used to set prompt words according to business scenarios. The prompt words are used to guide the large model to focus on the key information of different types of objects when generating descriptions. The input module is used to input multiple high-quality video clips and prompts into the visual language model to obtain text descriptions. The validation module validates the text description and obtains the validated text description. The construction module is used to fuse the verified text description with the corresponding video path and prompt words to build a training set; The fine-tuning module is used to fine-tune the multimodal visual language model using the training set and a full-parameter fine-tuning method, so as to obtain the fine-tuned multimodal visual language model. The deployment module is used to deploy the fine-tuned multimodal visual language model as a service interface through the vLLM inference engine; An import module is used to import object information into the prompt words to obtain inference prompt words; The inference module is used to input the surveillance video and the inference prompts into the service interface, call the fine-tuned multimodal visual language model to perform inference, and generate a preliminary natural language description. The optimization module is used to optimize the initial natural language description to obtain an optimized natural language description.

[0058] Furthermore, the configuration module is specifically used for: When the scene in the business scenario contains only a single person, the prompts guide the large model to focus on describing the number of people, their gender, clothing characteristics, behavior, and environment. When the scene in the business scenario contains only a single pet, the prompts guide the large model to focus on the pet's species, number, behavior, and environmental information. When the scene of a business scenario contains multiple people, the prompts guide the large model to focus on describing the number of people, group behavior characteristics, and environmental background. When both people and pets are present in the scene, prompts guide the large model to focus on the interaction between people and pets and the scene features. When the object of interest is not present in the visuals of the business scenario, prompts guide the large model to provide a brief description of the overall scenario.

[0059] Furthermore, the processing module is specifically used for: The original video data is divided according to the time length to obtain multiple segmented video clips; Data cleaning and quality screening were performed on multiple segmented video clips to obtain multiple high-quality video clips.

[0060] This invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method for generating user attention information for surveillance videos based on a large model provided in the above embodiments.

[0061] This invention also provides a computer-readable medium having processor-executable non-volatile program code, on which a computer program is stored. When the computer program is run by a processor, it executes the steps of the method for generating user attention information for surveillance videos based on a large model as described above.

[0062] The computer program product provided in this embodiment of the invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0063] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0064] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0065] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0066] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0067] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for generating user attention information in surveillance video based on a large model, characterized in that, The method includes: Identify the objects that the user is interested in, and collect the raw video data of the objects in the monitoring scenario. The original video data is segmented, cleaned, and quality-filtered to obtain multiple high-quality video clips; Based on the business scenario, prompt words are set. These prompt words are used to guide the large model to focus on key information of different types of objects when generating descriptions. The multiple high-quality video clips and the prompt words are input into the visual language model to obtain a text description. The text description is validated to obtain the validated text description; The verified text description is fused with the corresponding video path and the prompt words to construct a training set; Using the training set, the multimodal visual language model is fine-tuned using a full-parameter fine-tuning method to obtain the fine-tuned multimodal visual language model. The finely tuned multimodal visual language model is deployed as a service interface through the vLLM inference engine; By incorporating object information into the prompt words, inference prompt words are obtained; The monitoring video and the inference prompts are input into the service interface, and the fine-tuned multimodal visual language model is invoked to perform inference and generate a preliminary natural language description. The initial natural language description is optimized to obtain an optimized natural language description.

2. The method for generating user attention information in surveillance video based on a large model according to claim 1, characterized in that, Based on the business scenario, prompt words are set. These prompt words are used to guide the large model to focus on key information of different types of objects when generating descriptions, including: When the scene of the business scenario contains only a single person, the prompt words guide the large model to focus on describing the number of people, their gender, clothing characteristics, behavior, and environment. When the scene of the business scenario contains only a single pet, the prompt word guides the large model to focus on the pet's type, number, behavior status, and environmental information; When the scene of the business scenario contains multiple characters, the prompt words guide the large model to focus on describing the number of characters, group behavior characteristics, and environmental background; When both the character and the pet are present in the scene of the business scenario, the prompt word guides the large model to focus on the interactive behavior and scene features between the character and the pet; When the object of interest is not present in the scene of the business scenario, the prompt word guides the large model to provide a brief description of the overall scene.

3. The method for generating user attention information in surveillance video based on a large model according to claim 1, characterized in that, The original video data is segmented, cleaned, and quality-screened to obtain multiple high-quality video clips, including: The original video data is divided according to the time length to obtain multiple segmented video clips; The multiple segmented video segments are cleaned and quality filtered to obtain multiple high-quality video segments.

4. The method for generating user attention information in surveillance video based on a large model according to claim 1, characterized in that, The object information is obtained through the following methods: Acquire the surveillance video; Image frames extracted from the surveillance video are identified using a target detection algorithm to obtain the object information.

5. The method for generating user attention information in surveillance video based on a large model according to claim 1, characterized in that, The finely tuned multimodal visual language model is adapted to surveillance video scenarios and has the ability to generate the user-focused information.

6. A system for generating user attention information in surveillance video based on a large model, characterized in that, The system includes: The acquisition module is used to acquire the objects that the user is interested in and collect the raw video data of the objects in the monitoring scenario. The processing module is used to segment, clean, and quality-screen the raw video data to obtain multiple high-quality video clips; The setting module is used to set prompt words according to the business scenario. The prompt words are used to guide the large model to focus on the key information of different types of objects when generating descriptions. The input module is used to input the multiple high-quality video clips and the prompt words into the visual language model to obtain a text description; The verification module verifies the text description to obtain a verified text description. The construction module is used to fuse the verified text description with the corresponding video path and the prompt words to construct a training set; The fine-tuning module is used to fine-tune the multimodal visual language model using the training set and a full-parameter fine-tuning method, so as to obtain the fine-tuned multimodal visual language model. The deployment module is used to deploy the fine-tuned multimodal visual language model as a service interface through the vLLM inference engine; An import module is used to import object information into the prompt words to obtain inference prompt words; The inference module is used to input the surveillance video and the inference prompts into the service interface, call the fine-tuned multimodal visual language model to perform inference, and generate a preliminary natural language description. The optimization module is used to optimize the initial natural language description to obtain an optimized natural language description.

7. The system for generating user attention information for surveillance videos based on a large model according to claim 6, characterized in that, The setting module is specifically used for: When the scene of the business scenario contains only a single person, the prompt words guide the large model to focus on describing the number of people, their gender, clothing characteristics, behavior, and environment. When the scene of the business scenario contains only a single pet, the prompt word guides the large model to focus on the pet's type, number, behavior status, and environmental information; When the scene of the business scenario contains multiple characters, the prompt words guide the large model to focus on describing the number of characters, group behavior characteristics, and environmental background; When both the character and the pet are present in the scene of the business scenario, the prompt word guides the large model to focus on the interactive behavior and scene features between the character and the pet; When the object of interest is not present in the scene of the business scenario, the prompt word guides the large model to provide a brief description of the overall scene.

8. The system for generating user attention information for surveillance videos based on a large model according to claim 6, characterized in that, The processing module is specifically used for: The original video data is divided according to the time length to obtain multiple segmented video clips; The multiple segmented video segments are cleaned and quality filtered to obtain multiple high-quality video segments.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the method described in any one of claims 1 to 5.

10. A computer-readable medium having processor-executable non-volatile program code, characterized in that, The program code causes the processor to execute the method described in any one of claims 1 to 5.

Images