Multimodal large model processing method, device, storage medium and program product

Abstract

The present application provides a method, device, storage medium, and program product for processing a multimodal large model, wherein the method includes: obtaining a set of original images of a monitoring scene; identifying feature information of a target object in the original image set; generating a question-answer pair corresponding to the original image set based on the feature information; using the original image set and the corresponding question-answer pair as training data, fine-tuning the pre-trained multimodal large model to obtain a target multimodal large model adapted to the monitoring scene. The present application achieves a better adaptation of the multimodal large model to the needs of a specific monitoring scene, improves the performance and application effect of the multimodal large model in the monitoring scene, and thereby improves the intelligence level of the monitoring system.

Description

Technical Field

[0001] The present application relates to the field of information processing technology, and in particular to a method, device, storage medium and program product for processing a multimodal large model. Background Art

[0002] The intelligent video surveillance system adopts image processing, pattern recognition and computer vision technology. By adding an intelligent video analysis module to the monitoring system, it uses the computer's powerful data processing capabilities to filter out useless or interfering information in the video, automatically identify different objects, analyze and extract key useful information from the video source, quickly and accurately locate the accident scene, judge abnormal conditions in the monitoring image, and issue alarms or trigger other actions in the fastest and best way, thereby effectively carrying out pre-warning, in-process processing, and timely evidence collection afterwards. It is a fully automatic, all-weather, real-time monitoring intelligent system.

[0003] Existing intelligent monitoring systems rely on multiple expert models to achieve structured processing of video information. Each expert model requires specialized data collection and model training, resulting in high system complexity and maintenance costs. The response time to adapt to new scenarios or new demands is long.

[0004] Large multimodal models can integrate diverse information, including video, images, and text, improving their understanding of complex scenarios. Thanks to their ability to learn from massive amounts of data, they possess strong generalization capabilities. With their strong understanding of context and generalization, large models can perform a variety of tasks as needed and make reasonable inferences and responses to new scenarios or requirements. Therefore, large multimodal models hold enormous potential for application in intelligent surveillance.

[0005] However, when the current mainstream multimodal large models are directly applied to intelligent monitoring systems in specific scenarios, the scene adaptability is poor, resulting in low application performance of the large models in intelligent monitoring systems and difficulty in achieving practical application effects. Summary of the Invention

[0006] The main purpose of the embodiments of the present application is to provide a processing method, device, storage medium and program product for a multimodal large model, so as to enable the multimodal large model to better adapt to the needs of specific monitoring scenarios, improve the performance and application effect of the multimodal large model in the monitoring scenario, and thus improve the intelligence level of the monitoring system.

[0007] In a first aspect, an embodiment of the present application provides a method for processing a multimodal large model, comprising: obtaining an original image set of a monitoring scene; identifying feature information of a target object in the original image set; generating a question-answer pair corresponding to the original image set based on the feature information; using the original image set and the corresponding question-answer pair as training data, fine-tuning the pre-trained multimodal large model to obtain a target multimodal large model adapted to the monitoring scene.

[0008] In one embodiment, the identifying feature information of the target object in the original image set includes: determining the object category included in each original image in the original image set; and identifying the location information of at least one target object included in the original image based on the object category; wherein the feature information includes the object category and / or the location information.

[0009] In one embodiment, generating a question-answer pair corresponding to the original image set based on the feature information includes: generating a first prompt word of a preset large language model based on the feature information; inputting the first prompt word and the original image set into the large language model, so that the large language model generates a question-answer pair about the target object based on the first prompt word.

[0010] In one embodiment, generating a question-answer pair corresponding to the original image set based on the feature information includes: adding a position mark to the target object in the original image based on the feature information to generate a marked image; generating a second prompt word of a preset large language model based on the position mark; and inputting the marked image and the second prompt word into the preset large language model, so that the large language model generates a question-answer pair about the target object based on the second prompt word and the marked image.

[0011] In one embodiment, the original image set includes a human body image set; the feature information includes identification information of a human body object in each of the human body images; generating a question-answer pair corresponding to the original image set based on the feature information includes: screening out a first image pair having the same identification information and a similarity less than a first threshold and a second image pair having different identification information and a similarity greater than a second threshold from the human body image set, and marking the real identity labels of the first image pair and the second image pair, and determining the first image pair and the second image pair as a difficult sample pair; determining a pair of two images in the human body image set excluding the difficult sample pair as an easy sample pair, and marking the identity label of the easy sample pair according to the identification information of each human body image in the easy sample pair; generating question-answer pairs corresponding to the difficult sample pair and the easy sample pair according to preset questions and the identity labels.

[0012] In one embodiment, the original image set includes video data; the feature information includes a video foreground description and a video background description of the video data; generating a question-answer pair corresponding to the original image set based on the feature information includes: generating a third prompt word of a large language model based on the video foreground description, the video background description and preset key information about the video data; inputting the third prompt word and the video data into the large language model, so that the large language model generates at least one question-answer pair corresponding to the video data based on the third prompt word; wherein the preset key information includes: one or more of: event time, event sequence, object motion trajectory, number of human objects, and appearance features of human objects in the video.

[0013] In one embodiment, the multimodal large model includes a visual encoder, a visual-text feature mapper and a large language model; wherein: the visual encoder is used to extract the original visual features of the original image set; the visual-text mapper is used to map the original visual features to a feature format adapted to the large language model to obtain the converted target visual features; the large language model is used to generate predicted answers based on the target visual features and the question instructions in the question-answer pair; the original image set and the corresponding question-answer pair are used as training data to fine-tune the pre-trained multimodal large model to obtain a target multimodal large model adapted to the monitoring scenario, including: in the fine-tuning training process, the goal is to constrain the difference information between the true answer in the question-answer pair and the predicted answer.

[0014] In one embodiment, the multimodal large model includes a visual encoder, a local encoder, a visual-text feature mapper, and a large language model; the feature information includes position information of a target object in the original image set; wherein: the visual encoder is used to extract original visual features of the original image set; the local encoder is used to extract local features of the target object from the original visual features based on the position information of the target object, and splice the local features with the original visual features; the visual-text mapper is used to map the spliced ​​visual features to a feature format adapted to the large language model to obtain converted target visual features; the large language model is used to generate a predicted answer based on the target visual features and the question instruction in the question-answer pair;

[0015] In one embodiment, the original image set and the corresponding question-answer pairs are used as training data to fine-tune the pre-trained multimodal large model to obtain a target multimodal large model adapted to the monitoring scenario, including: during the fine-tuning training process, the goal is to constrain the difference information between the true answer and the predicted answer in the question-answer pair.

[0016] In one embodiment, the method further includes: in response to a human body comparison instruction, obtaining at least two images to be processed; inputting the at least two images to be processed into the target multimodal large model, so that the target multimodal large model outputs whether the human body objects in the at least two images to be processed are the same person; determining the detection accuracy of the target multimodal large model based on the output result of the target multimodal large model; if the comparison accuracy is less than a first preset threshold, adding a supplementary image and a question-answer pair of the supplementary image to the training data to generate new training data, and iteratively training the target multimodal large model based on the new training data.

[0017] In one embodiment, the method further includes: obtaining a surveillance video to be processed in response to a behavior recognition instruction; inputting the surveillance video to be processed into the target multimodal large model, so that the target multimodal large model outputs the specified behavior category present in the surveillance video to be processed according to the behavior detection prompt word; determining the recognition accuracy of the target multimodal large model based on the output result of the target multimodal large model; if the recognition accuracy is less than a second preset threshold, adding a supplementary video and a question-answer pair of the supplementary video to the training data to generate new training data, and iteratively training the target multimodal large model based on the new training data.

[0018] In a second aspect, an embodiment of the present application provides a method for identifying behavioral information, comprising: obtaining a surveillance video of a shopping mall in response to an identification instruction; inputting the surveillance video into a target multimodal large model, so that the target multimodal large model outputs an identification result about the surveillance video under the guidance of a preset prompt word; wherein, the target multimodal large model is obtained according to the method described in any of the above aspects.

[0019] In a third aspect, an embodiment of the present application provides a processing device for a multimodal large model, comprising:

[0020] An acquisition module is used to obtain a set of original images of the monitoring scene;

[0021] A recognition module, configured to recognize feature information of a target object in the original image set;

[0022] A generation module, configured to generate question-answer pairs corresponding to the original image set based on the feature information;

[0023] A fine-tuning module is used to use the original image set and the corresponding question-answer pairs as training data to fine-tune the pre-trained multimodal large model to obtain a target multimodal large model adapted to the monitoring scenario.

[0024] In one embodiment, the recognition module is used to determine the object category included in each original image in the original image set; and identify the location information of at least one target object included in the original image based on the object category; wherein the feature information includes the object category and / or the location information.

[0025] In one embodiment, the generation module is used to generate a first prompt word of a preset large language model based on the feature information; the first prompt word and the original image set are input into the large language model, so that the large language model generates a question-answer pair about the target object based on the first prompt word.

[0026] In one embodiment, the generation module is used to add a position mark to the target object in the original image based on the feature information to generate a marked image; generate a second prompt word of a preset large language model based on the position mark; and input the marked image and the second prompt word into the preset large language model so that the large language model generates a question-answer pair about the target object based on the second prompt word and the marked image.

[0027] In one embodiment, the original image set includes a human body image set; the feature information includes identification information of a human body object in each of the human body images; the generation module is used to screen out a first image pair having the same identification information and a similarity less than a first threshold and a second image pair having different identification information and a similarity greater than a second threshold from the human body image set, and mark the true identity labels of the first image pair and the second image pair, and determine the first image pair and the second image pair as a difficult sample pair; determine the pairwise image combinations other than the difficult sample pairs in the human body image set as easy sample pairs, and mark the identity labels of the easy sample pairs according to the identification information of each human body image in the easy sample pairs; and generate question-answer pairs corresponding to the difficult sample pairs and the easy sample pairs according to preset questions and the identity labels.

[0028] In one embodiment, the original image set includes video data; the feature information includes a video foreground description and a video background description of the video data; the generation module is used to generate a third prompt word of the large language model based on the video foreground description, the video background description and preset key information about the video data; the third prompt word and the video data are input into the large language model, so that the large language model generates at least one question-answer pair corresponding to the video data based on the third prompt word; wherein the preset key information includes: one or more of: event time, event sequence, object motion trajectory, number of human objects, and appearance features of human objects in the video.

[0029] In one embodiment, the multimodal large model includes a visual encoder, a visual-text feature mapper, and a large language model; wherein: the visual encoder is used to extract the original visual features of the original image set; the visual-text mapper is used to map the original visual features to a feature format adapted to the large language model to obtain converted target visual features; the large language model is used to generate predicted answers based on the target visual features and the question instructions in the question-answer pair; and a fine-tuning module is used to constrain the difference information between the true answer in the question-answer pair and the predicted answer during the fine-tuning training process.

[0030] In one embodiment, the multimodal large model includes a visual encoder, a local encoder, a visual-text feature mapper, and a large language model; the feature information includes position information of a target object in the original image set; wherein: the visual encoder is used to extract original visual features of the original image set; the local encoder is used to extract local features of the target object from the original visual features based on the position information of the target object, and splice the local features with the original visual features; the visual-text mapper is used to map the spliced ​​visual features to a feature format adapted to the large language model to obtain converted target visual features; the large language model is used to generate a predicted answer based on the target visual features and the question instruction in the question-answer pair;

[0031] In one embodiment, the fine-tuning module is configured to, during the fine-tuning training process, aim to constrain the difference information between the true answer and the predicted answer in the question-answer pair.

[0032] In one embodiment, the device also includes: a first iterative module, used to obtain at least two images to be processed in response to a human body comparison instruction; input the at least two images to be processed into the target multimodal large model, so that the target multimodal large model outputs whether the human body objects in the at least two images to be processed are the same person; determine the comparison accuracy of the target multimodal large model based on the output result of the target multimodal large model; if the comparison accuracy is less than a first preset threshold, add a supplementary image and a question-answer pair of the supplementary image to the training data to generate new training data, and iteratively train the target multimodal large model based on the new training data.

[0033] In one embodiment, the device also includes: a second iterative module, used to obtain the surveillance video to be processed in response to the behavior recognition instruction; input the surveillance video to be processed into the target multimodal large model, so that the target multimodal large model outputs the specified behavior category present in the surveillance video to be processed according to the behavior detection prompt word; determine the recognition accuracy of the target multimodal large model based on the output result of the target multimodal large model; if the recognition accuracy is less than a second preset threshold, add a supplementary video and a question-answer pair of the supplementary video to the training data to generate new training data, and iteratively train the target multimodal large model based on the new training data.

[0034] In a fourth aspect, an embodiment of the present application provides an electronic device, including:

[0035] at least one processor; and

[0036] a memory communicatively coupled to the at least one processor;

[0037] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the electronic device to execute the method described in any one of the above aspects.

[0038] In a fifth aspect, an embodiment of the present application provides a cloud device, including:

[0039] at least one processor; and

[0040] a memory communicatively coupled to the at least one processor;

[0041] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the cloud device to execute the method described in any one of the above aspects.

[0042] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method described in any one of the above aspects is implemented.

[0043] In a seventh aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements the method described in any of the above aspects when executed by a processor.

[0044] The processing method, device, storage medium and program product of the multimodal large model provided in the embodiment of the present application realizes fine-tuning training of the multimodal large model by processing and analyzing the original image set of the monitoring scene, thereby improving the adaptability and recognition accuracy of the model in a specific monitoring scene. Specifically, by obtaining the original image set of the monitoring scene, the actual visual information in the monitoring scene can be captured, and by identifying the feature information of the target object in the image set, the system can understand and extract the key elements in the scene, and then generate question-answer pairs based on the feature information, and use the original image set and the corresponding question-answer pairs as multimodal training data, thereby enriching the diversity and pertinence of the training data; by using the original image set and the question-answer pairs for fine-tuning training, the pre-trained multimodal large model is optimized to better adapt to the needs of the specific monitoring scene, and improve the performance and application effect of the model in the monitoring scene, thereby improving the intelligence level of the monitoring system. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are some embodiments of the present invention, and it is clear that those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0046] Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0047] Figure 2 A schematic diagram of an application scenario of a multimodal large model processing system provided in an embodiment of the present application;

[0048] Figure 3 A schematic flow chart of a method for processing a multimodal large model provided in an embodiment of the present application;

[0049] Figure 4 A schematic diagram of fine-tuning data provided in an embodiment of the present application;

[0050] Figure 5 A schematic diagram of feature information identification provided in an embodiment of the present application;

[0051] Figure 6 A schematic diagram of a marking image provided in an embodiment of the present application;

[0052] Figure 7 A schematic diagram of the model structure of a multimodal large model provided in an embodiment of the present application;

[0053] Figure 8 A schematic diagram of a model training process provided in an embodiment of the present application;

[0054] Figure 9 A flowchart of a behavior information identification method provided in an embodiment of the present application;

[0055] Figure 10 A schematic diagram of the structure of a multimodal large model processing device provided in an embodiment of the present application;

[0056] Figure 11 A schematic diagram of the structure of a cloud device provided in an embodiment of the present application.

[0057] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0058] Exemplary embodiments are described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different drawings represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with this application.

[0059] The term "and / or" in this article is used to describe the association relationship of associated objects, specifically indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0060] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0061] In order to clearly describe the technical solutions of the embodiments of the present application, the following definitions are given first:

[0062] Multimodal large model: A large-scale artificial intelligence system that can understand and integrate multiple types of data (such as text, images, video, voice, etc.). It achieves stronger information processing capabilities by jointly learning cross-modal representations.

[0063] Prompt words: Keywords or sentences used to guide the large model to accurately understand task requirements and respond accordingly.

[0064] Token: For text, it is the basic unit of data segmentation, which can be a word, character, or subword. For visual information, it is the feature vector of a piece of image segmentation after encoding.

[0065] Body ID: A unique identifier for a pedestrian within a day. All body images with the same ID belong to the same pedestrian. Body IDs do not change over the same day.

[0066] GroundingDINO: Grounding DETR with Improved deNoising anchOr boxes is a deep learning model that combines target detection and language understanding. The main goal of this model is to detect targets of any category based on human question input, that is, open-world target detection.

[0067] ReID (Re-identification), commonly referred to as "person re-identification," is a computer vision task that aims to identify and match pedestrians of the same identity across different camera views. In practical applications, this technology is often used in surveillance systems to track the movement of specific individuals across different camera coverage areas.

[0068] ‌CLIP: Contrastive Language-Image Pretraining, a multimodal contrastive learning model that aims to learn the matching relationship between images and text by contrasting their vector representations. ‌ During the pre-training phase, CLIP compares the vector representations of images and text, attempting to bring matching image and text vectors closer together and push mismatched vectors further apart, thereby capturing the deep semantic connection between images and text.‌

[0069] ‌SigLIP: Simultaneous Generative and Interpretive Language Processing, a simultaneous generative and interpretive language processing model, is a visual language pre-training model and an optimized version of the CLIP model.

[0070] MLP: Multilayer Perceptron.

[0071] Transformer: A deep learning model architecture that is primarily used to process sequence data, and is particularly effective in natural language processing tasks.

[0072] The processing method of the multimodal large model in the embodiment of the present application can be applied to any field that needs to process multimodal information.

[0073] The intelligent video surveillance system adopts image processing, pattern recognition and computer vision technology. By adding an intelligent video analysis module to the monitoring system, it uses the computer's powerful data processing capabilities to filter out useless or interfering information in the video, automatically identify different objects, analyze and extract key useful information from the video source, quickly and accurately locate the accident scene, judge abnormal conditions in the monitoring image, and issue alarms or trigger other actions in the fastest and best way, thereby effectively carrying out pre-warning, in-process processing, and timely evidence collection afterwards. It is a fully automatic, all-weather, real-time monitoring intelligent system.

[0074] Existing intelligent surveillance systems rely on multiple expert models to achieve structured processing of video information. Each expert model requires specialized data collection and model training, resulting in high system complexity and maintenance costs. Adapting to new scenarios or requirements also takes a long time to respond.

[0075] Large multimodal models can integrate diverse information, including video, images, and text, improving their understanding of complex scenarios. Thanks to their ability to learn from massive amounts of data, they possess strong generalization capabilities. With their strong understanding of context and generalization, large models can perform a variety of tasks as needed and make reasonable inferences and responses to new scenarios or requirements. Therefore, large multimodal models hold enormous potential for application in intelligent surveillance.

[0076] However, when the current mainstream multimodal large models are directly applied to intelligent monitoring systems in specific scenarios, the scene adaptability is poor, resulting in low application performance of the large models in intelligent monitoring systems and difficulty in achieving practical application effects.

[0077] In order to solve at least one of the above problems, the embodiment of the present application provides a processing solution for a multimodal large model. By obtaining the original image set of the monitoring scene, the actual visual information in the monitoring scene can be captured. By identifying the feature information of the target object in the image set, the system can understand and extract the key elements in the scene. Then, question-answer pairs are generated based on the feature information. The original image set and the corresponding question-answer pairs are used as multimodal training data, which enriches the diversity and pertinence of the training data. Ultimately, by using the original image set and question-answer pairs for fine-tuning training, the pre-trained multimodal large model is optimized, so that the multimodal large model can better adapt to the needs of specific monitoring scenarios, improve the performance and application effect of the multimodal large model in the monitoring scenario, and thus improve the intelligence level of the monitoring system.

[0078] The following detailed description of some embodiments of the present application is provided in conjunction with the accompanying drawings. The following embodiments and features thereof may be combined with one another unless they conflict with each other. Furthermore, the sequence of steps in the following method embodiments is provided for illustrative purposes only and is not intended to be a strict limitation.

[0079] like Figure 1 As shown, this embodiment provides an electronic device 1, including: at least one processor 11 and a memory 12, Figure 1 In the example, a processor 11 and a memory 12 are connected via a bus 10. Memory 12 stores instructions executable by processor 11. The instructions are executed by processor 11 so that electronic device 1 can execute all or part of the method described in the following embodiments. This allows the multimodal large model to better adapt to the needs of specific monitoring scenarios, improves the performance and application effects of the multimodal large model in monitoring scenarios, and thereby improves the intelligence level of the monitoring system.

[0080] In one embodiment, the electronic device 1 may be a mobile phone, a tablet computer, a laptop computer, a desktop computer, or a large computing system composed of multiple computers.

[0081] Figure 2 Schematic diagram of a multi-modal large model processing system application scenario 200 provided in an embodiment of the present application. Figure 2 As shown, the system includes: a server 210 and a terminal 220, wherein:

[0082] The server 210 can be a data platform that provides multimodal large model processing services, such as an intelligent monitoring system platform. In actual scenarios, an intelligent monitoring system platform may have multiple servers 210. Figure 2 Here, one server 210 is taken as an example.

[0083] The terminal 220 can be a computer, mobile phone, tablet or other device used by the user to log in to the intelligent monitoring system platform. There can also be multiple terminals 220. Figure 2 Two terminals 220 are used as an example for illustration.

[0084] The terminal 220 and the server 210 can transmit information via the Internet, so that the terminal 220 can access the data on the server 210. The terminal 220 and / or the server 210 can be implemented by the electronic device 1.

[0085] The processing solution of the multimodal large model in the embodiment of the present application can be deployed on the server 210, can also be deployed on the terminal 220, or can be deployed partially on the server 210 and partially on the terminal 220. In actual scenarios, the choice can be based on actual needs, and this embodiment does not limit it.

[0086] When the processing solution of the multimodal large model is fully or partially deployed on the server 210 , a calling interface can be opened to the terminal 220 to provide algorithm support to the terminal 220 .

[0087] The method provided in the embodiments of the present application can be implemented by executing corresponding software code on electronic device 1 and by interacting with a server. The electronic device 1 can be a local terminal device. When the method is run on a server, the method can be implemented and executed based on a cloud interaction system, which includes a server and a client device.

[0088] In a possible implementation, the method provided in the embodiment of the present application provides a graphical user interface through a terminal device, wherein the terminal device can be the local terminal device mentioned above, or it can be a client device in the cloud interaction system mentioned above.

[0089] Please see Figure 3 , which is a processing method for a multimodal large model according to an embodiment of the present application, which can be Figure 1 The electronic device 1 shown is used to perform and can be applied to Figure 2 In the monitoring application scenario shown in , the multimodal large model is made to better adapt to the needs of the specific monitoring scenario, thereby improving the performance and application effect of the multimodal large model in the monitoring scenario, thereby improving the intelligence level of the monitoring system. In this embodiment, terminal 220 is used as the execution end as an example, and the method includes the following steps:

[0090] Step 301: Acquire a set of original images of a monitoring scene.

[0091] In this step, the original image set can be multiple surveillance images or surveillance video clips in the surveillance scene. The specific area covered by the surveillance scene can be set according to actual needs. For example, the area covered by the surveillance scene includes but is not limited to shopping malls, schools, garages, traffic sections, libraries and other areas. The original images can be obtained by capturing videos of the surveillance area through surveillance cameras, or the original image set can be obtained by pulling the structured data returned by the surveillance system. For the video captured by the surveillance camera, human body detection can be performed on the video frames, and image frames and video clips containing human bodies can be extracted as the original image set. As for the structured data returned by the surveillance system, it is itself in the form of pictures and video clips, so it can be directly used as the original image set.

[0092] like Figure 4 The figure below shows a schematic diagram of fine-tuning data provided by an embodiment of this application. Using the images and video clips obtained through the two aforementioned methods as atomic data, combined with the characteristics of intelligent monitoring tasks, training data of various types, such as single-image text data, multi-image text data, and video text data, can be constructed. For details, please refer to the specific description of the following embodiment.

[0093] Step 302: Identify feature information of the target object in the original image set.

[0094] In this step, the original image set includes multiple original images, and the target object refers to the object contained in the original image, such as a vehicle, a human body, a pyramid, a chair, and so on. Feature information is used to characterize the basic characteristics of the target object in the original image. Feature information includes but is not limited to the target object's visual features, positional features, category information, and unique identifier. By identifying the feature information of the target object in the original image set, the system can understand and extract key elements of the scene, providing targeted auxiliary information for subsequent processing.

[0095] In one embodiment, step 302 may specifically include: determining an object category included in each original image in the original image set; and identifying location information of at least one target object included in the original image based on the object category. The feature information includes object category and / or location information.

[0096] In this embodiment, by determining the object category contained in each original image, the system can quickly classify and identify different objects in the image, providing a basis for subsequent processing. Identifying the location information of the target object based on the object category enables the system to not only identify the type of object, but also accurately locate the object's position in the image. For a single original image, its feature information includes but is not limited to the object category and location information contained in the original image, greatly enriching the descriptive ability of the image data, providing targeted data support for subsequent question-answer pair generation and fine-tuning training of multimodal large models, improving the system's recognition ability and response speed in complex monitoring scenarios, and enhancing the applicability and reliability of the model in practical applications.

[0097] like Figure 5As shown, a schematic diagram of feature information recognition provided by an embodiment of the present application can be used to obtain category labels for the original image based on a multimodal large model. For example, the prompt word for type label extraction can be: "Give the categories of specific objects involved in the image in the form of a list, without duplication between categories, only output deterministic category names, and give Chinese names and corresponding English names." For example, the multimodal large model outputs the object category labels contained in the original image as "car, person, traffic cone". Then, open target detection can be performed based on the category label of each original image. For example, a model such as GroundingDINO can be used to implement open target detection, and the circumscribed rectangular box of the target object in the original image is obtained. The rectangular box is used to represent the location information of the target object. GroundingDINO utilizes multimodal information of vision and language, and achieves more accurate and flexible target detection by combining language prompts with visual features. For example, we can detect target objects from an original image and give the coordinates of the rectangular boxes: "car [20, 30, 200, 130], car [200, 370, 450, 480], person [400, 500, 480, 628], traffic cone [420, 530, 475, 600]..." In an original image, each type label can include one or more target objects.

[0098] Optionally, you can randomly select a rectangular frame of a target object from the recognition results and use a pre-set large model to provide both a detailed and brief description. For example, a detailed description prompt might read: "As a visual expert, please describe this image in detail." A brief description prompt might read: "Describe this image in one sentence." These brief and detailed descriptions are included as part of the corresponding target object's feature information, providing more detailed and diverse data support for subsequent processing.

[0099] Step 303: Generate question-answer pairs corresponding to the original image set based on the feature information.

[0100] In this embodiment, a question-answer pair refers to a question about the details of an original image and the corresponding ground-truth answer. This pair is used to train a large multimodal model to understand images. Generating question-answer pairs based on feature information can enhance the pair's focus on the target object, thereby increasing the large multimodal model's attention to local features in the image, improving the model's accuracy in information perception and event understanding in surveillance scenarios.

[0101] In one embodiment, step 303 may specifically include generating a first prompt word for a preset large language model based on the feature information, inputting the first prompt word and the original image set into the large language model, so that the large language model generates a question-answer pair about the target object based on the first prompt word.

[0102] In this embodiment, corresponding single-image text data can be generated for each original image in the original image set. The single-image text data includes the original image and the question-answer pair data corresponding to the original image. The single-image text data can be mainly used to locate the target in the image, understand the target and understand the entire image. A part of the original image can be extracted from the original image set to generate single-image text data. When generating question-answer pairs, the feature information of the target object in the original image can be injected into the text instruction and / or the corresponding true value answer by text injection.

[0103] Specifically, text injection is to inject the acquired characteristic information of the target object as auxiliary information in the form of text into the first prompt word or answer of the preset large language model. The generated data type based on this preset larger than language model can include positioning data and target description data. The positioning data inquires about the location of the target object based on the description of the target object, that is, the target detection result. For example, the first prompt word can be: "Please output the following paragraph describing the rectangular box of the target object in the image: a description of a certain target." Target description data refers to inquiring about the description of the specific target object in the rectangular box based on the detection result of the target object. For example, the first prompt word at this time can be: "Please output the description of the target object in the target rectangular box [x0, y0, w0, h0]." The above-mentioned first prompt word is used as a question, and the data generated by the preset large language model according to the first prompt word is used as the true value answer to the question to generate a question and answer pair.

[0104] In this way, by generating the first prompt word for the preset large language model based on feature information, the input prompt is targeted and accurate, enabling the language model to better understand the key information of the target object in the image. The first prompt word and the original image set are input into the large language model, prompting the model to combine visual and linguistic information to generate question-answer pairs about the target object. This not only enriches the diversity of training data but also improves the relevance and quality of question-answer pairs, thereby more effectively optimizing the performance of the large language model during fine-tuning training. This enhances the model's intelligence in processing multimodal data, enabling it to more accurately generate question-answer pairs that meet actual needs in complex scenarios, thereby enhancing the model's practicality and application value.

[0105] In one embodiment, step 303 may specifically include: adding a location marker to the target object in the original image based on the feature information to generate a marked image; generating a second prompt word of a preset large language model based on the location marker; and inputting the marked image and the second prompt word into the preset large language model, so that the large language model generates a question-answer pair about the target object based on the second prompt word and the marked image.

[0106] In this embodiment, when generating a question-answer pair for a single image, the feature information of the target object in the original image can be used as auxiliary information. This auxiliary information can be injected into the question or ground truth answer of the question-answer pair using a visual injection method. Specifically, a preset large language model can be used to generate the question-answer pair. Injecting the auxiliary information into the input information of the preset large language model can be used to generate scene understanding data containing positioning information. Scene understanding data is an understanding of the entire image. In this case, the auxiliary information is all detected target objects in the original image. The specific process of generating the question-answer pair can include the following three steps:

[0107] i) For an original image, first calculate the center of the target object based on the rectangular frame of the target object, and mark the center of each target object with a numerical sequence (i.e., position mark), such as Figure 6 1 is a schematic diagram of a marking image provided in an embodiment of the present application, including: a human body (1), a car (2), a car (3), a traffic cone (4) and a fire hydrant (5).

[0108] Then, a pre-trained multimodal large model can be used to describe the numerically labeled image in detail. For example, the second prompt generated based on the labeled image could be: "You are a visual expert and have received an image with numerical numbers centered on some human figures and objects. Please describe this image in detail. Describe only the specific things and do not rely solely on numerical numbers."

[0109] ii) Based on the description containing the target object's numerical serial number obtained in step i), a large language model can be used to generate free question-answer pairs, and a special placeholder can be embedded in the generated answers. For example, the second prompt could be: "You are a visual expert who excels at generating questions and answers based on image descriptions. Please generate three question-answer pairs based on image descriptions containing partial target serial numbers. The generated questions directly reference the target serial number, and the generated answers use a special placeholder for the target serial number." <ref> Target Name< / ref> <box> [Target Number]< / box> Do not use numbers, marks, etc. Figure 6 For example, the following question-answer pairs can be generated (where Q represents the question and A represents the corresponding answer):

[0110] Q1: How many cars are there in the image?

[0111] A1: 2 cars.

[0112] Q2: Please give the location of the black car in the image?

[0113] A2: <red>black car <box> [3] <box>

[0114] Q3: Please describe [1]?

[0115] A3: A man wearing a black short-sleeved T-shirt and sneakers.

[0116] iii) Replace the target sequence number in the question-answer pair with the corresponding target rectangle. Figure 6 Taking the example question-answer pair as an example, after replacing the target sequence number in the question-answer pair with the corresponding target rectangle, the final question-answer pair is as follows:

[0117] Q2: Please give the location of the black car in the image?

[0118] A2: <red>black car <box> [[20, 30, 200, 130]] <box>

[0119] Q3: Please describe [400, 500, 480, 628]?

[0120] A3: A man wearing a black short-sleeved T-shirt and sneakers.

[0121] In this way, by combining image labeling and language prompts, the deep fusion of multimodal data and the ability to intelligently generate question-answer pairs are achieved. Specifically, by adding position labels to the target object based on feature information in the original image and generating a labeled image, the interpretability and positioning accuracy of the image data are enhanced. Then, based on the position labels, the second prompt word of the preset large language model is generated to ensure a high degree of relevance and consistency between the language input and the visual information. Inputting the labeled image and the second prompt word into the preset large language model enables the model to fully utilize the complementarity of visual and language information to generate high-quality question-answer pairs about the target object, thereby improving the accuracy and contextual relevance of the question-answer pair generation, and thus providing rich training data for the subsequent training of the multimodal large model, thereby enhancing the adaptability and expressiveness of the multimodal large model in multimodal tasks.

[0122] In one embodiment, the original image set includes a human body image set. The feature information includes identification information of a human body object in each human body image. Based on the feature information, a question-answer pair corresponding to the original image set is generated, including: screening out a first image pair with the same identification information and a similarity less than a first threshold, and a second image pair with different identification information and a similarity greater than a second threshold from the human body image set, and marking the real identity labels of the first image pair and the second image pair, and determining the first image pair and the second image pair as a difficult sample pair. The combination of any two images in the human body image set excluding the difficult sample pair is determined as an easy sample pair, and the identity label of the easy sample pair is marked according to the identification information of each human body image in the easy sample pair. Based on the preset questions and identity labels, question-answer pairs corresponding to the difficult sample pair and the easy sample pair are generated respectively.

[0123] In this embodiment, the fine-tuning data may include multi-image text data, which can be mainly used for tasks such as pedestrian retrieval and pedestrian tracking. The identification information is used to uniquely identify the target object in the image, such as the human ID assigned by the system to the human object. By identifying and screening the identification information in the human image, different human objects in the image can be effectively distinguished and classified. The first image pairs with the same identification information but low similarity and the second image pairs with different identification information but high similarity are screened out, and their true identity labels are marked to identify difficult sample pairs. This method can help the multimodal large model better understand and distinguish complex samples and enhance its sensitivity to subtle differences. At the same time, the image combination excluding the difficult sample pairs is determined as an easy sample pair, and the identity labels are marked according to the identification information, ensuring the diversity and representativeness of the sample pairs. By generating corresponding question-answer pairs based on preset questions and identity labels, the content and structure of the training data are enriched, thereby enabling the subsequent fine-tuned multimodal large model to perform better when processing diverse and complex data.

[0124] For example, based on the human images and their human IDs in the original image set returned by the monitoring system, the ReID expert model can be used to mine difficult samples and semi-automatically label them. First, the features of each human image are extracted based on the ReID expert model, and human image pairs with the same human ID and a similarity less than the first threshold th1 are selected as the first image pair. Human image pairs with different human IDs and a similarity greater than the second threshold th2 are selected as the second image pairs, and the first image pair and the second image pair are used as difficult sample pairs. The first threshold and the second threshold can be selected according to actual needs. For example, the first threshold th1 can be 0.9, and the second threshold th2 can be 0.8. The remaining human images in the original image set excluding the difficult sample pairs are combined into easy sample pairs.

[0125] For difficult pairs, we can label them based on the true identity information of the human subjects in the image. Pairs with the same true identity information are labeled with the same true identity label, while pairs with different true identity information are labeled with different true identities. These labeled pairs are then used as a piece of multi-image data. For easy pairs, the person ID returned by the system is used as the true identity label of the human subject in the image. From the easy pairs, we randomly select easy pairs with the same or different identity labels as a piece of multi-image data. We then construct a question-answer pair. For example, the predefined question might be, "Do these images show the same person?" The true identity labels of the images are used as the true answer to the predefined question.

[0126] In one embodiment, the original image set includes video data. Feature information includes a video foreground description and a video background description of the video data. Based on the feature information, question-answer pairs corresponding to the original image set are generated, including: generating a third prompt word of the large language model based on the video foreground description, the video background description, and preset key information about the video data. The third prompt word and the video data are input into the large language model, so that the large language model generates at least one question-answer pair corresponding to the video data based on the third prompt word. The preset key information includes: one or more of: event time, event sequence, object motion trajectory, number of human subjects, and appearance features of human subjects in the video.

[0127] In this embodiment, the original image set may include video data, which may be surveillance video frames within a specific area. The video data may include information about people's activities within the specific area. For example, surveillance video within a shopping mall may include images of consumers, store sales staff, and other relevant personnel. The video data may be a continuous video stream or multiple video frames sampled from the surveillance video in a certain time sequence. For example, the video data may be acquired in real time from a surveillance system, or from local storage or cloud storage.

[0128] The fine-tuning data for the multimodal large model can also include video text data. The video text data mainly includes video data and its corresponding question-answer pairs, which are used to help the multimodal large model understand the behavior and relationships of people in the video, character interactions, and scene information.

[0129] The preset key information includes but is not limited to event time, event sequence, object motion trajectory, number of human objects, appearance characteristics of human objects, and other information in the video. The preset key information is used to guide the large language model to focus on the specified information when generating question-answer pairs, thereby generating targeted question-answer pairs.

[0130] Optionally, based on the foreground and background descriptions of the video, the large language model can be guided to propose question-answer pairs regarding information such as event timing, event sequence, motion trajectory, number of characters, and appearance characteristics in the video. This can be divided into the following two steps:

[0131] a) First, obtain foreground and background descriptions of the video data. This can be done based on a mainstream multimodal large model to describe the video's foreground, including a detailed description of each person's clothing and behavior. For example, the prompt for obtaining foreground and background descriptions could be: "You are a video understanding expert. Please describe the video's content in detail, sequentially, based on the video's timestamps. This includes, but is not limited to, each person's behavior and clothing, as well as interactions between people and objects." Furthermore, given that surveillance footage is generally stationary, the first frame of the video can be selected as the background description. The corresponding prompt could be: "Please describe this image in detail."

[0132] b) Based on the foreground and background descriptions of the video generated in the previous step, guide the large language model to generate question-answer pairs. For example, multiple third prompts can be generated for questions about key information such as event time (i.e., the time when an event occurred), event sequence, object movement trajectory, number of people, and appearance characteristics of human objects. For each video, three third prompts are randomly selected to guide the large language model in generating question-answer pairs.

[0133] Optionally, the third prompt word can be divided into two parts: a prefix and a topic. For each third prompt word, the prefix part can be the same, for example: "You are a video understanding expert. Please generate three question-answer pairs based on the foreground description and background description of the video. Foreground description: xxxx (here is the foreground description of the video data generated in step a), background description: xxx (here is the background description of the video data generated in step a)". The topic part can be as follows:

[0134] •Please pay attention to the content of the events in the video and the time when they occurred to generate question and answer pairs.

[0135] •Please pay attention to the content and order of events in the video to generate question-answer pairs.

[0136] •Please pay attention to the movement trajectory of each human body in the video and generate question-answer pairs.

[0137] •Please pay attention to the number of characters in the video and their changes to generate question-answer pairs.

[0138] •Please pay attention to the clothing and posture of the human body in the video to generate question-answer pairs.

[0139] In this embodiment, by extracting foreground and background descriptions from video data, the system can fully capture the dynamic and static information in the video and form a deep understanding of the video content. Based on these descriptions and preset key information, a third prompt word is generated to ensure that the third prompt word can accurately reflect the key elements and logical relationships specified in the video. The third prompt word and video data are then input into the large language model, enabling it to generate question-answer pairs related to the preset key information in the video data based on multimodal information. In this way, the pertinence and contextual consistency of the question-answer pairs are improved, and the accuracy and practicality of the large language model in video analysis and question-answer pair generation are enhanced, enabling it to be more effectively applied to a variety of practical scenarios, such as video surveillance, content analysis, and intelligent question-answering systems.

[0140] Step 304: Use the original image set and the corresponding question-answer pairs as training data to fine-tune the pre-trained multimodal large model to obtain a target multimodal large model adapted to the monitoring scenario.

[0141] In this step, the original image set and the corresponding question-answer pairs are used as multimodal training data, enriching the diversity and specificity of the training data. By using the original image set and question-answer pairs for fine-tuning training, the pre-trained multimodal large model is optimized, making it more adaptable to the needs of specific monitoring scenarios. This improves the performance and application effectiveness of the multimodal large model in monitoring scenarios, thereby enhancing the intelligence level of the monitoring system.

[0142] In one embodiment, the multimodal large model includes a visual encoder, a visual-text feature mapper, and a large language model. The visual encoder is used to extract the original visual features of the original image set. The visual-text mapper is used to map the original visual features to a feature format compatible with the large language model to obtain converted target visual features. The large language model is used to generate predicted answers based on the target visual features and the question instructions in the question-answer pair. Step 304 may specifically include: during the fine-tuning training process, the difference information between the true answer and the predicted answer in the constrained question-answer pair is taken as the goal.

[0143] In this embodiment, the multimodal large model may include a visual encoder, a visual-text feature mapper, and a large language model. The visual encoder is responsible for extracting visual features from the original image set. The visual features here include but are not limited to color information, texture information, shape information, edge features, pixel gradient features, spatial features, and other features of the original image, ensuring that the multimodal large model can capture the visual details and key information in the original image. The visual-text feature mapper converts these original visual features into a format compatible with the large language model, ensuring that the large language model can accurately understand the multimodal information. The large language model uses the converted target visual features and the question instructions in the question-answer pair to generate a predicted answer. A question-answer pair includes a question and the corresponding true answer to the question. During the fine-tuning training process, the question in the question-answer pair is used as a text instruction (i.e., question instruction) to guide the large model to output the predicted answer. The goal is to constrain the difference information between the true answer and the predicted answer in the question-answer pair, ensuring that the model continuously optimizes the accuracy and consistency of its output during the training process, so that the target multimodal large model after fine-tuning training can gradually adapt to the needs of specific monitoring scenarios. This not only improves the adaptability of the multimodal large model to complex scenarios, but also enhances its reliability and effectiveness in practical applications.

[0144] In one embodiment, the multimodal large model includes a visual encoder, a local encoder, a visual text feature mapper and a large language model. The feature information includes the position information of the target object in the original image set. Among them: the visual encoder is used to extract the original visual features of the original image set. The local encoder is used to extract the local features of the target object from the original visual features based on the position information of the target object, and splice the local features with the original visual features. The visual text mapper is used to map the spliced ​​visual features to a feature format adapted to the large language model to obtain the converted target visual features. The large language model is used to generate predicted answers based on the target visual features and the question instructions in the question and answer pair. Step 304 may specifically include: in the fine-tuning training process, the difference information between the true answer and the predicted answer in the constrained question and answer pair is used as the target.

[0145] In this embodiment, compared with the aforementioned embodiment, the multimodal large model may further include a local encoder. The visual encoder is responsible for extracting the overall visual features of the original image set to ensure that the model has a comprehensive perception of global information. The local encoder uses the position information of the target object to extract the local features of the target object from the original visual features, such as extracting the local features of the target object from the shallow information output by the visual encoder, and splicing them with the deep information in the global visual features. The model can enhance its attention to detail capture and local information of the target object, enabling the model to understand the key elements in the image more accurately. The visual text feature mapper converts the spliced ​​visual features into a format compatible with the large language model to ensure the effective integration and transmission of multimodal information. The large language model generates predicted answers based on the converted target visual features and the question instructions in the question-answer pair, thereby realizing intelligent conversion from vision to language. During the fine-tuning training process, the goal is to constrain the difference information between the true answer and the predicted answer in the question-answer pair, ensuring that the multimodal large model continuously optimizes the accuracy and consistency of its output during training. This enables the multimodal large model to more accurately understand and process multimodal data in monitoring scenarios, providing more intelligent and efficient solutions to meet diverse practical application needs.

[0146] Optionally, in actual scenarios, model training for a large multimodal model can include three stages: the first stage is the pre-training stage for image-text modality alignment, which can be trained using a public pre-training dataset. The second stage is the single-image understanding and positioning capability improvement stage, which can also be trained using public image understanding question-answer pairs and target detection data. The third stage is the instruction fine-tuning stage, which can perform model fine-tuning training based on the single-image text data, multi-image text data, and video text data obtained in any of the above embodiments of the present application.

[0147] like Figure 7 As shown in FIG, a schematic diagram of the model structure of a multimodal large model provided in an embodiment of the present application mainly includes four parts: a visual encoder, a local encoder, a visual-text feature mapper, and a large language model, wherein:

[0148] The visual encoder can be implemented using the SigLIP model or the CLIP model, or other types of models to obtain the visual signal X V The local encoder is based on the offline extracted single-image target box (i.e., the rectangular box of the target object in a single original image), and the output token Z of the third layer Transformer of the visual encoder is obtained. v0 Extract the token Z corresponding to the target object v1 , token Z v0 The shallow features of the original image are included in the image. The shallow features are the direct reflection of the basic elements of the original image. They can retain more fine-grained information and provide an accurate data basis for the local feature extraction of the target object. For example, the coordinates of the target box in a single image are: car [20, 30, 200, 130], person [400, 500, 480, 628]..., the local encoding can be based on the single image target box, from the output token Z of the third layer Transformer of the visual encoder v0 Extract the car's token Z v1 and the character's token Z v1 .

[0149] For a single original image, the token Z of the target object can be v1 It is concatenated with the output of the penultimate layer Transformer of the visual encoder, and the concatenated visual feature Z v The final token representation of a single original image. The output of the penultimate Transformer layer of the visual encoder is the deep features of the original image. These features represent the overall structure and content of the original image, understanding object relationships and overall semantics, which are key to image recognition and classification tasks. The spliced ​​visual features can enhance local target objects. For example, a car might be missed in the pre-splicing model description, but after splicing, it can be described, improving accuracy.

[0150] For the video data in the original image set, all video frames can be processed, or some video frames in the video data can be extracted, such as 8 frames or 16 frames from each video. The rectangular frame of the target object in each frame of the video image is extracted through the local encoder, and the rectangular frames extracted from different frames of the same video data are clustered. The rectangular frames of the same target object are clustered into one category, and then spliced ​​with the output of the visual encoder to obtain the spliced ​​visual feature Zv as the final token expression of the video data.

[0151] The visual text feature mapper can be implemented using a two-layer MLP to map the concatenated visual feature token Zv to the target visual feature token H that can be accepted by the large language model. V .

[0152] The large language model can be implemented using Qwen2 or other large language models, which is not limited in this embodiment. The question in the question-answer pair is used as the text instruction X q , extract text instruction X q Text token H q ,

[0153] The target visual feature token H V With text token H q Input the large language model together, guide the large language model to output the predicted answer, the predicted answer in this embodiment is Figure 7 The text output Xa in the question-answer pair can be minimized during fine-tuning training. q The difference between the corresponding true answer and the predicted answer Xa is used as the optimization target, so that the multimodal large model can continuously optimize the accuracy and consistency of its output during the training process, so that the multimodal large model can more accurately understand and process the multimodal data in the monitoring scenario and meet the diverse practical application needs.

[0154] Optionally, for a text instruction X of length L q , the large language model can use the following formula to calculate the predicted answer X a The probability P is:

[0155]

[0156] Among them, X V is the visual signal, x i It is the token of the predicted answer to be output in round i, such as the complete predicted answer X a "This describes an outdoor square with two cars and a pedestrian. There are buildings around the square." The large language model outputs each word or phrase in turn, x i It is the token output in round i. (q,<i) The text instructions used for each round of prediction before round i, X (a,<i) It is the predicted answer token output in each round before round i.

[0157] In one embodiment, the method further includes: obtaining at least two images to be processed in response to a human body comparison instruction. Inputting the at least two images to be processed into a target multimodal large model so that the target multimodal large model outputs whether the human subjects in the at least two images to be processed are the same person. Determining the comparison accuracy of the target multimodal large model based on the output results of the target multimodal large model and / or the results of manual review. If the comparison accuracy is less than a first preset threshold, adding a supplementary image and a question-answer pair of the supplementary image to the training data to generate new training data, and iteratively training the target multimodal large model based on the new training data.

[0158] In this embodiment, the accuracy and reliability of the multimodal large model in human body detection tasks can be improved by evaluating the output of the multimodal large model, dynamically adjusting the training data, and iteratively optimizing the model. Specifically, in response to a human body comparison instruction, the system acquires at least two images to be processed and inputs them into the target multimodal large model. The system then leverages the multimodal analysis capabilities of the multimodal large model, combining visual and language information, to determine whether the human subjects in these images represent the same person. Based on the output of the multimodal large model or combined with manual review and confirmation, the model's comparison accuracy is evaluated to monitor the model's performance in practical applications. The human body comparison instruction can be triggered proactively by the user or automatically and periodically by the system. For example, an inspector can proactively trigger multiple human body detection tasks and collect the output of the multimodal large model. Alternatively, in a multimodal large model application, the system can collect relevant input data and output results of human body detection tasks reported by users and calculate the comparison accuracy based on the multiple output results. For example, the number of correct detections in the output results can be counted, and the ratio of the number of correct detections to the total number of detections can be used as the comparison accuracy. When the comparison accuracy is lower than the first preset threshold, the system will add supplementary images and their corresponding question-answer pairs to the training data to generate new training data. The first threshold can be set according to actual needs. For example, the first threshold can be set to 90%. If the comparison accuracy of the human body detection task is less than 90%, it means that the human body comparison accuracy of the multimodal large model is not accurate enough. You can add single-image text data and multi-image text data to the original training data, and iteratively train the multimodal large model based on the new training data. This dynamic training data enhancement strategy ensures that the model can be exposed to more diverse and challenging samples. By iteratively training the target multimodal large model, the system continuously optimizes the model parameters to make it more accurate and stable in human body detection tasks. Through closed-loop feedback and continuous optimization, this solution enhances the adaptability and practicality of the model in complex scenarios, ensuring that it can provide efficient and accurate detection results in practical applications.

[0159] In one embodiment, the method further includes: obtaining a surveillance video to be processed in response to a behavior recognition instruction. Inputting the surveillance video to be processed into the target multimodal large model so that the target multimodal large model outputs the specified behavior category present in the surveillance video to be processed based on the behavior detection prompt word. Determining the recognition accuracy of the target multimodal large model based on the output results of the target multimodal large model and / or the manual review results. If the recognition accuracy is less than a second preset threshold, adding a supplementary video and a question-answer pair of the supplementary video to the training data to generate new training data, and iteratively training the target multimodal large model based on the new training data.

[0160] In this embodiment, to meet the needs of a large multimodal model for action warning tasks, the accuracy and robustness of the large multimodal model in behavior detection tasks can be improved by evaluating its output, performing dynamic data augmentation, and iterative training. Recognition accuracy can include the precision (P) and / or recall (R) of the large multimodal model for action warning. In practical scenarios, the second threshold can include a threshold for precision (P) and a threshold for recall (R), respectively, to facilitate threshold comparisons for precision (P) and recall (R), and to determine the need for additional training data based on the threshold comparison results. Specifically, in response to a behavior recognition instruction, the system obtains the surveillance video to be processed and inputs it into the target large multimodal model. The model uses behavior detection prompts to analyze the video content and output the specified behavior categories present. The behavior recognition instruction can be triggered actively by the user or automatically and periodically by the system. For example, inspectors can proactively trigger multiple behavior detection tasks and collect the output results of a large multimodal model. Alternatively, in a large multimodal model application, they can collect relevant input data and output results of behavior detection tasks reported by users. Based on multiple output results and combined with manual review results, the recognition accuracy is statistically analyzed. Designated behavior categories are pre-set action categories associated with the monitoring scenario that pose safety risks. For example, in a shopping mall or supermarket monitoring scenario, designated behavior types include, but are not limited to, picking up and putting goods on shelves, smoking, eating, passing bags, and playing with mobile phones. By evaluating the model's output results for behavior detection tasks, the system can monitor its accuracy in these tasks.

[0161] For example, in a supermarket surveillance scenario, if the supermarket monitoring staff discovers that the system's action recognition is incorrect, they can report the relevant pending video. For each pending video reported by the user, they can obtain the results of a manual inspection of the pending video content and then determine whether the manual inspection results are consistent with the action type results output by the system. If the manual inspection results show that there are no actions of the specified behavior type in the pending video, or if there are actions of the specified behavior type but the action type is different from the system output, it will be recorded as a false positive. If some actions of the specified behavior type in the pending video are included in the manual inspection results but are not detected by the system, matching target action types can be selected for these actions of the specified behavior type from the candidate action type list, and these will be treated as missed data of the target action type.

[0162] Optionally, taking the recognition accuracy determination process of action warning as an example, the recognition accuracy here can be characterized by the precision (P) and recall (R) of the action warning. The precision (P) and recall (R) of the first type of action warning are defined as follows:

[0163] P = the number of correct actions reported by the system as Class I actions / the total number of Class I actions reported by the system.

[0164] R = the number of correct actions reported by the system as Class I actions / the total number of all true Class I actions in the manual inspection results.

[0165] Assume that five action types are defined. For the first action type (smoking), 10 videos were reviewed, two of which were false positives. While reviewing actions of other action types (such as eating), a video of eating was found to contain smoking. This video was marked as a false negative for the smoking action type.

[0166] The recognition accuracy can be compared with the second threshold periodically or in real time. When the recognition accuracy is lower than the second preset threshold, the system will add supplementary videos and their corresponding question-answer pairs to the training data to generate new training data. The second threshold can be set according to actual needs. For example, the second threshold can be set to 90%. This dynamic adjustment strategy ensures that the model can be exposed to more diverse and challenging behavior samples, improving its ability to recognize different behavior patterns. By iteratively training the target multimodal large model, the system continuously optimizes the model parameters to make it more accurate and stable in behavior detection tasks. Through closed-loop feedback and continuous optimization, this solution enhances the adaptability and practicality of the model in complex monitoring scenarios, ensuring that it can provide efficient and accurate behavior detection results in practical applications.

[0167] like Figure 8 FIG. 1 is a schematic diagram of a model training process provided in an embodiment of the present application, including:

[0168] Fine-tune data generation, model fine-tuning, evaluation feedback, and evaluate whether the model output meets the preset indicators (such as the first and second thresholds mentioned above). If the indicators are met, training can be terminated. If the indicators are not met, relevant training data is supplemented for iterative model training.

[0169] The above-mentioned processing method of the multimodal large model proposes a method for establishing a fine-tuning data set in intelligent monitoring scenarios and a multimodal large model structure that strengthens local target features. It also establishes an evaluation feedback and model iteration mechanism for the application of the multimodal large model in monitoring scenarios, which improves the accuracy of information perception and event understanding of the multimodal large model in monitoring scenarios, can achieve better performance in actual monitoring scenarios, and improve the implementation effect of the multimodal large model in monitoring scenarios.

[0170] Please see Figure 9 , which is a behavior information identification method of an embodiment of the present application, the method can be Figure 1 The electronic device 1 shown is used to perform and can be applied to Figure 2 Compared with the above embodiment, this embodiment takes the shopping mall monitoring scenario as an example, and the method includes the following steps:

[0171] Step 901: In response to the recognition instruction, obtain the surveillance video of the shopping mall.

[0172] Step 902: Input the surveillance video into the target multimodal large model, so that the target multimodal large model outputs a recognition result of the surveillance video under the guidance of a preset prompt word. The target multimodal large model is obtained according to the method of any of the above embodiments.

[0173] For details of each step of the above method, please refer to the relevant description of the above embodiment, which will not be repeated here.

[0174] Please see Figure 10 , which is a multimodal large model processing device 1000 of an embodiment of the present application, which can be applied to Figure 1 The electronic device 1 shown can be applied to Figure 2 The device includes: an acquisition module 1001, an identification module 1002, a generation module 1003 and a fine-tuning module 1004. The functional principles of each module are as follows:

[0175] The acquisition module 1001 is used to acquire a set of original images of a monitoring scene.

[0176] The recognition module 1002 is used to recognize feature information of a target object in the original image set.

[0177] The generation module 1003 is used to generate question-answer pairs corresponding to the original image set based on the feature information.

[0178] The fine-tuning module 1004 is used to use the original image set and the corresponding question-answer pairs as training data to fine-tune the pre-trained multimodal large model to obtain a target multimodal large model adapted to the monitoring scenario.

[0179] In one embodiment, the recognition module 1002 is configured to determine the object category of each original image in the original image set and, based on the object category, identify the location information of at least one target object in the original image. The feature information includes the object category and / or location information.

[0180] In one embodiment, the generation module 1003 is configured to generate a first prompt word for a preset large language model based on the feature information. The first prompt word and the original image set are input into the large language model so that the large language model generates a question-answer pair about the target object based on the first prompt word.

[0181] In one embodiment, generation module 1003 is configured to add a location marker to the target object in the original image based on the feature information to generate a marked image. A second prompt word of a preset large language model is generated based on the location marker. The marked image and the second prompt word are input into the preset large language model, so that the large language model generates a question-answer pair about the target object based on the second prompt word and the marked image.

[0182] In one embodiment, the original image set includes a human body image set. The feature information includes identification information of a human body object in each human body image. The generation module 1003 is used to screen out a first image pair having the same identification information and a similarity less than a first threshold and a second image pair having different identification information and a similarity greater than a second threshold from the human body image set, and mark the real identity labels of the first image pair and the second image pair, and determine the first image pair and the second image pair as a difficult sample pair. The combination of any two images in the human body image set excluding the difficult sample pairs is determined as an easy sample pair, and the identity label of the easy sample pair is marked according to the identification information of each human body image in the easy sample pair. According to the preset questions and identity labels, question-answer pairs corresponding to the difficult sample pairs and the easy sample pairs are generated respectively.

[0183] In one embodiment, the original image set includes video data. Feature information includes a video foreground description and a video background description of the video data. Generation module 1003 is configured to generate a third prompt word for the large language model based on the video foreground description, the video background description, and preset key information about the video data. The third prompt word and the video data are input into the large language model so that the large language model generates at least one question-answer pair corresponding to the video data based on the third prompt word. The preset key information includes one or more of: event time, event sequence, object motion trajectory, number of human subjects, and appearance characteristics of human subjects in the video.

[0184] In one embodiment, the multimodal large model includes a visual encoder, a visual-text feature mapper, and a large language model. The visual encoder is used to extract raw visual features from the original image set. The visual-text mapper is used to map the raw visual features to a feature format compatible with the large language model to obtain converted target visual features. The large language model is used to generate predicted answers based on the target visual features and the question instructions in the question-answer pair. The fine-tuning module 1004 is used to target the difference information between the true answer and the predicted answer in the constrained question-answer pair during the fine-tuning training process.

[0185] In one embodiment, the multimodal large model includes a visual encoder, a local encoder, a visual-text feature mapper, and a large language model. The feature information includes the position information of the target object in the original image set. Among them: the visual encoder is used to extract the original visual features of the original image set. The local encoder is used to extract the local features of the target object from the original visual features based on the position information of the target object, and splice the local features with the original visual features. The visual-text mapper is used to map the spliced ​​visual features to a feature format compatible with the large language model to obtain the converted target visual features. The large language model is used to generate a predicted answer based on the target visual features and the question instructions in the question-answer pair.

[0186] In one embodiment, the fine-tuning module 1004 is configured to target the difference information between the true answer and the predicted answer in the constrained question-answer pair during the fine-tuning training process.

[0187] In one embodiment, the device further includes: a first iterative module for obtaining at least two images to be processed in response to a human body comparison instruction. The at least two images to be processed are input into a target multimodal large model so that the target multimodal large model outputs whether the human body objects in the at least two images to be processed are the same person. The comparison accuracy of the target multimodal large model is determined based on the output result of the target multimodal large model. If the comparison accuracy is less than a first preset threshold, a supplementary image and a question-answer pair of the supplementary image are added to the training data to generate new training data, and the target multimodal large model is iteratively trained based on the new training data.

[0188] In one embodiment, the device further includes: a second iterative module for obtaining a surveillance video to be processed in response to a behavior recognition instruction. The surveillance video to be processed is input into the target multimodal large model so that the target multimodal large model outputs the specified behavior category present in the surveillance video to be processed based on the behavior detection prompt word. The recognition accuracy of the target multimodal large model is determined based on the output result of the target multimodal large model. If the recognition accuracy is less than a second preset threshold, a supplementary video and a question-answer pair of the supplementary video are added to the training data to generate new training data, and the target multimodal large model is iteratively trained based on the new training data.

[0189] For a detailed description of the processing device 1000 for the multimodal large model, please refer to the description of the relevant method steps in the above embodiment. Its implementation principles and technical effects are similar and will not be repeated here in this embodiment.

[0190] Figure 11 This is a schematic diagram of the structure of a cloud device 110 provided in an exemplary embodiment of the present application. The cloud device 110 can be used to run the method provided in any of the above embodiments. Figure 11 As shown, the cloud device 110 may include: a memory 1104 and at least one processor 1105, Figure 11 A processor is used as an example.

[0191] The memory 1104 is used to store computer programs and can be configured to store various other data to support operations on the cloud device 110. The memory 1104 can be an object storage service (OSS).

[0192] The memory 1104 may be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0193] The processor 1105 is coupled to the memory 1104 and is used to execute the computer program in the memory 1104 to implement the solution provided by any of the above method embodiments. The specific functions and technical effects that can be achieved are not repeated here.

[0194] Furthermore, if Figure 11 The cloud device also includes: a firewall 1101, a load balancer 1102, a communication component 1106, a power supply component 1103 and other components. Figure 11 Only some components are shown schematically, which does not mean that the cloud device only includes Figure 11 Components shown.

[0195] In one embodiment, the above Figure 11 The communication component 1106 is configured to facilitate wired or wireless communication between the device in which the communication component 1106 resides and other devices. The device in which the communication component 1106 resides can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, LTE (Long Term Evolution), 5G, or other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component 1106 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1106 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, or other technologies.

[0196] In one embodiment, the above Figure 11 The power supply component 1103 provides power to various components of the device where the power supply component 1103 is located. The power supply component 1103 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply component is located.

[0197] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method of any of the aforementioned embodiments is implemented.

[0198] An embodiment of the present application also provides a computer program product, including a computer program, which implements the method of any of the aforementioned embodiments when executed by a processor.

[0199] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the module division is only a logical function division. In actual implementation, other division methods may be used. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not implemented.

[0200] The integrated modules implemented in the form of software function modules can be stored in a computer-readable storage medium. The software function modules stored in a storage medium include a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute some of the steps of the methods of various embodiments of the present application.

[0201] It should be understood that the above-mentioned processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly implemented as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The memory may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile storage NVM (NVM), such as at least one disk storage, and can also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0202] The above-mentioned storage medium can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random-access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0203] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0204] It should be noted that, in this document, the terms "comprise," "include," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, apparel, or apparatus comprising a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, apparel, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a..." does not preclude the presence of other identical elements in the process, method, apparel, or apparatus comprising the element.

[0205] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0206] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of this application, or the part that contributes to the existing technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of each embodiment of this application.

[0207] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of user data and other information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0208] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.< / box> < / box> < / red> < / box> < / box> < / red>

Claims

1. A method for processing a multimodal large model, characterized in that: include: Obtaining a set of original images of the monitoring scene; Identify feature information of a target object in the original image set; Generating question-answer pairs corresponding to the original image set based on the feature information; Using the original image set and the corresponding question-answer pairs as training data, fine-tuning the pre-trained multimodal large model to obtain a target multimodal large model adapted to the monitoring scenario; The multimodal large model includes a visual encoder, a visual-text feature mapper, and a large language model; wherein: The visual encoder is used to extract original visual features of the original image set; The visual-text feature mapper is used to map the original visual features into a feature format adapted to the large language model to obtain a converted target visual feature; The large language model is used to generate a predicted answer based on the target visual feature and the question instruction in the question-answer pair; The original image set includes a human body image set; the feature information includes identification information of a human body object in each of the human body images; and generating question-answer pairs corresponding to the original image set based on the feature information includes: Screening out, from the human image set, a first image pair having the same identification information and a similarity less than a first threshold, and a second image pair having different identification information and a similarity greater than a second threshold, annotating the true identity labels of the first image pair and the second image pair, and determining the first image pair and the second image pair as a difficult sample pair; Determine the pair of images in the human body image set excluding the difficult sample pairs as easy sample pairs, and label the easy sample pairs with identity labels according to the identification information of each human body image in the easy sample pairs; According to the preset questions and the identity labels, question-answer pairs corresponding to the difficult sample pairs and the easy sample pairs are generated respectively.

2. The method according to claim 1, characterized in that The identifying feature information of the target object in the original image set includes: determining an object category included in each original image in the original image set; identifying, according to the object category, location information of at least one target object included in the original image; The feature information includes the object category and / or the location information.

3. The method according to claim 1, characterized in that Generating question-answer pairs corresponding to the original image set based on the feature information includes: generating a first prompt word of a preset large language model according to the feature information; The first prompt word and the original image set are input into the large language model, so that the large language model generates a question-answer pair about the target object according to the first prompt word.

4. The method according to claim 1, wherein Generating question-answer pairs corresponding to the original image set based on the feature information includes: adding a position mark to the target object in the original image according to the feature information to generate a marked image; generating a second prompt word of a preset large language model according to the position mark; The labeled image and the second prompt word are input into the preset large language model, so that the large language model generates a question-answer pair about the target object according to the second prompt word and the labeled image.

5. The method according to claim 1, wherein The original image set includes video data; the feature information includes a video foreground description and a video background description of the video data; and generating a question-answer pair corresponding to the original image set based on the feature information includes: generating a third prompt word of a large language model according to the video foreground description, the video background description, and preset key information about the video data; inputting the third prompt word and the video data into the large language model, so that the large language model generates at least one question-answer pair corresponding to the video data according to the third prompt word; The preset key information includes one or more of the following: event time, event sequence, object motion trajectory, number of human objects, and appearance features of human objects in the video.

6. The method according to claim 1, characterized in that The method uses the original image set and the corresponding question-answer pairs as training data to fine-tune the pre-trained multimodal large model to obtain a target multimodal large model adapted to the monitoring scenario, including: During the fine-tuning training process, the goal is to constrain the difference information between the true answer and the predicted answer in the question-answer pair.

7. The method according to claim 1, characterized in that The multimodal large model includes a visual encoder, a local encoder, a visual-text feature mapper, and a large language model; the feature information includes the position information of the target object in the original image set; wherein: The visual encoder is used to extract original visual features of the original image set; The local encoder is used to extract local features of the target object from the original visual features according to the position information of the target object, and splice the local features with the original visual features; The visual text mapper is used to map the concatenated visual features into a feature format adapted to the large language model to obtain converted target visual features; The large language model is used to generate a predicted answer based on the target visual feature and the question instruction in the question-answer pair; The method uses the original image set and the corresponding question-answer pairs as training data to fine-tune the pre-trained multimodal large model to obtain a target multimodal large model adapted to the monitoring scenario, including: During the fine-tuning training process, the goal is to constrain the difference information between the true answer and the predicted answer in the question-answer pair.

8. The method according to claim 1, characterized in that Also includes: In response to a human body comparison instruction, acquiring at least two images to be processed; Inputting the at least two images to be processed into the target multimodal large model, so that the target multimodal large model outputs whether the human objects in the at least two images to be processed are the same person; Determining the comparison accuracy of the target multimodal large model according to the output result of the target multimodal large model; If the comparison accuracy is less than a first preset threshold, a supplementary image and a question-answer pair of the supplementary image are added to the training data to generate new training data, and the target multimodal large model is iteratively trained based on the new training data.

9. The method according to claim 1, characterized in that Also includes: Responding to the behavior recognition instruction, obtaining the surveillance video to be processed; Inputting the surveillance video to be processed into the target multimodal large model, so that the target multimodal large model outputs the specified behavior category present in the surveillance video to be processed according to the behavior detection prompt word; Determining the recognition accuracy of the target multimodal large model according to the output result of the target multimodal large model; If the recognition accuracy is less than a second preset threshold, a supplementary video and a question-answer pair of the supplementary video are added to the training data to generate new training data, and the target multimodal large model is iteratively trained based on the new training data.

10. A behavior information recognition method, characterized in that: include: Responding to the recognition instruction, acquiring surveillance video of the shopping mall; Inputting the surveillance video into a target multimodal large model, so that the target multimodal large model outputs a recognition result of the surveillance video under the guidance of a preset prompt word; The target multimodal large model is obtained according to the method according to any one of claims 1 to 9.

11. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the electronic device to perform the method according to any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the method according to any one of claims 1 to 10 is implemented.

13. A computer program product, characterized in that The method comprises a computer program, which implements the method according to any one of claims 1 to 10 when the computer program is executed by a processor.

Images