Multi-modal large model processing method and device, storage medium and program product

Abstract

The invention provides a multi-modal large model processing method and device, a storage medium and a program product. The method comprises the steps of obtaining an original image set of a monitoring scene; identifying feature information of a target object in the original image set; according to the feature information, generating question and answer pairs corresponding to the original image set; and taking the original image set and the corresponding question and answer pair as training data, and performing fine tuning training on a pre-trained multi-modal large model to obtain a target multi-modal large model adaptive to the monitoring scene. According to the method and the device, the multi-modal large model can better adapt to the requirements of a specific monitoring scene, the performance and the application effect of the multi-modal large model in the monitoring scene are improved, and the intelligent level of a monitoring system can be improved.

Description

Technical Field

[0001] The present application relates to the field of information processing technology, and in particular to a processing method, device, storage medium and program product for a multimodal large model. Background Art

[0002] The intelligent video surveillance system adopts image processing, pattern recognition and computer vision technology. By adding an intelligent video analysis module to the monitoring system, it uses the powerful data processing capabilities of the computer to filter out useless or interfering information in the video, automatically identify different objects, analyze and extract key useful information in the video source, quickly and accurately locate the accident scene, judge abnormal situations in the monitoring screen, and issue alarms or trigger other actions in the fastest and best way, thereby effectively conducting pre-warning, in-process processing, and timely evidence collection afterwards. It is a fully automatic, all-weather, real-time monitoring intelligent system.

[0003] Existing intelligent monitoring systems rely on multiple expert models to achieve structured processing of video information. Each expert model requires special data collection and model training, so the system is complex and has high maintenance costs; the response time to adapt to new scenarios or new demands is long.

[0004] The multimodal large model can integrate multiple information such as videos, images, and texts to improve the ability to understand complex scenes. Thanks to the learning of massive data, it has good generalization ability. With strong environmental understanding and generalization, the large model can perform multiple tasks as needed, and can make reasonable inferences and responses when faced with new scenes or new needs. Therefore, the multimodal large model has great application potential in the field of intelligent monitoring.

[0005] However, when the current mainstream multimodal large models are directly applied to intelligent monitoring systems in specific scenarios, the scene adaptability is poor, resulting in low application performance of the large models in intelligent monitoring systems and difficulty in achieving practical use. Summary of the invention

[0006] The main purpose of the embodiments of the present application is to provide a processing method, device, storage medium and program product for a multimodal large model, so as to enable the multimodal large model to better adapt to the needs of specific monitoring scenarios, improve the performance and application effect of the multimodal large model in the monitoring scenario, and thus improve the intelligence level of the monitoring system.

[0007] In a first aspect, an embodiment of the present application provides a method for processing a multimodal large model, comprising: acquiring an original image set of a monitoring scene; identifying feature information of a target object in the original image set; generating a question-answer pair corresponding to the original image set based on the feature information; using the original image set and the corresponding question-answer pairs as training data, fine-tuning the pre-trained multimodal large model, and obtaining a target multimodal large model adapted to the monitoring scene.

[0008] In one embodiment, the identifying feature information of the target object in the original image set includes: determining the object category included in each original image in the original image set; and identifying the location information of at least one target object included in the original image based on the object category; wherein the feature information includes the object category and / or the location information.

[0009] In one embodiment, generating a question-answer pair corresponding to the original image set based on the feature information includes: generating a first prompt word of a preset large language model based on the feature information; inputting the first prompt word and the original image set into the large language model, so that the large language model generates a question-answer pair about the target object based on the first prompt word.

[0010] In one embodiment, generating a question-answer pair corresponding to the original image set based on the feature information includes: adding a position mark to the target object in the original image based on the feature information to generate a marked image; generating a second prompt word of a preset large language model based on the position mark; and inputting the marked image and the second prompt word into the preset large language model so that the large language model generates a question-answer pair about the target object based on the second prompt word and the marked image.

[0011] In one embodiment, the original image set includes a human image set; the feature information includes identification information of a human object in each of the human images; generating a question-answer pair corresponding to the original image set based on the feature information includes: screening out a first image pair having the same identification information and a similarity less than a first threshold and a second image pair having different identification information and a similarity greater than a second threshold from the human image set, and marking the real identity labels of the first image pair and the second image pair, and determining the first image pair and the second image pair as a difficult sample pair; determining a combination of any two images in the human image set excluding the difficult sample pair as an easy sample pair, and marking the identity label of the easy sample pair according to the identification information of each human image in the easy sample pair; and generating question-answer pairs corresponding to the difficult sample pair and the easy sample pair according to preset questions and the identity labels, respectively.

[0012] In one embodiment, the original image set includes video data; the feature information includes a video foreground description and a video background description of the video data; generating a question-answer pair corresponding to the original image set based on the feature information includes: generating a third prompt word of a large language model based on the video foreground description, the video background description and preset key information about the video data; inputting the third prompt word and the video data into the large language model, so that the large language model generates at least one question-answer pair corresponding to the video data based on the third prompt word; wherein the preset key information includes: one or more of event time, event sequence, object motion trajectory, number of human objects, and appearance features of human objects in the video.

[0013] In one embodiment, the multimodal large model includes a visual encoder, a visual-text feature mapper and a large language model; wherein: the visual encoder is used to extract the original visual features of the original image set; the visual-text mapper is used to map the original visual features to a feature format adapted to the large language model to obtain converted target visual features; the large language model is used to generate predicted answers based on the target visual features and the question instructions in the question-answer pair; the original image set and the corresponding question-answer pair are used as training data to fine-tune the pre-trained multimodal large model to obtain a target multimodal large model adapted to the monitoring scene, including: in the fine-tuning training process, the difference information between the true answer in the question-answer pair and the predicted answer is constrained as the goal.

[0014] In one embodiment, the multimodal large model includes a visual encoder, a local encoder, a visual text feature mapper and a large language model; the feature information includes the position information of the target object in the original image set; wherein: the visual encoder is used to extract the original visual features of the original image set; the local encoder is used to extract the local features of the target object from the original visual features according to the position information of the target object, and splice the local features with the original visual features; the visual text mapper is used to map the spliced ​​visual features to a feature format adapted to the large language model to obtain the converted target visual features; the large language model is used to generate a predicted answer according to the target visual features and the question instructions in the question-answer pair;

[0015] In one embodiment, the original image set and the corresponding question-answer pairs are used as training data to perform fine-tuning training on the pre-trained multimodal large model to obtain a target multimodal large model adapted to the monitoring scenario, including: during the fine-tuning training process, the goal is to constrain the difference information between the true answer and the predicted answer in the question-answer pair.

[0016] In one embodiment, the method also includes: in response to a human body comparison instruction, acquiring at least two images to be processed; inputting the at least two images to be processed into the target multimodal large model, so that the target multimodal large model outputs whether the human body objects in the at least two images to be processed are the same person; determining the detection accuracy of the target multimodal large model according to the output result of the target multimodal large model; if the comparison accuracy is less than a first preset threshold, adding a supplementary image and a question-answer pair of the supplementary image to the training data to generate new training data, and iteratively training the target multimodal large model according to the new training data.

[0017] In one embodiment, the method also includes: in response to a behavior recognition instruction, obtaining a surveillance video to be processed; inputting the surveillance video to be processed into the target multimodal large model, so that the target multimodal large model outputs a specified behavior category present in the surveillance video to be processed according to the behavior detection prompt word; determining the recognition accuracy of the target multimodal large model according to the output result of the target multimodal large model; if the recognition accuracy is less than a second preset threshold, adding a supplementary video and a question-answer pair of the supplementary video to the training data to generate new training data, and iteratively training the target multimodal large model according to the new training data.

[0018] In a second aspect, an embodiment of the present application provides a behavior information recognition method, comprising: obtaining a surveillance video of a shopping mall in response to a recognition instruction; inputting the surveillance video into a target multimodal large model, so that the target multimodal large model outputs a recognition result about the surveillance video under the guidance of a preset prompt word; wherein the target multimodal large model is obtained according to the method described in any of the above aspects.

[0019] In a third aspect, an embodiment of the present application provides a processing device for a multimodal large model, including:

[0020] An acquisition module is used to acquire a set of original images of the monitoring scene;

[0021] A recognition module, used to recognize feature information of a target object in the original image set;

[0022] A generation module, used to generate question-answer pairs corresponding to the original image set according to the feature information;

[0023] A fine-tuning module is used to use the original image set and the corresponding question-answer pairs as training data to fine-tune the pre-trained multimodal large model to obtain a target multimodal large model adapted to the monitoring scene.

[0024] In one embodiment, the recognition module is used to determine the object category included in each original image in the original image set; based on the object category, identify the location information of at least one target object included in the original image; wherein the feature information includes the object category and / or the location information.

[0025] In one embodiment, the generation module is used to generate a first prompt word of a preset large language model based on the feature information; input the first prompt word and the original image set into the large language model, so that the large language model generates a question-answer pair about the target object based on the first prompt word.

[0026] In one embodiment, the generation module is used to add a position mark to the target object in the original image according to the feature information to generate a marked image; generate a second prompt word of a preset large language model according to the position mark; input the marked image and the second prompt word into the preset large language model, so that the large language model generates a question-answer pair about the target object according to the second prompt word and the marked image.

[0027] In one embodiment, the original image set includes a human image set; the feature information includes identification information of a human object in each of the human images; the generation module is used to screen out a first image pair having the same identification information and a similarity less than a first threshold and a second image pair having different identification information and a similarity greater than a second threshold from the human image set, and mark the real identity labels of the first image pair and the second image pair, and determine the first image pair and the second image pair as a difficult sample pair; determine the combination of any two images in the human image set excluding the difficult sample pair as an easy sample pair, and mark the identity label of the easy sample pair according to the identification information of each human image in the easy sample pair; and generate question-answer pairs corresponding to the difficult sample pair and the easy sample pair according to preset questions and the identity labels.

[0028] In one embodiment, the original image set includes video data; the feature information includes a video foreground description and a video background description of the video data; the generation module is used to generate a third prompt word of the large language model according to the video foreground description, the video background description and preset key information about the video data; the third prompt word and the video data are input into the large language model so that the large language model generates at least one question-answer pair corresponding to the video data according to the third prompt word; wherein the preset key information includes: one or more of event time, event sequence, object motion trajectory, number of human objects, and appearance features of human objects in the video.

[0029] In one embodiment, the multimodal large model includes a visual encoder, a visual-text feature mapper and a large language model; wherein: the visual encoder is used to extract the original visual features of the original image set; the visual-text mapper is used to map the original visual features to a feature format adapted to the large language model to obtain converted target visual features; the large language model is used to generate predicted answers based on the target visual features and the question instructions in the question-answer pair; a fine-tuning module is used to constrain the difference information between the true answer in the question-answer pair and the predicted answer during the fine-tuning training process.

[0030] In one embodiment, the multimodal large model includes a visual encoder, a local encoder, a visual text feature mapper and a large language model; the feature information includes the position information of the target object in the original image set; wherein: the visual encoder is used to extract the original visual features of the original image set; the local encoder is used to extract the local features of the target object from the original visual features according to the position information of the target object, and splice the local features with the original visual features; the visual text mapper is used to map the spliced ​​visual features to a feature format adapted to the large language model to obtain the converted target visual features; the large language model is used to generate a predicted answer according to the target visual features and the question instructions in the question-answer pair;

[0031] In one embodiment, the fine-tuning module is used to constrain the difference information between the true answer and the predicted answer in the question-answer pair during the fine-tuning training process.

[0032] In one embodiment, the device also includes: a first iterative module, used to respond to a human body comparison instruction to obtain at least two images to be processed; input the at least two images to be processed into the target multimodal large model, so that the target multimodal large model outputs whether the human body objects in the at least two images to be processed are the same person; determine the comparison accuracy of the target multimodal large model according to the output result of the target multimodal large model; if the comparison accuracy is less than a first preset threshold, add a supplementary image and a question-answer pair of the supplementary image to the training data to generate new training data, and iteratively train the target multimodal large model according to the new training data.

[0033] In one embodiment, the device also includes: a second iterative module, used to obtain the surveillance video to be processed in response to the behavior recognition instruction; input the surveillance video to be processed into the target multimodal large model, so that the target multimodal large model outputs the specified behavior category present in the surveillance video to be processed according to the behavior detection prompt word; determine the recognition accuracy of the target multimodal large model according to the output result of the target multimodal large model; if the recognition accuracy is less than a second preset threshold, add a supplementary video and a question-answer pair of the supplementary video to the training data to generate new training data, and iteratively train the target multimodal large model according to the new training data.

[0034] In a fourth aspect, an embodiment of the present application provides an electronic device, including:

[0035] at least one processor; and

[0036] a memory communicatively coupled to the at least one processor;

[0037] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the electronic device to execute the method described in any one of the above aspects.

[0038] In a fifth aspect, an embodiment of the present application provides a cloud device, including:

[0039] at least one processor; and

[0040] a memory communicatively coupled to the at least one processor;

[0041] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the cloud device to execute the method described in any one of the above aspects.

[0042] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method described in any one of the above aspects is implemented.

[0043] In a seventh aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements the method described in any of the above aspects when executed by a processor.

[0044] The processing method, device, storage medium and program product of the multimodal large model provided in the embodiment of the present application realizes fine-tuning training of the multimodal large model by processing and analyzing the original image set of the monitoring scene, thereby improving the adaptability and recognition accuracy of the model in a specific monitoring scene. Specifically, by acquiring the original image set of the monitoring scene, the actual visual information in the monitoring scene can be captured, and by identifying the feature information of the target object in the image set, the system can understand and extract the key elements in the scene, and then generate question-answer pairs based on the feature information, and use the original image set and the corresponding question-answer pairs as multimodal training data, enriching the diversity and pertinence of the training data; by using the original image set and the question-answer pairs for fine-tuning training, the pre-trained multimodal large model is optimized to better adapt to the needs of specific monitoring scenes, improve the performance and application effect of the model in the monitoring scene, and then improve the intelligence level of the monitoring system. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application. Obviously, the drawings described below are some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor.

[0046] Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0047] Figure 2 A schematic diagram of an application scenario of a multimodal large model processing system provided in an embodiment of the present application;

[0048] Figure 3 A schematic diagram of a process flow of a multimodal large model processing method provided in an embodiment of the present application;

[0049] Figure 4 A schematic diagram of fine-tuning data provided in an embodiment of the present application;

[0050] Figure 5 A schematic diagram of feature information identification provided in an embodiment of the present application;

[0051] Figure 6 A schematic diagram of a marking image provided in an embodiment of the present application;

[0052] Figure 7 A schematic diagram of the model structure of a multimodal large model provided in an embodiment of the present application;

[0053] Figure 8 A schematic diagram of a model training process provided in an embodiment of the present application;

[0054] Fig. 9 A flowchart of a behavior information identification method provided in an embodiment of the present application;

[0055] Fig.10 A schematic diagram of the structure of a multi-modal large model processing device provided in an embodiment of the present application;

[0056] Fig.11 A schematic diagram of the structure of a cloud device provided in an embodiment of the present application.

[0057] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0058] Here, exemplary embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application.

[0059] The term "and / or" in this article is used to describe the association relationship of associated objects, specifically indicating that there may be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0060] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0061] In order to clearly describe the technical solution of the embodiment of the present application, the terms involved in the present application are first defined:

[0062] Multimodal large model: a large-scale artificial intelligence system that can understand and integrate multiple types of data (such as text, images, video, voice, etc.). It achieves stronger information processing capabilities by jointly learning cross-modal representations.

[0063] Prompt words: Keywords or sentences used to guide the large model to accurately understand task requirements and respond accordingly.

[0064] Token: For text, it is the basic unit of data segmentation, which can be a word, character or subword. For visual information, it is the feature vector of a piece of image after segmentation.

[0065] Human ID: A unique identifier of a pedestrian within a day. All human images with the same ID belong to the same pedestrian. Human IDs do not cross days.

[0066] GroundingDINO: Grounding DETR with Improved deNoising anchOr boxes is a deep learning model that combines target detection and language understanding. The main goal of this model is to detect targets of any category based on human question input, that is, open-world target detection.

[0067] ReID: Re-identification, usually refers to "Person Re-identification". ReID is a computer vision task that aims to identify and match pedestrians of the same identity across different camera views. In practical applications, this technology is often used in surveillance systems to track the movement of specific individuals in areas covered by different cameras.

[0068] ‌CLIP: Contrastive Language-Image Pretraining, a multimodal contrastive learning model, aims to learn the matching relationship between images and texts by contrasting their vector representations. ‌In the pre-training stage, CLIP compares the vector representations of images and texts, trying to bring matching image and text vectors closer and push mismatched vectors farther apart, thereby capturing the deep semantic connection between images and texts. ‌

[0069] ‌SigLIP: Simultaneous Generative and Interpretive Language Processing, a simultaneous generative and interpretive language processing model, is a visual language pre-training model, which is an optimized version of the CLIP model.

[0070] MLP: Multilayer Perceptron.

[0071] Transformer: It is a deep learning model architecture that is mainly used to process sequence data, especially in natural language processing tasks.

[0072] The processing method of the multimodal large model in the embodiment of the present application can be applied to any field that needs to process multimodal information.

[0073] The intelligent video surveillance system adopts image processing, pattern recognition and computer vision technology. By adding an intelligent video analysis module to the monitoring system, it uses the powerful data processing capabilities of the computer to filter out useless or interfering information in the video, automatically identify different objects, analyze and extract key useful information in the video source, quickly and accurately locate the accident scene, judge abnormal situations in the monitoring screen, and issue alarms or trigger other actions in the fastest and best way, thereby effectively conducting pre-warning, in-process processing, and timely evidence collection afterwards. It is a fully automatic, all-weather, real-time monitoring intelligent system.

[0074] The existing intelligent monitoring system relies on multiple expert models to achieve structured processing of video information. Each expert model needs to conduct special data collection and model training, so the system is complex and has high maintenance costs. It takes a long time to adapt to new scenarios or new requirements.

[0075] The multimodal large model can integrate multiple information such as videos, images, and texts to improve the ability to understand complex scenes. Thanks to the learning of massive data, it has good generalization ability. With strong environmental understanding and generalization, the large model can perform multiple tasks as needed, and can make reasonable inferences and responses when faced with new scenes or new needs. Therefore, the multimodal large model has great application potential in the field of intelligent monitoring.

[0076] However, when the current mainstream multimodal large models are directly applied to intelligent monitoring systems in specific scenarios, the scene adaptability is poor, resulting in low application performance of the large models in intelligent monitoring systems and difficulty in achieving practical use.

[0077] In order to solve at least one of the above problems, the embodiment of the present application provides a processing solution for a multimodal large model. By acquiring the original image set of the monitoring scene, the actual visual information in the monitoring scene can be captured. By identifying the feature information of the target object in the image set, the system can understand and extract the key elements in the scene, and then generate question-answer pairs based on the feature information. The original image set and the corresponding question-answer pairs are used as multimodal training data, enriching the diversity and pertinence of the training data. Finally, by using the original image set and question-answer pairs for fine-tuning training, the pre-trained multimodal large model is optimized, so that the multimodal large model can better adapt to the needs of specific monitoring scenarios, improve the performance and application effect of the multimodal large model in the monitoring scenario, and thus improve the intelligence level of the monitoring system.

[0078] Some embodiments of the present application are described in detail below in conjunction with the accompanying drawings. In the case where there is no conflict between the embodiments, the following embodiments and the features in the embodiments can be combined with each other. In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.

[0079] like Figure 1 As shown, this embodiment provides an electronic device 1, including: at least one processor 11 and a memory 12, Figure 1 A processor is taken as an example. The processor 11 and the memory 12 are connected via a bus 10. The memory 12 stores instructions that can be executed by the processor 11, and the instructions are executed by the processor 11 so that the electronic device 1 can execute all or part of the process of the method in the following embodiment, so as to make the multimodal large model better adapt to the needs of specific monitoring scenarios, improve the performance and application effect of the multimodal large model in the monitoring scenario, and thus improve the intelligence level of the monitoring system.

[0080] In one embodiment, the electronic device 1 may be a mobile phone, a tablet computer, a laptop computer, a desktop computer, or a large computing system composed of multiple computers.

[0081] Figure 2 A schematic diagram of an application scenario 200 of a multi-modal large model processing system provided in an embodiment of the present application. Figure 2 As shown, the system includes: a server 210 and a terminal 220, wherein:

[0082] The server 210 may be a data platform that provides multi-modal large model processing services, such as an intelligent monitoring system platform. In actual scenarios, an intelligent monitoring system platform may have multiple servers 210. Figure 2 In the figure, one server 210 is taken as an example.

[0083] The terminal 220 may be a computer, mobile phone, tablet or other device used by the user to log in to the intelligent monitoring system platform. There may be multiple terminals 220. Figure 2 Two terminals 220 are taken as an example for illustration.

[0084] The terminal 220 and the server 210 can transmit information via the Internet, so that the terminal 220 can access the data on the server 210. The terminal 220 and / or the server 210 can be implemented by the electronic device 1.

[0085] The processing scheme of the multimodal large model in the embodiment of the present application can be deployed on the server 210, or on the terminal 220, or partially on the server 210 and partially on the terminal 220. In actual scenarios, it can be selected based on actual needs, and this embodiment does not limit it.

[0086] When the processing solution of the multimodal large model is fully or partially deployed on the server 210 , a calling interface can be opened to the terminal 220 to provide algorithm support to the terminal 220 .

[0087] The method provided in the embodiment of the present application can be implemented by executing the corresponding software code by the electronic device 1, and can be implemented by exchanging data with the server. The electronic device 1 can be a local terminal device. When the method is run on the server, the method can be implemented and executed based on the cloud interaction system, wherein the cloud interaction system includes a server and a client device.

[0088] In a possible implementation, the method provided in the embodiment of the present application provides a graphical user interface through a terminal device, wherein the terminal device can be the local terminal device mentioned above, or can be the client device in the cloud interaction system mentioned above.

[0089] Please see Figure 3 , which is a processing method of a multimodal large model in an embodiment of the present application, the method can be Figure 1 The electronic device 1 shown is used to perform and can be applied to Figure 2 In the monitoring application scenario shown in , the multimodal large model is better adapted to the needs of a specific monitoring scenario, the performance and application effect of the multimodal large model in the monitoring scenario are improved, and the intelligence level of the monitoring system is improved. In this embodiment, the terminal 220 is used as an execution end as an example, and the method includes the following steps:

[0090] Step 301: Acquire a set of original images of a monitoring scene.

[0091] In this step, the original image set can be a plurality of surveillance images or surveillance video clips in the surveillance scene. The specific area covered by the surveillance scene can be set according to actual needs. For example, the area covered by the surveillance scene includes but is not limited to shopping malls, schools, garages, traffic sections, libraries and other areas. The original image can be obtained by collecting videos of the surveillance area through surveillance cameras, or by pulling the structured data returned by the surveillance system to obtain the original image set. For the video collected by the surveillance camera, human body detection can be performed on the video frames, and image frames and video clips containing human bodies can be extracted as the original image set. For the structured data returned by the surveillance system, it is itself in the form of pictures and video clips, so it can be directly used as the original image set.

[0092] like Figure 4 As shown, a schematic diagram of fine-tuning data provided by an embodiment of the present application is provided. The pictures and video clips obtained by the above two methods are used as atomic data, and combined with the characteristics of tasks in the field of intelligent monitoring, training data of types such as single-image text data, multi-image text data, and video text data can be constructed. For details, please refer to the specific description of the following embodiment.

[0093] Step 302: Identify feature information of the target object in the original image set.

[0094] In this step, the original image set includes multiple original images, and the target object refers to the object contained in the original image, such as vehicles, human bodies, triangular cones, chairs and other objects. Feature information is used to characterize the basic characteristics of the target object in the original image. The feature information includes but is not limited to the visual features, position features, category information and unique identification information of the target object. By identifying the feature information of the target object in the original image set, the system can understand and extract the key elements in the scene, and provide targeted auxiliary information for subsequent processing.

[0095] In one embodiment, step 302 may specifically include: determining the object category included in each original image in the original image set. According to the object category, identifying the location information of at least one target object included in the original image. The feature information includes the object category and / or location information.

[0096] In this embodiment, by determining the object category contained in each original image, the system can quickly classify and identify different objects in the image, providing a basis for subsequent processing. By identifying the location information of the target object according to the object category, the system can not only identify the type of object, but also accurately locate the object in the image. For a single original image, its feature information includes but is not limited to the object category and location information contained in the original image, which greatly enriches the description ability of the image data, provides targeted data support for the subsequent question-answer pair generation and fine-tuning training of the multimodal large model, improves the system's recognition ability and response speed in complex monitoring scenarios, and enhances the applicability and reliability of the model in practical applications.

[0097] like Figure 5As shown, a schematic diagram of feature information recognition provided by an embodiment of the present application can be used to obtain category labels for the original image based on a multimodal large model. For example, the prompt word for type label extraction can be: "Give the categories of specific objects involved in the image in the form of a list, without duplication between categories, only output deterministic category names, and give Chinese names and corresponding English names at the same time." For example, the multimodal large model outputs the object category labels contained in the original image as "car, person, traffic cone". Then, open target detection can be performed based on the category label of each original image. For example, models such as GroundingDINO can be used to implement open target detection, and the circumscribed rectangular frame of the target object in the original image is obtained. The rectangular frame is used to represent the location information of the target object. GroundingDINO utilizes multimodal information of vision and language, and achieves more accurate and flexible target detection by combining language prompts with visual features. For example, the target object is detected from the original image and the coordinates of the rectangular box are given: "car [20, 30, 200, 130], car [200, 370, 450, 480], person [400, 500, 480, 628], traffic cone [420, 530, 475, 600]..." In an original image, each type label can include one or more target objects.

[0098] Optionally, you can randomly select a rectangular frame of some target objects from the above recognition results, and use the preset large model to make a detailed description and a brief description. For example, the prompt for the detailed description can be: "You are a visual expert, please describe this image in detail." The prompt for the brief description can be: "Describe this image in one sentence." The brief description and the detailed description are used as part of the feature information of the corresponding target object, providing more detailed and diverse data support for subsequent processing.

[0099] Step 303: Generate question-answer pairs corresponding to the original image set based on the feature information.

[0100] In this embodiment, the question-answer pair refers to a question instruction about the original image detail information and the true value answer corresponding to the question instruction, which is used to train the multimodal large model to learn the ability to understand the image based on the question-answer pair. Generating question-answer pairs based on feature information can enhance the attention of the question-answer pair to the target object, thereby enhancing the attention of the multimodal large model to the local features of the image, and improving the accuracy of the model's information perception and event understanding in the monitoring scene.

[0101] In one embodiment, step 303 may specifically include: generating a first prompt word of a preset large language model according to the feature information, inputting the first prompt word and the original image set into the large language model, so that the large language model generates a question-answer pair about the target object according to the first prompt word.

[0102] In this embodiment, corresponding single image text data can be generated for each original image in the original image set, and the single image text data includes the original image and the question-answer pair data corresponding to the original image. The single image text data can be mainly used for locating the target in the image, understanding the target and understanding the whole image. A part of the original image can be extracted from the original image set to generate single image text data. When generating question-answer pairs, the feature information of the target object in the original image can be injected into the text instruction and / or the corresponding true value answer by text injection.

[0103] Specifically, text injection is to inject the acquired characteristic information of the target object as auxiliary information into the first prompt word or answer of the preset large language model in the form of text. The generated data type based on this preset larger than language model may include positioning data and target description data. The positioning data inquires about the location of the target object based on the description of the target object, that is, the target detection result. For example, the first prompt word can be: "Please output the following paragraph describing the rectangular box of the target object in the image: a description of a certain target." The target description data refers to the description of the specific target object in the rectangular box based on the detection result of the target object. For example, the first prompt word can be: "Please output the description of the target object in the target rectangular box [x0, y0, w0, h0]." The above-mentioned first prompt word is used as a question, and the data generated by the preset large language model according to the first prompt word is used as the true value answer to the question to generate a question and answer pair.

[0104] In this way, by generating the first prompt word of the preset large language model based on the feature information, the pertinence and accuracy of the input prompt are ensured, so that the language model can better understand the key information of the target object in the image. The first prompt word and the original image set are input into the large language model, prompting the model to generate question-answer pairs about the target object by combining visual and language information. Not only does it enrich the diversity of training data, but it also improves the relevance and quality of question-answer pairs, thereby more effectively optimizing the performance of the large language model in fine-tuning training, improving the model's intelligence level when processing multimodal data, and enabling it to more accurately generate question-answer pairs that meet actual needs in complex scenarios, enhancing the practicality and application value of the model.

[0105] In one embodiment, step 303 may specifically include: adding a position mark to the target object in the original image according to the feature information to generate a marked image. Generating a second prompt word of a preset large language model according to the position mark. Inputting the marked image and the second prompt word into the preset large language model, so that the large language model generates a question-answer pair about the target object according to the second prompt word and the marked image.

[0106] In this embodiment, when generating a question-answer pair for a single image, the feature information of the target object in the original image can be used as auxiliary information, and the auxiliary information can be injected into the question or true value answer of the question-answer pair by visual injection. Specifically, a preset large language model can be used to generate a question-answer pair, and the auxiliary information can be injected into the input information of the preset large language model, which can be used to generate scene understanding data containing positioning information. The scene understanding data is an understanding of the entire image. At this time, the auxiliary information is all the target objects detected in the original image. The specific generation process of the question-answer pair can include the following three steps:

[0107] i) For an original image, first calculate the center of the target object based on the rectangular frame of the target object, and mark the center of each target object with a numerical sequence (i.e., position mark), such as Figure 6 As shown, a schematic diagram of a marking image provided in an embodiment of the present application includes: a human body (1), a car (2), a car (3), a traffic cone (4) and a fire hydrant (5).

[0108] Then, the pre-trained multimodal large model can be used to describe the labeled images with digital numbers in detail. For example, the second prompt word generated based on the labeled image can be: "You are a visual expert and receive an image with digital numbers in the center of some human bodies and objects in the image. Please describe the image in detail. Only describe certain things and do not rely entirely on digital numbers."

[0109] ii) Based on the description containing the target object's numerical serial number obtained in i), a large language model can be used to generate free question-answer pairs, and a special placeholder can be embedded in the generated answer. For example, the second prompt can be: "You are a visual expert who is good at generating questions and answers based on image descriptions. Please generate three pairs of question-answer pairs based on the image description containing part of the target serial number. The target numerical serial number is directly referenced in the generated question, and the target serial number is embedded in the generated answer with a special placeholder. <ref> Target Name< / ref> <box> [Target Number]< / box> Do not include numbers, symbols, etc. Figure 6 For example, the following question-answer pairs can be generated (where Q represents the question and A represents the corresponding answer):

[0110] Q1: How many cars are there in the image?

[0111] A1:2 cars.

[0112] Q2: Please give the location of the black car in the image?

[0113] A2: <red>Black car <box> [3] <box>

[0114] Q3: Please describe [1]?

[0115] A3: A man wearing a black short-sleeved T-shirt and sneakers.

[0116] iii) Replace the target sequence number in the question-answer pair with the corresponding target rectangle. Figure 6 Taking the example question-answer pair as an example, after replacing the target sequence number in the question-answer pair with the corresponding target rectangular box, the final question-answer pair is as follows:

[0117] Q2: Please give the location of the black car in the image?

[0118] A2: <red>Black car <box> [[20, 30, 200, 130]] <box>

[0119] Q3: Please describe [400, 500, 480, 628]?

[0120] A3: A man wearing a black short-sleeved T-shirt and sneakers.

[0121] In this way, by combining image labeling and language prompts, the deep fusion of multimodal data and the ability to intelligently generate question-answer pairs are achieved. Specifically, by adding position labels to the target object based on feature information in the original image and generating a labeled image, the interpretability and positioning accuracy of the image data are enhanced, and then the second prompt word of the preset large language model is generated based on the position label to ensure the high correlation and consistency between the language input and the visual information. The labeled image and the second prompt word are input into the preset large language model, so that the model can make full use of the complementarity of visual and language information to generate high-quality question-answer pairs about the target object, improve the accuracy and contextual relevance of the question-answer pair generation, and then provide rich training data for the subsequent training of the multimodal large model, and enhance the adaptability and expressiveness of the multimodal large model in multimodal tasks.

[0122] In one embodiment, the original image set includes a human image set. The feature information includes identification information of a human object in each human image. Based on the feature information, a question-answer pair corresponding to the original image set is generated, including: selecting a first image pair having the same identification information and a similarity less than a first threshold, and a second image pair having different identification information and a similarity greater than a second threshold from the human image set, and marking the real identity labels of the first image pair and the second image pair, and determining the first image pair and the second image pair as a difficult sample pair. A combination of two images in the human image set excluding the difficult sample pair is determined as an easy sample pair, and the identity label of the easy sample pair is marked according to the identification information of each human image in the easy sample pair. According to the preset questions and identity labels, question-answer pairs corresponding to the difficult sample pairs and the easy sample pairs are generated respectively.

[0123] In this embodiment, the fine-tuning data may include multi-image text data, which can be mainly used for tasks such as pedestrian retrieval and pedestrian tracking. The identification information is used to uniquely identify the target object in the image, such as the human ID distributed by the system to the human object. By identifying and screening the identification information in the human image, different human objects in the image can be effectively distinguished and classified. The first image pair with the same identification information but low similarity and the second image pair with different identification information but high similarity are screened out, and their real identity labels are marked to identify the difficult sample pairs. This method can help the multimodal large model better understand and distinguish complex samples and enhance its sensitivity to subtle differences. At the same time, the image combination other than the difficult sample pair is determined as an easy sample pair, and the identity label is marked according to the identification information, ensuring the diversity and representativeness of the sample pair. By generating corresponding question-answer pairs according to preset questions and identity labels, the content and structure of the training data are enriched, so that the subsequent fine-tuned multimodal large model performs better when processing diversified and complex data.

[0124] For example, based on the human images and their human IDs in the original image set returned by the monitoring system, the ReID expert model can be used to perform difficult sample mining and semi-automatic labeling. First, based on the ReID expert model, the features of each human image are extracted, and human image pairs with the same human ID and a similarity less than the first threshold th1 are selected as the first image pair. Human image pairs with different human IDs and a similarity greater than the second threshold th2 are selected as the second image pairs, and the first image pair and the second image pair are used as difficult sample pairs. The first threshold and the second threshold can be selected according to actual needs. For example, the first threshold th1 can be 0.9, and the second threshold th2 can be 0.8. The remaining human images in the original image set except for the difficult sample pairs are combined into easy sample pairs.

[0125] For difficult sample pairs, they can be labeled according to the real identity information of the human object in the image. The hard sample pairs with the same real identity information are labeled with the same identity label truth value, and the hard sample pairs with different real identity information are labeled with different identity label truth values. The labeled hard sample pairs are used as a multi-image data. For easy sample pairs, the human ID returned by the system is determined as the true value of the identity label of the human object in the image. Randomly extract easy sample pairs with the same identity label or different identity labels from the easy sample pairs as a multi-image data. Then construct a question-answer pair. For example, the preset question can be: "Are these given images of the same person?" The true value of the identity label of these given images is used as the true value answer corresponding to the preset question.

[0126] In one embodiment, the original image set includes video data. The feature information includes a video foreground description and a video background description of the video data. Based on the feature information, a question-answer pair corresponding to the original image set is generated, including: generating a third prompt word of the large language model based on the video foreground description, the video background description and preset key information about the video data. The third prompt word and the video data are input into the large language model so that the large language model generates at least one question-answer pair corresponding to the video data according to the third prompt word. Among them, the preset key information includes: one or more of the event time in the video, the event sequence, the motion trajectory of the object, the number of human objects, and the appearance characteristics of the human objects.

[0127] In this embodiment, the original image set may include video data, which may be surveillance video frames in a specific area, and the video data may include information about people's activities in the specific area. For example, surveillance videos in a shopping mall may include activity images of consumers, store sales staff, and other relevant personnel. Video data may be a continuous video stream or a plurality of video frames sampled from surveillance videos with a certain time sequence. For example, video data may be obtained in real time through a monitoring system, or may be obtained from local storage or cloud storage.

[0128] The fine-tuning data for the multimodal large model can also include video text data, which mainly includes video data and its corresponding question-answer pairs, which are used to help the multimodal large model understand the behaviors and relationships of people in the video, character interactions, and scene information.

[0129] The preset key information includes but is not limited to event time, event sequence, object motion trajectory, number of human objects, appearance features of human objects and other information in the video. The preset key information is used to guide the large language model to focus on the specified information when generating question-answer pairs, thereby generating targeted question-answer pairs.

[0130] Optionally, based on the foreground and background descriptions of the video, the large language model can be guided to propose question-answer pairs about information such as event moments, event sequences, motion trajectories, number of characters, and appearance characteristics in the video. This can be divided into the following two steps:

[0131] a). First, obtain the foreground description and background description of the video data. Here, the foreground of the video can be described based on the mainstream multimodal large model, including a detailed description of the clothes and behavior of each person in the video. For example, the prompt words for obtaining the foreground description and background description can be: "You are a video understanding expert. Please describe the content of the video in detail in sequence based on the video timestamp, including but not limited to each person's behavior and clothing in the video, the interaction between people, and the interaction between people and objects, etc." In addition, considering that surveillance cameras are generally fixed, the first frame of the video can be selected as the background description, and the corresponding prompt words can be: "Please describe this image in detail."

[0132] b). Based on the foreground and background descriptions of the video generated in the previous step, guide the large language model to generate question-answer pairs. For example, multiple third prompt words can be generated for questions about key information such as event time (i.e., the time when the event occurred), event sequence, object movement trajectory, number of people, and appearance characteristics of human objects. For each video, three third prompt words are randomly selected to guide the large language model to generate question-answer pairs.

[0133] Optionally, the third prompt word can be divided into two parts: a prefix and a topic. For each third prompt word, the prefix part can be the same, for example: "You are a video understanding expert. Please generate three question-answer pairs based on the foreground description and background description of the video. Foreground description: xxxx (here is the foreground description of the video data generated in step a), background description: xxx (here is the background description of the video data generated in step a)". The topic parts can be as follows:

[0134] •Please pay attention to the content of the events in the video and the moments when they occurred to generate question-answer pairs.

[0135] •Please focus on the content and order of events in the video to generate question-answer pairs.

[0136] •Please pay attention to the movement trajectory of each human body in the video and generate question-answer pairs.

[0137] •Please pay attention to the number of characters in the video and their changes to generate question-answer pairs.

[0138] •Please pay attention to the clothes and postures of the people in the video and generate question-answer pairs.

[0139] In this embodiment, by extracting the foreground and background descriptions in the video data, the system can fully capture the dynamic and static information in the video and form a deep understanding of the video content. A third prompt word is generated based on these descriptions and preset key information to ensure that the third prompt word can accurately reflect the key elements and logical relationships specified in the video, and then the third prompt word and the video data are input into the large language model so that it can generate question-answer pairs related to the preset key information in the video data based on multimodal information. In this way, the pertinence and contextual consistency of the question-answer pairs are improved, and the accuracy and practicality of the large language model in video analysis and question-answer pair generation are improved, so that it can be more effectively applied to a variety of practical scenarios, such as video surveillance, content analysis, and intelligent question-answering systems.

[0140] Step 304: Use the original image set and the corresponding question-answer pairs as training data to fine-tune the pre-trained multimodal large model to obtain a target multimodal large model adapted to the monitoring scenario.

[0141] In this step, the original image set and the corresponding question-answer pairs are used as multimodal training data, enriching the diversity and pertinence of the training data. By using the original image set and question-answer pairs for fine-tuning training, the pre-trained multimodal large model is optimized, making the multimodal large model better adapted to the needs of specific monitoring scenarios, improving the performance and application effect of the multimodal large model in monitoring scenarios, and thus improving the intelligence level of the monitoring system.

[0142] In one embodiment, the multimodal large model includes a visual encoder, a visual text feature mapper and a large language model. Among them: the visual encoder is used to extract the original visual features of the original image set. The visual text mapper is used to map the original visual features to a feature format that is compatible with the large language model to obtain the converted target visual features. The large language model is used to generate a predicted answer based on the target visual features and the question instructions in the question and answer pair. Step 304 may specifically include: in the fine-tuning training process, the difference information between the true answer and the predicted answer in the constrained question and answer pair is targeted.

[0143] In this embodiment, the multimodal large model may include a visual encoder, a visual text feature mapper, and a large language model. The visual encoder is responsible for extracting the visual features of the original image set. The visual features here include but are not limited to the color information, texture information, shape information, edge features, pixel gradient features, spatial features and other features of the original image, ensuring that the multimodal large model can capture the visual details and key information in the original image. The visual text feature mapper converts these original visual features into a format compatible with the large language model to ensure that the large language model can accurately understand the multimodal information. The large language model uses the converted target visual features and the question instructions in the question-answer pair to generate a predicted answer. A question-answer pair includes a question and the true answer corresponding to the question. During the fine-tuning training process, the question in the question-answer pair is used as a text instruction (i.e., question instruction) to guide the large model to output a predicted answer, with the goal of constraining the difference information between the true answer and the predicted answer in the question-answer pair, ensuring that the model continuously optimizes the accuracy and consistency of its output during the training process, so that the target multimodal large model after fine-tuning training can gradually adapt to the needs of specific monitoring scenarios, which not only improves the adaptability of the multimodal large model to complex scenarios, but also enhances its reliability and effectiveness in practical applications.

[0144] In one embodiment, the multimodal large model includes a visual encoder, a local encoder, a visual text feature mapper and a large language model. The feature information includes the position information of the target object in the original image set. Among them: the visual encoder is used to extract the original visual features of the original image set. The local encoder is used to extract the local features of the target object from the original visual features according to the position information of the target object, and splice the local features with the original visual features. The visual text mapper is used to map the spliced ​​visual features to a feature format adapted to the large language model to obtain the converted target visual features. The large language model is used to generate a predicted answer based on the target visual features and the question instructions in the question and answer pair. Step 304 may specifically include: in the fine-tuning training process, the difference information between the true answer and the predicted answer in the constrained question and answer pair is taken as the target.

[0145] In this embodiment, compared with the aforementioned embodiment, the multimodal large model may further include a local encoder. The visual encoder is responsible for extracting the overall visual features of the original image set to ensure that the model has a comprehensive perception of global information. The local encoder uses the position information of the target object to extract the local features of the target object from the original visual features, such as extracting the local features of the target object from the shallow information output by the visual encoder, and splicing it with the deep information in the global visual features. The model can enhance the attention to detail capture and local information of the target object, so that the model can understand the key elements in the image more accurately. The visual text feature mapper converts the spliced ​​visual features into a format compatible with the large language model to ensure the effective integration and transmission of multimodal information. The large language model generates a predicted answer based on the converted target visual features and the question instructions in the question-answer pair, realizing the intelligent conversion from vision to language. During the fine-tuning training process, the goal is to constrain the difference information between the true answer and the predicted answer in the question-answer pair, ensuring that the multimodal large model continuously optimizes the accuracy and consistency of its output during the training process, so that the multimodal large model can more accurately understand and process the multimodal data in the monitoring scenarios, provide more intelligent and efficient solutions, and meet diverse practical application needs.

[0146] Optionally, in actual scenarios, model training for a large multimodal model may include three stages: the first stage is the pre-training stage for image-text modality alignment, which can be trained using a public pre-training data set. The second stage is the stage for improving single-image understanding and positioning capabilities, which can also be trained using public image understanding question-answer pairs and target detection data. The third stage is the instruction fine-tuning stage, which can perform model fine-tuning training based on the single-image text data, multi-image text data, and video text data obtained in any of the above embodiments of the present application.

[0147] like Figure 7 As shown, it is a schematic diagram of the model structure of a multimodal large model provided in an embodiment of the present application, which mainly includes four parts: a visual encoder, a local encoder, a visual text feature mapper and a large language model, wherein:

[0148] The visual encoder can be implemented using the SigLIP model or the CLIP model, or other types of models to obtain the visual signal X V The local encoder is based on the single-image target box extracted offline (i.e., the rectangular box of the target object in a single original image), and the output token Z of the third-layer Transformer of the visual encoder is obtained. v0 Extract the token Z corresponding to the target object v1 , token Z v0 The shallow features of the original image are included in the image. The shallow features are the direct reflection of the basic elements of the original image. They can retain more fine-grained information and provide an accurate data basis for the local feature extraction of the target object. For example, the coordinates of the single-image target box are: car [20,30,200,130], person [400,500,480,628]..., and the local encoding can be based on the single-image target box, from the output token Z of the third layer Transformer of the visual encoder v0 Extract the token Z of the car v1 and the character's token Z v1 .

[0149] For a single original image, the token Z of the target object can be v1 It is concatenated with the output of the penultimate layer Transformer of the visual encoder, and the concatenated visual feature Z v As the final token expression of a single original image. Among them, the output of the penultimate layer Transformer of the visual encoder is the deep features of the original image, which can represent the overall structure and content of the original image, understand the object relationship and overall semantics, and are the key to image recognition and classification tasks. The spliced ​​visual features can strengthen the local target object. For example, a car may be missed when describing the model before splicing, but it can be described after splicing, improving accuracy.

[0150] For the video data in the original image set, all video frames can be processed, or some video frames in the video data can be extracted, such as 8 frames or 16 frames from each video. The rectangular frame of the target object in each frame of the video image is extracted through a local encoder, and the rectangular frames extracted from different frames of the same video data are clustered. The rectangular frames of the same target object are clustered into one category, and then spliced ​​with the output of the visual encoder to obtain the spliced ​​visual feature Zv as the final token expression of the video data.

[0151] The visual-text feature mapper can be implemented using a two-layer MLP to map the concatenated visual feature token Zv to the target visual feature token H that the large language model can accept. V .

[0152] The large language model can be implemented using Qwen2 or other large language models, which is not limited in this embodiment. The question in the question-answer pair is used as the text instruction X q , extract text instruction X q Text token H q ,

[0153] The target visual feature token H V With text token H q Input the large language model together, guide the large language model to output the predicted answer. In this embodiment, the predicted answer is Figure 7 In the fine-tuning training process, the text instruction X in the question-answer pair can be minimized q The difference between the corresponding true answer and the predicted answer Xa is used as the optimization target, so that the multimodal large model can continuously optimize the accuracy and consistency of its output during the training process, so that the multimodal large model can more accurately understand and process the multimodal data in the monitoring scenario and meet the diverse practical application needs.

[0154] Optionally, for a text instruction X of length L q , the large language model can use the following formula to calculate the predicted answer X a The probability P is:

[0155]

[0156] Among them, X V is the visual signal, x i It is the token of the predicted answer to be output in round i, such as the complete predicted answer X a "This describes an outdoor square with two cars and a pedestrian. There are buildings around the square." The large language model outputs each character or word in turn, x i It is the token output in round i. (q,<i) is the text instruction used for prediction in each round before round i, X (a,<i) It is the predicted answer token output in each round before the i-th round.

[0157] In one embodiment, the method further includes: in response to a human body comparison instruction, obtaining at least two images to be processed. Inputting at least two images to be processed into a target multimodal large model so that the target multimodal large model outputs whether the human body objects in at least two images to be processed are the same person. Determine the comparison accuracy of the target multimodal large model based on the output result of the target multimodal large model and / or the manual review result. If the comparison accuracy is less than a first preset threshold, add a supplementary image and a question-answer pair of the supplementary image to the training data to generate new training data, and iteratively train the target multimodal large model based on the new training data.

[0158] In this embodiment, in view of the demand of the multimodal large model for the human body comparison task, the output results of the multimodal large model can be evaluated, the training data can be dynamically adjusted and the model can be iteratively optimized to improve the accuracy and reliability of the multimodal large model in the human body detection task. Specifically, in response to the human body comparison instruction, the system obtains at least two images to be processed and inputs them into the target multimodal large model, making full use of the multimodal analysis capability of the multimodal large model, combining visual and language information to determine whether the human body objects in these images to be processed are the same person. According to the output results of the multimodal large model, or in combination with the results of manual review and confirmation, the comparison accuracy of the model is evaluated, and the performance of the model in practical applications is monitored. The human body comparison instruction can be actively triggered by the user, or it can be automatically triggered periodically by the system. For example, the inspector can actively trigger multiple human body detection tasks, collect the output results of the multimodal large model, or in the multimodal large model application, collect the relevant input data and output results of the human body detection task reported by the user, and count the comparison accuracy based on multiple output results. For example, the number of correct detections in the output results can be counted, and the ratio of the number of detections with correct output results to the total number of detections can be used as the comparison accuracy. When the comparison accuracy is lower than the first preset threshold, the system will add supplementary images and their corresponding question-answer pairs to the training data to generate new training data. The first threshold can be set according to actual needs. For example, the first threshold can be set to 90%. If the comparison accuracy of the human body detection task is less than 90%, it means that the human body comparison accuracy of the multimodal large model is not accurate enough. You can add single-image text data and multi-image text data to the original training data, and iteratively train the multimodal large model based on the new training data. This dynamic training data enhancement strategy ensures that the model can be exposed to more diverse and challenging samples. By iteratively training the target multimodal large model, the system continuously optimizes the model parameters to make it more accurate and stable in human body detection tasks. Through closed-loop feedback and continuous optimization, this solution enhances the adaptability and practicality of the model in complex scenarios, ensuring that it can provide efficient and accurate detection results in practical applications.

[0159] In one embodiment, the method further includes: in response to a behavior recognition instruction, obtaining a surveillance video to be processed. Inputting the surveillance video to be processed into the target multimodal large model, so that the target multimodal large model outputs the specified behavior category present in the surveillance video to be processed according to the behavior detection prompt word. Determine the recognition accuracy of the target multimodal large model based on the output result of the target multimodal large model and / or the manual review result. If the recognition accuracy is less than a second preset threshold, add a supplementary video and a question-answer pair of the supplementary video to the training data to generate new training data, and iteratively train the target multimodal large model based on the new training data.

[0160] In this embodiment, in response to the needs of the multimodal large model for the action warning task, the accuracy and robustness of the multimodal large model in the behavior detection task can be improved by evaluating the output results of the multimodal large model, dynamic data enhancement and iterative training. The recognition accuracy may include the precision (P) and / or recall (R) of the multimodal large model on the action warning. In the actual scenario, the second threshold may include the threshold of the precision (P) and the threshold of the recall (R), so as to compare the thresholds for the precision (P) and the recall (R), respectively, and determine the training data that needs to be supplemented according to the threshold comparison results. Specifically, in response to the behavior recognition instruction, the system obtains the surveillance video to be processed and inputs it into the target multimodal large model. The model uses the behavior detection prompt word to analyze the video content and output the specified behavior category present therein. Here, the behavior recognition instruction can be actively triggered by the user or automatically triggered periodically by the system. For example, inspectors can actively trigger multiple behavior detection tasks and collect the output results of the multimodal large model, or in the multimodal large model application, collect the relevant input data and output results of the behavior detection tasks reported by users, and statistically analyze the recognition accuracy based on multiple output results combined with the results of manual review. The designated behavior category is a pre-set action category with potential safety hazards related to the monitoring scene. Taking the monitoring scene of a shopping mall or supermarket as an example, the designated behavior types include but are not limited to picking up goods from shelves, putting goods on shelves, smoking, eating, passing bags, playing with mobile phones, etc. By evaluating the output results of the model for the behavior detection task, the system can monitor its accuracy performance in the behavior detection task.

[0161] For example, in a supermarket monitoring scenario, if the supermarket monitoring staff finds that the system's action recognition is incorrect, they can report the relevant pending videos. For a pending video reported by the user, the manual inspection results of the pending video content can be obtained, and then it can be determined whether the manual inspection results are the same as the action type results output by the system. If the manual inspection results show that there is no action of the specified behavior type in the pending video, or there is an action of the specified behavior type but it is different from the action type output by the system, it is recorded as a false alarm. If some actions of the specified behavior type in the pending video are in the manual inspection results but are not detected by the system, matching target action types can be selected for these specified behavior types in the candidate action type list as missed data of the target action type.

[0162] Optionally, taking the recognition accuracy determination process of action warning as an example, the recognition accuracy here can be characterized by the precision (P) and recall (R) of the action warning. The precision (P) and recall (R) of the first type of action warning are defined as follows:

[0163] P = the number of correct actions reported by the system as Class I actions / the total number of Class I actions reported by the system.

[0164] R = the number of correct actions reported by the system as Class I actions / the total number of all true Class I actions in the manual inspection results.

[0165] Assume that a total of 5 action types of specified behavior types are defined. For the first type of action (smoking), a total of 10 videos were reviewed, of which 2 were false positives. When reviewing actions of other specified behavior types (such as eating), a video of eating was found to have a smoking action, and the video was marked as a false negative data of the smoking action type.

[0166] The recognition accuracy can be compared with the second threshold periodically or in real time. When the recognition accuracy is lower than the second preset threshold, the system will add supplementary videos and their corresponding question-answer pairs to the training data to generate new training data. The second threshold can be set according to actual needs. For example, the second threshold can be set to 90%. This dynamic adjustment strategy ensures that the model can be exposed to more diverse and challenging behavior samples and improves its ability to recognize different behavior patterns. By iteratively training the target multimodal large model, the system continuously optimizes the model parameters to make it more accurate and stable in behavior detection tasks. Through closed-loop feedback and continuous optimization, this solution enhances the adaptability and practicality of the model in complex monitoring scenarios, ensuring that it can provide efficient and accurate behavior detection results in practical applications.

[0167] like Figure 8 As shown, a schematic diagram of a model training process provided in an embodiment of the present application includes:

[0168] Fine-tune data generation, model fine-tuning, evaluation feedback, and evaluate whether the model output meets the preset indicators (such as the first threshold and the second threshold mentioned above). If the indicators are met, the training can be terminated. If the indicators are not met, relevant training data is supplemented for model iterative training.

[0169] The above-mentioned processing method of the multimodal large model proposes a method for establishing a fine-tuning data set in an intelligent monitoring scenario and a multimodal large model structure that strengthens local target features, and establishes an evaluation feedback and model iteration mechanism for the application of the multimodal large model in the monitoring scenario, which improves the accuracy of the multimodal large model in information perception and event understanding in the monitoring scenario, can achieve better performance in actual monitoring scenarios, and improve the implementation effect of the multimodal large model in the monitoring scenario.

[0170] Please see Fig. 9 , which is a behavior information identification method of an embodiment of the present application, the method can be Figure 1 The electronic device 1 shown is used to perform and can be applied to Figure 2 Compared with the above-mentioned embodiment, this embodiment takes the shopping mall monitoring scene as an example, and the method includes the following steps:

[0171] Step 901: In response to the identification instruction, obtain the surveillance video of the shopping mall.

[0172] Step 902: Input the surveillance video into the target multimodal large model, so that the target multimodal large model outputs the recognition result of the surveillance video under the guidance of the preset prompt words. The target multimodal large model is obtained according to the method of any of the above embodiments.

[0173] For details of each step of the above method, please refer to the relevant description of the above embodiment, which will not be repeated here.

[0174] Please see Fig.10 , which is a processing device 1000 of a multimodal large model according to an embodiment of the present application, and the device can be applied to Figure 1 The electronic device 1 shown can be applied to Figure 2 The device includes: an acquisition module 1001, an identification module 1002, a generation module 1003 and a fine-tuning module 1004, and the functional principles of each module are as follows:

[0175] The acquisition module 1001 is used to acquire a set of original images of a monitoring scene.

[0176] The recognition module 1002 is used to recognize feature information of a target object in an original image set.

[0177] The generation module 1003 is used to generate question-answer pairs corresponding to the original image set based on the feature information.

[0178] The fine-tuning module 1004 is used to use the original image set and the corresponding question-answer pairs as training data to fine-tune the pre-trained multimodal large model to obtain a target multimodal large model adapted to the monitoring scenario.

[0179] In one embodiment, the recognition module 1002 is used to determine the object category included in each original image in the original image set. According to the object category, the position information of at least one target object included in the original image is recognized. The feature information includes the object category and / or the position information.

[0180] In one embodiment, the generating module 1003 is used to generate a first prompt word of a preset large language model according to the feature information, and input the first prompt word and the original image set into the large language model so that the large language model generates a question-answer pair about the target object according to the first prompt word.

[0181] In one embodiment, the generating module 1003 is used to add a position mark to the target object in the original image according to the feature information to generate a marked image. A second prompt word of a preset large language model is generated according to the position mark. The marked image and the second prompt word are input into the preset large language model, so that the large language model generates a question-answer pair about the target object according to the second prompt word and the marked image.

[0182] In one embodiment, the original image set includes a human image set. The feature information includes identification information of a human object in each human image. A generation module 1003 is used to screen out a first image pair having the same identification information and a similarity less than a first threshold and a second image pair having different identification information and a similarity greater than a second threshold from the human image set, and mark the real identity labels of the first image pair and the second image pair, and determine the first image pair and the second image pair as difficult sample pairs. A combination of two images in the human image set excluding the difficult sample pairs is determined as an easy sample pair, and the identity label of the easy sample pair is marked according to the identification information of each human image in the easy sample pair. According to preset questions and identity labels, question-answer pairs corresponding to the difficult sample pairs and the easy sample pairs are generated respectively.

[0183] In one embodiment, the original image set includes video data. The feature information includes a video foreground description and a video background description of the video data. The generation module 1003 is used to generate a third prompt word of the large language model according to the video foreground description, the video background description and preset key information about the video data. The third prompt word and the video data are input into the large language model so that the large language model generates at least one question-answer pair corresponding to the video data according to the third prompt word. Among them, the preset key information includes: one or more of the event time in the video, the event sequence, the movement trajectory of the object, the number of human objects, and the appearance characteristics of the human objects.

[0184] In one embodiment, the multimodal large model includes a visual encoder, a visual text feature mapper and a large language model. Among them: the visual encoder is used to extract the original visual features of the original image set. The visual text mapper is used to map the original visual features to a feature format that is compatible with the large language model to obtain the converted target visual features. The large language model is used to generate a predicted answer based on the target visual features and the question instructions in the question and answer pair. The fine-tuning module 1004 is used to target the difference information between the true answer and the predicted answer in the constrained question and answer pair during the fine-tuning training process.

[0185] In one embodiment, the multimodal large model includes a visual encoder, a local encoder, a visual text feature mapper and a large language model. The feature information includes the position information of the target object in the original image set. Wherein: the visual encoder is used to extract the original visual features of the original image set. The local encoder is used to extract the local features of the target object from the original visual features according to the position information of the target object, and splice the local features with the original visual features. The visual text mapper is used to map the spliced ​​visual features to a feature format adapted to the large language model to obtain the converted target visual features. The large language model is used to generate a predicted answer based on the target visual features and the question instructions in the question-answer pair.

[0186] In one embodiment, the fine-tuning module 1004 is used to target the difference information between the true answer and the predicted answer in the constrained question-answer pair during the fine-tuning training process.

[0187] In one embodiment, the device further includes: a first iteration module, which is used to obtain at least two images to be processed in response to a human body comparison instruction. The at least two images to be processed are input into the target multimodal large model so that the target multimodal large model outputs whether the human body objects in at least two images to be processed are the same person. The comparison accuracy of the target multimodal large model is determined according to the output result of the target multimodal large model. If the comparison accuracy is less than a first preset threshold, a supplementary image and a question-answer pair of the supplementary image are added to the training data to generate new training data, and the target multimodal large model is iteratively trained according to the new training data.

[0188] In one embodiment, the device further includes: a second iteration module, which is used to obtain the surveillance video to be processed in response to the behavior recognition instruction. The surveillance video to be processed is input into the target multimodal large model, so that the target multimodal large model outputs the specified behavior category present in the surveillance video to be processed according to the behavior detection prompt word. The recognition accuracy of the target multimodal large model is determined according to the output result of the target multimodal large model. If the recognition accuracy is less than the second preset threshold, the supplementary video and the question-answer pair of the supplementary video are added to the training data to generate new training data, and the target multimodal large model is iteratively trained according to the new training data.

[0189] For a detailed description of the processing device 1000 for the multimodal large model, please refer to the description of the relevant method steps in the above embodiment. The implementation principle and technical effects are similar and will not be repeated in this embodiment.

[0190] Fig.11 The following is a schematic diagram of a cloud device 110 provided in an exemplary embodiment of the present application. The cloud device 110 can be used to run the method provided in any of the above embodiments. Fig.11 As shown, the cloud device 110 may include: a memory 1104 and at least one processor 1105, Fig.11 A processor is taken as an example.

[0191] The memory 1104 is used to store computer programs and can be configured to store various other data to support operations on the cloud device 110. The memory 1104 can be an object storage service (OSS).

[0192] The memory 1104 may be implemented by any type of volatile or nonvolatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0193] The processor 1105 is coupled to the memory 1104 and is used to execute the computer program in the memory 1104 to implement the solution provided by any of the above method embodiments. The specific functions and technical effects that can be achieved are not repeated here.

[0194] Furthermore, if Fig.11 The cloud device also includes: a firewall 1101, a load balancer 1102, a communication component 1106, a power supply component 1103 and other components. Fig.11 Only some components are shown schematically, which does not mean that the cloud device only includes Fig.11 Components shown.

[0195] In one embodiment, the above Fig.11 The communication component 1106 in the communication component 1106 is configured to facilitate wired or wireless communication between the device where the communication component 1106 is located and other devices. The device where the communication component 1106 is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, LTE (Long Term Evolution, Long Term Evolution, referred to as LTE), 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component 1106 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1106 also includes a near field communication (Near Field Communication, referred to as NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (Radio Frequency Identification, referred to as RFID) technology, Infrared Data Association (Infrared Data Association, referred to as IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (Bluetooth, referred to as BT) technology and other technologies.

[0196] In one embodiment, the above Fig.11 The power supply component 1103 provides power to various components of the device where the power supply component 1103 is located. The power supply component 1103 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply component is located.

[0197] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method of any of the aforementioned embodiments is implemented.

[0198] An embodiment of the present application also provides a computer program product, including a computer program, which implements the method of any of the aforementioned embodiments when executed by a processor.

[0199] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0200] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0201] It should be understood that the above-mentioned processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The memory may include a high-speed RAM (Random Access Memory) memory, and may also include non-volatile storage NVM (Nonvolatile memory, NVM for short), such as at least one disk storage, and can also be a USB flash drive, a mobile hard disk, a read-only memory, a disk or an optical disk, etc.

[0202] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0203] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0204] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, clothing or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, clothing or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, clothing or device including the element.

[0205] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0206] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0207] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of user data and other information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0208] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.< / box> < / box> < / red> < / box> < / box> < / red>

Claims

1. A method for processing a multimodal large model, characterized in that: include: Obtaining a set of original images of the monitoring scene; Identify feature information of a target object in the original image set; Generating a question-answer pair corresponding to the original image set according to the feature information; The original image set and the corresponding question-answer pairs are used as training data to fine-tune the pre-trained multimodal large model to obtain a target multimodal large model adapted to the monitoring scene.

2. The method according to claim 1, characterized in that The identifying feature information of the target object in the original image set includes: Determine the object category included in each original image in the original image set; According to the object category, identifying position information of at least one target object included in the original image; The feature information includes the object category and / or the location information.

3. The method according to claim 1, characterized in that: Generating a question-answer pair corresponding to the original image set according to the feature information includes: Generate a first prompt word of a preset large language model according to the feature information; The first prompt word and the original image set are input into the large language model, so that the large language model generates a question-answer pair about the target object according to the first prompt word.

4. The method according to claim 1, characterized in that: Generating a question-answer pair corresponding to the original image set according to the feature information includes: Adding a position mark to the target object in the original image according to the feature information to generate a marked image; Generate a second prompt word of a preset large language model according to the position mark; The labeled image and the second prompt word are input into the preset large language model, so that the large language model generates a question-answer pair about the target object according to the second prompt word and the labeled image.

5. The method according to claim 1, characterized in that The original image set includes a human body image set; the feature information includes identification information of a human body object in each of the human body images; and generating a question-answer pair corresponding to the original image set according to the feature information includes: Screening out a first image pair having the same identification information and a similarity less than a first threshold and a second image pair having different identification information and a similarity greater than a second threshold from the human image set, marking the real identity labels of the first image pair and the second image pair, and determining the first image pair and the second image pair as a difficult sample pair; Determine the combinations of any two images in the human image set excluding the difficult sample pairs as easy sample pairs, and annotate the identity labels of the easy sample pairs according to the identification information of each human image in the easy sample pairs; According to the preset question and the identity label, question-answer pairs corresponding to the difficult sample pair and the easy sample pair are generated respectively.

6. The method according to claim 1, characterized in that The original image set includes video data; the feature information includes a video foreground description and a video background description of the video data; and generating a question-answer pair corresponding to the original image set according to the feature information includes: generating a third prompt word of a large language model according to the video foreground description, the video background description and preset key information about the video data; Inputting the third prompt word and the video data into the large language model, so that the large language model generates at least one question-answer pair corresponding to the video data according to the third prompt word; The preset key information includes: one or more of event time, event sequence, object motion trajectory, number of human objects, and appearance features of human objects in the video.

7. The method according to claim 1, characterized in that The multimodal large model includes a visual encoder, a visual text feature mapper and a large language model; wherein: The visual encoder is used to extract original visual features of the original image set; The visual text mapper is used to map the original visual features to a feature format adapted to the large language model to obtain a converted target visual feature; The large language model is used to generate a predicted answer based on the target visual feature and the question instruction in the question-answer pair; The method uses the original image set and the corresponding question-answer pairs as training data, performs fine-tuning training on the pre-trained multimodal large model, and obtains a target multimodal large model adapted to the monitoring scene, including: During the fine-tuning training process, the goal is to constrain the difference information between the true answer in the question-answer pair and the predicted answer.

8. The method according to claim 1, characterized in that The multimodal large model includes a visual encoder, a local encoder, a visual text feature mapper and a large language model; the feature information includes the position information of the target object in the original image set; wherein: The visual encoder is used to extract original visual features of the original image set; The local encoder is used to extract the local features of the target object from the original visual features according to the position information of the target object, and to concatenate the local features with the original visual features; The visual text mapper is used to map the concatenated visual features to a feature format adapted to the large language model to obtain the converted target visual features; The large language model is used to generate a predicted answer based on the target visual feature and the question instruction in the question-answer pair; The method uses the original image set and the corresponding question-answer pairs as training data, performs fine-tuning training on the pre-trained multimodal large model, and obtains a target multimodal large model adapted to the monitoring scene, including: During the fine-tuning training process, the goal is to constrain the difference information between the true answer and the predicted answer in the question-answer pair.

9. The method according to claim 1, characterized in that: Also includes: In response to a human body comparison instruction, acquiring at least two images to be processed; Inputting the at least two images to be processed into the target multimodal large model, so that the target multimodal large model outputs whether the human objects in the at least two images to be processed are the same person; Determining the comparison accuracy of the target multimodal large model according to the output result of the target multimodal large model; If the comparison accuracy is less than a first preset threshold, a supplementary image and a question-answer pair of the supplementary image are added to the training data to generate new training data, and the target multimodal large model is iteratively trained according to the new training data.

10. The method according to claim 1, characterized in that Also includes: In response to the behavior recognition instruction, obtaining the surveillance video to be processed; Inputting the surveillance video to be processed into the target multimodal large model, so that the target multimodal large model outputs the specified behavior category present in the surveillance video to be processed according to the behavior detection prompt word; Determining the recognition accuracy of the target multimodal large model according to the output result of the target multimodal large model; If the recognition accuracy is less than a second preset threshold, a supplementary video and a question-answer pair of the supplementary video are added to the training data to generate new training data, and the target multimodal large model is iteratively trained based on the new training data.

11. A behavior information recognition method, characterized in that: include: In response to the identification instruction, acquiring the surveillance video of the shopping mall; Inputting the surveillance video into the target multimodal large model, so that the target multimodal large model outputs a recognition result about the surveillance video under the guidance of a preset prompt word; Wherein, the target multimodal large model is obtained according to the method described in any one of claims 1-10.

12. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the electronic device to perform the method described in any one of claims 1-11.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 11 is implemented.

14. A computer program product, characterized in that The method comprises a computer program, which implements the method according to any one of claims 1 to 11 when being executed by a processor.

Images