Abnormal event identification method and device based on multi-modal large model, and storage medium

Abstract

The invention discloses an abnormal event identification method and device based on a multi-modal large model and a storage medium, and belongs to the technical field of artificial intelligence, and the method comprises the steps: constructing a governance event library and various prompt templates corresponding to a city governance scene; the method comprises the following steps: acquiring a to-be-processed city scene image and mode selection information in a city governance process; generating a target prompt word according to the mode selection information, the governance event library and the prompt template; and event identification of the city scene image is performed according to the target prompt word and the multi-modal large model, so that abnormal problems occurring in the city governance process can be rapidly and accurately identified, the phenomena of false detection and missing detection are avoided, the method is suitable for various city governance scenes, model training and comparison image library setting are not needed, and the efficiency is improved. The method has good generalization ability.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and in particular, relates to an abnormal event recognition method, device and storage medium based on a multimodal large model. Background Art

[0002] With the development of globalization and industrialization, the urbanization process is accelerating. Rapid urbanization has brought a series of challenges, such as drying clothes on the street, occupying the street for business, overflowing garbage cans, floating objects on the water surface and other violations in urban governance. These problems not only affect the appearance of the city, but also restrict the sustainable development of the city.

[0003] At present, in the process of urban governance, we mainly rely on manual guard duty or manual monitoring of video surveillance to identify whether there are any violations, and notify the relevant departments in a timely manner to deal with the violations when violations are found. However, this method of relying on manual identification not only requires a certain number of grassroots staff, resulting in high labor costs and low identification efficiency, but also easily leads to missed detection and false detection of violations, resulting in poor identification effect. Summary of the invention

[0004] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes an abnormal event recognition method, device and storage medium based on a multimodal large model, which can quickly and accurately identify abnormal problems occurring in the process of urban governance, and has a good recognition effect.

[0005] In a first aspect, the present application provides a method for identifying abnormal events based on a multimodal large model, comprising:

[0006] Construct a governance event library and multiple prompt templates corresponding to urban governance scenarios. Each governance event in the governance event library includes an event category and event description content. Different prompt templates correspond to different recognition modes.

[0007] In the process of urban governance, obtain the urban scene images and mode selection information to be processed;

[0008] Generate a target prompt word according to the mode selection information, the governance event library and the prompt template;

[0009] Abnormal events in the urban scene image are identified according to the target prompt words and the multimodal large model.

[0010] In some embodiments, the prompt template includes an event option slot, and generating a target prompt word according to the mode selection information, the governance event library and the prompt template includes:

[0011] Determine a target prompt template corresponding to the mode selection information from the prompt templates;

[0012] Determine, according to the governance event library, a first filling content corresponding to the event option slot of the target prompt template;

[0013] A target prompt word is generated according to the first filling content and the target prompt template.

[0014] In some embodiments, determining a target prompt template corresponding to the mode selection information from the prompt template includes:

[0015] When the mode selection information indicates fast identification, the first prompt template is used as the target prompt template, and the first prompt template is used for direct matching prompt;

[0016] When the mode selection information indicates accurate recognition, the second prompt template is used as the target prompt template, and the second prompt template is used for multi-stage matching prompt;

[0017] When the mode selection information indicates verification identification, the third prompt template is used as the target prompt template, and the third prompt template is used for judgment prompting.

[0018] In some embodiments, determining the first filling content corresponding to the event option slot of the target prompt template according to the governance event library includes:

[0019] When the target prompt template is used for direct matching prompt or multi-stage matching prompt, the governance event library is used as the first filling content corresponding to the event option slot of the target prompt template;

[0020] When the target prompt template is used for judgment prompting, a standard event is determined from the governance event library, and the standard event is used as the first filling content corresponding to the event option slot of the target prompt template.

[0021] In some embodiments, when the target prompt template is used for multi-stage matching prompts, the target prompt template further includes an image description slot, the target prompt words include first-stage prompt words and second-stage prompt words, and generating target prompt words according to the first filling content and the target prompt template includes:

[0022] generating the first-stage prompt words, where the first-stage prompt words are used to instruct image description;

[0023] Inputting the first-stage prompt word and the city scene image into the multimodal large model to generate image description information corresponding to the city scene image, and using the image description information as second filling content corresponding to the image description slot;

[0024] The second-stage prompt word is generated according to the first filling content, the second filling content, the image description slot and the event option slot.

[0025] In some embodiments, before identifying abnormal events in the urban scene image according to the target prompt word and the multimodal large model, the method further includes:

[0026] Obtaining a number of sample images corresponding to each of the governance events, and constructing sample image description information corresponding to each of the sample images;

[0027] Constructing a fine-tuning prompt instruction set according to the prompt template, the governance event and the sample image description information;

[0028] The multimodal large model is fine-tuned according to the fine-tuning prompt instruction set, the sample image and the governance event.

[0029] In some embodiments, the fine-tuning prompt instruction set includes a first fine-tuning prompt instruction, a second fine-tuning prompt instruction, and a third fine-tuning prompt instruction, and constructing the fine-tuning prompt instruction set according to the prompt template, the governance event, and the sample image description information includes:

[0030] Determine a plurality of candidate event groups based on all of the governance events;

[0031] Constructing a corresponding first fine-tuning prompt instruction according to each candidate event group and the first prompt template, wherein each candidate event group corresponds to one first fine-tuning prompt instruction, and the first prompt template is used for performing a direct matching prompt;

[0032] According to each of the candidate event groups, the sample image description information and the second prompt template, a corresponding second fine-tuning prompt instruction is constructed, each of the candidate event groups corresponds to one second fine-tuning prompt instruction, and the second prompt template is used for performing multi-stage matching prompts;

[0033] According to each of the governance events and the third prompt template, a corresponding third fine-tuning prompt instruction is constructed, each of the governance events corresponds to one third fine-tuning prompt instruction, and the third prompt template is used for making judgment prompts.

[0034] In a second aspect, the present application provides an abnormal event recognition device based on a multimodal large model, comprising:

[0035] A construction unit, used to construct a governance event library and multiple prompt templates corresponding to urban governance scenarios, wherein each governance event in the governance event library includes an event category and event description content, and different prompt templates correspond to different recognition modes;

[0036] An acquisition unit, used to acquire the urban scene image and mode selection information to be processed during the urban governance process;

[0037] A generating unit, configured to generate a target prompt word according to the mode selection information, the governance event library and the prompt template;

[0038] The recognition unit is used to recognize abnormal events in the urban scene image according to the target prompt word and the multimodal large model.

[0039] In some embodiments, the prompt template includes an event option slot, and the generating unit is specifically used to:

[0040] Determine a target prompt template corresponding to the mode selection information from the prompt templates;

[0041] Determine, according to the governance event library, a first filling content corresponding to the event option slot of the target prompt template;

[0042] A target prompt word is generated according to the first filling content and the target prompt template.

[0043] In some embodiments, the generating unit is specifically used for:

[0044] When the mode selection information indicates fast identification, the first prompt template is used as the target prompt template, and the first prompt template is used for direct matching prompt;

[0045] When the mode selection information indicates accurate recognition, the second prompt template is used as the target prompt template, and the second prompt template is used for multi-stage matching prompt;

[0046] When the mode selection information indicates verification identification, the third prompt template is used as the target prompt template, and the third prompt template is used for judgment prompting.

[0047] In some embodiments, the generating unit is specifically used for:

[0048] When the target prompt template is used for direct matching prompt or multi-stage matching prompt, the governance event library is used as the first filling content corresponding to the event option slot of the target prompt template;

[0049] When the target prompt template is used for judgment prompting, a standard event is determined from the governance event library, and the standard event is used as the first filling content corresponding to the event option slot of the target prompt template.

[0050] In some embodiments, when the target prompt template is used for multi-stage matching prompts, the target prompt template further includes an image description slot, the target prompt words include first-stage prompt words and second-stage prompt words, and the generating unit is specifically used to:

[0051] generating the first-stage prompt words, where the first-stage prompt words are used to instruct image description;

[0052] Inputting the first-stage prompt word and the city scene image into the multimodal large model to generate image description information corresponding to the city scene image, and using the image description information as second filling content corresponding to the image description slot;

[0053] The second-stage prompt word is generated according to the first filling content, the second filling content, the image description slot and the event option slot.

[0054] In some embodiments, the abnormal event recognition device based on the multimodal large model further includes a model fine-tuning unit, which is used to:

[0055] Before the recognition unit recognizes abnormal events in the urban scene image according to the target prompt words and the multimodal large model, a plurality of sample images corresponding to each of the governance events are obtained, and sample image description information corresponding to each of the sample images is constructed;

[0056] Constructing a fine-tuning prompt instruction set according to the prompt template, the governance event and the sample image description information;

[0057] The multimodal large model is fine-tuned according to the fine-tuning prompt instruction set, the sample image and the governance event.

[0058] In some embodiments, the fine-tuning prompt instruction set includes a first fine-tuning prompt instruction, a second fine-tuning prompt instruction, and a third fine-tuning prompt instruction, and the model fine-tuning unit is specifically used to:

[0059] Determine a plurality of candidate event groups based on all of the governance events;

[0060] Constructing a corresponding first fine-tuning prompt instruction according to each candidate event group and the first prompt template, each candidate event group corresponds to one first fine-tuning prompt instruction, and the first prompt template is used for performing a direct matching prompt;

[0061] Constructing a corresponding second fine-tuning prompt instruction according to each candidate event group, the sample image description information and the second prompt template, each candidate event group corresponds to one second fine-tuning prompt instruction, and the second prompt template is used for multi-stage matching prompt;

[0062] According to each of the governance events and the third prompt template, a corresponding third fine-tuning prompt instruction is constructed, each of the governance events corresponds to one third fine-tuning prompt instruction, and the third prompt template is used for making judgment prompts.

[0063] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned methods for identifying abnormal events based on a multimodal large model.

[0064] In a fourth aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for identifying abnormal events based on a multimodal large model as described above is implemented.

[0065] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for identifying abnormal events based on a multimodal large model.

[0066] The abnormal event recognition method, device, storage medium, electronic device and computer program product based on the multimodal large model provided in the embodiments of the present application construct a governance event library and a variety of prompt templates corresponding to the urban governance scenarios; in the urban governance process, obtain the urban scene image and mode selection information to be processed; generate target prompt words according to the mode selection information, the governance event library and the prompt template; perform event recognition of the urban scene image according to the target prompt words and the multimodal large model, that is, recognize abnormal events in the urban scene image based on the multimodal large model and the prompt template, without relying on manual recognition, so that it can not only quickly and accurately identify abnormal problems occurring in the urban governance process and avoid false detection and missed detection, but also is applicable to various urban governance scenarios, does not require model training and setting up a comparison image library, and has good generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0068] Figure 1 It is a flowchart of an abnormal event identification method based on a multimodal large model provided in an embodiment of the present application;

[0069] Figure 2It is another flow chart of the abnormal event identification method based on the multimodal large model provided in the embodiment of the present application;

[0070] Figure 3 A schematic diagram of the structure of an abnormal event recognition device based on a multimodal large model provided in an embodiment of the present application;

[0071] Figure 4 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0072] Figure 5 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0073] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0074] Embodiments of the present application provide a method, device, storage medium, electronic device, and computer program product for identifying abnormal events based on a multimodal large model.

[0075] See also Figure 1 , Figure 1 1 is a flow chart of an abnormal event identification method based on a multimodal large model provided in an embodiment of the present application. The abnormal event identification method based on a multimodal large model is applied to an electronic device, and can be performed by an electronic device or a functional module or functional entity in the electronic device. The abnormal event identification method based on a multimodal large model includes the following steps 101-104, wherein:

[0076] 101. Construct a governance event library and multiple prompt templates corresponding to urban governance scenarios. Each governance event in the governance event library includes an event category and event description content. Different prompt templates correspond to different recognition modes.

[0077] 102. In the process of urban governance, obtain the urban scene images and mode selection information to be processed;

[0078] 103. Generate target prompt words according to the mode selection information, the governance event library and the prompt template;

[0079] 104. Identify abnormal events in urban scene images based on target prompt words and multimodal large models.

[0080] The abnormal event recognition method based on the multimodal large model provided in the embodiment of the present application constructs a governance event library and a variety of prompt templates corresponding to the urban governance scenario; in the urban governance process, obtains the urban scene image to be processed and the mode selection information; generates the target prompt word according to the mode selection information, the governance event library and the prompt template; performs event recognition of the urban scene image according to the target prompt word and the multimodal large model, that is, recognizes abnormal events in the urban scene image based on the multimodal large model and the prompt template, without relying on manual recognition, so that it can not only quickly and accurately identify abnormal problems occurring in the urban governance process and avoid false detection and missed detection, but also is applicable to various urban governance scenarios, does not require model training and setting of a comparison image library, and has good generalization ability.

[0081] The above steps 101 - 104 will be described in detail below.

[0082] 101. Construct a governance event library and multiple prompt templates corresponding to urban governance scenarios. Each governance event in the governance event library includes an event category and event description content. Different prompt templates correspond to different recognition modes.

[0083] The governance event library can be a summary of historical governance situations in urban governance scenarios, and event categories include but are not limited to drying clothes on the street, occupying the road for business, overflowing trash cans, floating objects on the water, etc. For example, the governance event library can be expressed as C = {e i :d i ,i=1,2,…,n}, n represents the number of event categories. Taking drying clothes on the street as an example, e i = drying along the street, d i =Hanging and drying clothes on roads and other public places; using trees, guardrails, road signs, electric poles and other facilities on roads and other public places to hang and dry clothes.

[0084] There are three recognition modes, such as fast recognition, precise recognition and verification recognition. Users can choose the appropriate recognition mode according to their needs. For example, for events in unknown scenes, the model needs to give specific event types and explanations. You can choose fast recognition or precise recognition. If you only want to determine whether there is a certain type of abnormal event in the graph, select verification recognition.

[0085] A prompt template is a rule that specifies the organization of prompt words. The prompt template corresponds to the recognition mode one by one, and presets different types of questioning methods to achieve different questioning effects. A prompt word is a short text instruction or question that guides a multimodal large model to generate a specific type of response or complete a specific task. Prompt templates usually contain some fixed text elements and replaceable variables or parameters, which will be replaced by corresponding values ​​when generating specific prompts. Prompt templates make the process of generating prompts more standardized and efficient, and are particularly suitable for scenarios where similar prompts need to be generated repeatedly.

[0086] 102. In the process of urban governance, obtain the urban scene images and mode selection information to be processed.

[0087] The mode selection information may be a recognition mode manually selected by the user or a recognition mode set by the system by default. An urban governance abnormal event analysis platform may be constructed, through which users may actively upload their captured urban scene images and select the required recognition mode for recognition. Alternatively, the platform may be connected to a variety of image acquisition devices, and collect urban scene images in real time for recognition through the connected devices.

[0088] 103. Generate a target prompt word according to the mode selection information, the governance event library and the prompt template.

[0089] In some embodiments, see Figure 2 , Figure 2 It is another flow chart of the abnormal event identification method based on the multimodal large model provided in the embodiment of the present application. The prompt template includes an event option slot. The above step 103 may include the following steps 1031-1033, wherein:

[0090] 1031. Determine a target prompt template corresponding to the mode selection information from the prompt templates.

[0091] Furthermore, the above step 1031 specifically includes:

[0092] When the mode selection information indicates fast identification, the first prompt template is used as the target prompt template, and the first prompt template is used for direct matching prompt;

[0093] When the mode selection information indicates accurate recognition, the second prompt template is used as the target prompt template, and the second prompt template is used for multi-stage matching prompt;

[0094] When the mode selection information indicates verification identification, the third prompt template is used as the target prompt template, and the third prompt template is used for judgment prompting.

[0095] Among them, the urban governance abnormal event analysis platform can provide an interactive interface, which includes options for various recognition modes. After logging into the platform, the user can enter the interactive interface to select the required recognition mode. Different recognition modes are associated with different prompt templates to achieve different prompt effects. By specifying the required recognition mode to adapt the corresponding prompt template, the appropriate prompt words are generated and provided to the multimodal large model.

[0096] 1032. Determine the first filling content corresponding to the event option slot of the target prompt template according to the governance event library.

[0097] In some embodiments, the above step 1032 specifically includes:

[0098] When the target prompt template is used for direct matching prompt or multi-stage matching prompt, the governance event library is used as the first filling content corresponding to the event option slot of the target prompt template;

[0099] When the target prompt template is used for judgment prompting, a standard event is determined from the governance event library, and the standard event is used as the first filling content corresponding to the event option slot of the target prompt template.

[0100] Among them, the slots in different prompt templates can be partially the same, for example, the first prompt template, the second prompt template and the third prompt template can all include event option slots, and the first prompt template and the second prompt template can also include basic question slots. The filling content of some slots in the prompt template can be directly set in the prompt template, and the filling content of some slots needs to be dynamically generated when the prompt is made, which depends on the actual situation.

[0101] The same slot in different prompt templates can correspond to the same filling content or different filling content. For example, for the first prompt template for direct matching prompts and the second prompt template for multi-stage matching prompts, both can design basic question slots, but the filling content of the basic question slots is different. In the first prompt template, it can be directly set to "Which of the following options is the most consistent with the content in the picture?", and in the second prompt template, it can be directly set to "Which option is closest to the meaning of the image description?". For event option slots, the first prompt template and the second prompt template can have the same first filling content. For example, all governance events in the governance event library are used as their first filling content in the form of data pairs, which can be expressed as {e i :d i},e i is the event category, d i For the third prompt template, a specified event needs to be selected from the governance event library as the standard event, and the standard event is used as its first filling content, for example, it is expressed as (ei ,d i ).

[0102] 1033. Generate a target prompt word according to the first filling content and the target prompt template.

[0103] Among them, for the first prompt template for direct matching prompt and the third prompt template for judging matching prompt, the target prompt word can be directly generated according to the first filling content and other content in the target template. For example, for the first prompt template, there is also a basic question slot The content is fixed as "Which of the following options is the most suitable in the picture?", event option slot The first fill content is {e i :d i}, then the basic question slot and event option slots The final target prompt word p1 can be obtained by splicing the respective filling contents; for the third prompt template, the event option slot The first filling content is (e i ,d i ), the structure of the target prompt word p3 generated is: Is there e in the graph? i ?d i For example, the target prompt word p3 is: Is there any drying clothes along the street in the picture? Hanging and drying clothes on roads and other public places; using trees, guardrails, road signs, electric poles and other facilities on roads and other public places to hang and dry clothes.

[0104] For the second prompt template, prompt words need to be generated in stages, and different prompt words prompt the multimodal large model to perform different tasks. For example, when the target prompt template is used for multi-stage matching prompts, the target prompt template also includes an image description slot, and the target prompt words include first-stage prompt words and second-stage prompt words. At this time, the above step 1033 specifically includes:

[0105] Generate a first-stage prompt word, where the first-stage prompt word is used to instruct image description;

[0106] Inputting the first-stage prompt word and the city scene image into the multimodal large model to generate image description information corresponding to the city scene image, and using the image description information as the second filling content corresponding to the image description slot;

[0107] A second-stage prompt word is generated according to the first filling content, the second filling content, the image description slot and the event option slot.

[0108] For example, the target prompt word generated by the second prompt template The first stage prompt word, is the second stage prompt word. The first stage is the image description stage, and its prompt word can be fixed, such as =Please describe the scene in the picture in detail. The second stage is the matching stage. Similar to the composition of p1 above, including basic question slots and event option slots The filling content of the basic question slot The fill content of the event option slot is changed to "Which option is closest to the image description?" The first fill content is {e i :d i}, It also includes the second filling content of the image description slot (i.e., the image description information generated by the multimodal large model in the image description stage), through Filling content, The filling content and image description information are spliced ​​to obtain the second stage prompt word

[0109] 104. Identify abnormal events in the urban scene image according to the target prompt word and the multimodal large model.

[0110] Among them, the target prompt words and urban scene images are input into the multimodal large model. The multimodal large model can identify abnormal events in the image based on the target prompt words, thereby realizing zero-sample abnormal event recognition of the multimodal large model in the urban governance process. It is suitable for various urban governance scenarios and has good generalization ability.

[0111] In other embodiments, it is common to train a deep learning target detection or segmentation model in advance and identify abnormal events based on the trained model. However, this training requires a large amount of sample data. When the abnormal event is a small sample type, data is difficult to collect, which will lead to the problem of poor model generalization ability. In addition, some prior information needs to be set for image comparison. When the scene changes, the prior information needs to be reset, which is less flexible.

[0112] In addition, in addition to realizing abnormal event recognition of the multimodal large model based on the design of the prompt template alone, the multimodal large model can also be fine-tuned through a small number of samples to further improve the scene understanding ability and instruction following ability of the multimodal large model. That is, before the above step 104, the abnormal event recognition method based on the multimodal large model can also include:

[0113] 1051. Obtain a number of sample images corresponding to each governance event, and construct sample image description information corresponding to each sample image;

[0114] 1052. Construct a fine-tuning prompt instruction set according to the prompt template, the governance event and the sample image description information;

[0115] 1053. Fine-tune the multimodal large model according to the fine-tuning prompt instruction set, the sample image, and the governance event.

[0116] The sample image description information is constructed manually, and includes elements such as scene type, description of abnormal events in the image, etc. It is necessary to design corresponding fine-tuning prompt instructions for each prompt template, that is, the number of types of fine-tuning instructions in the fine-tuning prompt instruction set is consistent with the number of prompt templates. For example, there are three prompt templates, and the fine-tuning prompt instruction set I = {t k ,k=1,2,3}, different types of fine-tuning prompt instructions may correspond to different numbers of instructions. The governance event corresponding to each sample image is used as annotation information, and the multimodal large model is fine-tuned in combination with the fine-tuning prompt instruction set to further enhance the scene understanding ability and instruction compliance ability of the multimodal large model and improve the model recognition accuracy.

[0117] In some embodiments, the fine-tuning prompt instruction set includes a first fine-tuning prompt instruction, a second fine-tuning prompt instruction, and a third fine-tuning prompt instruction. In this case, the above step 1052 specifically includes:

[0118] Determine multiple candidate event groups based on all governance events;

[0119] According to each candidate event group and the first prompt template, a corresponding first fine-tuning prompt instruction is constructed, each candidate event group corresponds to a first fine-tuning prompt instruction, and the first prompt template is used for direct matching prompt;

[0120] According to each candidate event group, the sample image description information and the second prompt template, a corresponding second fine-tuning prompt instruction is constructed, each candidate event group corresponds to a second fine-tuning prompt instruction, and the second prompt template is used for multi-stage matching prompt;

[0121] According to each governance event and the third prompt template, a corresponding third fine-tuning prompt instruction is constructed. Each governance event corresponds to a third fine-tuning prompt instruction. The third prompt template is used for making judgment prompts.

[0122] Among them, a certain number (for example, 5) of governance events can be randomly selected from the governance event library as a candidate event group. The first fine-tuning prompt instruction t1 is similar in structure to the prompt word p1 generated by the first prompt template, and is formed by splicing the filling content of the basic question slot and the event option slot, except that the filling content of the event option slot is modified to the candidate event group. The second fine-tuning prompt instruction t2 is similar in structure to the prompt word p2 generated by the second prompt template, including the first-stage instruction and the second-stage instruction, the first-stage instruction and the first-stage prompt word Similarly, the second stage instructions and the second stage prompt words Similarly, the second-stage instructions are formed by splicing the filling contents of the basic question slot, event option slot, and image description slot, except that the filling contents of the event option slot are modified to the candidate event group, and the filling contents of the image description slot are modified to the sample image description information. The third fine-tuning prompt instruction t3 has a similar structure to the prompt word p3 generated by the third prompt template. The event option slot is filled with a single urban governance event, and multiple third fine-tuning prompt instructions t3 are generated according to the number of events in the selected candidate event group.

[0123] It should be pointed out that by combining model fine-tuning and prompt templates, the multimodal large model can obtain more accurate analysis results of abnormal events in urban governance scenarios, effectively reduce missed detections and false detections, and can achieve different recognition requirements in practical applications, such as fast recognition, precise recognition and verification recognition.

[0124] From the above, it can be seen that the abnormal event recognition method based on the multimodal large model provided in the embodiment of the present application constructs a governance event library and a variety of prompt templates corresponding to the urban governance scenario; in the urban governance process, the urban scene image to be processed and the mode selection information are obtained; the target prompt words are generated according to the mode selection information, the governance event library and the prompt template; the event recognition of the urban scene image is performed according to the target prompt words and the multimodal large model, that is, the abnormal events in the urban scene image are recognized based on the multimodal large model and the prompt template, without relying on manual recognition, so that it can not only quickly and accurately identify abnormal problems occurring in the urban governance process, avoiding false detection and missed detection, but also is applicable to various urban governance scenarios, without the need for model training and setting up a comparison image library, and has good generalization ability.

[0125] According to the method described in the above embodiment, the present application embodiment also provides an abnormal event recognition device based on a multimodal large model, which is used to perform the steps in the above abnormal event recognition method based on a multimodal large model. Figure 3 , Figure 31 is a schematic diagram of the structure of an abnormal event recognition device 200 based on a multimodal large model provided in an embodiment of the present application. The abnormal event recognition device 200 based on a multimodal large model is applied to an electronic device, and the abnormal event recognition method based on a multimodal large model can be executed by the electronic device or a functional module or functional entity in the electronic device. Specifically, the abnormal event recognition device 200 based on a multimodal large model includes a construction unit 201, an acquisition unit 202, a generation unit 203 and an identification unit 204, wherein:

[0126] A construction unit 201 is used to construct a governance event library and multiple prompt templates corresponding to the urban governance scenario. Each governance event in the governance event library includes an event category and an event description content. Different prompt templates correspond to different recognition modes.

[0127] The acquisition unit 202 is used to acquire the urban scene image to be processed and the mode selection information during the urban governance process;

[0128] A generating unit 203, configured to generate a target prompt word according to the mode selection information, the governance event library and the prompt template;

[0129] The recognition unit 204 is used to recognize abnormal events in the urban scene image according to the target prompt word and the multimodal large model.

[0130] In some embodiments, the prompt template includes an event option slot, and the generating unit 203 is specifically used to:

[0131] Determine a target prompt template corresponding to the mode selection information from the prompt template;

[0132] Determine, according to the governance event library, a first filling content corresponding to the event option slot of the target prompt template;

[0133] A target prompt word is generated according to the first filling content and the target prompt template.

[0134] In some embodiments, the generating unit 203 is specifically used to:

[0135] When the mode selection information indicates fast identification, the first prompt template is used as the target prompt template, and the first prompt template is used for direct matching prompt;

[0136] When the mode selection information indicates accurate recognition, the second prompt template is used as the target prompt template, and the second prompt template is used for multi-stage matching prompt;

[0137] When the mode selection information indicates verification identification, the third prompt template is used as the target prompt template, and the third prompt template is used for judgment prompting.

[0138] In some embodiments, the generating unit 203 is specifically used to:

[0139] When the target prompt template is used for direct matching prompt or multi-stage matching prompt, the governance event library is used as the first filling content corresponding to the event option slot of the target prompt template;

[0140] When the target prompt template is used for judgment prompting, a standard event is determined from the governance event library, and the standard event is used as the first filling content corresponding to the event option slot of the target prompt template.

[0141] In some embodiments, when the target prompt template is used for multi-stage matching prompts, the target prompt template also includes an image description slot, the target prompt words include first-stage prompt words and second-stage prompt words, and the generating unit 203 is specifically used to:

[0142] Generate the first-stage prompt word, where the first-stage prompt word is used to instruct image description;

[0143] Inputting the first-stage prompt word and the city scene image into the multimodal large model to generate image description information corresponding to the city scene image, and using the image description information as second filling content corresponding to the image description slot;

[0144] The second-stage prompt word is generated according to the first filling content, the second filling content, the image description slot and the event option slot.

[0145] In some embodiments, the abnormal event identification device 200 based on the multimodal large model further includes a model fine-tuning unit for:

[0146] Before the recognition unit recognizes the abnormal event in the urban scene image according to the target prompt word and the multimodal large model, a plurality of sample images corresponding to each governance event are obtained, and sample image description information corresponding to each sample image is constructed;

[0147] Constructing a fine-tuning prompt instruction set according to the prompt template, the governance event and the sample image description information;

[0148] The multimodal large model is fine-tuned according to the fine-tuning prompt instruction set, the sample image and the governance event.

[0149] In some embodiments, the fine-tuning prompt instruction set includes a first fine-tuning prompt instruction, a second fine-tuning prompt instruction, and a third fine-tuning prompt instruction, and the model fine-tuning unit is specifically used to:

[0150] Determine multiple candidate event groups based on all governance events;

[0151] According to each candidate event group and the first prompt template, a corresponding first fine-tuning prompt instruction is constructed, each candidate event group corresponds to a first fine-tuning prompt instruction, and the first prompt template is used for direct matching prompt;

[0152] According to each candidate event group, the sample image description information and the second prompt template, a corresponding second fine-tuning prompt instruction is constructed, each candidate event group corresponds to a second fine-tuning prompt instruction, and the second prompt template is used for multi-stage matching prompt;

[0153] According to each governance event and the third prompt template, a corresponding third fine-tuning prompt instruction is constructed. Each governance event corresponds to a third fine-tuning prompt instruction. The third prompt template is used for making judgment prompts.

[0154] It should be noted that the specific details of each module unit in the above-mentioned abnormal event identification device 200 based on a multimodal large model have been described in detail in the embodiment of the above-mentioned abnormal event identification method based on a multimodal large model, and will not be repeated here.

[0155] From the above, it can be seen that the abnormal event recognition device 200 based on the multimodal large model provided in the embodiment of the present application constructs a governance event library and a variety of prompt templates corresponding to the urban governance scenario through the construction unit 201; in the urban governance process, the acquisition unit 202 acquires the urban scene image and mode selection information to be processed; the generation unit 203 generates the target prompt word according to the mode selection information, the governance event library and the prompt template; the recognition unit 204 performs event recognition of the urban scene image according to the target prompt word and the multimodal large model, that is, the abnormal events in the urban scene image are recognized based on the multimodal large model and the prompt template, without relying on manual recognition, so that it can not only quickly and accurately identify abnormal problems occurring in the urban governance process, avoiding false detection and missed detection, but also is suitable for various urban governance scenarios, without the need for model training and setting up a comparison image library, and has good generalization ability.

[0156] In some embodiments, the abnormal event identification device based on the multimodal large model in the embodiment of the present application can be an electronic device, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a server, which can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. The embodiment of the present application is not specifically limited.

[0157] In some embodiments, Figure 4 As shown, an embodiment of the present application also provides an electronic device 300, including a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the program is executed by the processor 301, the various processes of the above-mentioned abnormal event recognition method embodiment based on the multimodal large model are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0158] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0159] Figure 5 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0160] The electronic device 400 includes but is not limited to: a radio frequency unit 401, a network module 402, an audio output unit 403, an input unit 404, a sensor 405, a display unit 406, a user input unit 407, an interface unit 408, a memory 409, a processor 410 and other components.

[0161] In some embodiments, the processor 410 implements the following steps when executing the program:

[0162] Build a governance event library and multiple prompt templates corresponding to urban governance scenarios. Each governance event in the governance event library includes event category and event description content. Different prompt templates correspond to different recognition modes.

[0163] In the process of urban governance, obtain the urban scene images and mode selection information to be processed;

[0164] Generate a target prompt word according to the mode selection information, the governance event library and the prompt template;

[0165] Abnormal events in the urban scene image are identified based on the target prompt word and the multimodal large model.

[0166] In some embodiments, the prompt template includes an event option slot, and the processor 410 is specifically configured to:

[0167] Determine a target prompt template corresponding to the mode selection information from the prompt template;

[0168] Determine, according to the governance event library, a first filling content corresponding to the event option slot of the target prompt template;

[0169] A target prompt word is generated according to the first filling content and the target prompt template.

[0170] In some embodiments, the processor 410 is specifically configured to:

[0171] When the mode selection information indicates fast identification, the first prompt template is used as the target prompt template, and the first prompt template is used for direct matching prompt;

[0172] When the mode selection information indicates accurate recognition, the second prompt template is used as the target prompt template, and the second prompt template is used for multi-stage matching prompt;

[0173] When the mode selection information indicates verification identification, the third prompt template is used as the target prompt template, and the third prompt template is used for judgment prompting.

[0174] In some embodiments, the processor 410 is specifically configured to:

[0175] When the target prompt template is used for direct matching prompt or multi-stage matching prompt, the governance event library is used as the first filling content corresponding to the event option slot of the target prompt template;

[0176] When the target prompt template is used for judgment prompting, a standard event is determined from the governance event library, and the standard event is used as the first filling content corresponding to the event option slot of the target prompt template.

[0177] In some embodiments, when the target prompt template is used for multi-stage matching prompts, the target prompt template further includes an image description slot, the target prompt words include first-stage prompt words and second-stage prompt words, and the processor 410 is specifically configured to:

[0178] Generate the first-stage prompt word, where the first-stage prompt word is used to instruct image description;

[0179] Inputting the first-stage prompt word and the city scene image into the multimodal large model to generate image description information corresponding to the city scene image, and using the image description information as second filling content corresponding to the image description slot;

[0180] The second-stage prompt word is generated according to the first filling content, the second filling content, the image description slot and the event option slot.

[0181] In some embodiments, before identifying abnormal events in the urban scene image according to the target prompt word and the multimodal large model, the processor 410 is further configured to:

[0182] Obtain at least one sample image corresponding to each governance event, and construct sample image description information corresponding to each sample image;

[0183] Constructing a fine-tuning prompt instruction set according to the prompt template, the governance event and the sample image description information;

[0184] The multimodal large model is fine-tuned according to the fine-tuning prompt instruction set, the sample image and the governance event.

[0185] In some embodiments, the fine-tuning prompt instruction set includes a first fine-tuning prompt instruction, a second fine-tuning prompt instruction, and a third fine-tuning prompt instruction, and the processor 410 is specifically configured to:

[0186] Determine multiple candidate event groups based on all governance events;

[0187] According to each candidate event group and the first prompt template, a corresponding first fine-tuning prompt instruction is constructed, each candidate event group corresponds to a first fine-tuning prompt instruction, and the first prompt template is used for direct matching prompt;

[0188] According to each candidate event group, the sample image description information and the second prompt template, a corresponding second fine-tuning prompt instruction is constructed, each candidate event group corresponds to a second fine-tuning prompt instruction, and the second prompt template is used for multi-stage matching prompt;

[0189] According to each governance event and the third prompt template, a corresponding third fine-tuning prompt instruction is constructed. Each governance event corresponds to a third fine-tuning prompt instruction. The third prompt template is used for making judgment prompts.

[0190] Those skilled in the art will appreciate that the electronic device 400 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 410 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 5 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0191] It should be understood that in the embodiment of the present application, the input unit 404 may include a graphics processor (Graphics Processing Unit, GPU) 4041 and a microphone 4042, and the graphics processor 4041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 406 may include a display panel 4061, and the display panel 4061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 407 includes a touch panel 4071 and at least one of other input devices 4072. The touch panel 4071 is also called a touch screen. The touch panel 4071 may include two parts: a touch detection device and a touch controller. Other input devices 4072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0192] The memory 409 can be used to store software programs and various data. The memory 409 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 409 may include a volatile memory or a non-volatile memory, or the memory 409 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 409 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0193] The processor 410 may include one or more processing units; the processor 410 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 410.

[0194] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned abnormal event identification method embodiment based on a multimodal large model are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0195] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0196] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned abnormal event identification method based on a multimodal large model.

[0197] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0198] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0199] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0200] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0201] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0202] In the description of the present application, “plurality” means two or more.

[0203] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0204] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A method for identifying abnormal events based on a multimodal large model, characterized in that: include: Construct a governance event library and multiple prompt templates corresponding to urban governance scenarios. Each governance event in the governance event library includes an event category and event description content. Different prompt templates correspond to different recognition modes. In the process of urban governance, obtain the urban scene images and mode selection information to be processed; Generate a target prompt word according to the mode selection information, the governance event library and the prompt template; Abnormal events in the urban scene image are identified according to the target prompt words and the multimodal large model.

2. The abnormal event identification method based on multimodal large model according to claim 1 is characterized in that: The prompt template includes an event option slot, and the generating of a target prompt word according to the mode selection information, the governance event library and the prompt template includes: Determine a target prompt template corresponding to the mode selection information from the prompt templates; Determine, according to the governance event library, a first filling content corresponding to the event option slot of the target prompt template; A target prompt word is generated according to the first filling content and the target prompt template.

3. The abnormal event identification method based on multimodal large model according to claim 2 is characterized in that: The determining, from the prompt template, a target prompt template corresponding to the mode selection information, comprises: When the mode selection information indicates fast identification, the first prompt template is used as the target prompt template, and the first prompt template is used for direct matching prompt; When the mode selection information indicates accurate recognition, the second prompt template is used as the target prompt template, and the second prompt template is used for multi-stage matching prompt; When the mode selection information indicates verification identification, the third prompt template is used as the target prompt template, and the third prompt template is used for judgment prompting.

4. The abnormal event identification method based on multimodal large model according to claim 2 is characterized in that: The determining, according to the governance event library, a first filling content corresponding to the event option slot of the target prompt template includes: When the target prompt template is used for direct matching prompt or multi-stage matching prompt, the governance event library is used as the first filling content corresponding to the event option slot of the target prompt template; When the target prompt template is used for judgment prompting, a standard event is determined from the governance event library, and the standard event is used as the first filling content corresponding to the event option slot of the target prompt template.

5. The abnormal event identification method based on multimodal large model according to claim 2 is characterized in that: When the target prompt template is used for multi-stage matching prompts, the target prompt template further includes an image description slot, the target prompt words include first-stage prompt words and second-stage prompt words, and generating target prompt words according to the first filling content and the target prompt template includes: generating the first-stage prompt words, where the first-stage prompt words are used to instruct image description; Inputting the first-stage prompt word and the city scene image into the multimodal large model to generate image description information corresponding to the city scene image, and using the image description information as second filling content corresponding to the image description slot; The second-stage prompt word is generated according to the first filling content, the second filling content, the image description slot and the event option slot.

6. The abnormal event identification method based on a multimodal large model according to any one of claims 1 to 5, characterized in that: Before identifying abnormal events in the urban scene image according to the target prompt word and the multimodal large model, the method further includes: Acquire a number of sample images corresponding to each of the governance events, and construct sample image description information corresponding to each of the sample images; Constructing a fine-tuning prompt instruction set according to the prompt template, the governance event and the sample image description information; The multimodal large model is fine-tuned according to the fine-tuning prompt instruction set, the sample image and the governance event.

7. The abnormal event identification method based on multimodal large model according to claim 6 is characterized in that: The fine-tuning prompt instruction set includes a first fine-tuning prompt instruction, a second fine-tuning prompt instruction and a third fine-tuning prompt instruction. The fine-tuning prompt instruction set is constructed according to the prompt template, the governance event and the sample image description information, including: Determine a plurality of candidate event groups based on all of the governance events; Constructing a corresponding first fine-tuning prompt instruction according to each candidate event group and the first prompt template, wherein each candidate event group corresponds to one first fine-tuning prompt instruction, and the first prompt template is used for performing a direct matching prompt; According to each of the candidate event groups, the sample image description information and the second prompt template, a corresponding second fine-tuning prompt instruction is constructed, each of the candidate event groups corresponds to one second fine-tuning prompt instruction, and the second prompt template is used for performing multi-stage matching prompts; According to each of the governance events and the third prompt template, a corresponding third fine-tuning prompt instruction is constructed, each of the governance events corresponds to one third fine-tuning prompt instruction, and the third prompt template is used for making judgment prompts.

8. An abnormal event recognition device based on a multimodal large model, characterized in that: include: A construction unit, used to construct a governance event library and multiple prompt templates corresponding to urban governance scenarios, wherein each governance event in the governance event library includes an event category and event description content, and different prompt templates correspond to different recognition modes; An acquisition unit, used to acquire the urban scene image and mode selection information to be processed during the urban governance process; A generating unit, configured to generate a target prompt word according to the mode selection information, the governance event library and the prompt template; The recognition unit is used to recognize abnormal events in the urban scene image according to the target prompt word and the multimodal large model.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the abnormal event recognition method based on a multimodal large model as described in any one of claims 1 to 7 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the abnormal event identification method based on the multimodal large model as described in any one of claims 1-7 is implemented.

Images