Urban governance multi-modal large model evaluation method and device based on multiple levels of texts

Abstract

The invention discloses an urban governance multi-modal large model evaluation method and device based on multiple levels of texts. The method comprises the following steps: acquiring a real label text of a sample picture and an output text of an urban governance multi-modal large model; segmenting the text into independent vocabularies by adopting a city management word library, and calculating a vocabulary unit similarity score; creating a bag-of-words model by using a city management lexicon, converting the text data into a word frequency matrix, and calculating a low-layer word frequency similarity score; a trained language model is adopted to extract semantic features with contexts in the text, the semantic features are converted into vector representation, and a semantic similarity score is calculated; obtaining a high-level semantic similarity score corresponding to the text by adopting a formatting request general large language model with business guide words and evaluation requirements; and in combination with the similarity score of each level, calculating a comprehensive evaluation score of the multi-modal large model. According to the method, the similarity measurement under the multi-level space of the text is fully considered, and the accuracy of urban governance large model text output evaluation is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing and smart city technology, and in particular to a method and device for evaluating a multi-modal large model of urban governance based on multi-level text. Background Art

[0002] In recent years, big models have made breakthrough progress. Big models refer to complex artificial neural network models with ultra-large-scale parameters, usually containing hundreds of millions to trillions of parameters. Big models have shown wide application potential in many fields due to their powerful learning and generalization capabilities. For example, OpenAI's GPT (Generative Pre-trained Transformer, a trained language model based on the Transformer architecture) series of big models can generate high-quality text content such as articles, stories, and news reports. Multimodal big models are further expanded on the basis of big models. They not only process text data, but also process information of different modalities such as pictures, sounds, and videos at the same time. For example, a multimodal big model can be used to understand the content of a picture and generate corresponding text descriptions based on it. Evaluating the performance of multimodal big models has become a subsequent task. The urban governance multimodal big model is a multimodal application customized for solving urban management scenarios based on the general big model. The urban governance multimodal big model can recognize urban street scene pictures and output a description of urban management. For example, if a picture showing illegal out-of-store operations is input into the big model, the urban governance multimodal big model will output "the picture shows that the store business has expanded its business location outward, occupying public resources, which is an out-of-store operation problem."

[0003] At present, the existing evaluation methods of multimodal large models mostly adopt the visual question answering (VQA) evaluation method. The visual question answering (VQA) dataset is an open-ended question answering dataset containing pictures. These questions require the understanding of vision, language, and common sense to answer. VQA refers to generating a relevant answer given a picture and a natural language question related to the picture. The VQA dataset includes more than 200,000 pictures, each picture has an average of 5 questions, and each question has 10 true value answers. However, the answer form of VQA is relatively simple, and the answer types are divided into attributes, colors, whether, values, etc., and the output text is also relatively short. For example: the question is "Who is wearing glasses in the picture", and the answer is "man"; the question is "How many children are there on the bed", and the answer is "2". However, the output of large models is mostly long text. The VQA evaluation method aims to measure the model's ability to understand the content of the picture and give the correct answer based on the question. VQA accuracy is the proportion of the answer given by the model that completely matches the correct answer annotated by humans. VQA accuracy is scored by comparing the matching degree between the answers predicted by the model and multiple answers provided by a group of human respondents. If the predicted answer is consistent with the content of a human answer (including the consistency of the content after simple format conversion, such as the English word two and the Arabic number 2), it will be counted in the accuracy calculation. Obviously, the evaluation method of VQA does not take into account the similarity between texts. Even if the text cosine similarity metric is added, it will lead to the problem of inaccurate similarity. On the other hand, the output of large models is mostly long texts, and the similarity measurement between long texts is also more complicated. It is difficult to evaluate the semantic similarity between two texts by simply using the text cosine similarity metric. To give a simple example: the large model outputs the text "I like you", and the real label text is "I don't like you". If the text similarity evaluation is used, the index is as high as 0.8. Obviously, the semantics of these two sentences are opposite. Therefore, it is inaccurate to simply use the text similarity metric.

[0004] For smart city scenarios, the output text of the urban governance multimodal large model contains descriptions of the target and its attributes, and the text is relatively long. If only the low-level cosine similarity between the output text and the answer text is considered, and the output results of the model are not compared from a semantic perspective, the large model evaluation method will be unreliable. For example: the large model outputs the text "There are peddlers in the picture. They are operating on the road and causing crowds to gather. This may affect traffic and pose certain safety hazards.", while the real label text is "The picture shows unlicensed peddlers, but they are not selling goods, so they have no impact on the surrounding environment." If text similarity evaluation is used, the index is as high as 0.79, but there is still a large difference between the output text of the large model and the real text content. Therefore, it is inaccurate to use low-level text similarity to evaluate the performance of the multimodal large model text output in the urban governance scenario. Summary of the invention

[0005] The purpose of the present invention is to provide a method and device for evaluating a multimodal large model of urban governance based on multi-level text, which solves the problems existing in the prior art.

[0006] The present invention is achieved through the following technical solutions:

[0007] On the one hand, the present invention provides a method for evaluating a multimodal large model of urban governance based on multi-level text, comprising:

[0008] Obtain the real label text of the sample image and the output text of the urban governance multimodal large model;

[0009] The urban management vocabulary is used to segment the real label text into first independent words, the urban management vocabulary is used to segment the output text of the urban governance multimodal large model into second independent words, and the vocabulary unit similarity score between the first independent words and the second independent words is obtained;

[0010] Create a bag-of-words model based on the urban management vocabulary, convert the real label text into the first word frequency matrix, convert the output text of the urban governance multimodal large model into the second word frequency matrix, and obtain the low-level word frequency similarity score between the first word frequency matrix and the second word frequency matrix;

[0011] The trained language model is used to extract semantic features with context in the real label text and convert them into a first vector representation. The trained language model is used to extract semantic features with context in the output text of the urban governance multimodal large model and convert them into a second vector representation, and the semantic similarity score between the first vector representation and the second vector representation is obtained;

[0012] Based on the real label text and the output text of the urban governance multimodal large model, a formatted request general large language model with business guide words and evaluation requirements is used to obtain a high-level semantic similarity score;

[0013] The comprehensive evaluation score of the multimodal large model is obtained based on the vocabulary unit similarity score, low-level word frequency similarity, semantic similarity score, and high-level semantic similarity score.

[0014] Furthermore, the method of obtaining the real label text of the sample image and the output text of the urban governance multimodal large model includes:

[0015] Obtain sample images for managing multimodal large models and the real label text corresponding to the sample images;

[0016] The sample image is input into the multimodal large model of urban governance to obtain the output text of the multimodal large model of urban governance corresponding to the sample image.

[0017] Furthermore, the urban management vocabulary is used to segment the real label text into the first independent vocabulary, the urban management vocabulary is used to segment the output text of the urban governance multimodal large model into the second independent vocabulary, and the vocabulary unit similarity score between the first independent vocabulary and the second independent vocabulary is obtained, including:

[0018] Use professional urban management terminology to build an urban management vocabulary;

[0019] Based on the urban management vocabulary, the real label text is segmented into a first independent vocabulary, and the output text of the urban governance multimodal large model is segmented into a second independent vocabulary;

[0020] The Jaccard coefficient and / or edit distance method is used to obtain a vocabulary unit similarity score between the first independent vocabulary and the second independent vocabulary.

[0021] Furthermore, a bag-of-words model is created based on the urban management vocabulary, the real label text is converted into a first word frequency matrix, the output text of the urban governance multimodal large model is converted into a second word frequency matrix, and the low-level word frequency similarity score between the first word frequency matrix and the second word frequency matrix is ​​obtained, including:

[0022] Use the word frequency method or the word frequency-inverse document frequency method to build a bag-of-words model based on the urban management vocabulary;

[0023] Converting the real label text into a first word frequency matrix according to the bag-of-words model;

[0024] Converting the output text of the urban governance multimodal large model into a second word frequency matrix according to the bag-of-words model;

[0025] The cosine similarity method is used to obtain the low-level word frequency similarity score between the first word frequency matrix and the second word frequency matrix.

[0026] Furthermore, the trained language model is used to extract semantic features with context in the real label text and convert them into a first vector representation, the trained language model is used to extract semantic features with context in the output text of the urban governance multimodal large model and convert them into a second vector representation, and the semantic similarity score between the first vector representation and the second vector representation is obtained, including:

[0027] Use the trained language model as a word segmenter and load the trained language model;

[0028] The real label text and the output text of the urban governance multimodal large model are converted into vector representations through the trained language model to obtain a first vector representation corresponding to the real label text and a second vector representation corresponding to the output text of the urban governance multimodal large model;

[0029] Use the trained language model to extract the semantic features with context in the first vector representation corresponding to the real label text and the second vector representation corresponding to the output text of the urban governance multimodal large model, and obtain the first semantic features corresponding to the real label text and the second semantic features corresponding to the output text of the urban governance multimodal large model;

[0030] The cosine similarity method is used to obtain the semantic similarity score between the first vector representation and the second vector representation.

[0031] Furthermore, the trained language model includes a BERT layer, a semantic feature layer, and a classification result output layer connected in sequence.

[0032] Furthermore, based on the real label text and the output text of the urban governance multimodal large model, a formatted request general large language model with business guide words and evaluation requirements is used to obtain high-level semantic similarity scores, including:

[0033] According to the formatted request with business guide words and evaluation requirements, the real label text and the output text of the urban governance multimodal large model are spliced ​​into the target question;

[0034] Based on the target question, a question request is sent to the general large language model to obtain a response text result of the general large language model;

[0035] The response text result of the general large language model is parsed to extract the numerical value of the high-level semantic similarity in the response text to obtain the high-level semantic similarity score.

[0036] Furthermore, the universal large language model is a large language model LLMs having Chinese character understanding capability and text generation capability.

[0037] Furthermore, based on the vocabulary unit similarity score, low-level word frequency similarity, semantic similarity score, and high-level semantic similarity score, the comprehensive evaluation score of the multimodal large model is obtained as follows:

[0038]

[0039] Where N represents the number of sample images, i represents the index of the sample image, and α represents the influence coefficient of the vocabulary unit similarity. represents the similarity score of the vocabulary unit, β represents the influence coefficient of the low-level word frequency similarity, represents the low-level word frequency similarity, γ represents the semantic similarity influence coefficient, represents the semantic similarity score, δ represents the high-level semantic similarity influence coefficient, Represents the high-level semantic similarity score.

[0040] On the other hand, the present invention provides a multi-modal large model evaluation device for urban governance based on multi-level text, including: a text acquisition module, a vocabulary unit similarity score acquisition module, a low-level word frequency similarity score acquisition module, a semantic similarity score acquisition module, a high-level semantic similarity score acquisition module and a comprehensive evaluation score acquisition module;

[0041] The text acquisition module is used to obtain the real label text of the sample image and the output text of the urban governance multimodal large model;

[0042] The vocabulary unit similarity score acquisition module is used to use the trained language model to extract semantic features with context in the real label text and convert them into a first vector representation, use the trained language model to extract semantic features with context in the output text of the urban governance multimodal large model and convert them into a second vector representation, and obtain the semantic similarity score between the first vector representation and the second vector representation;

[0043] The low-level word frequency similarity score acquisition module is used to create a bag-of-words model based on the urban management vocabulary, convert the real label text into a first word frequency matrix, convert the output text of the urban governance multimodal large model into a second word frequency matrix, and obtain the low-level word frequency similarity score between the first word frequency matrix and the second word frequency matrix;

[0044] The semantic similarity score acquisition module is used to use the trained language model to convert the real label text and the output text of the urban governance multimodal large model into two vector representations, respectively, and extract semantic features with contextual context from the two vector representations, respectively, to obtain the first semantic feature corresponding to the real label text and the second semantic feature corresponding to the output text of the urban governance multimodal large model, and obtain the semantic similarity score between the first semantic feature and the second semantic feature;

[0045] The high-level semantic similarity score acquisition module is used to obtain a high-level semantic similarity score based on the real label text and the output text of the urban governance multimodal large model, using a formatted request general large language model with business guide words and evaluation requirements;

[0046] The comprehensive evaluation score acquisition module is used to obtain the comprehensive evaluation score of the multimodal large model based on the vocabulary unit similarity score, the low-level word frequency similarity, the semantic similarity score and the high-level semantic similarity score.

[0047] The present invention provides a method and device for evaluating a multimodal large model of urban governance based on multi-level texts. The method uses an urban management vocabulary to divide the text into independent words, calculates the similarity scores of word units, and considers the word units at the fine-grained level of the urban governance large model text output.

[0048] A bag-of-words model was created through the urban management vocabulary. The bag-of-words model was used to convert text data into a word frequency matrix, calculate the low-level word frequency similarity score, and measure the distance in the vector space of the text output sentence of the urban governance model.

[0049] By calculating the semantic similarity score through the trained language model, the context of the text output of the urban governance model is taken into account;

[0050] The high-level semantic similarity score was obtained through the general large language model, and the high-level semantic understanding ability of the urban governance large model was evaluated;

[0051] Through the comprehensive evaluation score of the multimodal large model, the vocabulary unit similarity, low-level word frequency similarity, semantic similarity and high-level semantic similarity of the output text are fully considered, effectively enhancing the accuracy of the text output evaluation of the urban governance large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without creative work. In the drawings:

[0053] Figure 1 It is a flowchart of a method for evaluating a multi-modal large model of urban governance based on multi-level text in an embodiment of the present invention;

[0054] Figure 2 A schematic diagram of multi-level spatial evaluation in an embodiment of the present invention;

[0055] Figure 3 A flowchart for obtaining a semantic similarity score in an embodiment of the present invention;

[0056] Figure 4 A flowchart for obtaining a high-level semantic similarity score in an embodiment of the present invention;

[0057] Figure 5 A schematic diagram of a formatted request with business guide words and evaluation requirements in an embodiment of the present invention;

[0058] Figure 6 It is a structural schematic diagram of a device for evaluating a multi-modal large model of urban governance based on multi-level text in an embodiment of the present invention;

[0059] Marks and corresponding parts names in the attached drawings:

[0060] 201-text acquisition module, 202-lexical unit similarity score acquisition module, 203-low-level word frequency similarity score acquisition module, 204-semantic similarity score acquisition module, 205-high-level semantic similarity score acquisition module, 206-comprehensive evaluation score acquisition module. DETAILED DESCRIPTION

[0061] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0062] like Figure 1 As shown, the embodiment of the present invention provides a method for evaluating a multimodal large model of urban governance based on multi-level text, including:

[0063] S101, obtaining the real label text of the sample image and the output text of the urban governance multimodal large model;

[0064] S102, using the urban management vocabulary to segment the real label text into first independent words, using the urban management vocabulary to segment the output text of the urban governance multimodal large model into second independent words, and obtaining the vocabulary unit similarity score between the first independent words and the second independent words;

[0065] S103. Create a bag-of-words model based on the urban management vocabulary, convert the real label text into a first word frequency matrix, convert the output text of the urban governance multimodal large model into a second word frequency matrix, and obtain the low-level word frequency similarity score between the first word frequency matrix and the second word frequency matrix;

[0066] S104, using the trained language model to extract semantic features with context in the real label text and converting them into a first vector representation, using the trained language model to extract semantic features with context in the output text of the urban governance multimodal large model and converting them into a second vector representation, and obtaining a semantic similarity score between the first vector representation and the second vector representation;

[0067] S105. Based on the real label text and the output text of the urban governance multimodal large model, a formatted request general large language model with business guide words and evaluation requirements is used to obtain a high-level semantic similarity score;

[0068] S106. Obtain a comprehensive evaluation score of the multimodal large model based on the vocabulary unit similarity score, the low-level word frequency similarity, the semantic similarity score, and the high-level semantic similarity score.

[0069] like Figure 2 As shown, it is used for Figure 1 For further explanation, the first level original text is the text content described by S101, the second level vocabulary structure is the independent vocabulary content of S102, the third level word frequency matrix is ​​the word frequency matrix content of S103, the fourth level semantic vector is the vector representation content of S104, and the fifth level high-level semantics is the high-level semantic content of S105.

[0070] In a possible implementation, the real label text of the sample image and the output text of the urban governance multimodal large model are obtained, including: obtaining a sample image used for the governance multimodal large model and the real label text corresponding to the sample image; inputting the sample image into the urban governance multimodal large model to obtain the urban governance multimodal large model output text corresponding to the sample image.

[0071] In a possible implementation, the urban governance multimodal large model is a multimodal large model obtained through fine-tuning training for a large amount of urban governance scene image data;

[0072] The multimodal large model receives visual images and question text inputs and generates answer text output results corresponding to the images.

[0073] The multimodal large model of urban governance adopted in this embodiment is based on the general large language model LLMs, and the model is obtained by parameter setting, training and processing.

[0074] In a possible implementation, the urban management vocabulary is used to segment the real label text into first independent words, the urban management vocabulary is used to segment the output text of the urban governance multimodal large model into second independent words, and the vocabulary unit similarity score between the first independent word and the second independent word is obtained, including:

[0075] Use professional urban management terminology to build an urban management vocabulary;

[0076] Based on the urban management vocabulary, the real label text is segmented into a first independent vocabulary, and the output text of the urban governance multimodal large model is segmented into a second independent vocabulary;

[0077] In a possible implementation, a Jaccard coefficient or an edit distance method is used to obtain a vocabulary unit similarity score between the first independent vocabulary and the second independent vocabulary.

[0078] Use professional urban management terms to build an urban management vocabulary, such as: "fallen bicycles", "drying items", "floating signs set up in squares", "slogan-type propaganda materials", "carport-type peddlers", "occupying roads for business", "packaged garbage", "overflowing garbage", "scattered garbage", "cracked and damaged", "randomly piled materials", "graffiti", "banner-type propaganda materials", "vender tricycles", "vender four-wheeled trucks", "occupying roads for business", "large areas of water accumulation", "scattered garbage", "abandoned furniture and equipment", "illegal street slopes", "outdoor advertising", "inflatable or arched doors", "doors of handover boxes not closed ”, "The bottom cover of the handover box is missing", "The door of the facility box is not closed", "The anti-collision barrel has fallen", "The sidewalk pile has fallen", "The guardrail has fallen off", "The road is occupied for decoration", "Medical waste", "The barrel is damaged", "The trees on the road are missing", "The fire hydrant is abnormal", "The bottom cover of the pole is missing", "Kitchen waste", "Tree protection pool", "The isolation body has fallen", "The sidewalk is damaged", "Kerbstone", "Construction site fencing", "Street seats", "Sprinkled soil", "Construction waste materials", "Traffic signs", "Scrap collection", "Poultry and livestock", "Green guardrail", "Missing manhole cover", "Missing grates".

[0079] A picture of a city street scene is input into the multimodal big model of urban governance in this implementation, and the text output of the multimodal big model is: This photo shows a vendor's four-wheeled truck parked on the side of the road, with some items on the truck and several people standing next to it, seemingly talking.

[0080] The real label text is: The picture shows a street scene at an intersection. Street vendors with carts are parked on the side of the road. The carts are full of fruits, and several passers-by are buying fruits.

[0081] For the input urban street scene picture, in this urban street scene, the first independent word extracted is the peddlers of the carport type, and the second independent word extracted is the peddlers' four-wheeled truck. Based on the extracted first independent word and the second independent word, the Jaccard coefficient is used to calculate the vocabulary unit similarity score It is 0.43.

[0082] The urban management bag-of-words model only focuses on the frequency of occurrence of terms, while ignoring the order and grammatical structure of words in the text, so the embodiment of the present invention further processes it.

[0083] In a possible implementation, a bag-of-words model is created based on the urban management vocabulary, the real label text is converted into a first word frequency matrix, the output text of the urban governance multimodal large model is converted into a second word frequency matrix, and the low-level word frequency similarity score between the first word frequency matrix and the second word frequency matrix is ​​obtained, including:

[0084] The term frequency method (TF) or term frequency-inverse document frequency method (TF-IDF) is used to construct a bag-of-words model based on the urban management vocabulary.

[0085] Converting the real label text into a first word frequency matrix according to the bag-of-words model;

[0086] Converting the output text of the urban governance multimodal large model into a second word frequency matrix according to the bag-of-words model;

[0087] The cosine similarity method is used to obtain the low-level word frequency similarity score between the first word frequency matrix and the second word frequency matrix.

[0088] If the word frequency matrix of the real label text and the output text of the urban governance multimodal large model is obtained separately, then the first word frequency matrix and the second word frequency matrix obtained are both vectors. It is worth noting that the real label text and the output text of the urban governance multimodal large model can also be used to obtain the word frequency matrix. For example, the label truth value is "the urban management problem shown in the picture is that there are tricycles of fruit and vegetable vendors engaged in mobile (non-motorized vehicle) business in public places", and the large model predicts the output text "the existing urban management problem is tricycles of fruit and vegetable vendors engaged in mobile business behavior", and the TF-IDF method is used to convert them into the word frequency matrix X, where the first row vector of the matrix corresponds to the true value of the label, and the second row vector corresponds to the output text of the large model, and then the cosine similarity method is used to calculate the similarity value.

[0089] For the above input urban street scene picture, in this urban street scene, the TF-IDF method is used to obtain the first word frequency matrix, and the TF-IDF method is used to obtain the second word frequency matrix, and the low-level word frequency similarity is calculated. is 0.51.

[0090] In a possible implementation, a trained language model is used to extract semantic features with context in real label text and convert them into a first vector representation, and a trained language model is used to extract semantic features with context in the output text of the urban governance multimodal large model and convert them into a second vector representation, and a semantic similarity score between the first vector representation and the second vector representation is obtained, including: using the trained language model as a word segmenter and loading the trained language model; converting the real label text and the output text of the urban governance multimodal large model into vector representations through the language model word segmenter to obtain a first vector representation corresponding to the real label text and a second vector representation corresponding to the output text of the urban governance multimodal large model; using the trained language model to extract semantic features with context in the first vector representation corresponding to the real label text and the second vector representation corresponding to the output text of the urban governance multimodal large model, to obtain a first semantic feature corresponding to the real label text and a second semantic feature corresponding to the output text of the urban governance multimodal large model; and using the cosine similarity method to obtain the semantic similarity score between the first vector representation and the second vector representation.

[0091] The specific calculation process of the above semantic similarity score includes:

[0092] The trained language model is used to extract semantic features with context from the real label text and convert it into the first vector representation, and then the semantic similarity score is calculated;

[0093] Among them, the trained language model adds a semantic feature layer and a classification result output layer based on the trained language model. The trained language model is a natural language processing model based on self-attention to extract long-distance dependencies of texts, and the classification labels of the training corpus data are the contents of the urban management vocabulary.

[0094] The pre-trained language model uses BERT (Bidirectional Encoder Representations from Transformers). The BERT model is built on the encoder part of the Transformer and uses a bidirectional training method. It can consider the contextual information before and after the words at the same time, which enables the model to obtain a richer contextual understanding. A semantic feature layer (feature length is 512) is added to the final output layer of BERT, and then a classification result output layer is added (the output result is classified into 50 categories, which is the content of the urban management vocabulary). The training corpus data uses the text data in the training urban governance model, and the classification label uses the content of the urban management vocabulary, including the following steps:

[0095] Use the trained language model as a tokenizer and load the trained language model;

[0096] The language model tokenizer encodes the text to be tested and converts the text into a vector representation;

[0097] Use the trained language model to extract the vector representation of the penultimate layer output of the model as a semantic feature with context;

[0098] Calculate the cosine similarity score for the semantic features.

[0099] In a possible implementation, the trained language model includes a BERT layer, a semantic feature layer, and a classification result output layer connected sequentially.

[0100] like Figure 3 As shown, the dotted line is above the training mode, the dotted line is below the inference mode, and the pre-trained language model uses BERT (Bidirectional Encoder Representations from Transformers). The BERT model is built based on the Encoder part of the Transformer and uses a bidirectional training method. It can consider the contextual information before and after the vocabulary at the same time, which enables the model to obtain a richer contextual understanding. A semantic feature layer (feature length is 512) is added to the final output layer of BERT, and then a classification result output layer is added (the output result is classified into 50 categories, that is, the content of the urban management vocabulary). The training corpus data uses the text data in the training urban governance model, and the classification label uses the content of the urban management vocabulary.

[0101] Based on the above input urban street scene pictures, in the urban street scene, the scene description is input into the trained language model, the semantic features with context are extracted respectively, and the semantic similarity score Sim is calculated. isema It is 0.78.

[0102] In a possible implementation, Figure 4 As shown, based on the real label text and the output text of the urban governance multimodal large model, a formatted request general large language model with business guide words and evaluation requirements is used to obtain high-level semantic similarity scores, including:

[0103] According to the formatted request with business guide words and evaluation requirements, the real label text and the output text of the urban governance multimodal large model are spliced ​​into the target question;

[0104] Based on the target question, a question request is sent to the general large language model to obtain a response text result of the general large language model;

[0105] The response text result of the general large language model is parsed to extract the numerical value of the high-level semantic similarity in the response text to obtain the high-level semantic similarity score.

[0106] In this implementation, the general large language model LLMs refers to a general large language model based on a generative pre-trained Transformer, which has the ability to understand Chinese characters and generate text. The general large language model LLMs can use OpenAI's GPT-4, Meta AI's LLaMA3, Tsinghua University's ChatGLM-6B, or Ali Qianwen Qwen.

[0107] The high-level semantic similarity score ranges from 0 to 1.0, including the boundary values, where 0 indicates completely opposite semantics and 1.0 indicates exactly the same semantics.

[0108] like Figure 5 As shown, the formatted request with business guide words and evaluation requirements is as follows: Sentence 1: A. Sentence 2: B. Please evaluate the semantic similarity of the above two sentences from the perspective of urban management business and give a score between 0 and 1.0, where 0 means completely different and 1 means completely the same. A and B represent the true value text of the label and the output text of the large model prediction, respectively.

[0109] In some possible implementations, a city street scene picture is input into the city governance multimodal big model in this implementation (the picture is the same city street scene picture mentioned above, and this sentence should be repeated four times), and the multimodal big model text output is: The city street scene picture shows a small truck parked on the side of the road, with some items on the truck, and several people standing next to it seem to be talking. The real label text is: The picture shows a street scene at an intersection, a silver pickup truck is parked on the side of the road, the truck bed is full of fruits, and several passers-by are buying fruits.

[0110] After processing the city street scene picture, we get the following formatted request with business guide words and evaluation requirements:

[0111] Sentence 1: This photo shows a pickup truck parked on the side of the road, carrying some items, and several people standing next to it, seemingly talking.

[0112] Sentence 2: The picture shows a street scene at an intersection. A silver pickup truck is parked on the side of the road. The truck bed is full of fruits, and several passers-by are buying fruits.

[0113] Please evaluate the semantic similarity of the above two sentences and give a score between 0 and 1.0, where 0 means completely different and 1 means exactly the same.

[0114] Based on the above input urban street scene picture, in the urban street scene, according to the formatted request with business guide words and evaluation requirements, the real label text and the output text of the urban governance multimodal large model are spliced ​​into the target question. Based on the target question, a question request is sent to the general large language model to obtain the response text result of the general large language model, and the response text result of the general large language model is parsed to extract the numerical value of the high-level semantic similarity in the response text to obtain the high-level semantic similarity score. is 0.8.

[0115] In a possible implementation, based on the vocabulary unit similarity score, the low-level word frequency similarity, the semantic similarity score, and the high-level semantic similarity score, the comprehensive evaluation score of the multimodal large model is obtained as follows:

[0116]

[0117] Among them, ACC represents the comprehensive evaluation score, N represents the number of sample images, i represents the index of the sample image, and α represents the influence coefficient of the vocabulary unit similarity. represents the similarity score of the vocabulary unit, β represents the influence coefficient of the low-level word frequency similarity, represents the low-level word frequency similarity, γ represents the semantic similarity influence coefficient, represents the semantic similarity score, δ represents the high-level semantic similarity influence coefficient, Represents the high-level semantic similarity score.

[0118] In some possible implementations, a specific numerical example is given, in which the weight of high-level semantics is the largest, and the high-level semantic similarity influence coefficient δ can be set to 0.5, the weight of semantic similarity is second, and the semantic similarity influence coefficient γ can be set to 0.3, the weight of low-level word frequency similarity is third, and the low-level word frequency similarity influence coefficient β can be set to 0.15, the weight of vocabulary unit similarity is the smallest, and the vocabulary unit similarity influence coefficient α can be set to 0.05; based on this, the vocabulary unit similarity score Sim can be obtained. word is 0.43, the low-level word frequency similarity score Sim freq is 0.51, and the semantic similarity score Sim sema is 0.78, and the high-level semantic similarity score Sim llms is 0.8.

[0119] Based on the above specific values, the result calculated by the formula of the comprehensive evaluation score is 0.732. If the low-level word frequency similarity is used alone as a measurement indicator, the similarity is only 0.51. The higher the comprehensive score, the higher the accuracy of the multimodal large model text output. Obviously, the calculation result of 0.732 in the embodiment of the present invention can better reflect the accuracy of the multimodal large model text output.

[0120] like Figure 6 As shown, the embodiment of the present invention provides a multi-modal large model evaluation device for urban governance based on multi-level text, including: a text acquisition module 201, a vocabulary unit similarity score acquisition module 202, a low-level word frequency similarity score acquisition module 203, a semantic similarity score acquisition module 204, a high-level semantic similarity score acquisition module 205 and a comprehensive evaluation score acquisition module 206;

[0121] The text acquisition module 201 is used to obtain the real label text of the sample image and the output text of the urban governance multimodal large model;

[0122] The vocabulary unit similarity score acquisition module 202 is used to use the urban management vocabulary to segment the real label text into first independent vocabulary, use the urban management vocabulary to segment the urban governance multimodal large model output text into second independent vocabulary, and obtain the vocabulary unit similarity score between the first independent vocabulary and the second independent vocabulary;

[0123] The low-level word frequency similarity score acquisition module 203 is used to create a bag-of-words model based on the urban management vocabulary, convert the real label text into a first word frequency matrix, convert the output text of the urban governance multimodal large model into a second word frequency matrix, and obtain the low-level word frequency similarity score between the first word frequency matrix and the second word frequency matrix;

[0124] The semantic similarity score acquisition module 204 is used to use the trained language model to convert the real label text and the output text of the urban governance multimodal large model into two vector representations, respectively, and extract semantic features with contextual context from the two vector representations, respectively, to obtain the first semantic feature corresponding to the real label text and the second semantic feature corresponding to the output text of the urban governance multimodal large model, and obtain the semantic similarity score between the first semantic feature and the second semantic feature;

[0125] The high-level semantic similarity score acquisition module 205 is used to obtain a high-level semantic similarity score based on the real label text and the output text of the urban governance multimodal large model, using a formatted request general large language model with business guide words and evaluation requirements;

[0126] The comprehensive evaluation score acquisition module 206 is used to obtain the comprehensive evaluation score of the multimodal large model according to the vocabulary unit similarity score, the low-level word frequency similarity, the semantic similarity score and the high-level semantic similarity score.

[0127] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The schemes in the embodiments of the present invention may be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0128] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0129] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0130] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0131] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0132] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A multi-modal large model evaluation method for urban governance based on multi-level text, characterized in that: include: Obtain the real label text of the sample image and the output text of the urban governance multimodal large model; The urban management vocabulary is used to segment the real label text into first independent words, the urban management vocabulary is used to segment the output text of the urban governance multimodal large model into second independent words, and the vocabulary unit similarity score between the first independent words and the second independent words is obtained; Create a bag-of-words model based on the urban management vocabulary, convert the real label text into the first word frequency matrix, convert the output text of the urban governance multimodal large model into the second word frequency matrix, and obtain the low-level word frequency similarity score between the first word frequency matrix and the second word frequency matrix; The trained language model is used to extract semantic features with context in the real label text and convert them into a first vector representation. The trained language model is used to extract semantic features with context in the output text of the urban governance multimodal large model and convert them into a second vector representation, and the semantic similarity score between the first vector representation and the second vector representation is obtained. Based on the real label text and the output text of the urban governance multimodal large model, a formatted request general large language model with business guide words and evaluation requirements is used to obtain a high-level semantic similarity score; The comprehensive evaluation score of the multimodal large model is obtained based on the vocabulary unit similarity score, low-level word frequency similarity, semantic similarity score, and high-level semantic similarity score.

2. According to the text-based multi-level urban governance multimodal large model evaluation method of claim 1, it is characterized by: The method of obtaining the real label text of the sample image and the output text of the urban governance multimodal large model includes: Obtain sample images for managing multimodal large models and the real label text corresponding to the sample images; The sample image is input into the multimodal large model of urban governance to obtain the output text of the multimodal large model of urban governance corresponding to the sample image.

3. The method for evaluating a multimodal large model of urban governance based on multi-level text according to claim 1 is characterized in that: The urban management vocabulary is used to segment the real label text into the first independent vocabulary, and the urban management vocabulary is used to segment the output text of the urban governance multimodal large model into the second independent vocabulary, and the vocabulary unit similarity score between the first independent vocabulary and the second independent vocabulary is obtained, including: Use professional urban management terminology to build an urban management vocabulary; Based on the urban management vocabulary, the real label text is segmented into a first independent vocabulary, and the output text of the urban governance multimodal large model is segmented into a second independent vocabulary; The Jaccard coefficient and / or edit distance method is used to obtain a vocabulary unit similarity score between the first independent vocabulary and the second independent vocabulary.

4. According to the text-based multi-level urban governance multimodal large model evaluation method of claim 1, it is characterized in that: A bag-of-words model is created based on the urban management vocabulary, the real label text is converted into the first word frequency matrix, the output text of the urban governance multimodal large model is converted into the second word frequency matrix, and the low-level word frequency similarity score between the first word frequency matrix and the second word frequency matrix is ​​obtained, including: Use the word frequency method or the word frequency-inverse document frequency method to build a bag-of-words model based on the urban management vocabulary; Converting the real label text into a first word frequency matrix according to the bag-of-words model; Converting the output text of the urban governance multimodal large model into a second word frequency matrix according to the bag-of-words model; The cosine similarity method is used to obtain the low-level word frequency similarity score between the first word frequency matrix and the second word frequency matrix.

5. According to the text-based multi-level urban governance multimodal large model evaluation method of claim 1, it is characterized by: The trained language model is used to extract the semantic features with context in the real label text and convert it into a first vector representation. The trained language model is used to extract the semantic features with context in the output text of the urban governance multimodal large model and convert it into a second vector representation. The semantic similarity score between the first vector representation and the second vector representation is obtained, including: Use the trained language model as a word segmenter and load the trained language model; The real label text and the output text of the urban governance multimodal large model are converted into vector representations through the trained language model to obtain a first vector representation corresponding to the real label text and a second vector representation corresponding to the output text of the urban governance multimodal large model; The cosine similarity method is used to obtain the semantic similarity score between the first vector representation and the second vector representation.

6. The method for evaluating a multimodal large model of urban governance based on multi-level text according to claim 5 is characterized in that: The trained language model includes a BERT layer, a semantic feature layer, and a classification result output layer connected in sequence.

7. The method for evaluating a multimodal large model of urban governance based on multi-level text according to claim 1 is characterized in that: Based on the real label text and the output text of the urban governance multimodal large model, a formatted request general large language model with business guide words and evaluation requirements is used to obtain high-level semantic similarity scores, including: According to the formatted request with business guide words and evaluation requirements, the real label text and the output text of the urban governance multimodal large model are spliced ​​into the target question; Based on the target question, a question request is sent to the general large language model to obtain a response text result of the general large language model; The response text result of the general large language model is parsed to extract the numerical value of the high-level semantic similarity in the response text to obtain the high-level semantic similarity score.

8. The method for evaluating a multimodal large model of urban governance based on multi-level text according to claim 7 is characterized in that: The universal large language model is a large language model LLMs with Chinese character understanding and text generation capabilities.

9. The method for evaluating a multimodal large model of urban governance based on multi-level text according to claim 1 is characterized in that: According to the vocabulary unit similarity score, low-level word frequency similarity, semantic similarity score and high-level semantic similarity score, the comprehensive evaluation score of the multimodal large model is obtained as follows: Among them, ACC represents the comprehensive evaluation score, N represents the number of sample images, i represents the index of the sample image, and α represents the influence coefficient of the vocabulary unit similarity. represents the similarity score of the vocabulary unit, β represents the influence coefficient of the low-level word frequency similarity, represents the low-level word frequency similarity, γ represents the semantic similarity influence coefficient, represents the semantic similarity score, δ represents the high-level semantic similarity influence coefficient, Represents the high-level semantic similarity score.

10. A multi-modal large model evaluation device for urban governance based on multi-level text, characterized in that: It includes: text acquisition module, vocabulary unit similarity score acquisition module, low-level word frequency similarity score acquisition module, semantic similarity score acquisition module, high-level semantic similarity score acquisition module and comprehensive evaluation score acquisition module; The text acquisition module is used to obtain the real label text of the sample image and the output text of the urban governance multimodal large model; The vocabulary unit similarity score acquisition module is used to use the urban management vocabulary to segment the real label text into first independent vocabulary, use the urban management vocabulary to segment the urban governance multimodal large model output text into second independent vocabulary, and obtain the vocabulary unit similarity score between the first independent vocabulary and the second independent vocabulary; The low-level word frequency similarity score acquisition module is used to create a bag-of-words model based on the urban management vocabulary, convert the real label text into a first word frequency matrix, convert the output text of the urban governance multimodal large model into a second word frequency matrix, and obtain the low-level word frequency similarity score between the first word frequency matrix and the second word frequency matrix; The semantic similarity score acquisition module is used to extract semantic features with context in the real label text using the trained language model and convert them into a first vector representation, extract semantic features with context in the output text of the urban governance multimodal large model using the trained language model and convert them into a second vector representation, and obtain the semantic similarity score between the first vector representation and the second vector representation; The high-level semantic similarity score acquisition module is used to obtain a high-level semantic similarity score based on the real label text and the output text of the urban governance multimodal large model, using a formatted request general large language model with business guide words and evaluation requirements; The comprehensive evaluation score acquisition module is used to obtain the comprehensive evaluation score of the multimodal large model based on the vocabulary unit similarity score, the low-level word frequency similarity, the semantic similarity score and the high-level semantic similarity score.

Images