Urban governance multi-modal large model evaluation method and device based on text tagging

Abstract

The invention discloses an urban governance multi-modal large model evaluation method and device based on text tagging, and relates to the technical field of data processing and smart cities. Sample data are obtained, sample pictures and corresponding text tags are obtained, the sample pictures are input into a model, an output text corresponding to the model is obtained, and the output text corresponding to the model is obtained; extracting object existence, classification labels and target position information in the output text to obtain a structured dictionary list, and obtaining an accuracy index of the model in a single sample picture according to data in the structured dictionary list and the text labels corresponding to the sample pictures; and obtaining a comprehensive evaluation index of the model on the sample data according to the accuracy indexes of the plurality of single sample pictures. According to the method, the object judgment capability, the label classification capability and the target positioning capability of the multi-modal urban governance multi-modal large model are fully considered, the problem that plain text similarity evaluation indexes are unreliable is solved, and the accuracy of urban governance multi-modal large model evaluation is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing and smart city technology, and in particular to a method and device for evaluating a multimodal large model of urban governance based on text labeling. Background Art

[0002] In recent years, big models have made breakthrough progress. Big models refer to complex artificial neural network models with ultra-large-scale parameters, usually containing hundreds of millions to trillions of parameters. Big models have shown wide application potential in many fields due to their powerful learning and generalization capabilities. For example, OpenAI's GPT series of big models can generate high-quality text content such as articles, stories, and news reports. Based on the application potential of big models, multimodal big models have been further developed on the basis of big models. Multimodal big models can not only process text data, but also process information of different modes such as pictures, sounds, and videos at the same time. For example, multimodal big models can be used to understand the content of a picture and generate corresponding text descriptions based on it. Based on the ability of multimodal big models to process complex pictures, a multimodal big model for urban governance is developed. Relying on the general big model as the foundation, a multimodal application customized for solving urban management scenarios is developed. For example, the multimodal big model for urban governance can recognize urban street scene pictures and output a description of urban management.

[0003] At present, the existing evaluation methods of urban governance multimodal large models mostly use text similarity-related methods to calculate the difference between the predicted output results of the urban governance multimodal large model and the real description. For example, the predicted output result of the urban governance multimodal large model is that the picture shows that there are peddlers selling goods [[188,411,488,871]], while the real description of the picture is that there are no illegal unlicensed peddlers in the picture [[190,425,480,866]]. If the calculation method of pure text similarity is used, the accuracy of the predicted result and the real result is 0.75. From a semantic point of view, there are problems in the output picture of the urban governance multimodal large model, but there are no problems in the picture in the real description. Obviously, the difference between the predicted result and the real result is huge. Therefore, using pure text similarity to evaluate the urban governance multimodal large model is unreliable in the urban governance scenario. Summary of the invention

[0004] The purpose of the present invention is to provide a method and device for evaluating a multimodal large model of urban governance based on text labeling, which solves the problems existing in the prior art.

[0005] The present invention is achieved through the following technical solutions:

[0006] On the one hand, the present invention provides a method for evaluating a multimodal large model of urban governance based on text labeling, comprising:

[0007] Obtain sample data for evaluating a large multimodal model of urban governance; wherein the sample data includes sample images and text labels corresponding to the sample images;

[0008] Input the sample images in the sample data into the urban governance multimodal large model to obtain the output text corresponding to the urban governance multimodal large model;

[0009] Extracting object existence, classification labels, and target location information from the output text, and obtaining a structured dictionary list based on the object existence, classification labels, and target location information;

[0010] According to the data in the structured dictionary list and the text labels corresponding to the sample images, the accuracy index of the urban governance multimodal large model in a single sample image is obtained;

[0011] Based on the accuracy indicators of multiple single sample images, comprehensive evaluation indicators for evaluating the multimodal large model of urban governance on sample data are obtained.

[0012] In the embodiment of the present invention, the urban governance multimodal large model is a multimodal large model obtained through fine-tuning training for a large amount of urban governance scene image data;

[0013] The multimodal large model receives visual images and question text inputs and generates text corresponding to the images.

[0014] In an embodiment of the present invention, extracting object existence, classification labels, and target location information in the output text includes:

[0015] For the description corresponding to any target in the output text, a keyword recognition method is used to identify the existence and classification label of the object corresponding to the target;

[0016] For the description corresponding to any target in the output text, an identifier recognition method is used to identify the target position information corresponding to the target;

[0017] Traverse each target in the output text to obtain the object existence, classification label and target location information corresponding to each target.

[0018] In the embodiment of the present invention, the structured dictionary list is:

[0019] Build a list, each element in the list corresponds to a picture, and the elements of the list are structured dictionaries;

[0020] The structured dictionary list stores image names, image size attributes, existence keywords, label keywords, and target boxes.

[0021] In an embodiment of the present invention, according to the data in the structured dictionary list and the text labels corresponding to the sample images, the accuracy index of the urban governance multimodal large model in a single sample image is obtained as follows:

[0022]

[0023] Among them, IOU i represents the target box intersection-union ratio of the i-th target in a single sample image, represents the true classification label of the i-th target in a single sample image, Represents the predicted classification label of the i-th target in the text label corresponding to the sample image, Represents the predicted classification label and the true classification label The cosine similarity of the texts between A quantitative value indicating whether the target exists in the real result. A quantitative value indicating whether the target exists in the prediction result corresponding to the text label. N represents the total number of targets in a single sample image.

[0024] In the embodiment of the present invention, the comprehensive evaluation index for evaluating the urban governance multimodal large model on sample data is obtained based on the accuracy index of multiple single sample images:

[0025]

[0026] Among them, M represents the total number of sample images in the sample data, λ represents the influence coefficient of single accuracy, Indicates the text label corresponding to the jth sample image in the sample data, The jth sample image in the sample data corresponds to the text description of the prediction result of the urban governance multimodal large model, Indicates the actual result text description And the prediction result text description The cosine similarity of the texts between them, η represents the influence coefficient of text similarity, is the accuracy of a single sample image.

[0027] On the other hand, the present invention provides an urban governance multimodal large model evaluation device based on text labeling, comprising: a data acquisition module, an output acquisition module, a structured dictionary acquisition module, a first indicator acquisition module, a second indicator acquisition module and an evaluation result acquisition module;

[0028] The data acquisition module is used to acquire sample data for evaluating the multimodal large model of urban governance; wherein the sample data includes sample images and text labels corresponding to the sample images;

[0029] The output acquisition module is used to input the sample images in the sample data into the urban governance multimodal large model to obtain the output text corresponding to the urban governance multimodal large model;

[0030] The structured dictionary acquisition module is used to construct a list, each element in the list corresponds to a picture, and the elements of the list are structured dictionaries; wherein the structured dictionary list stores picture names, picture size attributes, existence keywords, label keywords and target boxes;

[0031] The first indicator acquisition module is used to obtain the accuracy indicator of the urban governance multimodal large model in a single sample image according to the data in the structured dictionary list and the text label corresponding to the sample image;

[0032] The second indicator acquisition module is used to obtain a comprehensive evaluation indicator for evaluating the urban governance multimodal large model on sample data based on the accuracy indicator of the single sample image.

[0033] In the embodiment of the present invention, the structured dictionary acquisition module includes a data extraction unit and a dictionary acquisition unit;

[0034] The data extraction unit is used to identify the object existence and classification label corresponding to any target in the output text by using a keyword recognition method; identify the target location information corresponding to any target in the output text by using an identifier recognition method; traverse each target in the output text to obtain the object existence, classification label and target location information corresponding to each target;

[0035] The dictionary acquisition unit is used to construct a list, each list element corresponds to a picture, and the list element stores object existence, classification label and target location information, to obtain a structured dictionary stored in the list.

[0036] In the embodiment of the present invention, the first indicator acquisition module includes: a prediction data acquisition unit, a real data acquisition unit and an accuracy indicator acquisition unit;

[0037] The prediction data acquisition unit is used to determine whether there are quantized values ​​for the prediction box, the prediction classification label, and the prediction result in the single sample image according to the data in the structured dictionary, and obtain the prediction data corresponding to the single sample image;

[0038] The real data acquisition unit is used to determine whether there are quantized values ​​for the real prediction box, the real classification label, and the real result in the text label according to the text label corresponding to the sample image, and obtain the real data corresponding to the single sample image;

[0039] The accuracy index acquisition unit is used to obtain the accuracy index of a single sample image according to the predicted data corresponding to the single sample image and the real data corresponding to the single sample image:

[0040]

[0041] Among them, IOU i represents the target box intersection-union ratio of the i-th target in a single sample image, represents the true classification label of the i-th target in a single sample image, Represents the predicted classification label of the i-th target in the text label corresponding to the sample image, Represents the predicted classification label and the true classification label The cosine similarity of the texts between A quantitative value indicating whether the target exists in the real result. A quantitative value indicating whether the target exists in the prediction result corresponding to the text label. N represents the total number of targets in a single sample image.

[0042] In an embodiment of the present invention, the second indicator acquisition module includes a text similarity prediction unit and a comprehensive evaluation indicator acquisition unit;

[0043] The text similarity prediction unit is used to determine the text cosine similarity between the real result text description and the predicted result text description according to the sample pictures in the sample data and the text labels corresponding to the sample pictures;

[0044] The comprehensive evaluation index acquisition unit is used to determine the comprehensive evaluation index of the urban governance multimodal large model according to the text cosine similarity and accuracy index corresponding to multiple single sample images:

[0045]

[0046] Among them, M represents the total number of sample images in the sample data, λ represents the influence coefficient of single accuracy, Indicates the text label corresponding to the jth sample image in the sample data, The jth sample image in the sample data corresponds to the text description of the prediction result of the urban governance multimodal large model, Indicates the actual result text description And the prediction result text description The cosine similarity of the texts between them, η represents the influence coefficient of text similarity, is the accuracy of a single sample image.

[0047] The present invention provides a method and device for evaluating a multimodal large model of urban governance based on text labeling, which extracts object existence, classification labels and target location information from the output text of the multimodal large model of urban governance; stores the extraction results in a structured dictionary list, and obtains the accuracy index of a single image according to the data recorded in the structured dictionary list; then considers the pure text similarity and calculates the comprehensive evaluation index, which fully considers the object judgment ability, label classification ability and target positioning ability of the multimodal large model of multimodal urban governance, overcomes the problem of unreliable pure text similarity evaluation index, and greatly improves the accuracy of the evaluation of the multimodal large model of urban governance. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without creative work. In the drawings:

[0049] Figure 1 A flowchart of a method for evaluating a multimodal large model of urban governance based on text labeling provided in an embodiment of the present invention;

[0050] Figure 2 A schematic diagram of the structure of a large-scale urban governance multimodal model evaluation device based on text labeling provided by an embodiment of the present invention;

[0051] Figure 3 A schematic diagram of a structured dictionary list provided by an embodiment of the present invention;

[0052] Figure 4 A schematic diagram of calculating the accuracy index of a single image provided by an embodiment of the present invention.

[0053] Marks and corresponding parts names in the attached drawings:

[0054] Among them, 201 - data acquisition module, 202 - output acquisition module, 203 - structured dictionary acquisition module, 204 - first indicator acquisition module, 205 - second indicator acquisition module. DETAILED DESCRIPTION

[0055] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0056] like Figure 1As shown, an embodiment of the present invention provides a method for evaluating a multimodal large model of urban governance based on text labeling, comprising:

[0057] S101. Obtain sample data for evaluating a large multimodal model for urban governance; wherein the sample data includes sample images and text labels corresponding to the sample images;

[0058] S102, inputting the sample image in the sample data into the urban governance multimodal large model to obtain the output text corresponding to the urban governance multimodal large model;

[0059] S103, extracting object existence, classification labels and target location information in the output text, and obtaining a structured dictionary list according to the object existence, classification labels and target location information;

[0060] S104. Obtain the accuracy index of the urban governance multimodal large model in a single sample image according to the data in the structured dictionary list and the text label corresponding to the sample image, that is, the true value label information;

[0061] S105. Based on the accuracy indicators of multiple single sample images, obtain comprehensive evaluation indicators for evaluating the multimodal large model of urban governance on sample data.

[0062] In the embodiment of the present invention, the multiple single sample pictures in S105 include all pictures of the sample.

[0063] By taking the structured dictionary list, accuracy index and comprehensive evaluation index as the evaluation results, the staff can verify, check and trace the evaluation results.

[0064] The present embodiment provides a method for evaluating a multimodal large model of urban governance based on text labeling, which extracts object existence, classification labels and target location information from the output text of the multimodal large model of urban governance; stores the extraction results in a structured dictionary list, and obtains the accuracy index of a single image based on the data recorded in the structured dictionary list; then considers the pure text similarity and calculates the comprehensive evaluation index, which fully considers the object judgment ability, label classification ability and target positioning ability of the multimodal large model of multimodal urban governance, overcomes the problem of unreliable pure text similarity evaluation indicators, and greatly improves the accuracy of the evaluation of the multimodal large model of urban governance.

[0065] In the embodiment of the present invention, the structured dictionary list, the accuracy index and the comprehensive evaluation index are used to avoid data isolation, so the data in the structured dictionary list is used to obtain the index.

[0066] In the embodiment of the present invention, the urban governance multimodal large model is a multimodal large model obtained through fine-tuning training for a large amount of urban governance scene image data;

[0067] The multimodal large model receives visual images and question text inputs and generates text corresponding to the images.

[0068] This embodiment can adopt Qwen-VL and CogVLM-17B. Qwen-VL and CogVLM-17B are a kind of multimodal large model, which belongs to the base large model. The urban governance multimodal large model in this embodiment can be based on this model, and the model can be trained and optimized to obtain the urban governance multimodal large model.

[0069] In an embodiment of the present invention, extracting object existence, classification labels, and target location information in the output text includes:

[0070] For the description corresponding to any target in the output text, a keyword recognition method is used to identify the existence and classification label of the object corresponding to the target;

[0071] For the description corresponding to any target in the output text, an identifier recognition method is used to identify the target position information corresponding to the target;

[0072] Traverse each target in the output text to obtain the object existence, classification label and target location information corresponding to each target.

[0073] Optionally, in order for those skilled in the art to better understand the technical solution described in the embodiments of the present invention, the technical solution of the present invention is described below by way of examples.

[0074] The output text of the urban governance multimodal large model corresponds one-to-one to the sample images of the model input. The output text of the urban governance multimodal large model includes existence description, classification label description and target location description. If there are multiple targets and multiple labels, the descriptive text corresponds one-to-one to the number of targets or the number of labels. If the text output by the urban governance multimodal large model does not meet the requirements, exception handling is performed.

[0075] Each target and each label corresponds to a separate sentence. Taking the case of multiple targets as an example, the output text of the urban governance multimodal large model should be "The picture shows that the urban management problem is that there are shops (including various kiosks) on the sidewalk that place items outside the shops and cross the doors to occupy the road for business [[603,121,761,360]]", where [[603,121,761,360]] corresponds to [Xmin,Ymin,Xmax,Ymax]], which represents a rectangular positioning box, where Xmin is the x-coordinate of the upper left corner of the positioning box, Ymin is the y-coordinate of the upper left corner of the positioning box, Xmax is the x-coordinate of the lower right corner of the positioning box, and Ymax is the y-coordinate of the lower right corner of the positioning box. There are shops (including various kiosks) on the sidewalk that place items outside the shops and cross the doors to occupy the road for business [[367,175,497,325]]. It can be found that the prediction output of the urban governance multimodal model contains two targets (occupancy of road operations [[603,121,761,360]] and occupation of road operations [[367,175,497,325]]). The urban governance multimodal model outputs text with a period as a string delimiter. Each string after segmentation can only contain one target. If multiple labels or multiple positioning boxes appear in the segmented string, the urban governance multimodal model is forced to regenerate the text output according to the format requirements.

[0076] The embodiment of the present invention defines "existence" and "non-existence" as existence description keywords. For example, the urban governance multimodal large model outputs the text "The picture shows that the urban management problem is that there are unauthorized installation of illegal circular-framed long-base slogan-type propaganda materials [[271,473,346,773]]", then the existence information of a single object can be extracted based on the keyword "existence". For another example, the urban governance multimodal large model outputs the text "There is no packaged garbage that has not been poured into garbage containers in public places", then the existence information of a single object can be extracted based on the keyword "non-existence".

[0077] The embodiment of the present invention defines the label text as a classification label description keyword, such as: "fallen bicycle", "drying items", "ground signboards set up in the square", "slogan propaganda materials", "carport-type peddlers", "occupying the road for business", "packaged garbage", "overflowing garbage", "scattered garbage", "cracked and damaged", "randomly piled materials", "shared bicycles", "exposed loess", "graffiti", "banner-type propaganda materials", "peddlers' tricycles", "peddlers' four-wheeled trucks", "peddlers occupying the road for business", "randomly parked non-motor vehicles", "large areas of water accumulation". For example, if the output text of the urban governance multimodal large model is "there are carport-type peddlers engaged in mobile business in public places [[752,522,999,815]]", the single object classification label information "carport-type peddlers" can be extracted.

[0078] In the embodiment of the present invention, "[[]]" is defined as a target location description identifier. For example, if the urban governance multimodal large model outputs the text "there are box-like piles of materials [[445,735,538,907]] in public places", the single object target location information "445,735,538,907" can be extracted.

[0079] Continue to cycle the extraction process of a single target and extract the existence, classification labels and target location information of all targets from the output text of the urban governance multimodal large model.

[0080] In an embodiment of the present invention, the structured dictionary list is:

[0081] Build a list, each element in the list corresponds to a picture, and the elements of the list are structured dictionaries;

[0082] Among them, the structured dictionary list stores the image name, image size attribute, existence keyword, label keyword and target box;

[0083] Specific:

[0084] like Figure 3As shown, an embodiment of the present invention provides a schematic diagram of a structured dictionary list, where the number of pictures is N, and the length of the list is N. The elements of the list are structured dictionaries dict_elei, where the subscript i represents the picture index. The keys of the structured dictionary dict_ele include: picture name Name, picture size Size, existence Obj, label Label, and target box Box. If the target label does not exist in the picture, the existence keyword key-value pair is Obj: 0. Taking two pictures as an example to illustrate the structure of the structured dictionary list, the output of the multimodal large model of urban governance corresponding to Picture 1 is "The picture shows that the problem of urban management is that there is bare loess [[000,826,170,988]] that is not covered by green vegetation, then the structured dictionary dict_ele1 is {Name: "name1.jpg", Size: [1600,900], Obj: 1, Label: "bare loess", Box: [000,826,170,988]}. The output of the urban governance multimodal large model corresponding to Picture 2 is "The picture shows that the urban management problem is that there is packaged garbage [[245,833,305,916]]] in public places that has not been dumped into garbage containers", then the structured dictionary dict_ele2 is {Name: "name2.jpg", Size: [1600,900], Obj: 1, Label: "Packaged garbage", Box: [245,833,305,916]}. Then the output result of all pictures is a list [{Name:“name1.jpg”,Size:[1600,900],Obj:1,Label:“bare loess”,Box:[000,826,170,988]},{Name:“name2.jpg”,Size:[1600,900],Obj:1,Label:“packaged garbage”,Box:[245,833,305,916]}].

[0085] This invention facilitates the multimodal large model of urban governance by constructing a structured dictionary

[0086] In an embodiment of the present invention, according to the data in the structured dictionary list and the text labels corresponding to the sample images, the accuracy index of the urban governance multimodal large model in a single sample image is obtained as follows:

[0087]

[0088] Among them, IOU i represents the target box intersection-union ratio of the i-th target in a single sample image, represents the true classification label of the i-th target in a single sample image, Represents the predicted classification label of the i-th target in the text label corresponding to the sample image, Represents the predicted classification label and the true classification label The cosine similarity of the texts between A quantitative value indicating whether the target exists in the real result. A quantitative value indicating whether the target exists in the prediction result corresponding to the text label. If the target exists in the result, obj i = 1, if the target does not exist, then obj i =0, N represents the total number of targets in a single sample image.

[0089] like Figure 4 As shown, an embodiment of the present invention provides a schematic diagram for calculating the accuracy index of a single image. The true value of the image is "The image shows that there are no illegal unlicensed vendors [[190,425,480,866]].", and the output prediction value of the urban governance multimodal large model is "The image shows that there are vendors selling goods [[188,411,488,871]].". Since the existence description in the true value is "non-existent", the quantitative value obj of whether the target exists in the true result is t is 0. Since the existence description in the predicted value is "existence", the quantitative value obj of whether the target exists in the prediction result p is 1. Similarly, the true classification label lab t For unlicensed peddlers, predict the classification label lab p The target box intersection over union (IOU) of the true value coordinates "[190,425,480,866]" and the predicted value coordinates "[188,411,488,871]" is 0.55. The number of targets in the image N is 1. Substituting the above values ​​into the single image accuracy index formula, we can calculate A simple is 0.

[0090] In the embodiment of the present invention, the comprehensive evaluation index for evaluating the urban governance multimodal large model on sample data is obtained based on the accuracy index of multiple single sample images:

[0091]

[0092] Among them, M represents the total number of sample images in the sample data, λ represents the influence coefficient of single accuracy, Indicates the text label corresponding to the jth sample image in the sample data, The jth sample image in the sample data corresponds to the text description of the prediction result of the urban governance multimodal large model, Indicates the actual result text description And the prediction result text description The cosine similarity of the texts between them, η represents the influence coefficient of text similarity, is the accuracy of a single sample image.

[0093] Optional, accuracy of a single sample image It is relatively reliable, and the single accuracy influence coefficient λ is set to 0.8; the text cosine similarity Sim is not very reliable, and the text similarity influence coefficient η is set to 0.2.

[0094] The embodiment of the present invention provides a case, in which the true value label is: there are illegal unlicensed peddlers in the picture [316,319,613,456]. The prediction output result of the urban governance multimodal large model is: there are no peddlers selling in the picture [325,302,640,443]. If a pure text similarity measurement method is used, the accuracy of a single sample picture is 0.69. There is only one target object in the picture, the coordinate overlap IOU is 0.72, the existence quantization value of the true value label is 1, the existence quantization value of the urban governance multimodal large model prediction result is 0, and the classification accuracy (text similarity) of the classification label "" and "unlicensed peddlers" is 0.68. According to the single picture accuracy index formula, the existence quantization value XOR value is 0, so the accuracy of this picture is 0. Obviously, it is more accurate to calculate the index by text labeling.

[0095] The embodiment of the present invention provides another case, in which the truth label is: there are shops (including various kiosks) on the road surface that place goods outside the shop and cross the door to occupy the road for business [[066,507,178,707]]. The predicted text result of the urban governance multimodal large model is: there are vendors in public places who engage in road occupation for business [[273,587,423,926]]. According to the intersection over union (IOU) calculated by the positioning frame, which is 0, it can be known that the accuracy of the positioning frame calculation for a single image is 0 because the prediction frame cannot match the truth frame.

[0096] like Figure 2 As shown, the embodiment of the present invention provides an urban governance multimodal large model evaluation device based on text labeling, including: a data acquisition module 201, an output acquisition module 202, a structured dictionary acquisition module 203, a first indicator acquisition module 204, a second indicator acquisition module 205 and an evaluation result acquisition module 206;

[0097] The data acquisition module 201 is used to acquire sample data for evaluating the multimodal large model of urban governance; wherein the sample data includes sample images and text labels corresponding to the sample images;

[0098] The output acquisition module 202 is used to input the sample image in the sample data into the urban governance multimodal large model to obtain the output text corresponding to the urban governance multimodal large model;

[0099] The structured dictionary acquisition module 203 is used to construct a list, each element in the list corresponds to a picture, and the elements of the list are structured dictionaries; wherein the structured dictionary list stores picture names, picture size attributes, existence keywords, label keywords and target boxes;

[0100] The first indicator acquisition module 204 is used to obtain the accuracy indicator of the urban governance multimodal large model in a single sample image according to the data in the structured dictionary list and the text label corresponding to the sample image;

[0101] The second indicator acquisition module 205 is used to obtain a comprehensive evaluation indicator for evaluating the urban governance multimodal large model on sample data based on the accuracy indicator of the single sample image.

[0102] The urban governance multimodal large model evaluation device based on text labeling described in an embodiment of the present invention can execute the above method embodiment and produce the same beneficial effects, which will not be repeated here.

[0103] In the embodiment of the present invention, the structured dictionary acquisition module 203 includes a data extraction unit and a dictionary acquisition unit;

[0104] The data extraction unit is used to identify the object existence and classification label corresponding to any target in the output text by using a keyword recognition method; identify the target location information corresponding to any target in the output text by using an identifier recognition method; traverse each target in the output text to obtain the object existence, classification label and target location information corresponding to each target;

[0105] The dictionary acquisition unit is used to construct a list, each list element corresponds to a picture, and the list element stores object existence, classification label and target location information, to obtain a structured dictionary stored in the list.

[0106] In the embodiment of the present invention, the first indicator acquisition module 204 includes: a prediction data acquisition unit, a real data acquisition unit and an accuracy indicator acquisition unit;

[0107] The prediction data acquisition unit is used to determine whether there are quantized values ​​for the prediction box, the prediction classification label, and the prediction result in the single sample image according to the data in the structured dictionary, and obtain the prediction data corresponding to the single sample image;

[0108] The real data acquisition unit is used to determine whether there are quantized values ​​for the real prediction box, the real classification label, and the real result in the text label according to the text label corresponding to the sample image, and obtain the real data corresponding to the single sample image;

[0109] The accuracy index acquisition unit is used to obtain the accuracy index of a single sample image according to the predicted data corresponding to the single sample image and the real data corresponding to the single sample image:

[0110]

[0111] Among them, IOU i represents the target box intersection-union ratio of the i-th target in a single sample image, represents the true classification label of the i-th target in a single sample image, Represents the predicted classification label of the i-th target in the text label corresponding to the sample image, Represents the predicted classification label and the true classification label The cosine similarity of the texts between A quantitative value indicating whether the target exists in the real result. A quantitative value indicating whether the target exists in the prediction result corresponding to the text label. If the target exists in the result, obj i = 1, if the target does not exist in the result, then obj i =0, N represents the total number of targets in a single sample image.

[0112] In the embodiment of the present invention, the second indicator acquisition module 205 includes a text similarity prediction unit and a comprehensive evaluation indicator acquisition unit;

[0113] The text similarity prediction unit is used to determine the text cosine similarity between the real result text description and the predicted result text description according to the sample pictures in the sample data and the text labels corresponding to the sample pictures;

[0114] The comprehensive evaluation index acquisition unit is used to determine the comprehensive evaluation index of the urban governance multimodal large model according to the text cosine similarity and accuracy index corresponding to multiple single sample images:

[0115]

[0116] Among them, M represents the total number of sample images in the sample data, λ represents the influence coefficient of single accuracy, Indicates the text label corresponding to the jth sample image in the sample data, The jth sample image in the sample data corresponds to the text description of the prediction result of the urban governance multimodal large model, Indicates the actual result text description And the prediction result text description The cosine similarity of the texts between them, η represents the influence coefficient of text similarity, is the accuracy of a single sample image.

[0117] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The schemes in the embodiments of the present invention may be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0118] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0119] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0120] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0121] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0122] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for evaluating a large multimodal model of urban governance based on text labeling, characterized in that: include: Obtain sample data for evaluating a large multimodal model of urban governance; wherein the sample data includes sample images and text labels corresponding to the sample images; Input the sample images in the sample data into the urban governance multimodal large model to obtain the output text corresponding to the urban governance multimodal large model; Extracting object existence, classification labels, and target location information from the output text, and obtaining a structured dictionary list based on the object existence, classification labels, and target location information; According to the data in the structured dictionary list and the text labels corresponding to the sample images, the accuracy index of the urban governance multimodal large model in a single sample image is obtained; Based on the accuracy indicators of multiple single sample images, comprehensive evaluation indicators for evaluating the multimodal large model of urban governance on sample data are obtained.

2. According to the text labeling-based urban governance multimodal large model evaluation method of claim 1, it is characterized in that: The urban governance multimodal large model is a multimodal large model obtained through fine-tuning training for urban governance scene image data; The multimodal large model receives visual images and question text inputs and generates text corresponding to the images.

3. According to the text labeling-based urban governance multimodal large model evaluation method of claim 1, it is characterized in that: Extracting object existence, classification labels, and target location information from the output text includes: For the description corresponding to any target in the output text, a keyword recognition method is used to identify the existence and classification label of the object corresponding to the target; For the description corresponding to any target in the output text, an identifier recognition method is used to identify the target position information corresponding to the target; Traverse each target in the output text to obtain the object existence, classification label and target location information corresponding to each target.

4. According to the text labeling-based urban governance multimodal large model evaluation method of claim 3, it is characterized in that: The structured dictionary list is: Build a list, each element in the list corresponds to a picture, and the elements of the list are structured dictionaries; The structured dictionary list stores image names, image size attributes, existence keywords, label keywords, and target boxes.

5. According to the text labeling-based urban governance multimodal large model evaluation method of claim 1, it is characterized in that: According to the data in the structured dictionary list and the text labels corresponding to the sample images, the accuracy index of the urban governance multimodal large model in a single sample image is obtained as follows: Among them, IOU i represents the target box intersection-union ratio of the i-th target in a single sample image, represents the true classification label of the i-th target in a single sample image, Represents the predicted classification label of the i-th target in the text label corresponding to the sample image, Represents the predicted classification label and the true classification label The cosine similarity of the texts between A quantitative value indicating whether the target exists in the real result. A quantitative value indicating whether the target exists in the prediction result corresponding to the text label. N represents the total number of targets in a single sample image.

6. According to the text labeling-based urban governance multimodal large model evaluation method of claim 5, it is characterized in that: According to the accuracy index of multiple single sample images, the comprehensive evaluation index of evaluating the urban governance multimodal large model on the sample data is obtained as follows: Among them, M represents the total number of sample images in the sample data, λ represents the influence coefficient of single accuracy, Indicates the text label corresponding to the jth sample image in the sample data, The jth sample image in the sample data corresponds to the text description of the prediction result of the urban governance multimodal large model, Indicates the actual result text description And the prediction result text description The cosine similarity of the texts between them, η represents the influence coefficient of text similarity, is the accuracy of a single sample image.

7. A text-labeled urban governance multimodal large model evaluation device, characterized in that: include: A data acquisition module, an output acquisition module, a structured dictionary acquisition module, a first indicator acquisition module, a second indicator acquisition module and an evaluation result acquisition module; The data acquisition module is used to acquire sample data for evaluating the multimodal large model of urban governance; wherein the sample data includes sample images and text labels corresponding to the sample images; The output acquisition module is used to input the sample images in the sample data into the urban governance multimodal large model to obtain the output text corresponding to the urban governance multimodal large model; The structured dictionary acquisition module is used to extract the object existence, classification label and target location information in the output text, and obtain a structured dictionary list according to the object existence, classification label and target location information; The first indicator acquisition module is used to obtain the accuracy indicator of the urban governance multimodal large model in a single sample image according to the data in the structured dictionary list and the text label corresponding to the sample image; The second indicator acquisition module is used to obtain a comprehensive evaluation indicator for evaluating the urban governance multimodal large model on sample data based on the accuracy indicator of the single sample image.

8. The urban governance multimodal large model evaluation device based on text labeling according to claim 7 is characterized in that: The structured dictionary acquisition module includes a data extraction unit and a dictionary acquisition unit; The data extraction unit is used to use a keyword recognition method to identify the existence and classification label of the object corresponding to the target for the description corresponding to any target in the output text; and use an identifier recognition method to identify the target location information corresponding to the target for the description corresponding to any target in the output text; Traverse each target in the output text to obtain the object existence, classification label and target location information corresponding to each target; The dictionary acquisition unit is used to construct a list, each element in the list corresponds to a picture, and the elements of the list are structured dictionaries; wherein the structured dictionary list stores picture names, picture size attributes, existence keywords, label keywords and target boxes.

9. The urban governance multimodal large model evaluation device based on text labeling according to claim 8 is characterized in that: The first indicator acquisition module includes: a prediction data acquisition unit, a real data acquisition unit and an accuracy indicator acquisition unit; The prediction data acquisition unit is used to determine whether there are quantized values ​​for the prediction box, the prediction classification label, and the prediction result in the single sample image according to the data in the structured dictionary, and obtain the prediction data corresponding to the single sample image; The real data acquisition unit is used to determine whether there are quantized values ​​for the real prediction box, the real classification label, and the real result in the text label according to the text label corresponding to the sample image, and obtain the real data corresponding to the single sample image; The accuracy index acquisition unit is used to obtain the accuracy index of a single sample image according to the predicted data corresponding to the single sample image and the real data corresponding to the single sample image: Among them, IOU i represents the target box intersection-union ratio of the i-th target in a single sample image, represents the true classification label of the i-th target in a single sample image, Represents the predicted classification label of the i-th target in the text label corresponding to the sample image, Represents the predicted classification label and the true classification label The cosine similarity of the texts between A quantitative value indicating whether the target exists in the real result. A quantitative value indicating whether the target exists in the prediction result corresponding to the text label. N represents the total number of targets in a single sample image.

10. The urban governance multimodal large model evaluation device based on text labeling according to claim 9 is characterized in that: The second indicator acquisition module includes a text similarity prediction unit and a comprehensive evaluation indicator acquisition unit; The text similarity prediction unit is used to determine the text cosine similarity between the real result text description and the predicted result text description according to the sample pictures in the sample data and the text labels corresponding to the sample pictures; The comprehensive evaluation index acquisition unit is used to determine the comprehensive evaluation index of the urban governance multimodal large model according to the text cosine similarity and accuracy index corresponding to multiple single sample images: Among them, M represents the total number of sample images in the sample data, λ represents the influence coefficient of single accuracy, Indicates the text label corresponding to the jth sample image in the sample data, The jth sample image in the sample data corresponds to the text description of the prediction result of the urban governance multimodal large model, Indicates the actual result text description And the prediction result text description The cosine similarity of the texts between them, η represents the influence coefficient of text similarity, is the accuracy of a single sample image.

Images