Cascade mark-based urban multi-modal large model construction method and system

Abstract

The invention provides an urban multi-modal large model construction method and system based on cascade tagging, and relates to the technical field of smart cities, the urban multi-modal large model construction method comprises the following steps: S1, reasoning urban street scene pictures in an urban training data set through a universal multi-modal large model, and obtaining urban street scene pictures in the urban training data set; performing cascade marking on the reasoning result to obtain a text label with a mark; s2, inputting the city streetscape picture and the corresponding text label with the mark as a first training sample into the initial city multi-modal large model for first-stage training; s3, splitting the text label with the mark into four question and answer dialogues according to the mark, and inputting the four question and answer dialogues which are randomly disordered as second training samples into the city multi-modal large model after the first-stage training for second-stage training; and S4, performing model evaluation on the urban multi-modal large model trained in the second stage by taking the marked quantitative score as an evaluation index.

Description

Technical Field

[0001] This invention relates to the field of smart city-related technologies, specifically to a method and system for constructing a large multimodal urban model based on cascaded tags. Background Technology

[0002] In urban governance, multimodal large models are often used to identify urban street scene images. The multimodal large model outputs a text description related to urban management. For example, if a picture showing illegal outdoor business operations is input into the multimodal large model, the model will identify and output "The picture shows that the shop owner has expanded his business location outward, occupying public resources, which is a problem of outdoor business operations".

[0003] While general-purpose multimodal large models can effectively understand input images and output detailed descriptions, for urban governance, a more crucial aspect of multimodal large models is accurately identifying the problem type. To address this, current approaches incorporate thought chains into the multimodal large model, gradually deriving accurate problem type conclusions through a rigorous reasoning process. However, adding thought chains leads to cumbersome reasoning processes and lengthy output text, resulting in slower problem type conclusion deriving. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for constructing a multimodal urban model based on cascading tags. By adding cascading tags to the text in the reasoning process of the multimodal urban model, a progressive thinking chain is formed to construct a multimodal urban governance model, which can accurately and quickly draw conclusions about the problem type.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following solution:

[0006] A method for constructing a large multimodal urban model based on cascaded tags includes the following steps:

[0007] S1. Obtain the city training dataset, use a general multimodal large model to infer the city street view images in the city training dataset, and cascade the inference results to obtain labeled text tags.

[0008] S2. Construct an initial multimodal city model. Use city street view images and corresponding labeled text tags as the first training samples and input them into the initial multimodal city model for the first stage of training to obtain the multimodal city model after the first stage of training.

[0009] The initial urban multimodal model is constructed based on general multimodal large models, such as GLM-4V and Qwen-VL. Urban street view images and corresponding labeled text labels are used as the first training samples to train the initial urban multimodal model to generate the urban multimodal large model.

[0010] S3. The labeled text tags are split into four question-and-answer dialogues according to the tags. The four question-and-answer dialogues in random order are used as the second training samples and input into the city multimodal large model after the first stage of training for the second stage of training, so as to obtain the city multimodal large model after the second stage of training.

[0011] S4. The labeled quantization score is used as the evaluation index to evaluate the urban multimodal large model after the second stage of training. The urban multimodal large model is selected based on the evaluation results.

[0012] Furthermore, in S1, the process of inferring urban street view images in the urban training dataset using a general multimodal large model is as follows:

[0013] First, design corresponding image description prompts, problem reasoning prompts, problem cause prompts, and conclusion prompts for city street scene images. When the general multimodal large model performs reasoning on the city street scene images, it infers the corresponding image description text, problem reasoning text, problem cause text, and conclusion text based on the image description prompts, problem reasoning prompts, problem cause prompts, and conclusion prompts.

[0014] Then add description tags at the beginning and end of the image description text, reasoning tags at the beginning and end of the question reasoning text, cause tags at the beginning and end of the question cause text, and conclusion tags at the beginning and end of the conclusion text.

[0015] Furthermore, in S1, the process of cascading the inference results is as follows:

[0016] After obtaining the image description text, question reasoning text, question cause text, and conclusion text of the general multimodal large model for reasoning on urban street view images, the corresponding image description text, question reasoning text, question cause text, and conclusion text are concatenated in the order of description tag, reasoning tag, cause tag, and conclusion tag to generate a text label with tags. One text label corresponds to one urban street view image.

[0017] Furthermore, the process of generating text tags with markings is as follows:

[0018] Following the order of description markers, reasoning markers, cause markers, and conclusion markers, the corresponding image description text, problem reasoning text, problem cause text, and conclusion text are sequentially concatenated into tagged text tags. These tagged text tags are then input into a general multimodal large model for statement logic checks, resulting in cleaned and filtered text tags, which are then used as tagged text tags.

[0019] Furthermore, the marked text label refers to the text that is sequentially assembled from image description text marked with description tags at the beginning and end, question reasoning text marked with reasoning tags at the beginning and end, question reasoning text marked with reason tags at the beginning and end, and conclusion text marked with conclusion tags at the beginning and end.

[0020] Furthermore, the training process for the first phase of training is as follows:

[0021] The city street view images and corresponding labeled text labels are used as the first training samples and input into the initial city multimodal large model. The labeled text labels are trained in a single-turn dialogue manner, and the training loss function used is the cross-entropy loss function with a label format penalty term.

[0022] Furthermore, the cross-entropy loss function with a mark format penalty term is calculated by adjusting the correct marks in the text labels using a preset format penalty term adjustment coefficient.

[0023] Furthermore, the training process for the second phase of training is as follows:

[0024] First, the labeled text tags are split into image description text, question reasoning text, question cause text, and conclusion text according to the tags. Then, the randomly ordered image description text, question reasoning text, question cause text, and conclusion text are used as second training samples and input into the city multimodal large model after the first stage of training. The randomly ordered image description text, question reasoning text, question cause text, and conclusion text are trained in a multi-turn dialogue manner, and the training loss function used is the tag-weighted cross-entropy loss function.

[0025] Furthermore, the labeled weighted cross-entropy loss function is obtained by calculating the length of the corresponding text of the image description text with descriptive labels, the question reasoning text with inference labels, the question reasoning text with cause labels, and the conclusion text with conclusion text.

[0026] A system for constructing a large-scale multimodal urban model based on cascaded tags, employing the aforementioned method for constructing a large-scale multimodal urban model based on cascaded tags, includes:

[0027] Data Cascade Labeling Module: Obtain the city training dataset, perform inference on the city street view images in the city training dataset using a general multimodal large model, and perform cascade labeling on the inference results to obtain labeled text tags;

[0028] The first stage training module: Construct an initial urban multimodal large model, using urban street view images and corresponding labeled text tags as the first training samples, and inputting them into the initial urban multimodal large model for the first stage training, to obtain the urban multimodal large model after the first stage training;

[0029] The second-stage training module: The labeled text tags are split into four question-and-answer dialogues according to the tags. The four question-and-answer dialogues in random order are used as the second training samples and input into the city multimodal large model after the first stage of training for the second stage of training, so as to obtain the city multimodal large model after the second stage of training.

[0030] Model evaluation module: The labeled quantization score is used as the evaluation index to evaluate the urban multimodal large model after the second stage of training, and the urban multimodal large model is selected based on the evaluation results.

[0031] The beneficial effects of this invention are:

[0032] This invention provides a method and system for constructing a multimodal urban model based on cascaded tags. It improves upon existing multimodal models that incorporate thought chains by adding cascaded tags to the text during the reasoning process of the multimodal model, forming a progressive thought chain. Based on this progressive thought chain formed by cascaded tags, the multimodal urban governance model undergoes two stages of training and model evaluation. The optimal model parameters are then selected to construct the multimodal urban governance model, ensuring that the model can accurately and quickly derive conclusions about problem types. Attached Figure Description

[0033] Figure 1 This is a flowchart illustrating the urban multimodal large model construction method in Embodiment 1 of the present invention;

[0034] Figure 2 This is a schematic diagram illustrating the use of cascaded labeling to infer urban street view images in the urban training dataset through a general multimodal large model in Embodiment 1 of the present invention. Figure 3 This is a schematic diagram of a city street scene image in Embodiment 1 of the present invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention.

[0037] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0038] Additionally, descriptions of well-known structures, functions, and configurations may be omitted for clarity and conciseness. Those skilled in the art will recognize that various changes and modifications can be made to the examples described herein without departing from the spirit and scope of the present disclosure.

[0039] Technologies, methods and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, such technologies, methods and equipment should be considered part of the authorization specification.

[0040] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0041] The present invention will now be described in detail with reference to the accompanying drawings and embodiments:

[0042] Example 1

[0043] In this embodiment, to address the issue that adding thought chains to a multimodal large model in urban governance leads to a slow conclusion on the problem type, a novel multimodal large model construction method is proposed. This new method is primarily applied to urban governance, specifically the construction of an urban multimodal large model. This method involves adding cascading markers to the text during the multimodal large model's reasoning process to form a progressive thought chain, thereby constructing a multimodal large model for urban governance. This approach can accurately and quickly arrive at conclusions on the problem type.

[0044] The "Chain of Thought" (CoT) mentioned here refers to the "Chain of Thought" (CoT) in Large Language Models (LLMs), a technique designed to improve the reasoning ability of models. It mimics the human problem-solving process by having the model generate a series of intermediate reasoning steps before producing an answer. This method is particularly helpful for solving problems requiring multi-step reasoning, such as mathematical problems, logic puzzles, and complex question-and-answer tasks.

[0045] For reasoning chains, a common approach is to design specific prompts (prompt engineering) to guide the model in generating answers that include reasoning steps. For example, an instruction like "Let's think about this problem step by step" can be added after the question. Another method is to provide the model with a large dataset containing detailed problem-solving steps during the training phase, enabling it to learn how to generate similar reasoning chains.

[0046] like Figure 1 As shown, the method for constructing a large multimodal urban model includes the following steps:

[0047] S1. Obtain the city training dataset, use a general multimodal large model to infer the city street view images in the city training dataset, and perform cascade labeling on the inference results to obtain labeled text labels; the purpose is to realize the labeling of text labels, thereby facilitating the management of text labels.

[0048] S2. Construct an initial urban multimodal large model. Use urban street view images and corresponding labeled text tags as the first training samples and input them into the initial urban multimodal large model for the first stage of training to obtain the urban multimodal large model after the first stage of training. In step S2, after the first stage of training, the urban multimodal large model can output text content with cascading labels and has the ability of progressive thinking chain.

[0049] S3. The labeled text tags are split into four question-and-answer dialogues according to the tags. The four question-and-answer dialogues in random order are used as the second training samples and input into the city multimodal large model after the first stage of training for the second stage of training, so as to obtain the city multimodal large model after the second stage of training. In step S3, after the second stage of training, the city multimodal large model can achieve decoupling between cascaded tags and can quickly output the response text of split questions.

[0050] S4. The labeled quantization score is used as an evaluation index to evaluate the urban multimodal large model after the second stage of training. The urban multimodal large model is selected based on the evaluation results. In step S4, the labeled quantization score guides the model to pay more attention to the problem type conclusion, which helps to build a model with correct conclusion orientation.

[0051] In one embodiment, in S1, the process of inferring urban street view images in the urban training dataset using a general multimodal large model is as follows:

[0052] First, design corresponding image description prompts, problem reasoning prompts, problem cause prompts, and conclusion prompts for city street scene images. When the general multimodal large model performs reasoning on the city street scene images, it infers the corresponding image description text, problem reasoning text, problem cause text, and conclusion text based on the image description prompts, problem reasoning prompts, problem cause prompts, and conclusion prompts.

[0053] Then add description tags at the beginning and end of the image description text, reasoning tags at the beginning and end of the question reasoning text, cause tags at the beginning and end of the question cause text, and conclusion tags at the beginning and end of the conclusion text.

[0054] like Figure 2 As shown, this invention employs cascaded tagging to infer urban street view images in a city training dataset using a general multimodal large model. Specifically, the urban street view images are input into the general multimodal large model (MLLM). First, image description prompts are added to the MLLM to obtain corresponding image description text. Then, description tags are added to the beginning and end of the image description text, and the image description text is input into the MLLM. Question reasoning prompts are added to the MLLM as constraints to obtain corresponding question reasoning text. Then, reasoning tags are added to the beginning and end of the question reasoning text, and the question reasoning text is input into the MLLM. Question cause prompts are added to the MLLM as constraints. The process involves obtaining the corresponding problem cause text, then adding cause markers to the beginning and end of the problem cause text, and inputting the problem cause text into a general multimodal large model. Conclusion prompts are added to the general multimodal large model as constraints to obtain the corresponding conclusion text. Conclusion markers are then added to the beginning and end of the conclusion text. In summary, through a progressive approach, image description text with descriptive markers, image reasoning text with inference markers, image cause text with cause markers, and conclusion text with conclusion markers are obtained sequentially, thus forming a labeled text tag. The labeled text tag consists of image description text with descriptive markers, image reasoning text with inference markers, image cause text with cause markers, and conclusion text with conclusion markers.

[0055] Specifically, for a city street view image, the city street view image is as follows: Figure 3 As shown, where Figure 3The image contains a target recognition bounding box for object detection. After designing corresponding image description prompts, the answer text corresponding to the image description prompts is obtained, i.e., the image description text, specifically:

[0056] This photo shows a scene of a city street. Specifically, on the left side of the picture, there is a row of green trees, a black street lamp and a silver van next to the trees, many electric vehicles parked on the street, and a person walking. On the right side is a gray residential building with blue window frames. There are two air conditioner outdoor units on the outside of the first floor, as well as two clothes racks covered with clothes, which look somewhat messy. In addition, there is a red fire hydrant in front of the residential building. On one of the balconies of this residential building, there are several pieces of clothing hanging on the balcony, including black, pink and blue ones. One of the pink ones may be a nightgown.

[0057] Then, description tags are added to the beginning and end of the above image description text to obtain the image description text with added description tags, specifically:

[0058] <Description>This photo shows a scene of a city street. Specifically, on the left side of the picture, there is a row of green trees, next to which is a black street lamp and a silver van. Many electric vehicles are parked on the street, and a person is walking. On the right is a gray residential building with blue window frames. On the outside of the first floor, there are two air conditioner units and two clothes racks, covered with clothes, appearing somewhat cluttered. In addition, there is a red fire hydrant in front of the building. On one of the balconies of this building, several pieces of clothing—black, pink, and blue—are hanging, including a pink one that may be a nightgown. <Description>

[0059] Here, <description> represents a description tag.

[0060] Similarly, by designing corresponding image description prompts, problem reasoning prompts, problem cause prompts, and conclusion prompts for urban street scene images, the corresponding image description text, problem reasoning text, problem cause text, and conclusion text can be obtained. Furthermore, corresponding markers are added to the beginning and end of the image description text, problem reasoning text, problem cause text, and conclusion text to obtain the marked text. This invention can perform cascading marking based on the marked text to form a progressive thought chain.

[0061] The image description prompts, question reasoning prompts, question reasoning prompts, and conclusion prompts can all be designed by staff based on the image content. For example, the question reasoning prompt could be: "Based on the image content, determine whether the clothing in the city street scene image is hung out to dry along the street or sold outside a store, and provide a detailed reasoning process." The question reasoning prompt could be: "Explain the reason for the judgment in one sentence, using a 'because therefore' sentence structure." The conclusion prompt could be: "The question options include selling clothes outside a store and hanging clothes out to dry along the street, and the output sentence could be 'The problem reflected in the image is...'". The general multimodal large model can output the corresponding text based on the above prompts.

[0062] In one embodiment, in S1, the process of cascading the inference results is as follows:

[0063] After obtaining the image description text, question reasoning text, question cause text, and conclusion text of the general multimodal large model for reasoning on urban street view images, the corresponding image description text, question reasoning text, question cause text, and conclusion text are concatenated in the order of description tag, reasoning tag, cause tag, and conclusion tag to generate a text label with tags. One text label corresponds to one urban street view image.

[0064] The labeled text tag refers to a text tag composed of image description text marked with description tags at the beginning and end, question reasoning text marked with reasoning tags at the beginning and end, question reasoning text marked with reason tags at the beginning and end, and conclusion text marked with conclusion tags at the beginning and end, arranged sequentially.

[0065] If we design corresponding image description prompts, problem reasoning prompts, problem cause prompts, and conclusion prompts for a city street scene image, resulting in corresponding image description text, problem reasoning text, problem cause text, and conclusion text, and then concatenate these texts based on the tags at the beginning and end, the resulting tagged text labels are as follows:

[0066] <Description>This photo shows a scene of a city street. Specifically, there is a large tree on the left side of the picture, with its trunk painted white. Several pedestrians are walking on the street, some wearing green jackets and others wearing black clothes. There is a shop on the left with a white trash can in front of it. In the middle is a car detailing shop, with a red banner in front that reads "Car Problem Solutions, Air Conditioner Wash, Healthy Deep Breathing". On the right is a car repair shop with a red car parked inside, and a red cabinet with three tires on it placed in front of it. There are several buildings in the background, some with security grilles on their exterior walls. <Description><Inference>Based on the content of the picture, it can be inferred that the banner in the picture is a hanging banner rather than a hand-held banner. The following is a detailed reasoning process: Banner position: As can be seen from the picture, the banner is fixed to two posts, which are located on the left and right sides of the picture respectively. This method of fixing usually involves hanging the banner using some kind of bracket or hook. Banner condition: The banner is flat and there are no wrinkles or twists. This indicates that the banner has been fully unfurled and is hanging stably. Surrounding Environment: No one is seen holding the ends of the banner in the picture. If it were a hand-held banner, someone would typically be holding both ends to keep it flat and visible. Banner Content: Although the text on the banner cannot be read directly, we can observe that it is a striking yellow, often used in outdoor events to attract attention. Hand-held banners are more likely used for indoor events or smaller gatherings. In conclusion, considering the banner's fixation to the pillar, its flatness, and the surrounding environment and intended use, we can reasonably infer that the banner in the picture is a hanging banner. <Inference> <Reason> Because the banner is fixed to the pillar and flat, the banner in the picture is a hanging banner. <Reason> <Conclusion> The problem depicted in the picture is a hanging banner. <Conclusion>

[0067] In this context, <description> represents the description marker, <reasoning> represents the reasoning marker, <cause> represents the cause marker, and <conclusion> represents the conclusion marker.

[0068] In one embodiment, the process of generating text tags with markings is as follows:

[0069] Following the order of description markers, reasoning markers, cause markers, and conclusion markers, the corresponding image description text, problem reasoning text, problem cause text, and conclusion text are sequentially concatenated into tagged text tags. These tagged text tags are then input into a general multimodal large model for statement logic checks, resulting in cleaned and filtered text tags, which are then used as tagged text tags.

[0070] In summary, when constructing a multimodal urban model, adding cascading markers to the reasoning text of the multimodal model can form a progressive thought chain. This allows the progressive cascading markers to directly and quickly respond to the conclusion question during the reasoning process, thus making the conclusion of the question type accurate and fast.

[0071] Large Language Models (LLMs) extensively use the cross-entropy loss function to optimize model parameters during training. These models aim to learn the probability distribution of text data, thereby generating coherent and meaningful sequences of text. Cross-entropy loss is particularly useful in this context because it effectively measures the difference between the probability distribution predicted by the model and the true labels.

[0072] Therefore, in this embodiment, after forming a progressive thinking chain, the urban multimodal large model construction method designs a two-stage training and model evaluation based on the progressive thinking chain. In this way, the urban multimodal large model is trained to obtain the model parameters with the optimal evaluation index to construct an urban governance multimodal large model, ensuring that the problem type conclusion can be accurately and quickly drawn through the urban governance multimodal large model.

[0073] In one embodiment, the training process of the first stage of training is as follows:

[0074] The city street view images and corresponding labeled text labels are used as the first training samples and input into the initial city multimodal large model. The labeled text labels are trained in a single-turn dialogue manner, and the training loss function used is the cross-entropy loss function with a label format penalty term.

[0075] The cross-entropy loss function with the labeled format penalty term is:

[0076]

[0077] in, This indicates summing over each time step (or each word) in the sequence; This indicates summing over each word in the vocabulary; y t,i Let represent a one-hot vector, indicating whether the i-th word is the correct answer at time step t (1 indicates yes, 0 indicates no); This represents the probability of the model predicting the i-th word at time step t; This indicates that the sequence length is normalized, and the average loss is calculated; n represents the statistical number of correctly marked text, Max indicates taking the maximum value, and α represents the adjustment coefficient of the format penalty term.

[0078] In one embodiment, the training process of the second stage of training is as follows:

[0079] First, the labeled text tags are split into image description text, question reasoning text, question cause text, and conclusion text according to the tags. Then, the randomly ordered image description text, question reasoning text, question cause text, and conclusion text are used as second training samples and input into the city multimodal large model after the first stage of training. The randomly ordered image description text, question reasoning text, question cause text, and conclusion text are trained in a multi-turn dialogue manner, and the training loss function used is the tag-weighted cross-entropy loss function.

[0080] The labeled weighted cross-entropy loss function is calculated by taking the text lengths of the image description text with descriptive labels, the question reasoning text with inference labels, the question reasoning text with cause labels, and the conclusion text with conclusion text. Specifically:

[0081]

[0082] Among them, T summ T represents the length of the text with the conclusion marker. reason T represents the length of the text with reason markers. proc T represents the length of the text with inference markers. desc Indicates the length of the text with descriptive tags.

[0083] In one embodiment, in S4, the labeled quantization score is used as an evaluation metric to evaluate the urban multimodal large model trained in the second stage. The labeled quantization score is:

[0084]

[0085] in, The text indicating the reason for the large model's prediction. The true reason in the real label is represented by Sim, and the semantic similarity function is represented by . The text indicating the conclusions of the large model's predictions includes the quantified value of the target.

[0086] Example 2

[0087] A system for constructing a large-scale multimodal urban model based on cascaded tags, employing the aforementioned method for constructing a large-scale multimodal urban model based on cascaded tags, includes:

[0088] Data Cascade Labeling Module: Obtain the city training dataset, perform inference on the city street view images in the city training dataset using a general multimodal large model, and perform cascade labeling on the inference results to obtain labeled text tags;

[0089] The first stage training module: Construct an initial urban multimodal large model, using urban street view images and corresponding labeled text tags as the first training samples, and inputting them into the initial urban multimodal large model for the first stage training, to obtain the urban multimodal large model after the first stage training;

[0090] The second-stage training module: The labeled text tags are split into four question-and-answer dialogues according to the tags. The four question-and-answer dialogues in random order are used as the second training samples and input into the city multimodal large model after the first stage of training for the second stage of training, so as to obtain the city multimodal large model after the second stage of training.

[0091] Model evaluation module: The labeled quantization score is used as the evaluation index to evaluate the urban multimodal large model after the second stage of training, and the urban multimodal large model is selected based on the evaluation results.

[0092] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Based on the technical essence of the present invention and within the spirit and principles of the present invention, any simple modification, equivalent replacement and improvement of the above embodiment shall still fall within the scope of protection of the technical solution of the present invention.

Claims

1. A method for constructing a large multimodal urban model based on cascaded tags, characterized in that, The method for constructing a large multimodal urban model includes the following steps: S1. Obtain the city training dataset, use a general multimodal large model to infer the city street view images in the city training dataset, and cascade the inference results to obtain labeled text tags. S2. Construct an initial multimodal city model. Use city street view images and corresponding labeled text tags as the first training samples and input them into the initial multimodal city model for the first stage of training to obtain the multimodal city model after the first stage of training. S3. The labeled text tags are split into four question-and-answer dialogues according to the tags. The four question-and-answer dialogues in random order are used as the second training samples and input into the city multimodal large model after the first stage of training for the second stage of training, so as to obtain the city multimodal large model after the second stage of training. S4. The labeled quantization score is used as the evaluation index to evaluate the urban multimodal large model after the second stage of training. The urban multimodal large model is selected based on the evaluation results.

2. The method for constructing a large multimodal urban model based on cascaded tags according to claim 1, characterized in that, In S1, the process of inferring urban street view images in the urban training dataset using a general multimodal large model is as follows: First, corresponding image description prompts, problem reasoning prompts, problem cause prompts, and conclusion prompts are designed for urban street scene images. When performing progressive reasoning on urban street scene images using a general multimodal model, the progressive reasoning process is as follows: First, the image description prompts are used as input to the general multimodal model, and the corresponding image description text is output through reasoning; then, the image description text and problem reasoning prompts are used as input to the general multimodal model, and the corresponding problem reasoning text is output through reasoning; next, the problem reasoning text and problem cause prompts are used as input to the general multimodal model, and the corresponding problem cause text is output through reasoning; finally, the problem cause text and conclusion prompts are used as input to the general multimodal model, and the corresponding conclusion text is output through reasoning. Add description markers at the beginning and end of the image description text, reasoning markers at the beginning and end of the question reasoning text, cause markers at the beginning and end of the question cause text, and conclusion markers at the beginning and end of the conclusion text.

3. The method for constructing a large multimodal urban model based on cascaded tags according to claim 2, characterized in that, In S1, the process of cascading the inference results is as follows: After obtaining the image description text, question reasoning text, question cause text, and conclusion text of the general multimodal large model for reasoning on urban street view images, the corresponding image description text, question reasoning text, question cause text, and conclusion text are concatenated in the order of description tag, reasoning tag, cause tag, and conclusion tag to generate a text label with tags. One text label corresponds to one urban street view image.

4. The method for constructing a large multimodal urban model based on cascaded tags according to claim 3, characterized in that, The process of generating text tags with labels is as follows: Following the order of description markers, reasoning markers, cause markers, and conclusion markers, the corresponding image description text, problem reasoning text, problem cause text, and conclusion text are sequentially concatenated into tagged text tags. These tagged text tags are then input into a general multimodal large model for statement logic checks, resulting in cleaned and filtered text tags, which are then used as tagged text tags.

5. The method for constructing a large multimodal urban model based on cascaded tags according to claim 1, characterized in that, The labeled text tag refers to a text tag composed of image description text marked with description tags at the beginning and end, question reasoning text marked with reasoning tags at the beginning and end, question reasoning text marked with reason tags at the beginning and end, and conclusion text marked with conclusion tags at the beginning and end, arranged sequentially.

6. The method for constructing a large multimodal urban model based on cascaded tags according to claim 5, characterized in that, The training process for the first phase of training is as follows: The city street view images and corresponding labeled text labels are used as the first training samples and input into the initial city multimodal large model. The labeled text labels are trained in a single-turn dialogue manner, and the training loss function used is the cross-entropy loss function with a label format penalty term.

7. The method for constructing a large multimodal urban model based on cascaded tags according to claim 6, characterized in that, The cross-entropy loss function with a mark formatting penalty term is calculated by adjusting the correct mark in the text label using a preset formatting penalty term adjustment coefficient.

8. The method for constructing a large multimodal urban model based on cascaded tags according to claim 5, characterized in that, The training process for the second phase of training is as follows: First, the labeled text tags are split into image description text, question reasoning text, question cause text, and conclusion text according to the tags. Then, the randomly ordered image description text, question reasoning text, question cause text, and conclusion text are used as second training samples and input into the city multimodal large model after the first stage of training. The randomly ordered image description text, question reasoning text, question cause text, and conclusion text are trained in a multi-turn dialogue manner, and the training loss function used is the tag-weighted cross-entropy loss function.

9. The method for constructing a large multimodal urban model based on cascaded tags according to claim 8, characterized in that, The labeled weighted cross-entropy loss function is obtained by calculating the length of the corresponding texts: image description text with descriptive labels, question reasoning text with inference labels, question reasoning text with cause labels, and conclusion text with conclusion text.

10. A system for constructing a large multimodal urban model based on cascaded tags, characterized in that, The method for constructing a large multimodal urban model based on cascaded tags as described in any one of claims 1-9 includes: Data Cascade Labeling Module: Obtain the city training dataset, perform inference on the city street view images in the city training dataset using a general multimodal large model, and perform cascade labeling on the inference results to obtain labeled text tags; The first stage training module: Construct an initial urban multimodal large model, using urban street view images and corresponding labeled text tags as the first training samples, and inputting them into the initial urban multimodal large model for the first stage training, to obtain the urban multimodal large model after the first stage training; The second-stage training module: The labeled text tags are split into four question-and-answer dialogues according to the tags. The four question-and-answer dialogues in random order are used as the second training samples and input into the city multimodal large model after the first stage of training for the second stage of training, so as to obtain the city multimodal large model after the second stage of training. Model evaluation module: The labeled quantization score is used as the evaluation index to evaluate the urban multimodal large model after the second stage of training, and the urban multimodal large model is selected based on the evaluation results.

Images