Urban multi-modal large model training method and device, equipment and medium

Abstract

The invention provides an urban multi-modal large model training method, device and equipment and a medium, and relates to the technical field of artificial intelligence. Constructing an image-text interlaced data set according to the satellite image description text corresponding to each satellite image and the streetscape image description text corresponding to each streetscape image; and training the city multi-modal large model based on the image-text interlaced data set, and adjusting perceptron parameters corresponding to the perceptron in the city multi-modal large model and large language model parameters corresponding to the large language model to obtain a trained target city multi-modal large model. According to the technical scheme, the problems that in the prior art, a training data source is single, and images and texts are independently processed are solved, the trained target city multi-mode large model has good understanding on the city environment, and the space structure and function characteristics of a city can be comprehensively captured.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and medium for training a large multimodal urban model. Background Art

[0002] With the acceleration of urbanization, the demand for understanding urban spatial structure, function, and socioeconomic status is increasing. Advances in computer vision and multimodal learning technologies have driven the application of multimodal data in urban research, for tasks such as scene classification, geographic localization, and urban indicator prediction.

[0003] Existing large multimodal urban models mainly rely on a single data source for training, such as training only through satellite images or only through street view images. As a result, the trained large multimodal urban models have a poor understanding of the urban environment and find it difficult to fully capture the spatial structure and functional characteristics of the city. Summary of the Invention

[0004] The present invention provides a method, device, equipment and medium for training a large urban multimodal model, which is used to solve the problem in the prior art that the large urban multimodal model mainly relies on a single data source for training, resulting in the trained large urban multimodal model having a poor understanding of the urban environment and difficulty in fully capturing the spatial structure and functional characteristics of the city. The technical solution of the present invention uses a combination of satellite images, street view images and text to form an interleaved image and text data set to train the large urban multimodal model, solving the problem of a single training data source and separate processing of images and text in the prior art. In addition, compared with the traditional large multimodal model, a perceptron is added, so that the trained target large urban multimodal model has a good understanding of the urban environment and can fully capture the spatial structure and functional characteristics of the city.

[0005] The present invention provides a method for training a large multimodal urban model, comprising the following steps.

[0006] Constructing an image-text interleaved dataset based on a plurality of satellite images and a plurality of street view images corresponding to a target urban area, as well as satellite image description texts corresponding to the satellite images and street view image description texts corresponding to the street view images; The urban multimodal large model is trained based on the interleaved image-text dataset, and the perceptron parameters corresponding to the perceptron in the urban multimodal large model and the large language model parameters corresponding to the large language model are adjusted to obtain a trained target urban multimodal large model.

[0007] According to a method for training a large multimodal urban model provided by the present invention, the method comprises constructing an image-text interleaved dataset based on multiple satellite images and multiple street view images corresponding to a target urban area, as well as satellite image description texts corresponding to each of the satellite images and street view image description texts corresponding to each of the street view images, including: Acquire multiple satellite images and multiple street view images corresponding to the target city area, as well as a first geographic tag corresponding to each of the satellite images and a second geographic tag corresponding to each of the street view images; Inputting each of the satellite images into a preset text to generate a multimodal model, and obtaining a first text description corresponding to each of the satellite images; Inputting each of the street view images into the text to generate a multimodal model, thereby obtaining a second text description corresponding to each of the street view images; For each of the satellite images, determining a satellite image description text corresponding to the satellite image based on the first geographic tag and the first text description corresponding to the satellite image; For each of the street view images, determining a street view image description text corresponding to the street view image based on the second geographic tag and the second text description corresponding to the street view image; For each of the satellite images, construct an image interleaved subset based on the satellite image and a plurality of street view images corresponding to the satellite image; The image-text interleaved dataset is constructed based on the image interleaved subsets corresponding to all the satellite images, the satellite image text descriptions corresponding to all the satellite images, and the street view image description texts corresponding to all the street view images.

[0008] According to a method for training a large urban multimodal model provided by the present invention, the method trains the large urban multimodal model based on a text-image interleaved dataset, adjusts perceptron parameters corresponding to perceptrons in the large urban multimodal model and large language model parameters corresponding to a large language model, and obtains a trained target large urban multimodal model, comprising: Inputting the interleaved image and text dataset into a visual encoder in the urban multimodal large model, and obtaining satellite image visual features corresponding to each of the satellite images and street view image visual features corresponding to each of the street view images output by the visual encoder; Inputting satellite image visual features corresponding to each of the satellite images and street view image visual features corresponding to each of the street view images into the sensor, obtaining a plurality of image tags output by the sensor and text tags corresponding to each of the image tags; Based on each of the image tags and the text tags corresponding to each of the image tags, the perceptron parameters corresponding to the perceptron in the city multimodal large model and the large language model parameters corresponding to the large language model are adjusted to obtain a trained target city multimodal large model.

[0009] According to a method for training a large multimodal urban model provided by the present invention, the visual encoder determines the satellite image visual features corresponding to each of the satellite images and the street view image visual features corresponding to each of the street view images in the following manner: For each of the image interleaved subsets in the image-text interleaved dataset, the satellite image and each of the street view images in the image interleaved subset are encoded respectively, and satellite image visual features corresponding to the satellite image and street view image visual features corresponding to each of the street view images are output.

[0010] According to a method for training a large multimodal urban model provided by the present invention, the perceptron determines each image tag and the text tag corresponding to each image tag in the following manner: For each of the image interleaved subsets in the image-text interleaved dataset, using satellite image visual features corresponding to the satellite image as a query, performing cross-attention processing on the street view image visual features corresponding to each of the street view images, to obtain first fusion features corresponding to each of the street view images; Using the street view image visual features corresponding to each of the street view images as a query, performing cross-attention processing on the satellite image visual features corresponding to the satellite image, to obtain second fusion features corresponding to each of the street view images; determining a first visual embedding corresponding to the satellite image based on the second fusion features corresponding to each of the street view images and the satellite image visual features corresponding to the satellite image, and determining a first aligned text for the first visual embedding; determining a second visual embedding corresponding to each of the street view images based on the first fusion features and the street view image visual features respectively corresponding to each of the street view images, and determining a second aligned text for each of the second visual embeddings; Determining each of the image tags based on all of the first visual embeddings and all of the second visual embeddings corresponding to the image-text interleaved dataset; The text marks are determined based on all the first aligned texts and all the second aligned texts corresponding to the text-image interleaved dataset.

[0011] According to a method for training a large urban multimodal model provided by the present invention, adjusting the perceptron parameters corresponding to the perceptron and the large language model parameters corresponding to the large language model in the large urban multimodal model based on the image tags and the text tags corresponding to the image tags, includes: in, represents the objective function, represents the parameters that need to be adjusted for the urban multimodal model, Express The expectation operator of represents the logarithmic probability under preset historical information and conditions, represents the perceptron parameters, represents the large language model parameters, represents the first aligned text and the second aligned text, represents the image collection in the image-text interleaved dataset, Indicates the The text mark corresponding to the aligned text, Indicates the A text tag before the aligned text, Indicates the The image index corresponding to the aligned text Image tags, Indicates the The image index corresponding to the aligned text The previous image tag, represents the perceptron, represents the visual encoder, Indicates the The image index corresponding to the aligned text A collection of corresponding satellite and street view images.

[0012] According to a method for training a large multimodal urban model provided by the present invention, the method further includes: Acquire multiple fine-tuned satellite images and multiple fine-tuned street view images corresponding to the target urban area, and construct an instruction fine-tuning dataset based on all the fine-tuned satellite images and all the fine-tuned street view images; Based on the instruction fine-tuning data set, the perceptron parameters corresponding to the perceptron in the target city multimodal large model and the large language model parameters corresponding to the large language model are optimized to obtain a fine-tuned optimized city multimodal large model.

[0013] The present invention also provides a large-scale urban multimodal model training device, comprising the following modules: A construction module, configured to construct an image-text interleaved dataset based on a plurality of satellite images and a plurality of street view images corresponding to a target urban area, as well as satellite image description texts corresponding to each of the satellite images and street view image description texts corresponding to each of the street view images; The training module is used to train the urban multimodal large model based on the interleaved image and text dataset, adjust the perceptron parameters corresponding to the perceptron in the urban multimodal large model and the large language model parameters corresponding to the large language model, and obtain the trained target urban multimodal large model.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for training a large multimodal urban model as described above is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described urban multimodal large model training methods.

[0016] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described urban multimodal large model training methods.

[0017] The present invention provides a method, device, equipment, and medium for training a large urban multimodal model. The method, device, equipment, and medium construct an interleaved image-text dataset based on multiple satellite images and multiple street view images corresponding to the target urban area, as well as satellite image description text corresponding to each satellite image and street view image description text corresponding to each street view image. The method trains the large urban multimodal model based on the interleaved image-text dataset, adjusts the perceptron parameters corresponding to the perceptron in the large urban multimodal model, and adjusts the large language model parameters corresponding to the large language model to obtain a trained large urban multimodal model. The technical solution of the present invention uses a combination of satellite images, street view images, and text to form an interleaved image-text dataset to train the large urban multimodal model. This solves the problem of the existing technology of a single training data source and separate processing of images and text. Compared with the traditional large multimodal model, the method also adds a perceptron, so that the trained large urban multimodal model has a good understanding of the urban environment and can fully capture the spatial structure and functional characteristics of the city. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 It is a flow chart of the urban multimodal large model training method provided by the present invention.

[0020] Figure 2It is a schematic diagram of the process of constructing a text-image interleaved data set provided by the present invention.

[0021] Figure 3 It is a schematic diagram of the urban multimodal large model provided by the present invention.

[0022] Figure 4 It is a task execution diagram of fine-tuning and optimizing the urban multimodal large model provided by the present invention.

[0023] Figure 5 It is a structural schematic diagram of the urban multimodal large model training device provided by the present invention.

[0024] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0025] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0026] In view of the above problems in the prior art, the present invention provides a method for training a large multimodal urban model. Figure 1 This is a flow chart of the urban multimodal large model training method provided by the present invention, such as Figure 1 As shown, the method includes the following steps 110 and 120.

[0027] Step 110: constructing an image-text interleaved dataset based on a plurality of satellite images and a plurality of street view images corresponding to the target urban area, as well as satellite image description texts corresponding to the satellite images and street view image description texts corresponding to the street view images.

[0028] Specifically, the target city area can be an area corresponding to any number of cities in any country. A text-image interleaved dataset can be constructed based on multiple satellite images and multiple street view images, as well as satellite image description text corresponding to each satellite image and street view image description text corresponding to each street view image.

[0029] In one embodiment, constructing the image-text interleaved dataset based on multiple satellite images and multiple street view images corresponding to the target city area, as well as satellite image description text corresponding to each satellite image and street view image description text corresponding to each street view image, includes: Acquire multiple satellite images and multiple street view images corresponding to the target city area, as well as a first geographic tag corresponding to each of the satellite images and a second geographic tag corresponding to each of the street view images; Inputting each of the satellite images into a preset text to generate a multimodal model, and obtaining a first text description corresponding to each of the satellite images; Inputting each of the street view images into the text to generate a multimodal model, thereby obtaining a second text description corresponding to each of the street view images; For each of the satellite images, determining a satellite image description text corresponding to the satellite image based on the first geographic tag and the first text description corresponding to the satellite image; For each of the street view images, determining a street view image description text corresponding to the street view image based on the second geographic tag and the second text description corresponding to the street view image; For each of the satellite images, construct an image interleaved subset based on the satellite image and a plurality of street view images corresponding to the satellite image; The image-text interleaved dataset is constructed based on the image interleaved subsets corresponding to all the satellite images, the satellite image text descriptions corresponding to all the satellite images, and the street view image description texts corresponding to all the street view images.

[0030] Specifically, multiple satellite images and street view images corresponding to the target urban area can be obtained. Satellite images can be obtained from satellites (for example, from Beidou satellites), and street view images can be obtained from open-source image databases. It is easy to understand that both satellite and street view images are geotagged. The geotags may include, for example, the city / county where the image is located, as well as the longitude and latitude. After obtaining the satellite images, each satellite image can be input into a preset text generation multimodal model to obtain a first text description corresponding to each satellite image. The text generation multimodal model is a pre-configured model for generating text descriptions for images. For example, the text generation multimodal model may be the InternVL model, a series of multimodal large language models (MLLMs) developed by the OpenGVLab team at the Shanghai Artificial Intelligence Laboratory. The InternVL model is designed to process and understand data input from multiple modalities. After obtaining the street view images, each street view image can be input into the text generation multimodal model to obtain a second text description corresponding to each street view image. Furthermore, the geographic tags and text descriptions corresponding to the satellite image and the street view image can be fused to obtain satellite image description text for the satellite image and street view image description text for the street view image.

[0031] Furthermore, for each satellite image, an image interleaved subset can be constructed based on the satellite image and multiple street view images corresponding to the satellite image. The image interleaved subset can be expressed as ,in, represents a satellite image, Indicates the first street view image corresponding to the satellite image. Indicates the satellite image corresponding to Street View images, The set of street view images corresponding to the satellite image is represented. Furthermore, a text-image interleaved dataset can be constructed based on the image interleaved subsets corresponding to all satellite images, the satellite image text descriptions corresponding to all satellite images, and the street view image description text corresponding to all street view images.

[0032] For example, Figure 2 This is a schematic diagram of the process of constructing a text-image interleaved dataset provided by the present invention. Figure 2 As shown, taking a satellite image and its corresponding three street view images as an example, the four images are input into the text generation multimodal model (i.e. Figure 2 By combining the multimodal large model in

[15] , we can obtain text descriptions for images. After fusing them with geographic tags, we can obtain complete text descriptions for satellite images and street view images.

[0033] In the above embodiment, the singleness of the training dataset is avoided by constructing a text-image interleaved dataset. Training the urban multimodal large model based on the text-image interleaved dataset can improve the performance of the urban multimodal large model in complex urban environment tasks.

[0034] Step 120: Training the urban multimodal large model based on the image-text interleaved dataset, adjusting the perceptron parameters corresponding to the perceptron in the urban multimodal large model and the large language model parameters corresponding to the large language model, to obtain a trained target urban multimodal large model.

[0035] Specifically, the urban multimodal large model can be trained based on the image-text interleaved dataset. It is easy to understand that in the process of training the urban multimodal large model, it is also necessary to input the labels corresponding to the image-text interleaved dataset, and then the perceptron parameters corresponding to the perceptron in the urban multimodal large model and the large language model parameters corresponding to the large language model can be adjusted to obtain the trained target urban multimodal large model.

[0036] It should be noted that the urban multimodal large model includes a visual encoder, a perceptron and a large language model. In the technical solution of the present invention, the parameters corresponding to the visual encoder are frozen and no training is performed. Only the perceptron parameters corresponding to the perceptron and the large language model parameters corresponding to the large language model are adjusted.

[0037] In one embodiment, the training of the urban multimodal large model based on the image-text interleaved dataset, adjusting the perceptron parameters corresponding to the perceptrons in the urban multimodal large model and the large language model parameters corresponding to the large language model to obtain the trained target urban multimodal large model includes: Inputting the interleaved image and text dataset into a visual encoder in the urban multimodal large model, and obtaining satellite image visual features corresponding to each of the satellite images and street view image visual features corresponding to each of the street view images output by the visual encoder; Inputting satellite image visual features corresponding to each of the satellite images and street view image visual features corresponding to each of the street view images into the sensor, obtaining a plurality of image tags output by the sensor and text tags corresponding to each of the image tags; Based on each of the image tags and the text tags corresponding to each of the image tags, the perceptron parameters corresponding to the perceptron in the city multimodal large model and the large language model parameters corresponding to the large language model are adjusted to obtain a trained target city multimodal large model.

[0038] Specifically, Figure 3 This is a schematic diagram of the urban multimodal large model provided by the present invention, referring to Figure 3 The interleaved image-text dataset can be input into the visual encoder in the urban multimodal large model to obtain the satellite image visual features corresponding to each satellite image and the street view image visual features corresponding to each street view image output by the visual encoder. The satellite image visual features corresponding to each satellite image and the street view image visual features corresponding to each street view image can also be input into the perceptron to obtain multiple image tags and text tags corresponding to each image tag output by the perceptron. Furthermore, the perceptron parameters corresponding to the perceptron in the urban multimodal large model and the large language model parameters corresponding to the large language model can be adjusted based on each image tag and the text tags corresponding to each image tag to obtain the trained target urban multimodal large model.

[0039] In existing technology, images and text are typically encoded separately before training a large multimodal model. However, in the above embodiment, the interleaved image and text dataset is directly fed into the visual encoder, effectively integrating complementary information from different data sources and improving the performance of the urban multimodal model in complex urban environment tasks. The output of the visual encoder is then fed into a perceptron, which trains the large urban multimodal model based on its output, thereby improving its accuracy.

[0040] In one embodiment, the visual encoder determines the satellite image visual features corresponding to each of the satellite images and the street view image visual features corresponding to each of the street view images in the following manner: For each of the image interleaved subsets in the image-text interleaved dataset, the satellite image and each of the street view images in the image interleaved subset are encoded respectively, and satellite image visual features corresponding to the satellite image and street view image visual features corresponding to each of the street view images are output.

[0041] Specifically, the visual encoder processes each image interleaved subset in the image-text interleaved dataset separately, encodes the satellite image and each street view image in the image interleaved subset separately, and outputs the satellite image visual features corresponding to the satellite image. and the visual features of the street view images corresponding to each street view image .

[0042] In the above embodiment, the visual features of the satellite image and the street view image are extracted through the visual encoder, which lays the foundation for the subsequent training process.

[0043] In one embodiment, the sensor determines each image tag and the text tag corresponding to each image tag in the following manner: For each of the image interleaved subsets in the image-text interleaved dataset, using satellite image visual features corresponding to the satellite image as a query, performing cross-attention processing on the street view image visual features corresponding to each of the street view images, to obtain first fusion features corresponding to each of the street view images; Using the street view image visual features corresponding to each of the street view images as a query, performing cross-attention processing on the satellite image visual features corresponding to the satellite image, to obtain second fusion features corresponding to each of the street view images; determining a first visual embedding corresponding to the satellite image based on the second fusion features corresponding to each of the street view images and the satellite image visual features corresponding to the satellite image, and determining a first aligned text for the first visual embedding; determining a second visual embedding corresponding to each of the street view images based on the first fusion features and the street view image visual features respectively corresponding to each of the street view images, and determining a second aligned text for each of the second visual embeddings; Determining each of the image tags based on all of the first visual embeddings and all of the second visual embeddings corresponding to the image-text interleaved dataset; The text marks are determined based on all the first aligned texts and all the second aligned texts corresponding to the text-image interleaved dataset.

[0044] Specifically, refer to Figure 3 For each image interleaved subset in the image-text interleaved dataset, the satellite image visual features corresponding to the satellite image can be used as a query, and the street view image visual features corresponding to each street view image can be cross-attention processed to obtain the street view image. The corresponding first fusion feature You can also perform average pooling on the visual features of the street view images corresponding to all street view images, use the average pooling result as a query, and perform cross-attention processing on the visual features of the satellite image corresponding to the satellite image to obtain the street view image. The corresponding second fusion feature The first fusion feature and the second fusion feature can be expressed by the following formula: in, represents a multilayer perceptron, represents the result of average pooling, represents cross-attention processing, express feature dimension.

[0045] Furthermore, the fusion features can be selectively output through the gating module, and the first visual embedding corresponding to the satellite image can be determined based on the second fusion features corresponding to each street view image and the satellite image visual features corresponding to the satellite image. , and determine the first aligned text of the first visual embedding through semantic alignment. First visual embedding It can be expressed by the following formula: in, Represents the gating mechanism.

[0046] The gating module can also be used to determine the street view image based on the second fusion features corresponding to each street view image and the street view image visual features. The corresponding second visual embedding And determine the second aligned text of each second visual embedding through semantic alignment. Second visual embedding It can be expressed by the following formula: After determining the first visual embedding and the second visual embedding, each image tag can be determined based on all first visual embeddings and all second visual embeddings corresponding to the image-text interleaved dataset, and each text tag can also be determined based on all first aligned texts and all second aligned texts corresponding to the image-text interleaved dataset.

[0047] In the above embodiment, the setting of the sensor can enhance the understanding and perception of the city image during the visual encoding process.

[0048] In one embodiment, adjusting the perceptron parameters corresponding to the perceptron and the language model parameters corresponding to the language model in the urban multimodal large model based on the image tags and the text tags corresponding to the image tags includes: in, represents the objective function, represents the parameters that need to be adjusted for the urban multimodal model, Express The expectation operator of represents the logarithmic probability under preset historical information and conditions, represents the perceptron parameters, represents the large language model parameters, represents the first aligned text and the second aligned text, represents the image collection in the image-text interleaved dataset, Indicates the The text mark corresponding to the aligned text, Indicates the A text tag before the aligned text, Indicates the The image index corresponding to the aligned text Image tags, Indicates the The image index corresponding to the aligned text The previous image tag, represents the perceptron, represents the visual encoder, Indicates the The image index corresponding to the aligned text A collection of corresponding satellite and street view images.

[0049] Specifically, the text-graphic interleaved data set can be set to include images, the image set in the image-text interleaved dataset can be expressed as , the set With length Align text It is easy to understand that the first aligned text and the second aligned text are collectively referred to as aligned text.

[0050] The training goal of the large multimodal urban model is to accurately predict the next word token. In the above embodiment, setting the above objective function not only improves the model's ability to understand urban images, but also enhances its performance in processing complex text information, ultimately achieving a deep understanding of the urban environment.

[0051] The present invention provides a method for training a large urban multimodal model. The method constructs a text-image interleaved dataset based on multiple satellite images and multiple street view images corresponding to the target urban area, as well as satellite image description text corresponding to each satellite image and street view image description text corresponding to each street view image. The method then trains the large urban multimodal model based on the text-image interleaved dataset, adjusts the perceptron parameters corresponding to the perceptron in the large urban multimodal model and the large language model parameters corresponding to the large language model, and obtains a trained large urban multimodal model. The technical solution of the present invention uses a combination of satellite images, street view images, and text to form a text-image interleaved dataset to train the large urban multimodal model. This solves the problem of the existing technology of a single training data source and separate processing of images and text. Compared with the traditional large multimodal model, the method also incorporates a perceptron, so that the trained large urban multimodal model has a good understanding of the urban environment and can fully capture the spatial structure and functional characteristics of the city.

[0052] In one embodiment, the urban multimodal large model training method further includes: Acquire multiple fine-tuned satellite images and multiple fine-tuned street view images corresponding to the target urban area, and construct an instruction fine-tuning dataset based on all the fine-tuned satellite images and all the fine-tuned street view images; Based on the instruction fine-tuning data set, the perceptron parameters corresponding to the perceptron in the target city multimodal large model and the large language model parameters corresponding to the large language model are optimized to obtain a fine-tuned optimized city multimodal large model.

[0053] Specifically, multiple fine-tuning satellite images and multiple fine-tuning street view images corresponding to the target city area can also be obtained, and an instruction fine-tuning dataset can be constructed based on all the fine-tuning satellite images and all the fine-tuning street view images. The process of constructing the instruction fine-tuning dataset can be similar to the process of constructing the image-text interleaved dataset in step 110. Further, the perceptron parameters corresponding to the perceptron in the target city multimodal large model and the large language model parameters corresponding to the large language model can be optimized based on the instruction fine-tuning dataset to obtain a fine-tuned and optimized urban multimodal large model. The parameter optimization process for the target city multimodal large model is similar to the parameter adjustment process for the urban multimodal large model in step 120, and this embodiment of the present invention will not be repeated here.

[0054] It should be noted that the instruction fine-tuning dataset improves the ability of the target city multimodal large model to perform the following tasks.

[0055] (1) Satellite mission: Scene category: Select the scene to which the image belongs.

[0056] Object Reasoning: Answering questions about the location, existence, or quantity of specific objects.

[0057] Spatial Relationship Reasoning: Selecting the correct object relationships shown in an image.

[0058] Geolocation: Select the county to which this image belongs.

[0059] Indicator Prediction: Predict the population density or night light intensity in the image, ranging from 0.0 to 9.9.

[0060] (2) Street View Task: Scene Category: Select the scene or scenes to which the image belongs.

[0061] Object Reasoning: Answering questions about the location, existence, or quantity of specific objects.

[0062] Landmark Recognition: Select the correct landmark name displayed in the image.

[0063] Spatial Relationship Reasoning: Selecting the correct object relationships shown in an image.

[0064] Geolocation: Select the county to which this image belongs.

[0065] Index prediction: Predict the beauty, richness, and depression of the image, ranging from 0.0 to 9.9.

[0066] (3) Cross-view tasks combining satellite and street view: Indicator Prediction: Predict median income, poverty rate, total population, and depression rate for a collection of images within the same census tract, ranging from 0.0 to 9.9.

[0067] Spatial relationship reasoning: Determine where Street View images are located in satellite imagery of the same area.

[0068] For example, Figure 4 This is a task execution diagram of fine-tuning and optimizing the urban multimodal large model provided by the present invention, such as Figure 4 As shown in the figure, the fine-tuned and optimized urban multimodal large model has good output effects for street view tasks, remote sensing (satellite) tasks and cross-view tasks.

[0069] In the above embodiment, the target city multimodal big model is optimized by fine-tuning the data set through instructions, and the resulting fine-tuned optimized city multimodal big model further improves the understanding of the urban environment and improves the accuracy and applicability of the city multimodal big model in tasks such as urban socioeconomic indicator prediction and geographic location determination.

[0070] The urban multimodal large model training device provided by the present invention is described below. The urban multimodal large model training device described below and the urban multimodal large model training method described above can be referenced to each other.

[0071] Figure 5This is a schematic diagram of the structure of the urban multimodal large model training device provided by the present invention. Figure 5 As shown, the urban multimodal large model training device 500 includes the following modules: A construction module 510 is configured to construct an image-text interleaved dataset based on a plurality of satellite images and a plurality of street view images corresponding to a target urban area, as well as satellite image description texts corresponding to the satellite images and street view image description texts corresponding to the street view images. The training module 520 is used to train the urban multimodal large model based on the interleaved image and text dataset, adjust the perceptron parameters corresponding to the perceptron in the urban multimodal large model and the large language model parameters corresponding to the large language model, and obtain a trained target urban multimodal large model.

[0072] In one embodiment, the construction module 510 is specifically configured to: Acquire multiple satellite images and multiple street view images corresponding to the target city area, as well as a first geographic tag corresponding to each of the satellite images and a second geographic tag corresponding to each of the street view images; Inputting each of the satellite images into a preset text to generate a multimodal model, and obtaining a first text description corresponding to each of the satellite images; Inputting each of the street view images into the text to generate a multimodal model, thereby obtaining a second text description corresponding to each of the street view images; For each of the satellite images, determining a satellite image description text corresponding to the satellite image based on the first geographic tag and the first text description corresponding to the satellite image; For each of the street view images, determining a street view image description text corresponding to the street view image based on the second geographic tag and the second text description corresponding to the street view image; For each of the satellite images, construct an image interleaved subset based on the satellite image and a plurality of street view images corresponding to the satellite image; The image-text interleaved dataset is constructed based on the image interleaved subsets corresponding to all the satellite images, the satellite image text descriptions corresponding to all the satellite images, and the street view image description texts corresponding to all the street view images.

[0073] In one embodiment, the training module 520 is specifically configured to: Inputting the interleaved image and text dataset into a visual encoder in the urban multimodal large model, and obtaining satellite image visual features corresponding to each of the satellite images and street view image visual features corresponding to each of the street view images output by the visual encoder; Inputting satellite image visual features corresponding to each of the satellite images and street view image visual features corresponding to each of the street view images into the sensor, obtaining a plurality of image tags output by the sensor and text tags corresponding to each of the image tags; Based on each of the image tags and the text tags corresponding to each of the image tags, the perceptron parameters corresponding to the perceptron in the city multimodal large model and the large language model parameters corresponding to the large language model are adjusted to obtain a trained target city multimodal large model.

[0074] In one embodiment, the visual encoder determines the satellite image visual features corresponding to each of the satellite images and the street view image visual features corresponding to each of the street view images in the following manner: For each of the image interleaved subsets in the image-text interleaved dataset, the satellite image and each of the street view images in the image interleaved subset are encoded respectively, and satellite image visual features corresponding to the satellite image and street view image visual features corresponding to each of the street view images are output.

[0075] In one embodiment, the sensor determines each image tag and the text tag corresponding to each image tag in the following manner: For each of the image interleaved subsets in the image-text interleaved dataset, using satellite image visual features corresponding to the satellite image as a query, performing cross-attention processing on the street view image visual features corresponding to each of the street view images, to obtain first fusion features corresponding to each of the street view images; Using the street view image visual features corresponding to each of the street view images as a query, performing cross-attention processing on the satellite image visual features corresponding to the satellite image, to obtain second fusion features corresponding to each of the street view images; determining a first visual embedding corresponding to the satellite image based on the second fusion features corresponding to each of the street view images and the satellite image visual features corresponding to the satellite image, and determining a first aligned text for the first visual embedding; determining a second visual embedding corresponding to each of the street view images based on the first fusion features and the street view image visual features respectively corresponding to each of the street view images, and determining a second aligned text for each of the second visual embeddings; Determining each of the image tags based on all of the first visual embeddings and all of the second visual embeddings corresponding to the image-text interleaved dataset; The text marks are determined based on all the first aligned texts and all the second aligned texts corresponding to the text-image interleaved dataset.

[0076] In one embodiment, the training module 520 is further configured to: in, represents the objective function, represents the parameters that need to be adjusted for the urban multimodal model, Express The expectation operator of represents the logarithmic probability under preset historical information and conditions, represents the perceptron parameters, represents the large language model parameters, represents the first aligned text and the second aligned text, represents the image collection in the image-text interleaved dataset, Indicates the The text mark corresponding to the aligned text, Indicates the A text tag before the aligned text, Indicates the The image index corresponding to the aligned text Image tags, Indicates the The image index corresponding to the aligned text The previous image tag, represents the perceptron, represents the visual encoder, Indicates the The image index corresponding to the aligned text A collection of corresponding satellite and street view images.

[0077] In one embodiment, the urban multimodal large model training device 500 further includes a fine-tuning module, which is specifically used to: Acquire multiple fine-tuned satellite images and multiple fine-tuned street view images corresponding to the target urban area, and construct an instruction fine-tuning dataset based on all the fine-tuned satellite images and all the fine-tuned street view images; Based on the instruction fine-tuning data set, the perceptron parameters corresponding to the perceptron in the target city multimodal large model and the large language model parameters corresponding to the large language model are optimized to obtain a fine-tuned optimized city multimodal large model.

[0078] The present invention provides a large urban multimodal model training device. The device constructs a text-image interleaved dataset based on multiple satellite images and multiple street view images corresponding to the target urban area, as well as satellite image description text corresponding to each satellite image and street view image description text corresponding to each street view image. The device trains the large urban multimodal model based on the text-image interleaved dataset, adjusts the perceptron parameters corresponding to the perceptron in the large urban multimodal model and the large language model parameters corresponding to the large language model, and obtains a trained large urban multimodal model. The technical solution of the present invention uses a combination of satellite images, street view images, and text to form a text-image interleaved dataset to train the large urban multimodal model. This solves the problem of the existing technology of a single training data source and separate processing of images and text. Compared with the traditional large multimodal model, the device also incorporates a perceptron, so that the trained large urban multimodal model has a good understanding of the urban environment and can fully capture the spatial structure and functional characteristics of the city.

[0079] Figure 6 An example of a physical structure diagram of an electronic device is shown below. Figure 6 As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the urban multimodal large model training method, which includes: Constructing an image-text interleaved dataset based on a plurality of satellite images and a plurality of street view images corresponding to a target urban area, as well as satellite image description texts corresponding to the satellite images and street view image description texts corresponding to the street view images; The urban multimodal large model is trained based on the interleaved image-text dataset, and the perceptron parameters corresponding to the perceptron in the urban multimodal large model and the large language model parameters corresponding to the large language model are adjusted to obtain a trained target urban multimodal large model.

[0080] Furthermore, the logic instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0081] On the other hand, the present invention further provides a computer program product, comprising a computer program, which may be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the urban multimodal large model training method provided by the above methods, which includes: Constructing an image-text interleaved dataset based on a plurality of satellite images and a plurality of street view images corresponding to a target urban area, as well as satellite image description texts corresponding to the satellite images and street view image description texts corresponding to the street view images; The urban multimodal large model is trained based on the interleaved image-text dataset, and the perceptron parameters corresponding to the perceptron in the urban multimodal large model and the large language model parameters corresponding to the large language model are adjusted to obtain a trained target urban multimodal large model.

[0082] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the urban multimodal large model training method provided by the above methods, the method comprising: Constructing an image-text interleaved dataset based on a plurality of satellite images and a plurality of street view images corresponding to a target urban area, as well as satellite image description texts corresponding to the satellite images and street view image description texts corresponding to the street view images; The urban multimodal large model is trained based on the interleaved image-text dataset, and the perceptron parameters corresponding to the perceptron in the urban multimodal large model and the large language model parameters corresponding to the large language model are adjusted to obtain a trained target urban multimodal large model.

[0083] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0084] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for training a large multimodal urban model, characterized in that: include: Constructing an image-text interleaved dataset based on a plurality of satellite images and a plurality of street view images corresponding to a target urban area, as well as satellite image description texts corresponding to the satellite images and street view image description texts corresponding to the street view images; The urban multimodal large model is trained based on the interleaved image-text dataset, and the perceptron parameters corresponding to the perceptron in the urban multimodal large model and the large language model parameters corresponding to the large language model are adjusted to obtain a trained target urban multimodal large model.

2. The urban multimodal large model training method according to claim 1 is characterized in that: The constructing of the image-text interleaved dataset based on the multiple satellite images and the multiple street view images corresponding to the target city area, as well as the satellite image description texts corresponding to the satellite images and the street view image description texts corresponding to the street view images, includes: Acquire multiple satellite images and multiple street view images corresponding to the target city area, as well as a first geographic tag corresponding to each of the satellite images and a second geographic tag corresponding to each of the street view images; Inputting each of the satellite images into a preset text to generate a multimodal model, and obtaining a first text description corresponding to each of the satellite images; Inputting each of the street view images into the text to generate a multimodal model, thereby obtaining a second text description corresponding to each of the street view images; For each of the satellite images, determining a satellite image description text corresponding to the satellite image based on the first geographic tag and the first text description corresponding to the satellite image; For each of the street view images, determining a street view image description text corresponding to the street view image based on the second geographic tag and the second text description corresponding to the street view image; For each of the satellite images, construct an image interleaved subset based on the satellite image and a plurality of street view images corresponding to the satellite image; The image-text interleaved dataset is constructed based on the image interleaved subsets corresponding to all the satellite images, the satellite image text descriptions corresponding to all the satellite images, and the street view image description texts corresponding to all the street view images.

3. The urban multimodal large model training method according to claim 2 is characterized in that: The method of training the urban multimodal large model based on the image-text interleaved dataset, adjusting the perceptron parameters corresponding to the perceptron in the urban multimodal large model and the large language model parameters corresponding to the large language model to obtain the trained target urban multimodal large model includes: Inputting the interleaved image and text dataset into a visual encoder in the urban multimodal large model, and obtaining satellite image visual features corresponding to each of the satellite images and street view image visual features corresponding to each of the street view images output by the visual encoder; Inputting satellite image visual features corresponding to each of the satellite images and street view image visual features corresponding to each of the street view images into the sensor, obtaining a plurality of image tags output by the sensor and text tags corresponding to each of the image tags; Based on each of the image tags and the text tags corresponding to each of the image tags, the perceptron parameters corresponding to the perceptron in the city multimodal large model and the large language model parameters corresponding to the large language model are adjusted to obtain a trained target city multimodal large model.

4. The urban multimodal large model training method according to claim 3 is characterized in that: The visual encoder determines the satellite image visual features corresponding to each of the satellite images and the street view image visual features corresponding to each of the street view images in the following manner: For each of the image interleaved subsets in the image-text interleaved dataset, the satellite image and each of the street view images in the image interleaved subset are encoded respectively, and satellite image visual features corresponding to the satellite image and street view image visual features corresponding to each of the street view images are output.

5. The urban multimodal large model training method according to claim 3 is characterized in that: The sensor determines each image tag and the text tag corresponding to each image tag in the following manner: For each of the image interleaved subsets in the image-text interleaved dataset, using satellite image visual features corresponding to the satellite image as a query, performing cross-attention processing on the street view image visual features corresponding to each of the street view images, to obtain first fusion features corresponding to each of the street view images; Using the street view image visual features corresponding to each of the street view images as a query, performing cross-attention processing on the satellite image visual features corresponding to the satellite image, to obtain second fusion features corresponding to each of the street view images; determining a first visual embedding corresponding to the satellite image based on the second fusion features corresponding to each of the street view images and the satellite image visual features corresponding to the satellite image, and determining a first aligned text for the first visual embedding; determining a second visual embedding corresponding to each of the street view images based on the first fusion features and the street view image visual features respectively corresponding to each of the street view images, and determining a second aligned text for each of the second visual embeddings; Determining each of the image tags based on all of the first visual embeddings and all of the second visual embeddings corresponding to the image-text interleaved dataset; The text marks are determined based on all the first aligned texts and all the second aligned texts corresponding to the text-image interleaved dataset.

6. The urban multimodal large model training method according to claim 5 is characterized in that: The adjusting, based on each of the image tags and the text tags corresponding to each of the image tags, the perceptron parameters corresponding to the perceptron in the urban multimodal large model and the large language model parameters corresponding to the large language model, includes: in, represents the objective function, represents the parameters that need to be adjusted for the urban multimodal model, Express The expectation operator of represents the logarithmic probability under preset historical information and conditions, represents the perceptron parameters, represents the large language model parameters, represents the first aligned text and the second aligned text, represents the image collection in the image-text interleaved dataset, Indicates the The text mark corresponding to the aligned text, Indicates the A text tag before the aligned text, Indicates the The image index corresponding to the aligned text Image tags, Indicates the The image index corresponding to the aligned text The previous image tag, represents the perceptron, represents the visual encoder, Indicates the The image index corresponding to the aligned text A collection of corresponding satellite and street view images.

7. The urban multimodal large model training method according to any one of claims 1 to 6, characterized in that: The method further comprises: Acquire multiple fine-tuned satellite images and multiple fine-tuned street view images corresponding to the target urban area, and construct an instruction fine-tuning dataset based on all the fine-tuned satellite images and all the fine-tuned street view images; Based on the instruction fine-tuning data set, the perceptron parameters corresponding to the perceptron in the target city multimodal large model and the large language model parameters corresponding to the large language model are optimized to obtain a fine-tuned optimized city multimodal large model.

8. A large urban multimodal model training device, characterized in that: include: A construction module, configured to construct an image-text interleaved dataset based on a plurality of satellite images and a plurality of street view images corresponding to a target urban area, as well as satellite image description texts corresponding to each of the satellite images and street view image description texts corresponding to each of the street view images; The training module is used to train the urban multimodal large model based on the interleaved image and text dataset, adjust the perceptron parameters corresponding to the perceptron in the urban multimodal large model and the large language model parameters corresponding to the large language model, and obtain the trained target urban multimodal large model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the urban multimodal large model training method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the urban multimodal large model training method as described in any one of claims 1 to 7 is implemented.

Images