Multi-modal urban region representation learning method based on long text perception and noise suppression

Abstract

The invention discloses a multi-modal city region representation learning method based on long text perception and noise suppression, and belongs to the field of city calculation, and the method comprises the steps: obtaining a streetscape image in a city region and a plurality of corresponding initial long text descriptions; refining and removing illusion contents through image salient visual elements and a phrase level filtering strategy to obtain candidate texts, and screening the candidate text with the highest score based on a consensus evaluation mechanism; the method comprises the following steps: constructing a cross-modal pre-training framework, adopting a contrast language-image model to realize image and text representation alignment, introducing an extension interpolation strategy and a self-distillation mechanism based on momentum updating to obtain a pre-trained image encoder, extracting features of a plurality of images in the same area based on the pre-trained image encoder, and performing image feature extraction based on the extracted features. And carrying out average pooling to obtain urban region characterization. According to the method, noise interference in long text modeling can be effectively relieved, and the accuracy and robustness of urban region representation are improved.

Description

Technical Field

[0001] This invention relates to the field of urban computing, and in particular to a multimodal urban area representation learning method based on long text perception and noise suppression. Background Technology

[0002] Cities, as vital spaces for people's lives, work, and leisure, are becoming increasingly complex and diverse with the acceleration of urbanization. Utilizing multimodal urban data and combining it with deep learning methods to model and infer urban regional attributes has become an important research direction in the field of urban computing. This process enables a more comprehensive understanding of urban space, encompassing social, economic, and environmental aspects, thereby providing support for urban planning, sustainable development, and policy formulation.

[0003] Early research primarily relied on task-specific supervised learning methods, such as regional function identification, user income prediction, and economic status inference. However, these "one-to-one" methods not only depend on large amounts of labeled data, but the representations learned in one task are often difficult to transfer to other tasks, lacking generalizability. Therefore, in recent years, academia has begun to focus on self-supervised urban region representation learning, which obtains generalizable region representations through a "one-to-many" approach, avoiding the need to train models separately for each downstream task, thereby improving efficiency and applicability. For example, the paper "Urban Region Embedding via Multi-View Contrastive Prediction" proposes a method to mine common features between human movement data and point-of-interest data through multi-view contrastive prediction to model regional attributes; the paper "UrbanCLIP: Learning Text-Enhanced Urban Region Profiling with Contrastive Language-Image Pretraining from the Web" combines multimodal language models with a contrastive learning framework to align model-generated image captions with satellite images to enhance the multimodal understanding of urban regions.

[0004] While existing methods have improved the effectiveness of multimodal urban area representation to some extent, significant shortcomings remain. On one hand, multimodal language models have limited capabilities when processing long text descriptions; text encoders struggle to effectively model extremely long semantic sequences, leading to image encoders failing to fully capture internal details and multi-attribute relationships in complex street scene scenarios. On the other hand, large language models are prone to "illusion" phenomena during text generation, where generated descriptions do not match or deviate from the actual content of the image, such as including irrelevant words or omitting important entities. These problems reduce the accuracy and robustness of urban area representation results. Therefore, effectively suppressing noise interference while fully utilizing the rich semantic information of long texts remains a pressing technical challenge in this field. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a multimodal urban area representation learning method based on long text perception and noise suppression. This method utilizes a multimodal large language model to generate detailed long text descriptions for street view images and performs feature modeling based on a cross-modal pre-training framework. This invention can more comprehensively model the attribute features of urban areas, fully utilize the world knowledge contained in the large language model, and effectively improve the accuracy and robustness of cross-modal representations in complex scenarios, thereby overcoming the deficiencies of existing technologies in long text processing and noise resistance.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0007] Firstly, this invention proposes a multimodal urban area representation learning method based on long text perception and noise suppression, comprising the following steps:

[0008] S1, divide the city into different areas, collect street view images in each city area, and input them into several multimodal large language models to generate several corresponding initial long texts;

[0009] S2, refine the initial long text based on the salient visual elements of the street view image to generate candidate text;

[0010] S3 employs a consensus evaluation mechanism to filter candidate texts from different multimodal large language models and selects the candidate text with the highest consensus score to form a city street view multimodal dataset that matches street view images. This dataset contains street view images and candidate texts that correspond one-to-one with the street view images.

[0011] S4. Construct a cross-modal pre-training framework. The framework uses the contrastive language-image pre-trained CLIP model as the backbone network and introduces a momentum-update-based self-distillation mechanism. The pseudo-target generated by the momentum teacher model is used to constrain the student model. The urban street scene multimodal dataset is input into the cross-modal pre-training framework. The objective function is optimized by combining the contrastive learning loss and the self-distillation loss until the network converges, thus obtaining the pre-trained image encoder.

[0012] S5: Based on the pre-trained image encoder, visual features are extracted from all street view images within the target city area. Taking the city area as a unit, the visual features of all street view images within the area are averaged and pooled to obtain the representation of the city area.

[0013] Furthermore, the refinement process includes:

[0014] Extract salient visual elements from street view images;

[0015] Different multimodal large language models are invoked to generate initial short text based on the salient visual elements;

[0016] A phrase-level filtering strategy is used to remove the illusionary content of the initial long and short texts, retaining the valid text;

[0017] Candidate text is generated by fusing salient visual elements from street view images, effective text from the initial short text, and effective text from the initial long text derived from the same multimodal large language model.

[0018] Furthermore, the pre-training strategy of the cross-modal pre-training framework is as follows: the student model updates its parameters through backpropagation, and the teacher model updates its parameters by moving exponential averaging of the student model parameters.

[0019] Furthermore, the momentum-update-based self-distillation mechanism includes:

[0020] Street view image representations are generated using the image encoder and text encoder of the student model. and long text representation The momentum representation of street view images is generated by the momentum image encoder and momentum text encoder in the teacher model. Momentum representation of long text ;

[0021] Construct a dynamic queue of length K to store the latest K street view image momentum representations and long text momentum representations generated by the teacher model. The queue is updated using a first-in-first-out strategy.

[0022] Based on street view image representation Long text representation and their respective street view image momentum representations in the dynamic queue Momentum representation of long text Calculate the street view image similarity in the student model Similarity with long text Momentum representation based on street view images Momentum representation of long text and their respective street view image momentum representations in the dynamic queue Momentum representation of long text Calculate the street view image similarity in momentum representation. Similarity of long texts under momentum representation , as a pseudo-target of the teacher model;

[0023] Calculation of self-distillation loss based on KL divergence:

[0024]

[0025] in, This is due to self-distillation loss. For the preset batch size, For Kullback-Leibler divergence, This represents the similarity between the street view image and the momentum representations of all long texts in the dynamic queue. For long texts Similarity to the momentum representations of all street view images in the dynamic queue. Street View Images The pseudo-targets for similarity to the momentum representations of all long texts in the dynamic queue. for Similarity pseudo-targets corresponding to the momentum representations of all street view images in the dynamic queue.

[0026] Furthermore, the similarity calculation formulas for the student model and the pseudo-target calculation formulas for the teacher model are as follows:

[0027] ;

[0028] ;

[0029] ;

[0030] ;

[0031] in, Street view image representation Momentum representation of long text in dynamic queues cosine similarity, Momentum representation of street view images in dynamic queues cosine similarity, It is an exponential function. image With the dynamic queue The similarity between K long texts, Stacking to form a three-dimensional tensor ; For long texts With the dynamic queue The similarity between K street view images Stacking to form a three-dimensional tensor ; Momentum representation of street view images Momentum representation of long text in dynamic queues cosine similarity, Momentum representation for long text Momentum representation of street view images cosine similarity, To match street view images and the first in the dynamic queue K pseudo-targets corresponding to each long text Stacking to form a three-dimensional tensor ; For long texts and the first in the dynamic queue K pseudo-targets corresponding to street view images Stacking to form a three-dimensional tensor .

[0032] Furthermore, the process of using a consensus evaluation mechanism to screen candidate texts includes:

[0033] (3.1) Use a fact parser to extract core semantic information from each candidate text, including objects, attributes and their interrelationships;

[0034] (3.2) For the core semantic information of each pair of candidate texts, perform exact matching, synonym matching and fuzzy matching in sequence;

[0035] (3.3) Based on the core semantic information in (3.1) and the three matching results in (3.2), calculate the F1 score of the core semantic information according to object, attribute, and relationship type, and generate the consensus score C-score through weighted fusion. The calculation formula is as follows:

[0036]

[0037] in, , , The F1 scores are categorized into three types: objects, attributes, and relationships. Scale factor;

[0038] (3.4) Calculate the average consensus score of each candidate text with all other candidate texts as the final consensus score to select the optimal text.

[0039] Further, in step (3.2), exact matching refers to checking the complete consistency of the core elements in two candidate texts, determining whether the core elements are completely matched, and recording the first matching element; synonym matching refers to removing the first matching element from the core semantic information, determining whether there is a semantically equivalent matching relationship between the remaining core elements in the two candidate texts based on the synonym set, and recording the second matching element; fuzzy matching refers to vectorizing the remaining core elements that did not pass the first two steps of matching, and calculating the matching score between the core elements based on the vectorized encoding to evaluate their semantic similarity, and recording the third matching element.

[0040] Furthermore, the contrast learning loss is as follows:

[0041] ;

[0042] in, To compare the losses, For the preset batch size, Street View image Similarity between the momentum representations of all long texts in the dynamic queue and the similarity between the two representations. It is a long text Similarity between the momentum representations of all street view images in the dynamic queue and the momentum representations of all street view images.

[0043] Furthermore, the CLIP model utilizes an information-preserving extended interpolation strategy to extend the positional embeddings of the text encoder, including retaining the first 20 positional embeddings unchanged and extending the remaining 57 positional embeddings using a preset interpolation ratio to support the input of long text sequences.

[0044] Furthermore, the phrase-level filtering strategy includes:

[0045] Visual element phrases are extracted from the initial long text and the initial short text using a fact parser. The OWLv2 open vocabulary object detection model is then used to score the visual element phrases, and phrases with scores below a preset threshold are removed. This yields the effective information from the initial long text and the effective information from the initial short text, respectively.

[0046] Secondly, this invention proposes a multimodal urban area representation learning system based on long text perception and noise suppression, which is used to implement the above-mentioned interactive front-end generation method.

[0047] The beneficial effects of this invention are:

[0048] (1) This invention utilizes the world knowledge contained in the pre-trained multimodal large language model to perform deep modeling of street view image data, which can fully explore the detailed features inside the urban space and improve the comprehensiveness of regional representation.

[0049] (2) This invention supplements long text descriptions by introducing prominent visual elements from street view images, and combines phrase-level filtering strategies with consensus-based evaluation mechanisms to effectively reduce semantic omissions and illusionary content that occur during text generation, thereby improving the accuracy and reliability of long text descriptions.

[0050] (3) This invention proposes an extended interpolation strategy to extend the position embedding of the text encoder, which breaks through the limitation of the input length of the traditional pre-trained model, enabling the model to process longer text sequences, thereby more fully mining and utilizing the multi-attribute semantic information in complex street scene scenarios.

[0051] (4) In the cross-modal pre-training process, the present invention introduces a self-distillation mechanism based on momentum update, and uses the pseudo-target generated by the teacher model to constrain the student model, which can effectively improve the representation robustness under noisy text supervision and ensure the stability of learning results. Attached Figure Description

[0052] Figure 1 This is an overall framework diagram of a specific embodiment provided by the present invention;

[0053] Figure 2 This is a flowchart of the data preprocessing process in a specific embodiment provided by the present invention;

[0054] Figure 3 This is a cross-modal pre-training framework diagram provided in a specific embodiment of the present invention. Detailed Implementation

[0055] The present invention will be further described and illustrated below with reference to specific embodiments. The embodiments described are merely examples of the content of this disclosure and do not limit the scope of the invention. The technical features of each embodiment in the present invention can be combined accordingly, provided that there is no mutual conflict.

[0056] like Figure 1As shown, to address the problem of insufficient capture of high-level semantics and potential urban knowledge in urban area representation modeling due to the complexity of street scenes, this invention provides a multimodal urban area representation learning method based on long text perception and noise suppression. This method utilizes a multimodal large language model to generate detailed long text descriptions for street view images, extracts significant visual elements from the street view images to supplement missing semantic information in the text, uses phrase-level filtering strategies to remove illusory information inconsistent with the image content, and uses a consensus mechanism to screen high-quality candidate descriptions, thereby solving the problems of semantic omissions and illusions in the generated descriptions. Figure 2 As shown; to support long text input, the position embedding of the text encoder is further extended through an extended interpolation strategy; in the feature modeling process based on the cross-modal pre-training framework, a self-distillation mechanism based on momentum update is introduced, using pseudo-targets generated by the teacher model to constrain the student model, thereby improving the representation robustness under noisy supervision conditions, such as Figure 3 As shown, the visual features of the regional street view images are extracted using a pre-trained image encoder, and multiple image features of the same region are aggregated to obtain a stable and accurate representation of the urban area.

[0057] The specific steps include:

[0058] Step 1: Obtain street view images of the urban area and their corresponding initial long text.

[0059] First, based on the latitude and longitude information of the urban space, a grid division method is used to evenly divide the urban space into multiple urban areas of equal size. Within each urban area, street view images are acquired based on the urban road network at a preset sampling interval (in this embodiment, the sampling interval is set to 500m). The street view images are analyzed, and a detailed description of urban infrastructure, traffic, human activities, natural elements, and the overall function of the streets in the images is provided. Predefined prompts are generated and input accordingly. Different multimodal large language models generate different outputs from the same street view image. The first candidate text description is used to form the initial long text.

[0060] Step 2: Given the risk of illusions and missing visual elements in the text generated by multimodal large language models, a multi-model collaboration method is used to refine the initial long text in order to obtain a more comprehensive and reliable long text description.

[0061] The specific steps include:

[0062] (1) Extract salient visual elements from street view images and generate initial short text descriptions based on salient visual elements and multimodal large language models;

[0063] First, the street view image is segmented using a Segmentation Arbitrary Model (SAM), and the maximum rectangle algorithm is combined to reduce the overlap between segmented regions, ultimately generating a set of cropped bounding boxes for extracting salient visual elements from the street view image. Second, for each detected salient visual element, the multimodal large language model is invoked to generate... An initial short text is used to supplement visual details missing in the initial long text.

[0064] (2) A phrase-level filtering strategy is used to filter the hallucinatory content in the initial long text and the initial short text;

[0065] To suppress illusory content generated by the large language model, this invention introduces a phrase-level filtering mechanism to further filter the initial long and short texts. First, a fact parser based on the T5 model is used to extract visual element phrases from the initial long and short texts. This model is a natural language processing tool specifically designed for accurately extracting, verifying, and organizing factual information from text. The principle of the fact parser is not elaborated here. Then, the Open Lexical Object Detection Model (OWLv2) is used to score the visual element phrases, removing phrases with scores below 0.01. This removes illusory information inconsistent with the image content from the initial long and short texts, yielding the effective information from the initial long and short texts respectively. Finally, the multimodal large language model is used to fuse the salient visual elements, the effective information from the initial long text, and the effective information from the initial short text to generate a more semantically complete, detailed, and consistent image with the street view. A long text description was selected as the second candidate text.

[0066] Step 3: Design a consensus-based evaluation mechanism to screen second candidate texts from different multimodal large language models, and select the second candidate text with the highest consensus score to form a city street scene multimodal dataset that matches street scene images one by one. This method can use the consistency between texts as the evaluation criterion in an unsupervised environment and determine the reliability of generated texts in the absence of manual annotation.

[0067] The specific steps include:

[0068] (1) The core semantic information is extracted from each second candidate text using a fact parser based on the T5 model. The core semantic information includes candidate elements such as objects (obj), attributes (attr), and their interrelationships (rel). The set of candidate elements is denoted as . and ,in and These represent the second candidate texts from two different multimodal large language models. Subsequently, for the core semantic information of the two different second candidate text descriptions, the following three matching strategies are used in sequence for filtering: 1) Check whether there are candidate elements that are exactly matched between the two core semantic information, and obtain a set of candidate elements that are exactly matched. and 2) After removing the matching candidate elements from 1), the WordNet semantic database is used to obtain the synonym set of the remaining candidate elements in the two core semantic information. Then, the remaining candidate elements are matched when the synonym sets overlap, resulting in a set of candidate elements for synonym matching. and 3) After removing the matching candidate elements in 2), the remaining candidate elements in the two core semantic information are vectorized and encoded using the Sentence-BERT model. Based on the encoding, the matching score between the remaining candidate elements in the two core semantic information is calculated. The calculation formula is as follows:

[0069]

[0070]

[0071] in, For the Sentence-BERT model, and The second candidate text after removing the matching elements of 1) and 2), respectively. and The remaining candidate elements include three types: objects (obj), attributes (attr), and their relationships (rel). Finally, the highest value in each category is taken as the matching score. and .

[0072] (2) After obtaining the matching results in (1), calculate the precision of the core semantic information according to the three types of objects (obj), attributes (attr) and their interrelationships (rel). and recall rate The specific calculation formula is as follows:

[0073]

[0074]

[0075] in, This represents the number of elements of the corresponding type in the set.

[0076] Based on this result, consensus scores are generated for each pair of candidate texts. The calculation formula is as follows:

[0077]

[0078]

[0079] in, and As a scale factor, The F1 scores are the corresponding scores for the three types of core semantic information.

[0080] Repeat step three to obtain the consensus score between each second candidate text and all other second candidate texts corresponding to the same street view image, and take the average to obtain the consensus score of the second candidate text.

[0081] After obtaining the consensus scores of the second candidate texts corresponding to all multimodal large language models, the second candidate text with the highest consensus score is selected as the final output long text, thus forming a city street scene multimodal dataset with one-to-one pairing of street scene images and long texts. Where S represents the number of street view images, Represents the i-th street view image, This represents the long text corresponding to the i-th street view image.

[0082] Step 4: Construct a cross-modal pre-training framework.

[0083] (1) Construct student and teacher models respectively. In order to effectively transfer the semantic knowledge contained in long text descriptions to visual representations, this invention uses the contrastive language-image pre-trained CLIP model as the backbone network of the student model. This model contains an image encoder and a text encoder to achieve the representation alignment of long texts and street view images in multimodal datasets. At the same time, considering that some urban communities have high similarity in visual features, it may be possible that some negative sample long texts can still reasonably describe the content of street view images that are not paired with themselves. This invention introduces a self-distillation mechanism based on momentum update, that is, the momentum version of the student model is used as the teacher model. The teacher model contains a momentum image encoder and a momentum text encoder. The parameters are updated by backpropagation of the student model. The parameters of the teacher model are updated by moving exponential averaging of the parameters of the student model until the network converges.

[0084] Since the input sequence length of the text encoder in the CLIP model is limited to 77 tokens, it is difficult to directly process long text descriptions with an average length exceeding 100 words. Therefore, this invention proposes an information-preserving extended interpolation strategy to support long text input. This involves retaining the embeddings at the first 20 positions unchanged, while applying an interpolation ratio to the remaining 57 positions. Extended interpolation is performed to expand the positional embedding of the text encoder, calculated as follows:

[0085]

[0086] Where P is the position embedding of the original CLIP model. For the position embedding of the CLIP model after interpolation, To mark the location, To round down, This is for rounding up.

[0087] (2) Input the city street scene multimodal dataset into the student model and the teacher model respectively.

[0088] (3) The street view image representation is generated by encoding the image encoder and text encoder of the student model. and long text representation The momentum representation of street view images is generated by the momentum image encoder and momentum text encoder of the teacher model. Momentum representation of long text Furthermore, two dynamic queues are constructed in the teacher model: one for street view image momentum representation and the other for long text momentum representation. , where K represents the dynamic queue length, and the queue is updated using a first-in-first-out strategy.

[0089] (4) Calculate the similarity between the street view image and the long text. The calculation formula is as follows:

[0090]

[0091]

[0092] in, Street view image representation Momentum representation of long text in dynamic queues cosine similarity, , Momentum representation of street view images in dynamic queues cosine similarity, It is an exponential function. image With the dynamic queue Similarity between long texts With the dynamic queue Similarity between street view images.

[0093] (5) A contrastive learning mechanism is used to align the representations of street view images and long texts, and the contrastive loss is output. The calculation formula is as follows:

[0094]

[0095] in, The batch size is preset by the user. Street View image The similarity between the dynamic queue and all long text momentum representations is determined by K elements. Stacked together It is a long text The similarity between the momentum representations of all street view images in the dynamic queue and the K similarities is determined by the following: The results are stacked and the comparison loss is output in batches.

[0096] (6) By using a momentum-based self-distillation mechanism (i.e., using the momentum version of the student model as the teacher model), the pseudo-target generated by the teacher model is transmitted to the student model, which can provide additional supervision signals for the street view image-long text representation in the student model. The self-distillation loss constrains the student model to maintain semantic consistency under noisy conditions, thereby improving the robustness of representation learning under noisy text self-supervision conditions.

[0097] Specifically, the pseudo-target generated by the teacher model is first obtained, which is an image-text similarity calculated based on street view image representation, long text representation and their respective momentum representations. The calculation formula is as follows:

[0098]

[0099]

[0100] in, Momentum representation of street view images Momentum representation of long text in dynamic queues cosine similarity, Momentum representation for long text Momentum representation of street view images cosine similarity, To match street view images and the first in the dynamic queue A pseudo-target corresponding to a long text. For long texts and the first in the dynamic queue The pseudo-targets corresponding to the street view images.

[0101] Self-distillation loss The calculation formula is as follows:

[0102]

[0103] in, Street View Images The pseudo-targets corresponding to the momentum representations of all long texts in the dynamic queue consist of K elements. Stacked together The pseudo-targets corresponding to the momentum representations of all street view images in the dynamic queue consist of K elements. Stacked together For the Kullback-Leibler divergence. When A value that is too large indicates that the current parameter optimization direction of the student model may be biased; conversely, when... If the value is too small, it means that the current parameter optimization direction is basically correct.

[0104] This invention designs a pre-training strategy for the cross-modal pre-training framework, whereby the student model is based on an objective function. Backpropagation updates parameters; the teacher model's parameters are updated by a moving exponential average of the student model's parameters, and the objective function is optimized. The process continues until the network converges, ultimately yielding the pre-trained image encoder.

[0105] objective function By contrast loss With self-distillation loss The components together are calculated using the following formula:

[0106]

[0107] in, This is the balance coefficient.

[0108] Step 5: Based on the pre-trained image encoder, visual features are extracted from all street view images. Taking urban areas as units, the visual features of all street view images within an area are averaged and pooled to obtain the representation of the urban area. This representation can effectively capture the essential features and functional attributes of the urban area and can serve as the basic representation to support various downstream urban computing tasks, providing technical support for sustainable urban development.

[0109] In this embodiment, the street view data comes from the Beijing street view image dataset. The multimodal large language models used are LLaMA-Adapter V2, ShareGPT4V-7B, Qwen2.5-VL-7B, DeepSeekVL2-tiny, and InternVL3-8B. The CLIP model uses ViT-B / 16 as the basic visual encoding structure. Population, GDP, night light data, number of Dianping reviews, and carbon emissions are selected as downstream task indicators. The representations of different urban areas in Beijing are obtained through steps one to five. The training rounds are set to 1, the batch size is set to 64, the optimizer is Adam, and the initial learning rate is 0.001.

[0110] The generated regional representations were evaluated based on the aforementioned task indicators. The results are shown in Table 1. It can be observed that, compared to other traditional urban regional representation learning models, the proposed method achieves higher decision coefficients in the five tasks. The values ​​were 0.705, 0.440, 0.514, 0.591, and 0.677, respectively, and the root mean square errors (RMSE) were 0.686, 1.373, 0.562, 1.976, and 0.496, respectively. Both the RMSE and the RMSE are superior to the traditional learning models mentioned above, proving that the method of the present invention can accurately characterize the attribute features of urban areas.

[0111] Table 1 Comparison Results

[0112]

[0113] Based on the same inventive concept, this invention also provides a multimodal urban area representation learning system based on long text perception and noise suppression, comprising:

[0114] The text generation module is used to divide the city into different areas, collect street view images of each city area, and input them into several multimodal large language models to generate several corresponding initial long texts.

[0115] The text refinement module is used to refine the initial long text based on the salient visual elements of the street view image and then generate candidate text.

[0116] The text filtering module is used to filter candidate texts from different multimodal large language models and select the candidate text with the highest consensus score to form a city street scene multimodal dataset that matches street scene images.

[0117] The cross-modal pre-training module is used to build a cross-modal pre-training framework to obtain a pre-trained image encoder;

[0118] The region pooling module is used to extract visual features from all street view images based on the pre-trained image encoder, and to perform average pooling on the visual features of all street view images within a city region to obtain the representation of the city region.

[0119] For the system embodiments, since they basically correspond to the method embodiments, relevant details can be found in the descriptions of the method embodiments; the implementation methods of the modules will not be repeated here. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0120] The system embodiments of the present invention can be applied to any device with data processing capabilities, such as a computer or other similar device. The system embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution.

[0121] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A multimodal urban area representation learning method based on long text perception and noise suppression, characterized in that, Includes the following steps: S1, divide the city into different areas, collect street view images in each city area, and input them into several multimodal large language models to generate several corresponding initial long texts; S2, refine the initial long text based on the salient visual elements of the street view image to generate candidate text; S3 employs a consensus evaluation mechanism to filter candidate texts from different multimodal large language models and selects the candidate text with the highest consensus score to form a city street view multimodal dataset that matches street view images. This dataset contains street view images and candidate texts that correspond one-to-one with the street view images. S4. Construct a cross-modal pre-training framework. The framework uses the contrastive language-image pre-trained CLIP model as the backbone network and introduces a momentum-update-based self-distillation mechanism. The pseudo-target generated by the momentum teacher model is used to constrain the student model. The urban street scene multimodal dataset is input into the cross-modal pre-training framework. The objective function is optimized by combining the contrastive learning loss and the self-distillation loss until the network converges, thus obtaining the pre-trained image encoder. S5: Based on the pre-trained image encoder, visual features are extracted from all street view images within the target city area. Taking the city area as a unit, the visual features of all street view images within the area are averaged and pooled to obtain the representation of the city area.

2. The multimodal urban region representation learning method based on long text perception and noise suppression according to claim 1, characterized in that, The refinement process includes: Extract salient visual elements from street view images; Different multimodal large language models are invoked to generate initial short text based on the salient visual elements; A phrase-level filtering strategy is used to remove the illusionary content of the initial long and short texts, retaining the valid text; Candidate text is generated by fusing salient visual elements from street view images, effective text from the initial short text, and effective text from the initial long text derived from the same multimodal large language model.

3. The multimodal urban area representation learning method based on long text perception and noise suppression according to claim 1, characterized in that, The pre-training strategy of the cross-modal pre-training framework is as follows: the student model updates its parameters through backpropagation, and the teacher model updates its parameters by moving exponential averaging of the student model parameters.

4. The multimodal urban area representation learning method based on long text perception and noise suppression according to claim 1, characterized in that, The self-distillation mechanism based on momentum update includes: Street view image representations are generated using the image encoder and text encoder of the student model. and long text representation The momentum representation of street view images is generated by the momentum image encoder and momentum text encoder in the teacher model. Momentum representation of long text ; Construct a dynamic queue of length K to store the latest K street view image momentum representations and long text momentum representations generated by the teacher model. The queue is updated using a first-in-first-out strategy. Based on street view image representation Long text representation and their respective street view image momentum representations in the dynamic queue Momentum representation of long text Calculate the street view image similarity in the student model Similarity with long text Momentum representation based on street view images Momentum representation of long text and their respective street view image momentum representations in the dynamic queue Momentum representation of long text Calculate the street view image similarity in momentum representation. Similarity of long texts under momentum representation , as a pseudo-target of the teacher model; Calculation of self-distillation loss based on KL divergence: ； in, This is due to self-distillation loss. For the preset batch size, For Kullback-Leibler divergence, This represents the similarity between the street view image and the momentum representations of all long texts in the dynamic queue. For long texts Similarity between the momentum representations of all street view images in the dynamic queue and the momentum representation of the images. Street View Images The pseudo-targets for similarity to the momentum representations of all long texts in the dynamic queue. for Similarity pseudo-targets corresponding to the momentum representations of all street view images in the dynamic queue.

5. The multimodal urban area representation learning method based on long text perception and noise suppression according to claim 4, characterized in that, The similarity calculation formulas for the student model and the pseudo-target calculation formulas for the teacher model are as follows: ； ； ； ； in, Street view image representation Momentum representation of long text in dynamic queues cosine similarity, Momentum representation of street view images in dynamic queues cosine similarity, It is an exponential function. image With the dynamic queue The similarity between K long texts, Stacking to form a three-dimensional tensor ; For long texts With the dynamic queue The similarity between K street view images Stacking to form a three-dimensional tensor ; Momentum representation of street view images Momentum representation of long text in dynamic queues cosine similarity, Momentum representation for long text Momentum representation of street view images cosine similarity, To match street view images and the first in the dynamic queue K pseudo-targets corresponding to each long text Stacking to form a three-dimensional tensor ; For long texts and the first in the dynamic queue K pseudo-targets corresponding to street view images Stacking to form a three-dimensional tensor .

6. The multimodal urban area representation learning method based on long text perception and noise suppression according to claim 1, characterized in that, The process of using a consensus evaluation mechanism to screen candidate texts includes: (3.1) Use a fact parser to extract core semantic information from each candidate text, including objects, attributes and their interrelationships; (3.2) For the core semantic information of each pair of candidate texts, perform exact matching, synonym matching and fuzzy matching in sequence; (3.3) Based on the core semantic information in (3.1) and the three matching results in (3.2), calculate the F1 score of the core semantic information according to object, attribute, and relationship type, and generate the consensus score C-score through weighted fusion. The calculation formula is as follows: ； in, , , The F1 scores are categorized into three types: objects, attributes, and relationships. Scale factor; (3.4) Calculate the average consensus score of each candidate text with all other candidate texts as the final consensus score to select the optimal text.

7. The multimodal urban area representation learning method based on long text perception and noise suppression according to claim 6, characterized in that, In step (3.2), exact matching refers to checking the complete consistency of the core elements in two candidate texts, determining whether the core elements are completely matched, and recording the first matching element; synonym matching refers to removing the first matching element from the core semantic information, determining whether there is a semantically equivalent matching relationship between the remaining core elements in the two candidate texts based on the synonym set, and recording the second matching element; fuzzy matching refers to vectorizing the remaining core elements that did not pass the first two steps of matching, and calculating the matching score between the core elements based on the vectorized encoding to evaluate their semantic similarity, and recording the third matching element.

8. The multimodal urban area representation learning method based on long text perception and noise suppression according to claim 1, characterized in that, The CLIP model uses an information-preserving extended interpolation strategy to extend the positional embeddings of the text encoder, including keeping the first 20 positional embeddings unchanged and extending the remaining 57 positional embeddings using a preset interpolation ratio to support the input of long text sequences.

9. A multimodal urban region representation learning method based on long text perception and noise suppression according to claim 2, characterized in that, The phrase-level filtering strategy includes: Visual element phrases are extracted from the initial long text and the initial short text using a fact parser. The OWLv2 open vocabulary object detection model is then used to score the visual element phrases, and phrases with scores below a preset threshold are removed. This yields the effective information from the initial long text and the effective information from the initial short text, respectively.

10. A multimodal urban area representation learning system based on long text perception and noise suppression, used to implement the multimodal urban area representation learning method based on long text perception and noise suppression as described in claim 1, characterized in that the system... include: The text generation module is used to divide the city into different areas, collect street view images of each city area, and input them into several multimodal large language models to generate several corresponding initial long texts. The text refinement module is used to refine the initial long text based on the salient visual elements of the street view image and then generate candidate text. The text filtering module is used to filter candidate texts from different multimodal large language models and select the candidate text with the highest consensus score to form a city street scene multimodal dataset that matches street scene images. The cross-modal pre-training module is used to build a cross-modal pre-training framework to obtain a pre-trained image encoder; The region pooling module is used to extract visual features from all street view images based on the pre-trained image encoder, and then perform average pooling on the visual features of all street view images within a city region to obtain the representation of that city region.

Images