Multi-modal basic model training method, urban area identification method and device
Through the training and knowledge distillation technology of multimodal basic model, the problem of low accuracy and efficiency in urban area identification is solved, and high accuracy and generalization capabilities are achieved under limited data, which is suitable for diversified urban area identification. Task.
Patent Information
- Application Number
- CN202411171380.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-12-03
AI Technical Summary
The existing technology is difficult to obtain high accuracy and efficiency in urban area identification. It is mainly due to the lack of geospatial characteristics design and generalization capabilities for urban data, which leads to poor generalization of models and cannot be effectively applied to diversified urban area identification tasks. .
Using the training method of multimodal basic model, by obtaining data fragments of multiple data modes, using pre-trained general basic models for knowledge distillation, generate vector representations that can fully express urban regional characteristics, and improve the generalization of the model and Understanding ability.
It realizes high accuracy and efficiency of urban area identification under limited unlabeled data, can be applied to a variety of downstream tasks, improves the accuracy and efficiency of urban area identification, and avoids complex model design and large-scale annotation for different tasks. Data requirements.
Smart Images

Figure CN119067237A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence technology, especially to technical fields such as smart cities, natural language processing, computer vision, and basic models, and specifically to a training method and device for a multimodal basic model, a method and device for identifying urban areas, an electronic device, a computer-readable storage medium, and a computer program product. Background Art
[0002] As the global urbanization process continues to accelerate, the complexity and diversity of regions as the basic units of cities are becoming increasingly prominent. Regions not only carry people's living, working and entertainment activities, but also reflect the socio-economic status and vitality of cities. Therefore, it is increasingly important to quantitatively represent and understand urban regions, which helps us gain a deeper insight into the internal structure and dynamic changes of cities, and thus provide decision-making support for scientific and reasonable urban planning and governance.
[0003] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention
[0004] The present disclosure provides a method and device for training a multimodal basic model, a method and device for identifying an urban area, an electronic device, a computer-readable storage medium, and a computer program product.
[0005] According to one aspect of the present disclosure, a training method for a multimodal base model is provided, comprising: obtaining first city data of a first sample urban area, the first city data comprising a plurality of first data segments respectively corresponding to a plurality of data modalities; inputting the first city data into the multimodal base model to obtain prediction vector representations of each of the plurality of first data segments output by the multimodal base model; obtaining a plurality of pre-trained general base models, each of the plurality of general base models corresponding to at least one of the plurality of data modalities; for each of the plurality of general base models: using the general base model, generating a vector representation label of a first data segment of a corresponding data modality; determining a knowledge distillation loss of the general base model based on the vector representation label and the prediction vector representation of the first data segment; determining an overall loss of the multimodal base model based at least on the knowledge distillation losses of the plurality of general base models; and adjusting parameters of the multimodal base model based on the overall loss.
[0006] According to one aspect of the present disclosure, a method for identifying urban areas is provided, comprising: obtaining urban data of a target urban area to be identified, the urban data comprising a plurality of data segments corresponding to a plurality of data modalities respectively; inputting the urban data into a multimodal basic model to obtain a segment vector representation of each of the plurality of data segments output by the multimodal basic model, the multimodal basic model being obtained by the above-mentioned training method; determining a regional vector representation of the target urban area based on the respective segment vector representations of the plurality of data segments; and identifying target information of the target urban area using a prediction network based on the regional vector representation.
[0007] According to one aspect of the present disclosure, a training device for a multimodal base model is provided, comprising: a first acquisition module, configured to acquire first city data of a first sample urban area, the first city data comprising a plurality of first data segments respectively corresponding to a plurality of data modalities; a prediction module, configured to input the first city data into the multimodal base model to obtain a prediction vector representation of each of the plurality of first data segments output by the multimodal base model; a second acquisition module, configured to acquire a plurality of pre-trained universal base models, each universal base model in the plurality of universal base models corresponding to at least one data modality in the plurality of data modalities; a knowledge distillation module, configured to: for each universal base model in the plurality of universal base models: generate a vector representation label of a first data segment of a corresponding data modality using the universal base model; and determine a knowledge distillation loss of the universal base model based on the vector representation label and the prediction vector representation of the first data segment; a determination module, configured to determine an overall loss of the multimodal base model based at least on the knowledge distillation losses of each of the plurality of universal base models; and an adjustment module, configured to adjust parameters of the multimodal base model based on the overall loss.
[0008] According to one aspect of the present disclosure, there is provided an urban area identification device, comprising: an acquisition module, configured to acquire urban data of a target urban area to be identified, the urban data comprising a plurality of data segments corresponding to a plurality of data modalities; a representation module, configured to input the urban data into a multimodal basic model to obtain a segment vector representation of each of the plurality of data segments output by the multimodal basic model, the multimodal basic model being obtained by the above-mentioned training device; a determination module, configured to determine a regional vector representation of the target urban area based on the respective segment vector representations of the plurality of data segments; and an identification module, configured to identify target information of the target urban area using a prediction network based on the regional vector representation.
[0009] According to one aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor so that the at least one processor executes any of the methods described above.
[0010] According to one aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute any of the above methods.
[0011] According to one aspect of the present disclosure, a computer program product is provided, comprising computer program instructions, which implement any of the above methods when executed by a processor.
[0012] According to one or more embodiments of the present disclosure, the efficiency and accuracy of urban area recognition can be improved.
[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.
[0015] Figure 1 A schematic diagram showing an exemplary system in which various methods described herein may be implemented according to an embodiment of the present disclosure;
[0016] Figure 2 A flowchart of a method for training a multimodal base model according to an embodiment of the present disclosure is shown;
[0017] Figure 3 A structural block diagram of a multimodal basic model according to an embodiment of the present disclosure is shown;
[0018] Figure 4 A schematic diagram showing grid division methods of different granularities and their sorting results according to an embodiment of the present disclosure is shown;
[0019] Figure 5 A schematic diagram of a multimodal basic model and a pre-training framework thereof according to an embodiment of the present disclosure is shown;
[0020] Figure 6A flow chart of a method for identifying an urban area according to an embodiment of the present disclosure is shown;
[0021] Figure 7 A structural block diagram of a training device for a multimodal basic model according to an embodiment of the present disclosure is shown;
[0022] Figure 8 A structural block diagram of a device for identifying urban areas according to an embodiment of the present disclosure is shown; and
[0023] Fig. 9 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION
[0024] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0025] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.
[0026] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations. "Multiple" refers to two or more.
[0027] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are all subject to the provisions of relevant laws and regulations and do not violate public order and good morals.
[0028] In the related art, machine learning technology is usually used to solve urban area recognition tasks, such as urban village recognition, demolition area recognition, high population density area recognition, economic index prediction, etc. At present, most methods are designed for specific tasks. Specifically, it is necessary to design a neural network model for a specific urban area recognition task, use a large amount of labeled data to perform supervised training on the model, and then use the trained model to obtain the recognition result of the urban area. However, in practice, it is often difficult to obtain a large amount of labeled data, resulting in the model being unable to learn sufficient urban knowledge from the labeled data, so the accuracy of the recognition result is low. In addition, the model obtained by supervised training using labeled data for specific tasks has poor generalization and cannot be applied to other tasks (for example, the model trained to identify urban villages cannot be used to predict population density). For other tasks, it is necessary to obtain new labeled data to retrain a new model, resulting in low efficiency in urban area recognition and difficulty in meeting diverse urban area recognition needs.
[0029] Foundation Models (FMs) are models with strong representation capabilities that are widely used in various artificial intelligence tasks and are obtained through large-scale data pre-training. In recent years, foundation models have achieved remarkable results in the fields of natural language processing and computer vision. Through self-supervised pre-training technology, these foundation models can learn general knowledge from large-scale unlabeled data and then adapt to a wide range of downstream tasks.
[0030] However, the basic models in general fields (such as GPT, ViT and CLIP, etc.) usually adopt the traditional Transformer architecture and are trained based on plain text and images. These models do not consider how to model the geospatial characteristics of urban data in their structural design, and do not specifically design optimization targets for training on urban data. As a result, they lack knowledge in the geospatial field, which leads to poor performance and low accuracy in related applications in the field of urban area recognition.
[0031] In addition, the general knowledge and generalization ability of the basic model depends on pre-training it with massive data. However, the available urban data is limited. An urban area can usually only be divided into thousands to tens of thousands of areas. Even if data is collected from multiple cities, the data scale is far less than the data scale used for pre-training of the general basic model. If a basic model specifically for the task of urban area recognition is retrained based on urban data, the generalization ability of the model will be greatly limited due to the insufficient scale of pre-training data, resulting in low accuracy in urban area recognition.
[0032] In view of this, the embodiment of the present disclosure constructs a multimodal basic model for the field of urban area identification that can solve a wide range of downstream tasks, and adopts the paradigm of self-supervised pre-training to provide a pre-training method for the multimodal basic model. By performing knowledge distillation on the existing general basic models for different data modalities, the multimodal basic model of the embodiment of the present disclosure can fully learn multimodal knowledge under the guidance of multiple existing general basic models, thereby improving its generalization, enabling it to accurately understand multimodal urban data, and then generate a general vector representation that can comprehensively and accurately express the characteristics of urban areas. This comprehensive and accurate vector representation can be widely used in a variety of downstream tasks of urban area identification, thereby improving the accuracy and efficiency of urban area identification.
[0033] The disclosed embodiment can obtain a general basic model with good generalization and comprehension capabilities for the field of urban area recognition by using only limited unlabeled data. There is no need to obtain a large amount of labeled data, nor is there a need to design and train complex models for different urban area recognition tasks, thereby improving the efficiency and accuracy of urban area recognition.
[0034] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.
[0035] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1 , the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.
[0036] In an embodiment of the present disclosure, the client devices 101 , 102 , 103 , 104 , 105 , and 106 and the server 120 may run one or more services or software applications that enable execution of a training method for a multimodal base model.
[0037] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtualized environments and virtualized environments. In some embodiments, these services may be provided as web-based services or cloud services, such as provided to users of client devices 101, 102, 103, 104, 105, and / or 106 under a software as a service (SaaS) model.
[0038] exist Figure 1In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.
[0039] Client devices 101, 102, 103, 104, 105, and / or 106 may provide an interface that enables a user of the client device to interact with the client device. The client device may also output information to the user via the interface. Figure 1 Only six client devices are depicted, but one skilled in the art will appreciate that the present disclosure may support any number of client devices.
[0040] Client devices 101, 102, 103, 104, 105 and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, vehicle-mounted devices, game systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computer devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems; or include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices may include cellular phones, smart phones, tablet computers, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Game systems may include various handheld game devices, Internet-enabled game devices, etc. Client devices are capable of executing various different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and may use various communication protocols.
[0041] The network 110 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including but not limited to TCP / IP, SNA, IPX, etc. By way of example only, the one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, Wi-Fi), and / or any combination of these and / or other networks.
[0042] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.
[0043] The computing units in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.
[0044] In some implementations, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and / or 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and / or 106.
[0045] In some embodiments, the server 120 may be a server of a distributed system, or a server combined with a blockchain. The server 120 may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in a cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.
[0046] The system 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. The databases 130 may reside in various locations. For example, the database used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The databases 130 may be of different types. In some embodiments, the databases used by the server 120 may be, for example, relational databases. One or more of these databases may store, update, and retrieve data to and from the databases in response to commands.
[0047] In some embodiments, one or more of the databases 130 may also be used by applications to store application data. The databases used by the applications may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.
[0048] Figure 1 The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.
[0049] According to some embodiments, the server 120 may execute the training method of the multimodal basic model according to the embodiment of the present disclosure to obtain a pre-trained multimodal basic model that can solve a wide range of urban area recognition tasks. In some cases, the training method of the multimodal basic model of the embodiment of the present disclosure may also be executed by the client devices 101-106. This generally requires the client devices 101-106 to have higher hardware configuration and computing power.
[0050] The urban area recognition method of the embodiment of the present disclosure can be implemented by using the trained multimodal basic model. Specifically, the user can specify the urban area to be recognized through the client device 101-106. The client device 101-106 or the server 120 can obtain the city data of the urban area, and use the trained multimodal basic model to obtain the recognition result of the urban area. Further, the client device 101-106 can output the recognition result of the urban area to the user.
[0051] Figure 2 2 shows a flow chart of a method 200 for training a multimodal basic model according to an embodiment of the present disclosure. The execution subject of the method 200 may be a server, for example Figure 1 The server 120 shown in FIG. 1 may also be a client device, such as Figure 1Client devices 101-106 are shown in FIG.
[0052] like Figure 2 As shown, method 200 includes steps S210 - S270 .
[0053] In step S210, first city data of a first sample city area is obtained. The first city data includes a plurality of first data segments corresponding to a plurality of data modalities.
[0054] In step S220, the first city data is input into the multimodal basic model to obtain a prediction vector representation of each of the plurality of first data segments output by the multimodal basic model.
[0055] In step S230, a plurality of pre-trained universal base models are obtained, wherein each universal base model in the plurality of universal base models corresponds to at least one data modality in the plurality of data modalities.
[0056] For each of the plurality of common base models, the following steps S240 and S250 are performed.
[0057] In step S240, a vector representation label of a first data segment of a corresponding data modality is generated using a common base model.
[0058] In step S250, a knowledge distillation loss of the common base model is determined based on the vector representation label and the predicted vector representation of the first data segment.
[0059] In step S260, the overall loss of the multimodal base model is determined based at least on the knowledge distillation losses of each of the plurality of common base models.
[0060] In step S270, the parameters of the multimodal base model are adjusted based on the overall loss.
[0061] According to an embodiment of the present disclosure, a multimodal basic model for the field of urban area identification that can solve a wide range of downstream tasks is provided, and a pre-training method for the multimodal basic model is provided by adopting a self-supervised pre-training paradigm. By performing knowledge distillation on existing general basic models for different data modalities, the multimodal basic model of the embodiment of the present disclosure can fully learn multimodal knowledge under the guidance of multiple existing general basic models, thereby improving its generalization, enabling it to accurately understand multimodal urban data, and then generate a general vector representation that can comprehensively and accurately express the characteristics of urban areas. This comprehensive and accurate vector representation can be widely used in a variety of downstream tasks of urban area identification, thereby improving the accuracy and efficiency of urban area identification.
[0062] The disclosed embodiment can obtain a general basic model with good generalization and comprehension capabilities for the field of urban area recognition by using only limited unlabeled data. There is no need to obtain a large amount of labeled data, nor is there a need to design and train complex models for different urban area recognition tasks, thereby improving the efficiency and accuracy of urban area recognition.
[0063] The following is a detailed description of each step of method 200.
[0064] In step S210, first city data of a first sample city area is obtained. The first city data includes a plurality of first data segments corresponding to a plurality of data modalities.
[0065] According to some embodiments, the first sample cities, such as city A and city B, which are training data for the multimodal basic model can be determined first. For each first sample city, the geographical area where the first sample city is located is divided according to a preset division strategy to obtain multiple first sample city areas. According to some embodiments, the geographical area where the first sample city is located can be divided into non-overlapping grids of the same size (e.g., L r ×L r ) of multiple grids R = {R1, R2, …, R n}, each grid R i (i=1, 2, ..., n) is taken as a first sample city region. In other embodiments, other methods may be used to divide the geographical region where the first sample city is located, for example, dividing the first sample city into a plurality of first sample city regions according to the road network structure of the first sample city.
[0066] Different first sample urban areas have different urban data and thus have different characteristics. In the embodiment of the present disclosure, the urban data of the first sample urban area is recorded as first urban data.
[0067] The first city data may be multimodal, that is, the first city data may include a plurality of first data segments, each first data segment corresponding to a data modality.
[0068] Data modality refers to the form in which data exists, such as text, image, audio, etc. In some embodiments, the data modality may also be a data modality in an industrial scenario, such as sensor data, electrical signals, infrared signals, etc. The same object can be described using data of different modalities. Data of different modalities may have the same or similar semantics. For example, the name (text) of a restaurant, "xx restaurant", and a satellite image (or photo of the restaurant) containing the restaurant are both used to describe the restaurant, that is, the name text of the restaurant and the satellite image containing the restaurant have similar semantics.
[0069] According to some embodiments, the first city data may include point of interest (POI) data in a textual mode and image data in an image mode. The point of interest data and the image data are respectively a first data segment.
[0070] Points of interest refer to places on the map that provide various services, such as restaurants, hospitals, schools, etc. i It usually includes a series of points of interest P i = {P i1 ,P i2 ,…,P im}, where m is the first sample city area R i The number of points of interest within. Each point of interest P ij (j=1,2,…,m) can have name name ij , category id ij and geographic location loc ij The three attributes provide functional and spatial information of the points of interest, which are helpful for describing possible human activities in the first sample urban area. Accordingly, the points of interest data may include the name, category and geographical location of each point of interest located in the first sample urban area.
[0071] The image data of the first sample urban area may be, for example, a satellite image. The satellite image is a bird's-eye view of the first sample urban area, which contains rich geographic spatial information, such as the spatial distribution of buildings and roads in the area, and the type of land use. In other embodiments, the image data of the first sample urban area may also be photos taken at different locations within the first sample urban area. i The image data is represented as S i .
[0072] According to some embodiments, in addition to point of interest data and image data, the first city data may also include first data segments of other modalities, such as sound signals, sensor signals (such as temperature, humidity, atmospheric particulate matter concentration), etc. within the first sample city area.
[0073] After obtaining the first city data of the first sample city area through step S210, step S220 may be executed to input the first city data into the multimodal basic model to obtain prediction vector representations of each of the plurality of first data segments output by the multimodal basic model.
[0074] In the embodiments of the present disclosure, the multimodal basic model is a general basic model for the field of urban area recognition. The model can learn vector representations of semantically rich urban areas, which can be used to solve various urban area recognition tasks.
[0075] According to some embodiments, the multimodal basic model may include a plurality of unimodal encoders and a multimodal transformer. Each unimodal encoder corresponds to a data modality of the first city data and is used to encode the first data segment of the data modality. The input end of the multimodal transformer is respectively connected to the output end of each unimodal encoder, and is used to fuse and update the codes output by each unimodal encoder to obtain a vector representation of each first data segment.
[0076] According to some embodiments, Figure 3 As shown, in the case where the first city data includes interest point data in text mode and image data in image mode, the multimodal basic model 300 may include an interest point encoder 310 for encoding the interest point data, an image encoder 320 for encoding the image data, and a multimodal transformer 330 for fusing and updating the interest point encoding result and the image encoding result. Accordingly, step S220 may include steps S221-S223.
[0077] In step S221, the interest point data is input into the interest point encoder 310, and the encoding result of the interest point data, that is, the interest point embedding sequence, can be obtained. The interest point embedding sequence includes the embedding of each interest point data unit in the interest point data.
[0078] In step S222, the image data is input into the image encoder 320 to obtain the encoding result of the image data, that is, the image embedding sequence. The image embedding sequence includes the embedding of each image data unit in the image data.
[0079] In step S223, the interest point embedding sequence and the image embedding sequence are input into the multimodal transformer 330 so that the multimodal transformer 330 fuses and updates the interest point embedding sequence and the image embedding sequence to obtain a predicted vector representation of the interest point data and a predicted vector representation of the image data.
[0080] In the embodiments of the present disclosure, the data unit is the basic unit for data processing by the multimodal basic model. For the interest point data in the text mode, the interest point data unit can be each token obtained by segmenting the interest point text, for example, each token obtained by segmenting the interest point name sequence. For the image data in the image mode, the image data unit can be each image block (patch) obtained by dividing the image.
[0081] According to some embodiments, the POI data includes the name, geographic location and category of each of the plurality of POIs located in the first sample city area. Accordingly, step S221 may include: inputting the POI data into the POI encoder so that the POI encoder performs the following steps S2211-S2212.
[0082] In step S2211, the names of the respective points of interest are represented as a sequence of points of interest names, and each token in the sequence of points of interest is regarded as a point of interest data unit.
[0083] In step S2212, for each token in the POI name sequence, an embedding of the token is generated based on the token, at least one of the geographical location and category of the POI corresponding to the token.
[0084] According to the above embodiment, the embedding of each token in the name of each point of interest can be flexibly generated according to the name, geographical location and category of each point of interest.
[0085] In step S2211, by combining the names of the various points of interest, a sequence of points of interest names may be obtained.
[0086] According to some embodiments, in step S2211, the points of interest in the first city area may be sorted according to a preset point of interest sorting strategy to obtain a point of interest sequence. The names of the points of interest in the point of interest sequence are then combined and segmented to obtain a point of interest name sequence. The preset point of interest sorting strategy may be, for example, random sorting, sorting by the first letter of the point of interest name, sorting by the size of the geographical location coordinates of the point of interest, etc.
[0087] According to some embodiments, step S2211 may include steps S22111-S22113.
[0088] In step S22111, based on the relative distances between the multiple interest points, the multiple interest points are sorted to obtain an interest point sequence.
[0089] In step S22112, the names of the points of interest in the point of interest sequence are combined to obtain a pseudo-sentence of the point of interest name.
[0090] In step S22113, the POI name pseudo-sentence is segmented to obtain a POI name sequence.
[0091] According to the above embodiment, the points of interest are first sorted based on the relative distance between them, and then a point of interest name sequence is generated based on the names of the sorted points of interest, so that the point of interest name sequence can express both the spatial position relationship (relative distance) of the points of interest and the function of the points of interest (reflected by the names of the points of interest), thereby increasing the amount of information in the point of interest name sequence, which is conducive to enabling the point of interest encoder to generate a point of interest embedding sequence that can comprehensively and accurately express the characteristics of the regional points of interest, thereby improving the expression and understanding capabilities of the multimodal basic model for urban areas.
[0092] According to some embodiments, step S22111 may include: dividing the first sample urban area into multiple grids of the same size, wherein the multiple grids form a plurality of grid pairs, and each grid pair in the plurality of grid pairs includes two grids that are adjacent in position; sorting the multiple grids using a preset sorting strategy to obtain a grid sequence, wherein two grids in some grid pairs in the plurality of grid pairs are adjacent in the grid sequence; and respectively acquiring points of interest located in each grid in the grid sequence to obtain a sequence of points of interest.
[0093] According to the above embodiment, the sorting of interest points is achieved by dividing and serializing the grid, which can improve the calculation efficiency of the sorting of interest points.
[0094] The size of the grid can be set as needed. According to some embodiments, in order to improve the accuracy of the ordering of the points of interest, the first sample urban area can be divided into fine-grained grids, for example, the size of the grid can be 1m*1m, 1m*2m, 2m*2m, etc.
[0095] Each grid pair includes two grids that are adjacent in geographical location. For different grid division methods, the identification method of the grid pair may be different.
[0096] For example, when the first sample urban area is divided into multiple non-overlapping rectangular grids, the adjacent grids of a certain grid can be determined based on the four neighborhoods, that is, the adjacent grids of a certain grid are four grids located at the top (north), bottom (south), left (west), and right (east) of the grid, thereby obtaining four grid pairs; the adjacent grids of a certain grid can also be determined based on the eight neighborhoods, that is, the adjacent grids of a certain grid are eight grids located at the top (north), bottom (south), left (west), right (east), upper left (northwest), lower left (southwest), upper right (northeast), and lower right (southeast) of the grid, thereby obtaining eight grid pairs.
[0097] When the first sample urban area is divided into multiple non-overlapping grids based on the road network data, two grids whose number of intersections passed by the connected path is less than the intersection threshold can be regarded as two adjacent grids. For example, grid 1 and grid 2 are connected by path path, which passes through 3 intersections, which is less than the intersection threshold 5. Therefore, grid 1 and grid 2 are determined to be adjacent, and the two form a grid pair (grid 1, grid 2).
[0098] After the first sample urban area is divided into multiple grids, the multiple grids are further serialized, that is, the multiple grids are sorted using a preset sorting strategy to obtain a grid sequence. In the grid sequence, two grids with close geographical distances often have adjacent sorting. It should be noted that any current geographic sorting strategy cannot completely maintain the adjacent relationship in the two-dimensional geographic space, that is, it cannot guarantee that any two adjacent grids in the geographic space are adjacent in the sorting result. In fact, in the grid sorting result, only two grids in a part of the grid pairs (including two grids that are actually geographically adjacent) remain adjacent in the grid sequence; while the two grids in another part of the grid pairs are not adjacent in the grid sequence, or even far apart.
[0099] The grid ordering strategy can be set as needed. For example, a Z-ordering strategy or a Hilbert-ordering strategy can be used.
[0100] Figure 4 A schematic diagram showing grid division methods A and B of different granularities and corresponding Z-shaped sorting results.
[0101] In grid division method A, the first sample urban area is divided into four grids of size a*a, namely grids 401-404. Grids 401-404 are sorted according to zigzag curve 405, and the resulting grid sequence is {401, 402, 403, 404}. Grids 401 and 402 are adjacent in the actual geographic space and are also adjacent in the grid sequence; while grids 401 and 403 are adjacent in the actual geographic space, but are not adjacent in the grid sequence.
[0102] In grid division method B, the same first sample urban area is divided into 16 grids of size (a / 2)*(a / 2), namely grids 411-426. Grids 411-426 are sorted according to the zigzag curve 427, and the resulting grid sequence is {411, 412, 415, 416, 413, 414, 417, 418, 419, 420, 423, 424, 421, 422, 425, 426}. Grids 415 and 416 are adjacent in the actual geographic space and are also adjacent in the grid sequence; while grids 415 and 419 are adjacent in the actual geographic space, but are not adjacent in the grid sequence, and are even far apart (with 5 grids in between).
[0103] After obtaining the grid sequence, the various points of interest in the first sample urban area are sorted according to the grids in which they are located, and the obtained point of interest sequence can reflect the relative distances between the points of interest. Specifically, the point of interest sequence can be initialized to an empty sequence. For each grid in the grid sequence, the point of interest located in the grid is obtained, and the point of interest is added to the point of interest sequence until the grids in the grid sequence are traversed. In some cases, multiple points of interest with very close geographical locations may be located in the same grid. In this case, these points of interest can be randomly sorted and added to the point of interest sequence. Since the grid division granularity is usually very fine (for example, 1m*1m), randomly sorting multiple points of interest in the same grid will hardly affect the vector representation of the urban area.
[0104] After obtaining the interest point sequence in step S22111, in step S22112, the names of the interest points in the interest point sequence are combined to obtain a pseudo sentence of the interest point name.
[0105] Then, in step S22113, the interest point name pseudo-sentence is segmented to obtain an interest point name sequence.
[0106] According to some embodiments, a separator [SEP] may be further added after the last token of each point of interest based on the word segmentation result. The separator [SEP] is a preset special token used to separate different points of interest. When the separator [SEP] is added after the last token of each point of interest, the separator [SEP] is regarded as belonging to the point of interest located before it.
[0107] In some other embodiments, a separator [SEP] may be added before the first token of each point of interest. In this case, the separator [SEP] is regarded as belonging to the point of interest following it.
[0108] After obtaining the POI name sequence through step S2211 (including steps S22111-S22113), step S2212 may be performed. In step S2212, for each token in the POI name sequence, an embedding of the token is generated based on the token, the geographic location of the POI corresponding to the token, and at least one of the category.
[0109] According to some embodiments, in step S2212, for each token in the POI name sequence, an embedding of the token may be generated based on the token, the geographic location of the POI corresponding to the token, and the category. Specifically, step S2212 may include steps S22121-S22124.
[0110] In step S22121, the token is encoded to obtain a word embedding of the token.
[0111] In step S22122, based on the geographical location of the point of interest corresponding to the token, a geographical location embedding of the token is generated.
[0112] In step S22123, the category of the interest point corresponding to the token is encoded to obtain the category embedding of the token.
[0113] In step S22124, the embedding of the token is determined based on word embedding, geographic location embedding and category embedding.
[0114] According to the above embodiment, the embedding of each token can fully and accurately express the data of the corresponding point of interest, thereby improving the accuracy of the embedding of the token and thus improving the accuracy of urban area recognition.
[0115] According to some embodiments, in step S22121, the word embedding table E may be used. w () Map the word tokens to word embeddings in latent space. Word embedding table E w () stores the mapping relationship between word tokens and word embeddings. Word embedding table E w () can be a trainable parameter of the multimodal base model.
[0116] According to some embodiments, in step S22122, multiple location information of different granularities of the token may be obtained based on the geographic location of the point of interest corresponding to the token; and based on the multiple location information, the geographic location embedding of the token is generated. According to this embodiment, by encoding location information of different granularities, the geographic location embedding of each token can fully and accurately express its location information, thereby improving the accuracy of the embedding of the token.
[0117] According to some embodiments, multiple location information of different granularities of a token include the first position of the token in the name of the corresponding point of interest, the second position of the point of interest in the point of interest sequence corresponding to the point of interest name sequence, and the grid where the point of interest is located. The grid is obtained by evenly dividing the first sample city area. Accordingly, the geographic location embedding of the token can be obtained by the following steps: encoding the above-mentioned first position to obtain the word position embedding of the token; encoding the above-mentioned second position to obtain the point of interest position embedding of the token; encoding the identifier of the above-mentioned grid to obtain the grid position embedding of the token; and fusing the above-mentioned word position embedding, point of interest position embedding and grid position embedding to obtain the geographic location embedding of the token.
[0118] According to the above embodiment, the geographic location embedding of each word token is generated from the word level (Word-level), the point of interest level (POI-level) and the grid level (Grid-level), so that the geographic location embedding can express both the complex sequence relationship between the word tokens and the spatial position relationship of the points of interest, thereby improving the accuracy and comprehensiveness of the geographic location embedding.
[0119] According to some embodiments, the word position embedding table E may be used wp () Map the first position of the token in the corresponding POI name to the word position embedding. Word Position Embedding Table E wp () stores the mapping relationship between the first position and the word position embedding. Word position embedding table E wp () can be a trainable parameter of the multimodal base model.
[0120] According to some embodiments, the point of interest location can be used to embed the table E pp () Map the second position of the interest point to which the token belongs in the interest point sequence to the interest point position embedding. Interest point position embedding table E pp () stores the mapping relationship between the second position and the point of interest position embedding. Point of interest position embedding table E pp () can be a trainable parameter of the multimodal base model.
[0121] According to some embodiments, the grid position embedding table E may be used gp () Map the identity of the grid where the interest point to which the token belongs is located to the grid position embedding. Grid position embedding table E gp () stores the mapping relationship between the grid identifier and the grid position embedding. Grid position embedding table E gp () can be a trainable parameter of the multimodal base model.
[0122] The word position embedding, the point of interest position embedding and the grid position embedding of a token are fused to obtain the geographic location embedding of the token. There are many ways to fuse the word position embedding, the point of interest position embedding and the grid position embedding. For example, the word position embedding, the point of interest position embedding and the grid position embedding of a token can be fused into the geographic location embedding of the token by summing, averaging, weighted averaging, concatenating, etc.
[0123] According to some embodiments, in step S22123, the category embedding table E may be used. c () Map the category of the interest point to which the token belongs to into category embedding. Category Embedding Table E c () stores the mapping relationship between the categories of interest points and category embeddings. Category Embedding Table E c () can be a trainable parameter of the multimodal base model.
[0124] According to some embodiments, in step S22124, the sum of the word embedding, geographic location embedding, and category embedding of the token may be used as the embedding of the token. In other embodiments, the average, weighted sum, or concatenation of the word embedding, geographic location embedding, and category embedding may be used as the embedding of the token.
[0125] According to some embodiments, in step S22124, the modality embedding table E may be used. m () Map the data modality (text modality, which can also be understood as POI modality) corresponding to the token into modality embedding. Modality Embedding Table E m () stores the mapping relationship between data modality and modality embedding. Modality embedding table E m () can be a trainable parameter of the multimodal base model. Further, the embedding of the token can be determined based on the word embedding, geographic location embedding, category embedding and modal embedding of the token. For example, the sum, average value, concatenation result, etc. of the word embedding, geographic location embedding, category embedding and modal embedding can be used as the embedding of the token.
[0126] According to some embodiments, step S222 may include: inputting the image data into an image encoder so that the image encoder performs the following steps S2221 and S2222.
[0127] In step S2221, the image data is split into multiple image blocks of the same size, each image block being an image data unit.
[0128] In step S2222, for each image block in the image data, a linear transformation is performed on the image block to obtain an embedding of the image block.
[0129] For example, the image data may be a satellite image with a height of H and a width of W and containing three color channels of R, G, and B, that is, the size of the image data is H*W*3. The size of the image block may be, for example, s*s*3. Accordingly, the image data may be divided into (H*W) / (s*s) image blocks, each of which includes the pixel values of s*s pixels in the three channels of R, G, and B, and may be represented as a vector of s*s*3 dimensions. The vector is mapped to a d-dimensional vector using a linear transformation matrix, and the d-dimensional vector is used as the embedding of the image block. The linear transformation matrix (including multiple weights) may be a trainable parameter of the multimodal base model.
[0130] According to some embodiments, after the embedding of each image block is obtained, 1-D position encoding and modality embedding may be further added to each image block, and the embedding of each image block, 1-D position encoding and modality embedding may be combined into an image embedding sequence.
[0131] The 1-D position encoding method is an encoding method that takes into account one-dimensional position information. The 1-D position encoding of an image block indicates the position (sequence number) of the image block in the image embedding sequence. According to some embodiments, the 1-D position embedding table E can be used. 1D () Map the serial number of the image block in the image embedding sequence to a 1-D position code. 1-D position embedding table E 1D () stores the mapping relationship between the serial number of the image block and its 1-D position encoding. 1-D position embedding table E 1D () can be a trainable parameter of the multimodal base model.
[0132] The modality embedding indicates the data modality of the image block, i.e., the image modality. According to some embodiments, the modality embedding table E may be used. m () Map the data modality (i.e., image modality) corresponding to the image block into modality embedding. Modality Embedding Table E m () stores the mapping relationship between data modality and modality embedding. Modality embedding table E m () can be a trainable parameter of the multimodal base model.
[0133] In step S230, a plurality of pre-trained universal base models are obtained, wherein each universal base model in the plurality of universal base models corresponds to at least one data modality in the plurality of data modalities.
[0134] General-purpose FMs refer to basic models that are commonly used in various application fields and are used to process and characterize data of a specific modality. The data modality corresponding to the general-purpose FM is the data modality that the general-purpose FM can process.
[0135] According to some embodiments, the data modality set formed by the data modalities corresponding to each universal basic model covers multiple data modalities included in the first city data. Thus, multiple universal basic models can be used to comprehensively enhance the understanding and characterization capabilities of the multimodal basic model of the embodiment of the present disclosure for multimodal city data.
[0136] According to some embodiments, the universal basic model obtained in step S230 may include a unimodal universal basic model, such as a language basic model corresponding to a text modality, a visual basic model corresponding to an image modality, etc. The universal basic model may also include a cross-modal universal basic model, such as a visual language basic model, etc. The cross-modal universal basic model may map data of different modalities to the same feature space.
[0137] Language FMs are large language models (LLM, also known as large language models or large models). Large language models are deep learning models trained using a large amount of text data, which can realize the understanding and generation of natural language text. The number of model parameters of a large language model can reach tens of billions, hundreds of billions or even higher. According to some embodiments, a large language model can adopt a decoder structure (Decoder-Only). The large language model of the decoder structure only includes a decoder (Decoder) in a transformer (Transformer) structure, but does not include an encoder (Encoder). The large language model of the decoder structure usually has good performance in text generation tasks. According to other embodiments, the large language model can also adopt a transformer structure including an encoder and a decoder. The large language model can be, for example, Wenxin Yiyan, GPT (Generative Pre-trained Transformer) and other models.
[0138] Visual FMs are used to understand an image to generate a vector representation of the image. The visual FMs may be, for example, a ViT (Vision Transformer) model.
[0139] The visual language basic model is used to understand text and images, map text and images to the same encoding feature space, and obtain vector representations of text and images. The visual language basic model can be, for example, a CLIP (Contrastive Language-Image Pre-training) model.
[0140] According to some embodiments, for each data modality in the first city data, a unimodal universal basic model for processing the data modality is obtained. For each data modality pair (including two data modalities) in the first city data, a cross-modal universal basic model for processing the data modality pair is obtained. In this way, not only can the unimodal universal basic models be used to enhance the understanding and characterization capabilities of the multimodal basic model for each modality of urban data, but also the cross-modal universal basic models can be used to enhance the semantic alignment capabilities of the multimodal basic model for multimodal urban data.
[0141] For each of the plurality of common base models acquired in step S230 , the following steps S240 and S250 are performed.
[0142] In step S240, a vector representation label of a first data segment of a corresponding data modality is generated using a common base model.
[0143] According to some embodiments, for the language base model of the text modality, the vector representation label of the point of interest data of the text modality can be generated using the language base model. Specifically, at least a portion of the point of interest data can be input into the language base model (for example, only the name of each point of interest is input, or all the point of interest data consisting of the name, category and geographical location of each point of interest is input) to obtain the description text of the first sample urban area generated by the language base model. The description text is used to describe the functions and possible resident activities of the first sample urban area. Subsequently, the description text is encoded to obtain the vector representation label of the point of interest data.
[0144] For example, given a set of points of interest in a first sample urban area Generate prompt text (prompt) of the language-based model based on the name of each point of interest and the preset prompt template Prompt text For example, it could be "There are the following facilities in the area: POI1, POI2, ..., please infer the function of this area and possible resident activities." Input the language base model, and the language base model uses the prompt text to generate a description text of the first sample city area Then, the text embedding model can be used to encode the description text into a vector, which is the vector representation label of the point of interest data. The process of encoding the description text can be described as follows:
[0145]
[0146] Among them, u PThe vector representation label of the interest point data, SenEmb() represents the encoding process of the text embedding model.
[0147] According to some embodiments, for a visual basis model of an image modality, a vector representation label of image data of the image modality may be generated using the visual basis model. Specifically, the image data may be input into the visual basis model to obtain a vector representation label of the image data output by the visual basis model. The process of generating a vector representation label of image data using the visual basis model may be expressed as follows:
[0148] u S =VFM(S) (2)
[0149] Among them, u S is the vector representation label of the image data, and VFM() is the encoding process of the visual basis model.
[0150] According to some embodiments, for the visual language basic model of the image-text modality, the visual language basic model can be used to generate vector representation labels of interest point data of the text modality and vector representation labels of image data of the image modality. Specifically, the interest point data and the image data can be input into the visual language basic model respectively to obtain the vector representation labels of the interest point data and the image data output by the visual language basic model.
[0151] In step S250, a knowledge distillation loss of the common base model is determined based on the vector representation label and the predicted vector representation of the first data segment.
[0152] According to some embodiments, for a unimodal universal base model (eg, a language base model and a visual base model), its knowledge distillation loss may be determined through steps S251-S253.
[0153] In step S251, a linear transformation is performed on the prediction vector representation of the first data segment to obtain a transformed prediction vector representation.
[0154] In step S252, the similarity between the transformed prediction vector representation and the vector representation label is calculated. The similarity may be, for example, cosine similarity.
[0155] In step S253, based on the above similarity, the unimodal knowledge distillation loss of the unimodal general basic model is determined, wherein the unimodal knowledge distillation loss is negatively correlated with the above similarity. That is, the greater the similarity between the transformed predicted vector representation and the vector representation label, the smaller the value of the unimodal knowledge distillation loss.
[0156] According to the above embodiment, the predicted vector representation can be mapped to the same feature space as the vector representation label through linear transformation. Based on the similarity between the transformed predicted vector representation and the vector representation label, the unimodal knowledge distillation loss is determined, and the unimodal general basic model can be used to enhance the multimodal basic model's ability to understand and express unimodal urban data.
[0157] According to some embodiments, the unimodal knowledge distillation loss of the language base model It can be calculated according to the following formula:
[0158]
[0159] Among them, h [P] The predicted vector representation of the interest point data generated by the multimodal base model, u P The vector representation label of the interest point data generated by the language-based model, σ POI () is the linear transformation function of the interest point data, and Cos() is the calculation function of the cosine similarity.
[0160] According to some embodiments, the unimodal knowledge distillation loss of the visual base model It can be calculated according to the following formula:
[0161]
[0162] Among them, h [S] The predicted vector representation of the image data generated by the multimodal base model, u S The vector representation label of the image data generated by the visual base model, σ Sate () is the linear transformation function of image data, and Cos() is the calculation function of cosine similarity.
[0163] According to some embodiments, for a cross-modal universal base model (eg, a visual language base model), its knowledge distillation loss may be determined through steps S254-S257. The data modalities corresponding to the cross-modal universal base model are recorded as the first data modality and the second data modality.
[0164] In step S254, a first prediction vector representation and a first vector representation label of first data of the first sample urban area, and a second prediction vector representation and a second vector representation label of second data are obtained, wherein the first data is a first data segment corresponding to the first data modality, and the second data is a first data segment corresponding to the second data modality.
[0165] The first prediction vector representation and the second prediction vector representation can be obtained from the output of the multimodal base model. The first vector representation label and the second vector representation label can be obtained from the output of the cross-modal general base model. That is, by inputting the first data and the second data into the cross-modal general base model, the first vector representation label and the second vector representation label can be obtained.
[0166] In step S255, a prediction similarity matrix is generated based on the first prediction vector representation and the second prediction vector representation of each of the plurality of first sample city regions. The element in the i-th row and j-th column of the prediction similarity matrix indicates the similarity between the first prediction vector representation of the first sample city region i and the second prediction vector representation of the first sample city region j.
[0167] For each of the first sample city regions, the first prediction vector representation and the second prediction vector representation can be obtained as described in step S254. The number of first sample city regions is denoted as B. By calculating the similarity between the first prediction vector representation of the first sample city region i (i=1, 2, ..., B) and the second prediction vector representation of the first sample city region j (j=1, 2, ..., B), a prediction similarity matrix of size B*B can be obtained. The above similarity may be, for example, cosine similarity.
[0168] In step S256, based on the first vector representation labels and the second vector representation labels of the plurality of first sample city regions, a similarity matrix label is generated, wherein the element in the i-th row and the j-th column of the similarity matrix label indicates the similarity between the first vector representation label of the first sample city region i and the second vector representation label of the first sample city region j.
[0169] For each of the first sample city regions among the plurality of first sample city regions, its first vector representation label and second vector representation label can be obtained as described in step S254. The number of first sample city regions is recorded as B, and by calculating the similarity between the first vector representation label of the first sample city region i (i=1,2,…,B) and the second vector representation label of the first sample city region j (j=1,2,…,B), a similarity matrix label M of size B*B can be obtained. The above similarity can be, for example, cosine similarity.
[0170] In step S257, based on the difference between the predicted similarity matrix and the similarity matrix label, the cross-modal knowledge distillation loss of the cross-modal general basic model is determined, wherein the cross-modal knowledge distillation loss is positively correlated with the above-mentioned difference, that is, the greater the difference between the predicted similarity matrix and the similarity matrix label, the greater the cross-modal knowledge distillation loss; the smaller the difference between the predicted similarity matrix and the similarity matrix label, the smaller the cross-modal knowledge distillation loss. The difference between the predicted similarity matrix and the similarity matrix label can be represented by the KL (Kullback-Leibler) divergence of the corresponding rows and columns in the two matrices, for example.
[0171] According to the above embodiments, the cross-modal general basic model can be used to enhance the semantic alignment capability of the multimodal basic model for multimodal urban data.
[0172] According to some embodiments, the cross-modal general base model is a visual language base model, and accordingly, the first data modality is a text modality, the first data is interest point data of the text modality, the second data modality is an image modality, and the second data is image data of the image modality. Cross-modal knowledge distillation loss of the visual language base model It can be calculated by the following formula:
[0173]
[0174] Among them, B is the number of first sample urban areas, M and are the similarity matrix label and the predicted similarity matrix, M i and are the i-th row in the similarity matrix label and predicted similarity matrix, respectively. and are the j-th column in the similarity matrix label and predicted similarity matrix respectively, ρ() is the softmax function, which is used to transform the rows or columns of the matrix into a probability distribution, and KL() is the calculation function of the KL divergence.
[0175] In step S260, the overall loss of the multimodal base model is determined based at least on the knowledge distillation losses of each of the plurality of common base models.
[0176] According to some embodiments, the overall loss of the multimodal base model may be, for example, a weighted sum of the knowledge distillation losses of the respective universal base models.
[0177] For example, in the case of multiple common base models including a language base model, a visual base model, and a visual-language base model, the overall loss of the multimodal base model It can be the sum of the knowledge distillation losses of each common base model (the weights of the three are all 1), that is:
[0178]
[0179] in, They are the knowledge distillation losses of the language-based model, the vision-based model, and the vision-language-based model. The calculation methods of the three can refer to the above formulas (3)-(5).
[0180] According to some embodiments, in addition to the knowledge distillation loss, the overall loss of the multimodal base model may also include a loss for a self-supervised learning task.
[0181] As described above, each first data segment in the first city data can be split into multiple data units. For example, the point of interest data can be split into multiple point of interest data units (i.e., tokens), and the image data can be split into multiple image data units (i.e., image blocks, patches).
[0182] According to some embodiments, the loss of the self-supervised learning task may include a mask loss. The mask loss may be calculated by following the steps S281-284.
[0183] In step S281, for each of the first data segments in the plurality of first data segments, at least one data unit in the first data segment is replaced with a preset mask data unit to obtain masked first city data. For example, some tokens (e.g., 15% of the tokens) in the point of interest data may be replaced with preset mask tokens [M], and some image blocks (e.g., 40% of the image blocks) in the image data may be replaced with blank image blocks, thereby obtaining masked first city data.
[0184] In step S282, the masked first city data is input into the multimodal basic model to obtain a vector representation of each data unit in the masked first city data output by the multimodal basic model.
[0185] In step S283, for each mask data unit, a first classifier is used to determine a first probability of correctly predicting a data unit before masking corresponding to the mask data unit based on the vector representation of the mask data unit. Specifically, by inputting the vector representation of the mask data unit into the first classifier, a first probability output by the first classifier can be obtained. The first classifier can be composed of, for example, a two-layer multi-layer perceptron (MLP) and a softmax layer.
[0186] In step S284, the mask loss of the multimodal base model is determined based on the first probability corresponding to each mask data unit.
[0187] According to some embodiments, in the case where the first city data includes interest point data and image data, the mask loss It can be calculated by the following formula:
[0188]
[0189] in, and Respectively represent the positions of the masked interest point tokens and image blocks, represents the vector representation h based on the masked i-th interest point token i Able to correctly predict the masked token at that position The first probability, Represents the vector representation h based on the masked i-th image block i The image block before masking that can be correctly predicted The first probability is obtained by the first classifier.
[0190] According to some embodiments, in step S260, the overall loss of the multimodal base model may be determined based on the knowledge distillation loss and mask loss of each of the plurality of common base models. The overall loss of the multimodal base model may be, for example, a weighted sum of the knowledge distillation loss and mask loss of each of the common base models.
[0191] For example, in the case of multiple common base models including a language base model, a visual base model, and a visual-language base model, the overall loss of the multimodal base model It can be the sum of the knowledge distillation loss and mask loss of each common base model (the weights of the four are all 1), that is:
[0192]
[0193] in, are the knowledge distillation losses of the language-based model, the vision-based model, and the vision-language-based model, respectively. is the mask loss. The calculation methods of the four can refer to the above formulas (3)-(5) and (7).
[0194] According to the above embodiment, the semantic understanding ability of the multimodal basic model for multimodal urban data can be enhanced through the self-supervised learning task based on mask loss.
[0195] According to some embodiments, the loss of the self-supervised learning task may include a cross-modal spatial alignment loss. The cross-modal spatial alignment loss may be calculated by following the steps S291-294.
[0196] In step S291, at least one of the multiple interest point data units is replaced with a preset interest point mask data unit, and at least one of the multiple image data units is replaced with a preset image mask data unit to obtain masked first city data.
[0197] For example, some tokens (e.g., 15% of the tokens) in the point of interest data may be replaced with preset mask tokens [M], and some image blocks (e.g., 40% of the image blocks) in the image data may be replaced with blank image blocks, thereby obtaining masked first city data.
[0198] In step S292, the masked first city data is input into the multimodal basic model to obtain a vector representation of each data unit in the masked first city data output by the multimodal basic model.
[0199] In step S293, for each unmasked point of interest data unit, based on the vector representation of the point of interest data unit, a second probability that the point of interest corresponding to the point of interest data unit is located in the masked image data unit is determined using a second classifier. Specifically, by inputting the vector representation of the unmasked point of interest data unit into the second classifier, a second probability output by the second classifier can be obtained. The second classifier can be, for example, a sigmoid classifier.
[0200] In step S294, the cross-modal spatial alignment loss of the multimodal base model is determined based on the second probability corresponding to each unmasked interest point data unit.
[0201] According to some embodiments, when the first city data includes interest point data and image data, the cross-modal spatial alignment loss It can be calculated by the following formula:
[0202]
[0203] Among them, L P Indicates the maximum position (maximum sequence number) of the interest point token, Indicates the position of the masked interest point token, y i is a binary classification label, indicating whether the i-th unmasked interest point token is located in the masked image block. If so, y i =1; if not, then y i = 0. p(y i |h i ) represents the second probability output by the second classifier.
[0204] According to some embodiments, in step S260, the overall loss of the multimodal base model may be determined based on the knowledge distillation loss and the cross-modal spatial alignment loss of each of the plurality of common base models. The overall loss of the multimodal base model may be, for example, a weighted sum of the knowledge distillation loss and the cross-modal spatial alignment loss of each of the common base models.
[0205] For example, in the case of multiple common base models including a language base model, a visual base model, and a visual-language base model, the overall loss of the multimodal base model It can be the sum of the knowledge distillation loss of each common base model and the cross-modal spatial alignment loss (the weights of the four are all 1), that is:
[0206]
[0207] in, are the knowledge distillation losses of the language-based model, the vision-based model, and the vision-language-based model, respectively. is the cross-modal spatial alignment loss. The calculation methods of the four can refer to the above formulas (3)-(5) and (9).
[0208] According to the above embodiment, through the self-supervised learning task based on the cross-modal spatial alignment loss, the multimodal basic model can perceive the positional relationship between the interest points and the image (satellite image) content, thereby achieving semantic alignment of the interest points and the image.
[0209] According to some embodiments, in step S260, the overall loss of the multimodal base model may be determined based on the knowledge distillation loss, mask loss, and cross-modal spatial alignment loss of each of the plurality of common base models. The overall loss of the multimodal base model may be, for example, a weighted sum of the knowledge distillation loss, mask loss, and cross-modal spatial alignment loss of each of the common base models.
[0210] For example, in the case of multiple common base models including a language base model, a visual base model, and a visual-language base model, the overall loss of the multimodal base model It can be the sum of the knowledge distillation loss, mask loss and cross-modal spatial alignment loss of each common base model (the weights of all five are 1), that is:
[0211]
[0212] in, are the knowledge distillation losses of the language-based model, the vision-based model, and the vision-language-based model, respectively. is the mask loss, is the cross-modal spatial alignment loss. The calculation methods of the four can refer to the above formulas (3)-(5), (7), and (9).
[0213] In step S270, the parameters of the multimodal base model are adjusted to reduce the overall loss.
[0214] It is understandable that steps S210-S270 can be repeatedly performed multiple times until a preset termination condition is reached. The multimodal basic model at this time is the trained first multimodal basic model. The preset termination condition can be, for example, the overall loss converges, reaches a preset number of iterations, the overall loss is less than a preset threshold, etc.
[0215] According to some embodiments, the trained first multimodal base model is used to generate a regional vector representation of an urban area, and the regional vector representation can be used as an input of a prediction network so that the prediction network identifies target information of the urban area based on the regional vector representation.
[0216] The prediction network is a neural network used to solve the downstream task of urban area recognition. The prediction network can adopt any structure, for example, it can include a multi-layer perceptron and a classification layer (such as a softmax classification layer, a sigmoid classification layer, etc.).
[0217] According to some embodiments, the target information may include any of the following information: type, population, traffic flow, economic index. The type of urban area may include, for example, urban villages, demolition areas, high population density areas, commercial areas, residential areas, industrial areas, etc. The economic index may indicate, for example, economic conditions such as housing prices and economic prosperity in the urban area.
[0218] The trained first multimodal base model can be used to solve any downstream task in the field of urban area recognition. According to some embodiments, a prediction network for downstream tasks can be trained based on the trained first multimodal base model using second city data of a second sample urban area.
[0219] According to some embodiments, a prediction network for downstream tasks may be trained through the following steps S201 - S205 .
[0220] In step S201, second city data of a second sample city area, information labels of the second sample city area, and a trained first multimodal basic model are obtained, wherein the second city data includes a plurality of second data segments corresponding to a plurality of data modalities, respectively.
[0221] In step S202, the second city data is input into the first multimodal basic model to obtain a segment vector representation of each of the plurality of second data segments output by the first multimodal basic model.
[0222] In step S203, based on the segment vector representations of each of the plurality of second data segments, a region vector representation of the second sample urban area is determined.
[0223] In step S204, the region vector representation of the second sample city region is input into the prediction network to obtain prediction information of the second sample city region output by the prediction network.
[0224] In step S205, at least parameters of the prediction network are adjusted based on the prediction information and the information label.
[0225] According to the above embodiment, the second city data of the second sample urban area and its information labels are used to perform supervised training on the prediction network, which can improve the prediction ability of the prediction network for downstream tasks and improve the accuracy of urban area recognition.
[0226] According to some embodiments, the second sample city area may be the same city area as the first sample city area described above. Similar to the first city data of the first sample city area, the second city data of the second sample city area may also include point of interest data and image data (e.g., satellite image). Further, the point of interest data may include the name, category, and geographic location of each point of interest in the second sample city area.
[0227] According to some embodiments, in step S203, the segment vector representations of the plurality of second data segments may be fused to obtain the region vector representation of the second sample urban area.
[0228] For example, the average value of the fragment vector representations of multiple second data fragments can be determined as the regional vector representation of the second sample urban area, or the sum, weighted sum or splicing result of the fragment vector representations of multiple second data fragments can be used as the regional vector representation of the second sample urban area.
[0229] According to some embodiments, in step S204, the prediction information of the second sample urban area may be, for example, the type, population size, traffic flow, economic index, etc. of the second sample urban area.
[0230] According to some embodiments, in step S205, the joint loss of the first multimodal basic model and the prediction network can be determined based on the prediction information and the information label; and then based on the joint loss, the parameters of the first multimodal basic model and the prediction network are adjusted simultaneously. This achieves supervised fine-tuning of the first multimodal basic model, so that the second multimodal basic model obtained after fine-tuning is more suitable for the urban area recognition task corresponding to the prediction network, thereby improving the recognition accuracy of the urban area recognition task. The above-mentioned joint loss can be calculated, for example, by a cross entropy loss function.
[0231] According to some embodiments, in step S205, the loss of the prediction network can be determined based on the prediction information and the information label; and then based on the loss, the parameters of the prediction network are adjusted. In this embodiment, the parameters of the first multimodal basic model are fixed and only the parameters of the prediction network are adjusted. This can improve the training efficiency of the prediction network.
[0232] Figure 5 A schematic diagram of a multimodal basic model and its pre-training framework according to an embodiment of the present disclosure is shown.
[0233] like Figure 5 As shown, the multimodal basic model for urban area recognition in the embodiment of the present disclosure (hereinafter referred to as the model or the present model) includes a multimodal geospatial data embedding (Multi-modal Geospatial Data Embedding) module and a mixture of geographic experts transformer (Mixture-of-Geospatial-Experts Transformer, MoGE Transformer), such as Figure 5 The modules (a) and (b) in the figure are shown in Figure 1. The multimodal geospatial data embedding module transforms the original POI data and satellite image data into an embedding sequence in the latent space, comprehensively models and integrates the text, visual and geospatial information in these urban data. The hybrid geographic expert transformer is used to perform deep interaction and fusion of multimodal geographic data and generate multimodal regional representations.
[0234] The pre-training framework of the multimodal base model includes the joint knowledge distillation from generic FMs and the self-supervised learning task on geospatial data, as shown in Figure 5 As shown in modules (c) and (d).
[0235] The following is a detailed introduction to the multimodal geospatial data embedding module, the hybrid geographic expert transformer, the joint distillation strategy based on multiple common base models, and the self-supervised learning task based on multimodal geographic data. and S represents the i-th city region R i POI collection and satellite images.
[0236] 1. Multi-modal Geospatial Data Embedding
[0237] This module encodes the original POI data and satellite image data into compact vector embeddings, and fully considers their geospatial context in the process. Multimodal geospatial data embedding includes the embedding of POI data (POIEmbedding) and the embedding of satellite image data (Satellite Image Embedding).
[0238] 1.1 POI Embedding
[0239] In order to comprehensively model POIs in urban areas, the disclosed embodiment proposes a joint encoding method, including word embedding, geo-aware position embedding and category embedding for POIs. The POI embedding obtained by this joint encoding method can not only effectively model the textual toponym knowledge of POIs, but also capture their spatial distribution relationship.
[0240] 1.1.1 Word Embedding
[0241] Given a city area, we first construct a sequence of POI names using all POIs in the area. This POI name sequence can be considered a pseudo sentence that matches the input of the Transformer model. We perform word embedding encoding on the POI name sequence. Specifically, for the POI set in the area, The sequence of their names (or pseudo sentence) is: name1name2…name m . Tokenize the name sequence and use the special token [SEP] to separate different POIs, and get the following tokenization results: Among them, t jk Indicates P j Then, we use a word embedding table E w () Map each token into a d-dimensional latent space word embedding: Word Embedding Table E w () are the trainable parameters of the multimodal base model.
[0242] 1.1.2 Geo-aware Position Embedding
[0243] In order to consider the positional relationship between tokens, the embodiment of the present disclosure designs a geo-aware position encoding to replace the conventional position encoding used in BERT (Bidirectional Encoder Representations from Transformers) and GPT. In the pseudo-sentence, many tokens come from different POIs, and the distribution of POIs in urban areas is irregular. Therefore, tokens from different POIs do not follow the rules in human language (such as grammar and discourse structure), but have certain spatial relationships according to the geographical distribution of POIs, such as some POIs are close to each other, and some POIs are far apart. This feature makes conventional position encoding not applicable. To this end, the embodiment of the present disclosure designs geo-aware position encoding to handle the complex sequence relationships and spatial relationships between tokens from three aspects:
[0244] (1) Word-level
[0245] For a POI P in the area j , we first use general position encoding to represent the order of each token in the POI name, which is consistent with the position encoding in BERT and GPT. j Token t in jk , using word-level position embedding table E wp (), token t jk In POI name j The order k in is mapped to the word-level positional encoding E wp (k). Word-level position embedding table E wp () are the trainable parameters of the multimodal base model.
[0246] (2) POI-level
[0247] The location encoding at the POI level is used to model the spatial position relationship between the tokens of different POIs. Inspired by the first law of geography (close things are more related), we hope that this location encoding can reflect the relative distance between POIs. Specifically, given a city area, we first divide it into fine-grained grids (size 1m*1m), and then serialize the grids using the Z-ordering strategy. In the resulting grid sequence, grids with similar distances will be arranged together. On this basis, the POIs in the area are sorted according to the grids in which they are located, and the resulting POI sequence can reflect the relative distance between POIs. In some cases, multiple POIs with very close distances may be located in the same grid, and these POIs can be randomly sorted. Let's assume that in In, P j The subscript j represents the position of the POI in the sequence. Using the position of the POI level, Table E is embedded pp (), P j The order j in the POI sequence is mapped to the position code E at the POI level pp (j). POI-level location embedding Table E pp () are the trainable parameters of the multimodal base model.
[0248] (3) Grid-level
[0249] Figure 5 The embodiment shown further considers the distribution of POIs in two-dimensional space. Specifically, the urban area is first divided into G grids and numbered as g = 1, 2, ..., G. Then, for the POIs located in the grid g, the grid-level position embedding table E is used. gp (), maps the grid number g to the grid level position code E gp (g) Grid-level position embedding Table E gp () are the trainable parameters of the multimodal base model.
[0250] Combining the location codes at the above three levels, the overall geographic spatial location code E of each token can be calculated. p (t jk ):
[0251] E P (t jk )=E wp (k)+E pp (j)+E gp (g) (12)
[0252] 1.1.3 Category Embedding
[0253] Considering that the type of POI reflects its functional information and plays an important role in describing the characteristics of an area, we also encode the category id of POI. j , we can use a learnable embedding table E c () its category idc j Mapped to a d-dimensional embedding in the latent space: Tokens of the same POI will share the same category embedding.
[0254] Finally, we combine word embedding, geo-aware location encoding, category embedding, and an additional modality embedding E m (P), get token t jk The final POI embedding E P (t jk ):
[0255] E P (t jk )=E w (t jk )+E p (t jk )+E c (c j )+E m (P) (13)
[0256] Modal Embedding Table E m () is used to map the interest point data modality P or satellite image data modality S into a d-dimensional modality embedding. Modality Embedding Table E m () are the trainable parameters of the multimodal base model.
[0257] In summary, by inputting the POI data of a region into the POI embedding layer in the multimodal geospatial data embedding module, the POI embedding sequence X corresponding to the POI data can be generated. P :
[0258]
[0259] in, It is the embedding of the i-th token in the POI name sequence obtained by calculating the above formula (13), L P Indicates the maximum sequence length allowed by the model, x [P] Corresponds to the CLS token [P] at the beginning of the sequence.
[0260] 1.2 Satellite Image Embedding
[0261] Similar to the general visual base model, the satellite image can be divided into patches and the patches can be encoded using a linear transformation. Transformed into a patch sequence, the size of the patch is s×s×3, and the length of the patch sequence is L S =HW / (s*s). Then, each patch in the patch sequence is mapped into a d-dimensional vector representation through a linear transformation. A learnable CLS token [S] will be inserted into the head of the sequence, and then a learnable 1-D position code E will be added to each patch. 1D () and modal embedding E m (S).
[0262] In summary, by inputting the satellite image of a region into the satellite image embedding layer, we can generate the satellite image embedding sequence X S :
[0263]
[0264] Finally, the POI embedding sequence and the satellite image embedding sequence are concatenated to obtain the multimodal embedding sequence X = [X P ;X S ], the sequence length is L P +L S +1, this sequence will be used as input to the Hybrid Geographic Expert Transformer.
[0265] 2. Hybrid Geographic Expert Transformer (MoGE Transformer)
[0266] After the embedding module, a hybrid geographic expert (MoGE) Transformer module is connected. This module generates contextualized region representations based on multimodal inputs. The MoGE Transformer adopts the structure of Multiway Transformer, and its core idea is to apply specialized sub-networks to process different types of urban data. This strategy can consider the specific patterns of two different modalities, POI and satellite images, and capture the correlation between different modalities (Cross-ModalDependency).
[0267] like Figure 5As shown in the module (b) in Figure 1, the MoGE Transformer replaces the single feed-forward network (FFN) in the standard Transformer model with a set of sub-networks, each of which uses different parameters. These sub-networks are designated as geospatial experts to process different data types, including a sub-network for processing POI data (P-FFN), a sub-network for processing satellite images (S-FFN), and a sub-network for processing two data types at the same time (PS-FFN).
[0268] This modular structure has two advantages: first, different forms of geospatial data exhibit unique characteristics, and it is difficult for a single network to effectively represent them. In MoGE Transformer, different parts of the input sequence are routed to corresponding sub-networks according to their modalities, and these sub-networks can process specific patterns of different modalities respectively; second, the shared multi-head attention layer (Shared MHA (Multi-Head Attention) layer) in MoGE Transformer adopts a cross-modal shared multi-head self-attention (Multi-head Self-attention, MSA) mechanism, which enables multimodal data to interact and fuse deeply and capture cross-modal dependencies, which is crucial for multimodal region representation learning.
[0269] like Figure 5 As shown in module (b) in , MoGE Transformer includes L MoGE Transformer processing units. Each processing unit includes a shared multi-head attention layer (Shared MHA layer), a residual connection and normalization layer (Add&Norm), a geographic expert sub-network (P-FFN, S-FFN and PS-FFN) and a residual connection and normalization layer (Add&Norm) connected in sequence. In the bottom (L-2) MoGE Transformer processing units, the sub-network P-FFN and the sub-network S-FFN are used to internally fuse the POI embedding sequence and the satellite image embedding sequence respectively. In the top two MoGE Transformer processing units, the sub-network PS-FFN is used to simultaneously process the POI embedding sequence and the satellite image embedding sequence to achieve deep interaction and fusion of multimodal data.
[0270] The MoGE Transformer model takes the multimodal embedding sequence X output by the multimodal geospatial data embedding module as input and outputs a processed vector representation sequence H = {h0,h1,h2,...,h L}, where h[P] =h0 and There are two CLS tokens, which are used to represent the pooled POI representation and the pooled satellite image representation respectively.
[0271] For the above multimodal basic model, two types of target tasks, namely knowledge distillation tasks and self-supervised learning tasks, are set to pre-train the model. The knowledge distillation task introduces three knowledge distillation objectives to transfer the common knowledge of multiple basic models to this model to improve its generalization ability and cope with downstream tasks in the field of urban area recognition. In addition, considering that geospatial domain knowledge is crucial for urban area recognition, two self-supervised learning tasks are set to train the model to capture useful features from multimodal urban data.
[0272] 3. Joint Knowledge Distillation from Generic FMs
[0273] In order to make the multimodal base model universally effective in various urban area recognition tasks, the powerful representation ability of the universal base model is used to enhance the model. Since the model needs to process multimodal information including text data (POI names) and visual data (satellite images), three knowledge distillation objectives are set to transfer the rich knowledge of the language base model (LFM), the image representation ability of the visual base model (VFM), and the semantic alignment ability of the visual-language base model (VLFM). Knowledge distillation follows the teacher-student paradigm, in which this model is a student model, which is jointly guided by three teacher models: LFM, VFM and VLFM. Figure 5 Module (c) in shows this teacher-student relationship.
[0274] 3.1 Distillation from Language FM (DLFM)
[0275] The ability of the model to understand regional functions based on POI data is enhanced by distilling a large language model (LLM). The LLM can generate a regional description based on POI data, infer the regional functions and possible resident activities, and encode the regional description as knowledge into LLM features. Subsequently, feature-based distillation methods are used to guide the model to extract relevant semantic information from POI data.
[0276] Specifically, given a set of POIs in a region Generate prompt text based on POI name The format of the prompt text is: "There are the following facilities in the area: POI1, POI2, ... Please infer the function of this area and possible resident activities:". LLM uses this prompt to generate a functional description of the area. Next, the generated function description is transformed into Encoded into a vector, called LLM feature. The process can be expressed as:
[0277]
[0278] Among them, u P It is an LLM feature, and SenEmb() represents the encoding process of the text embedding model.
[0279] Then, the cosine similarity between the LLM features and the POI features generated by this model is used to define the optimization objective of knowledge distillation:
[0280]
[0281] Among them, h [P] is the vector representation of the CLS token [P] position generated by this model, representing the representation of the POI modality, σ POI () is a linear transformation function used to map the POI modal representation to the LLM feature space. This optimization goal enables this model to learn semantic information related to regional functions from POI data under the guidance of LLM features. POI () can be a preset or a trainable parameter of this model.
[0282] The embodiments of the present disclosure do not limit the specific types of LLM and text embedding models. For example, LLM can use the ChatGLM-6B model, and the text embedding model can use the M3E (Moka Massive Mixed Embedding)-base model.
[0283] 3.2 Distillation from Visual FM (DVFM)
[0284] The model’s ability to represent satellite images is enhanced by distilling the visual basis model (VFM). Given a satellite image S of a region, the pre-trained visual basis model VFM() is used as the teacher model to extract the semantic features of the satellite image: S= VFM(S). Then, this feature is used to guide the model (student model) to learn satellite image representation through the distillation objective function based on cosine similarity:
[0285]
[0286] Among them, h [s] is the satellite image representation generated by this model, σ Sate () represents the linear transformation function used for this distillation task. σ Sate () can be a preset or a trainable parameter of this model.
[0287] 3.3 Distillation from Visual-Language FM (DVLFM)
[0288] In order to further enhance the semantic alignment capability of this model for multimodal urban data, the visual-language grounded model (VLFM) is used to perform knowledge distillation on this model. The visual-language grounded model has shown a very powerful ability in joint understanding and alignment of text-image content.
[0289] The core idea of this distillation task is that VLFM, which is pre-trained with a large amount of data, can encode text and images into the same latent space, so that the semantic similarity between the two can be compared by calculating the cosine similarity of the vector representations of different modal data in the latent space. We hope that this model can compare the semantic similarity of different modal data like VLFM, so that the cosine similarity between semantically similar POIs and satellite images is higher.
[0290] Therefore, for a group of urban areas, VLFM can be used to obtain the vector representations of the POIs and satellite images of these areas, and a cross-modal cosine similarity matrix of POI-satellite images between regions can be calculated based on these vector representations. This similarity matrix reflects the relative relationship between the semantic similarity between POIs and satellite images in different regions. Similarly, a cosine similarity matrix can also be calculated using this model. In order for this model to learn the cross-modal semantic alignment capability of VLFM, the cosine similarity matrix generated by this model can be matched to the matrix generated by VLFM.
[0291] Specifically, given a batch size B of region samples Use VLFM to encode their POI and satellite imagery data into semantic representations: and And calculate the cosine similarity matrix in In this matrix, the i-th row (column) reflects the region Ri The semantic relationship between the POI (satellite image) of the area and the satellite images (POI) of other areas in the batch. At the same time, this model also generates vector representations of two modalities and And calculate the cosine similarity matrix The elements in this matrix μ POI () and μ Sate () are two linear transformation functions, which can be preset or trainable parameters of this model. Then, the optimization goal of knowledge distillation for VLFM is defined as minimizing the KL divergence of the corresponding rows and columns of the two matrices, that is, to achieve the transformation of the matrix Matching to matrix M:
[0292]
[0293] Among them, ρ represents the softmax function, which is used to transform the rows and columns of the cosine similarity matrix into a probability distribution.
[0294] 4. Self-Supervised Learning on Geospatial Data
[0295] In order to enable the model to learn domain knowledge from urban data, this embodiment uses two self-supervised tasks: the first is Mask Geospatial Data Modeling (MGDM), which enables the model to understand the semantics of POI and satellite image data; the second is Cross-Modal Spatial Alignment (CMSA), which uses the spatial relationship between POI and satellite images to promote the alignment of the two modalities. Figure 5 As shown in module (d) in .
[0296] 4.1 Mask Geospatial Data Modeling (MGDM)
[0297] The Mask-then-predict paradigm is widely used in the pre-training of language, vision, and multimodal basic models. In the pre-training framework of this model, pre-training is also performed by mask modeling of multimodal urban data.
[0298] The general idea is to mask a portion of the POI and satellite image input data, and require the model to understand the text content, visual content, and spatial relationship of the unmasked part, and restore the masked part based on this information. For the input POI data, randomly mask 15% of the tokens and replace them with a special mask token [M], then let the model learn how to complete this part of the information. For satellite images, mask 40% of the image patches and predict the discrete visual tokens at these locations.
[0299] use and Represent the positions of the masked POI token and image patch respectively. The masked sequence is input into the model to obtain the vector representation sequence H. The training goal of this task is to minimize and Negative log-likelihood of the correct POI token and the correct visual token at the position:
[0300]
[0301] Among them, the first Indicates that the masked location POI token can be correctly predicted The prediction is done by a two-layer multi-layer perceptron (MLP) classifier with a softmax function. Indicates the correct prediction of the masked position visual token probability.
[0302] 4.2 Cross-Modal Spatial Alignment (CMSA)
[0303] Although POIs and satellite images describe urban areas from completely different perspectives, each POI corresponds to a location in the satellite image. This geospatial relationship can serve as a link to align the semantics between the two modalities. The CMSA task is based on this idea, allowing the model to perceive the positional relationship between POI and satellite image content, thereby promoting the semantic alignment between the two modalities.
[0304] Specifically, in the MGDM task, a portion of the image patch is masked, and in the CMSA task, the model is required to determine whether each POI is located in the masked area of the satellite image, that is, to perform binary classification on the vector representation of the POI and optimize the model with a binary cross entropy loss function. It is worth noting that the masked POI tokens in the MGDM task (these tokens are replaced with [M]) are not included in the calculation of the loss function to avoid the model learning a mapping that simply predicts [M] as the positive class. This training objective can be expressed as:
[0305]
[0306] Among them, the two-category label y i Indicates whether the POI at this location is located in a covered patch, y i = 1 means the POI is located in the masked patch, and y i = 0 means that the POI is not in the masked patch. i |h i ) represents the output probability of the sigmoid classifier.
[0307] 5. Pre-training
[0308] Three knowledge distillation tasks and two self-supervision tasks are used to pre-train this model, and the total loss function is:
[0309]
[0310] 6. Downstream Prediction
[0311] After obtaining the pre-trained model, the POI vector output by the model can be represented as h [P] and the satellite image vector h [S] Taking the average, we get the vector representation of the urban area. Subsequently, this model can be applied to the downstream task of urban area recognition in the following two ways.
[0312] 6.1 Fine-tuning the pre-trained model
[0313] A prediction network for downstream tasks (such as linear regression, logistic regression classification network, etc.) is connected to the output end of this model, and then the pre-trained model and the prediction network are trained together using the training data of the downstream task (i.e., the parameters of both are adjusted at the same time).
[0314] 6.2 Feature-based Prediction
[0315] Different from the above-mentioned fine-tuning method, the feature-based prediction method fixes the weights of the pre-trained model and only optimizes the prediction network in downstream tasks (i.e., only adjusts the parameters of the prediction network).
[0316] The multimodal basic model and its pre-training framework for the field of urban area recognition proposed in the embodiments of the present disclosure can effectively integrate the domain features in multimodal geographic data and the general knowledge of multiple general domain basic models, and make better use of multimodal data to effectively solve a wide range of urban area recognition tasks. Specifically, first, compared with directly applying the general basic model, we designed a special model structure to jointly model the text content, visual content and spatial information of multimodal geographic data, and pre-trained it with geographic data, so that the model has the domain knowledge that is crucial to the regional recognition task. Second, compared with training a special basic model from scratch based on urban data, we distilled the general domain large model and utilized the general knowledge of the large model, so that the model can obtain good generalization ability when the training data is limited. Third, combining the method of a single general basic model and urban data, we distilled multiple language, vision, and visual-language basic models at the same time to enhance the model's text understanding ability, image representation ability, and cross-modal alignment ability, so that it can better utilize multimodal geographic data and solve a wider range of regional recognition tasks.
[0317] Based on the trained multimodal basic model (ie, the first multimodal basic model mentioned above), the embodiment of the present disclosure also provides a method for identifying urban areas. Figure 6 FIG. 6 is a flow chart of a method 600 for identifying an urban area according to an embodiment of the present disclosure. The execution subject of the method 600 may be a server (eg Figure 1 The server 120 in FIG. 1 may also be a client device (eg, Figure 1 101-106 in the client device). Figure 6 As shown, method 600 includes steps S610-S640.
[0318] In step S610, city data of a target city area to be identified is obtained. The city data includes a plurality of data segments corresponding to a plurality of data modalities.
[0319] In step S620, the city data is input into the multimodal basic model to obtain segment vector representations of each of the multiple data segments output by the multimodal basic model. The multimodal basic model is obtained by the multimodal basic model training method 200 described above.
[0320] In step S630, a region vector representation of the target city region is determined based on the segment vector representations of each of the plurality of data segments.
[0321] In step S640, based on the region vector representation, a prediction network is used to identify target information of the target city region.
[0322] According to the embodiments of the present disclosure, a regional vector representation that can fully and accurately express the characteristics of urban areas can be generated by using a trained multimodal basic model. This comprehensive and accurate regional vector representation is applied to downstream tasks of urban area identification, thereby improving the accuracy and efficiency of urban area identification.
[0323] According to some embodiments, in step S610, the city data of the target city area includes, for example, point of interest data and image data. The point of interest data includes, for example, the name, category, and geographic location of each point of interest located in the target city area. The image data includes, for example, a satellite image of the target city area.
[0324] According to some embodiments, in step S630, the region vector representation of the target urban region is obtained by fusing the respective segment vector representations of the plurality of data segments.
[0325] For example, the average value of the fragment vector representations of multiple data fragments can be determined as the regional vector representation of the target city area, or the sum, weighted sum or splicing result of the fragment vector representations of multiple data fragments can be used as the regional vector representation of the target city area.
[0326] According to some embodiments, in step S640, the prediction network is a neural network that has been trained for the urban area recognition task for target information prediction. The prediction network can be obtained by training, for example, through steps S201-S205 described above.
[0327] According to some embodiments, in step S640, the prediction information of the second sample urban area may be, for example, information such as the type, population size, traffic flow, economic index, etc. of the second sample urban area.
[0328] According to an embodiment of the present disclosure, a training device for a multimodal basic model is also provided. Figure 7 FIG. 7 shows a structural block diagram of a multimodal basic model training device 700 according to an embodiment of the present disclosure. Figure 7 As shown, the apparatus 700 includes a first acquisition module 710 , a prediction module 720 , a second acquisition module 730 , a knowledge distillation module 740 , a determination module 750 and an adjustment module 760 .
[0329] The first acquisition module 710 is configured to acquire first city data of a first sample city area. The first city data includes a plurality of first data segments corresponding to a plurality of data modalities.
[0330] The prediction module 720 is configured to input the first city data into the multimodal basic model to obtain a prediction vector representation of each of the multiple first data segments output by the multimodal basic model.
[0331] The second acquisition module 730 is configured to acquire a plurality of pre-trained universal base models, wherein each universal base model in the plurality of universal base models corresponds to at least one data modality in a plurality of data modalities.
[0332] The knowledge distillation module 740 is configured to perform the following operations for each common base model among multiple common base models: using the common base model, generating a vector representation label for the first data segment of the corresponding data modality; and determining the knowledge distillation loss of the common base model based on the vector representation label and the predicted vector representation of the first data segment.
[0333] The determination module 750 is configured to determine the overall loss of the multimodal base model based at least on the knowledge distillation losses of the respective common base models.
[0334] The adjustment module 760 is configured to adjust the parameters of the multimodal base model based on the overall loss.
[0335] According to an embodiment of the present disclosure, a multimodal basic model for the field of urban area identification that can solve a wide range of downstream tasks is provided, and a pre-training method for the multimodal basic model is provided by adopting a self-supervised pre-training paradigm. By performing knowledge distillation on existing general basic models for different data modalities, the multimodal basic model of the embodiment of the present disclosure can fully learn multimodal knowledge under the guidance of multiple existing general basic models, thereby improving its generalization, enabling it to accurately understand multimodal urban data, and then generate a general vector representation that can comprehensively and accurately express the characteristics of urban areas. This comprehensive and accurate vector representation can be widely used in a variety of downstream tasks of urban area identification, thereby improving the accuracy and efficiency of urban area identification.
[0336] The disclosed embodiment can obtain a general basic model with good generalization and comprehension capabilities for the field of urban area recognition by using only limited unlabeled data. There is no need to obtain a large amount of labeled data, nor is there a need to design and train complex models for different urban area recognition tasks, thereby improving the efficiency and accuracy of urban area recognition.
[0337] It should be understood that Figure 7 The various modules and units of the apparatus 700 shown in FIG. 7 can be compared with those of the reference Figure 2The steps in the method 200 described above correspond to each other. Therefore, the operations, features and advantages described above for the method 200 are also applicable to the device 700 and the modules and units included therein. For the sake of brevity, some operations, features and advantages are not repeated here.
[0338] According to an embodiment of the present disclosure, a device for identifying an urban area is also provided. Figure 8 FIG. 8 is a block diagram showing a structure of an urban area identification device 800 according to an embodiment of the present disclosure. Figure 8 As shown, the apparatus 800 includes an acquisition module 810 , a representation module 820 , a determination module 830 and an identification module 840 .
[0339] The acquisition module 810 is configured to acquire the city data of the target city area to be identified. The city data includes a plurality of data segments corresponding to a plurality of data modalities.
[0340] The representation module 820 is configured to input the city data into the multimodal basic model to obtain a segment vector representation of each of the multiple data segments output by the multimodal basic model. The multimodal basic model is trained by the multimodal basic model training device 700.
[0341] The determination module 830 is configured to determine a region vector representation of the target city region based on the segment vector representations of each of the plurality of data segments.
[0342] The identification module 840 is configured to identify target information of the target urban area using a prediction network based on the area vector representation.
[0343] According to the embodiments of the present disclosure, a regional vector representation that can fully and accurately express the characteristics of urban areas can be generated by using a trained multimodal basic model. This comprehensive and accurate regional vector representation is applied to downstream tasks of urban area identification, thereby improving the accuracy and efficiency of urban area identification.
[0344] It should be understood that Figure 8 The various modules and units of the apparatus 800 shown in FIG. 8 can be compared with those of the reference Figure 6 The steps in the method 600 described above correspond to each other. Therefore, the operations, features and advantages described above for the method 600 are also applicable to the device 800 and the modules and units included therein. For the sake of brevity, some operations, features and advantages are not repeated here.
[0345] Although specific functionality is discussed above with reference to specific modules, it should be noted that the functionality of the various modules discussed herein may be separated into multiple modules, and / or at least some functionality of multiple modules may be combined into a single module.
[0346] It should also be understood that various techniques may be described herein in the general context of software hardware elements or program modules. Figure 7 , Figure 8 The various units described can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these units can be implemented as computer program codes / instructions, which are configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, these units can be implemented as hardware logic / circuits. For example, in some embodiments, one or more of modules 710-760, 810-840 can be implemented together in a system on chip (System on Chip, SoC). SoC can include an integrated circuit chip (which includes a processor (e.g., a central processing unit (CPU), a microcontroller, a microprocessor, a digital signal processor (DSP), etc.), a memory, one or more communication interfaces, and / or one or more components in other circuits), and can optionally execute the received program code and / or include embedded firmware to perform functions.
[0347] According to an embodiment of the present disclosure, there is also provided an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor so that the at least one processor executes the multimodal basic model training method and / or the urban area recognition method of the embodiment of the present disclosure.
[0348] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is also provided, and the computer instructions are used to enable a computer to execute the multimodal basic model training method and / or urban area recognition method of the embodiment of the present disclosure.
[0349] According to an embodiment of the present disclosure, a computer program product is also provided, including computer program instructions, which, when executed by a processor, implement the multimodal basic model training method and / or urban area recognition method of the embodiment of the present disclosure.
[0350] refer to Fig. 9, a block diagram of an electronic device 900 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0351] like Fig. 9 As shown, the electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.
[0352] Multiple components in the electronic device 900 are connected to the I / O interface 905, including: an input unit 906, an output unit 907, a storage unit 908, and a communication unit 909. The input unit 906 can be any type of device that can input information to the electronic device 900. The input unit 906 can receive input digital or character information and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 907 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 908 can include but is not limited to a disk, an optical disk. The communication unit 909 allows the electronic device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, an 802.11 device, a Wi-Fi device, a WiMAX device, a cellular communication device, and / or the like.
[0353] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as method 200 and method 600. For example, in some embodiments, method 200 and method 600 may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of method 200 and method 600 described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute the method 200 and / or the method 600 in any other appropriate manner (eg, by means of firmware).
[0354] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0355] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0356] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0357] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0358] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.
[0359] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0360] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be performed in parallel, sequentially or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document is not limited here.
[0361] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present disclosure is not limited by these embodiments or examples, but is only limited by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples may be omitted or replaced by their equivalent elements. In addition, each step may be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples may be combined in various ways. It is important that with the evolution of technology, many of the elements described herein may be replaced by equivalent elements that appear after the present disclosure.
Claims
1. A method for training a multimodal basic model, comprising: Acquire first city data of a first sample city area, wherein the first city data includes a plurality of first data segments respectively corresponding to a plurality of data modalities; Inputting the first city data into the multimodal basic model to obtain a prediction vector representation of each of the plurality of first data segments output by the multimodal basic model; Acquire a plurality of pre-trained universal base models, wherein each universal base model in the plurality of universal base models corresponds to at least one data modality among the plurality of data modalities; For each common base model in the plurality of common base models: generating a vector representation label of a first data segment of a corresponding data modality using the common base model; and determining a knowledge distillation loss for the common base model based on the vector representation label and the predicted vector representation of the first data segment; determining an overall loss of the multimodal base model based at least on the knowledge distillation losses of each of the plurality of common base models; and Based on the overall loss, parameters of the multimodal base model are adjusted.
2. The method according to claim 1, wherein: The plurality of first data segments include interest point data corresponding to a text modality, the plurality of universal base models include a language base model corresponding to a text modality, and the step of generating a vector representation label of a first data segment of a corresponding data modality by using the universal base model includes: Inputting at least a portion of the POI data into the language base model to obtain a description text of the first sample city area generated by the language base model; and The description text is encoded to obtain a vector representation label of the interest point data.
3. The method according to claim 1, wherein: The plurality of first data segments include image data corresponding to an image modality, the plurality of universal base models include a visual base model corresponding to an image modality, and using the universal base model to generate a vector representation label of the first data segment of the corresponding data modality includes: The image data is input into the visual basis model to obtain a vector representation label of the image data output by the visual basis model.
4. The method according to any one of claims 1 to 3, wherein: The plurality of universal base models include a unimodal universal base model corresponding to one of the plurality of data modalities, and the determining the knowledge distillation loss of the universal base model based on the vector representation label and the predicted vector representation of the first data segment includes: performing a linear transformation on the prediction vector representation of the first data segment to obtain a transformed prediction vector representation; Calculating the similarity between the transformed predicted vector representation and the vector representation label; and Based on the similarity, a unimodal knowledge distillation loss of the unimodal universal basic model is determined, wherein the unimodal knowledge distillation loss is negatively correlated with the similarity.
5. The method according to any one of claims 1 to 4, wherein: The multiple common base models include a cross-modal common base model, the cross-modal common base model corresponds to a first data modality and a second data modality among the multiple data modalities, and the determining the knowledge distillation loss of the common base model based on the vector representation label and the predicted vector representation of the first data segment includes: Obtaining a first predicted vector representation and a first vector representation label of first data of the first sample urban area, and a second predicted vector representation and a second vector representation label of second data; generating a prediction similarity matrix based on the first prediction vector representation and the second prediction vector representation of each of the plurality of first sample city regions; Generate a similarity matrix label based on the first vector representation label and the second vector representation label of each of the plurality of first sample city areas; and Based on the difference between the predicted similarity matrix and the similarity matrix label, a cross-modal knowledge distillation loss of the cross-modal general basic model is determined, wherein the cross-modal knowledge distillation loss is positively correlated with the difference.
6. The method according to claim 5, wherein: The first data modality is a text modality, the first data is interest point data, the second data modality is an image modality, the second data is image data, and the cross-modal universal basic model is a visual language basic model.
7. The method according to any one of claims 1 to 6, wherein: Each of the plurality of first data segments is split into a plurality of data units, and the method further comprises: For each first data segment of the plurality of first data segments, at least one data unit in the first data segment is replaced with a preset mask data unit to obtain masked first city data; Inputting the masked first city data into the multimodal basic model to obtain a vector representation of each data unit in the masked first city data output by the multimodal basic model; For each mask data unit, using a first classifier to determine a first probability of correctly predicting a pre-masked data unit corresponding to the mask data unit based on the vector representation of the mask data unit; and Determining a mask loss of the multimodal base model based on a first probability corresponding to each mask data unit, The determining of the overall loss of the multimodal base model based at least on the knowledge distillation losses of the respective common base models comprises: An overall loss of the multimodal base model is determined based on the knowledge distillation losses of the respective common base models and the mask loss.
8. The method according to any one of claims 1 to 6, wherein: The first city data includes point of interest data in a text mode and image data in an image mode, the point of interest data is split into a plurality of point of interest data units, and the image data is split into a plurality of image data units, and the method further includes: Replacing at least one of the plurality of interest point data units with a preset interest point mask data unit, and replacing at least one of the plurality of image data units with a preset image mask data unit, to obtain masked first city data; Inputting the masked first city data into the multimodal basic model to obtain a vector representation of each data unit in the masked first city data output by the multimodal basic model; For each unmasked interest point data unit, based on the vector representation of the interest point data unit, using a second classifier to determine a second probability that the interest point corresponding to the interest point data unit is located in the masked image data unit; and Based on the second probability corresponding to each unmasked interest point data unit, a cross-modal spatial alignment loss of the multimodal base model is determined.
9. The method according to any one of claims 1 to 8, wherein: The first city data includes interest point data in a text modality and image data in an image modality, the multimodal base model includes an interest point encoder, an image encoder, and a multimodal transformer, and the step of inputting the first city data into the multimodal base model to obtain a prediction vector representation of each of the plurality of first data segments output by the multimodal base model includes: Inputting the interest point data into the interest point encoder to obtain an interest point embedding sequence of the interest point data, wherein the interest point embedding sequence includes an embedding of each interest point data unit in the interest point data; inputting the image data into the image encoder to obtain an image embedding sequence of the image data, the image embedding sequence comprising an embedding of each image data unit in the image data; and The interest point embedding sequence and the image embedding sequence are input into the multimodal transformer so that the multimodal transformer fuses and updates the interest point embedding sequence and the image embedding sequence to obtain a predicted vector representation of the interest point data and a predicted vector representation of the image data.
10. The method according to claim 9, wherein: The point of interest data includes the names, geographic locations and categories of the plurality of points of interest located in the first sample city area, and the step of inputting the point of interest data into the point of interest encoder to obtain a point of interest embedding sequence for the point of interest data includes: The interest point data is input into the interest point encoder so that the interest point encoder performs the following operations: Representing the names of the plurality of points of interest as a point of interest name sequence, wherein each token in the point of interest name sequence is a point of interest data unit; and For each token in the sequence of POI names, an embedding of the token is generated based on the token and at least one of a geographic location and a category of the POI to which the token corresponds.
11. The method according to claim 10, wherein: The step of representing the names of the plurality of points of interest as a sequence of points of interest names comprises: sorting the plurality of interest points based on relative distances between the plurality of interest points to obtain an interest point sequence; Combining the names of the points of interest in the sequence of points of interest to obtain a pseudo sentence of the name of the points of interest; and The interest point name pseudo sentence is segmented to obtain the interest point name sequence.
12. The method according to claim 11, wherein: The step of sorting the plurality of interest points based on the relative distances between the plurality of interest points to obtain an interest point sequence comprises: Dividing the first sample urban area into a plurality of grids of the same size, wherein the plurality of grids form a plurality of grid pairs, and each grid pair in the plurality of grid pairs includes two grids located adjacently; Sorting the plurality of grids using a preset sorting strategy to obtain a grid sequence, wherein two grids in some grid pairs among the plurality of grid pairs are adjacent in the grid sequence; and The interest points located in each grid in the grid sequence are respectively acquired to obtain the interest point sequence.
13. The method according to any one of claims 10 to 12, wherein: The step of generating an embedding of the token based on the token and at least one of a geographical location and a category of a point of interest corresponding to the token comprises: Encode the token to obtain a word embedding of the token; Generate a geographic location embedding of the token based on the geographic location of the point of interest corresponding to the token; Encoding the category of the interest point corresponding to the token to obtain a category embedding of the token; and Based on the word embedding, the geographic location embedding and the category embedding, an embedding of the token is determined.
14. The method according to claim 13, wherein: The generating the geographic location embedding of the token based on the geographic location of the point of interest corresponding to the token comprises: Based on the geographical location of the point of interest corresponding to the token, obtaining multiple location information of different granularities of the token; and Based on the multiple location information, a geographic location embedding of the token is generated.
15. The method according to claim 14, wherein: The multiple location information includes a first position of the token in the name of the point of interest, a second position of the point of interest in a sequence of points of interest corresponding to the sequence of names of the points of interest, and a grid where the point of interest is located, the grid being obtained by evenly dividing the first sample city area, and the generating the geographic location embedding of the token based on the multiple location information includes: encoding the first position to obtain a word position embedding of the token; Encoding the second position to obtain an interest point position embedding of the token; Encoding the identity of the grid to obtain a grid position embedding of the token; and The word position embedding, the point of interest position embedding and the grid position embedding are fused to obtain the geographic location embedding of the word token.
16. The method according to any one of claims 9 to 15, wherein: The step of inputting the image data into the image encoder to obtain an image embedding sequence of the image data comprises: The image data is input to the image encoder so that the image encoder performs the following operations: Splitting the image data into a plurality of image blocks of the same size, wherein each of the plurality of image blocks is an image data unit; and For each image block in the image data, a linear transformation is performed on the image block to obtain an embedding of the image block.
17. The method according to any one of claims 1 to 16, wherein: The trained first multimodal base model is used to generate a regional vector representation of an urban area, and the regional vector representation is used as an input of a prediction network so that the prediction network recognizes target information of the urban area based on the regional vector representation.
18. The method according to claim 17, wherein: The target information includes any one of the following information: type, population, traffic flow, and economic index.
19. The method according to any one of claims 1 to 18, further comprising: Acquire second city data of a second sample city area, an information label of the second sample city area, and a trained first multimodal basic model, wherein the second city data includes a plurality of second data segments respectively corresponding to a plurality of data modalities; Inputting the second city data into the first multimodal base model to obtain a segment vector representation of each of the plurality of second data segments output by the first multimodal base model; determining a region vector representation of the second sample city region based on the segment vector representations of each of the plurality of second data segments; Inputting the region vector representation of the second sample city region into a prediction network to obtain prediction information of the second sample city region output by the prediction network; Based on the prediction information and the information label, at least a parameter of the prediction network is adjusted.
20. The method according to claim 19, wherein: The adjusting at least a parameter of the prediction network based on the prediction information and the information label comprises: Determining a joint loss of the first multimodal base model and the prediction network based on the prediction information and the information label; and Based on the joint loss, parameters of the first multimodal base model and the prediction network are adjusted.
21. The method according to claim 19, wherein: The adjusting at least a parameter of the prediction network based on the prediction information and the information label comprises: Determining a loss of the prediction network based on the prediction information and the information label; and Based on the loss, parameters of the prediction network are adjusted.
22. A method for identifying an urban area, comprising: Acquire city data of a target city area to be identified, wherein the city data includes a plurality of data segments corresponding to a plurality of data modalities; Inputting the city data into a multimodal basic model to obtain a segment vector representation of each of the multiple data segments output by the multimodal basic model, wherein the multimodal basic model is trained by the method according to any one of claims 1 to 21; Determining a region vector representation of the target city region based on the segment vector representations of each of the plurality of data segments; and Based on the region vector representation, a prediction network is used to identify target information of the target urban region.
23. The method according to claim 22, wherein: The target information includes any one of the following information: type, population, traffic flow, and economic index.
24. A training device for a multimodal basic model, comprising: A first acquisition module is configured to acquire first city data of a first sample city area, wherein the first city data includes a plurality of first data segments respectively corresponding to a plurality of data modalities; a prediction module configured to input the first city data into the multimodal basic model to obtain a prediction vector representation of each of the plurality of first data segments output by the multimodal basic model; A second acquisition module is configured to acquire a plurality of pre-trained universal base models, wherein each universal base model in the plurality of universal base models corresponds to at least one data modality in the plurality of data modalities; The knowledge distillation module is configured to, for each common base model of the plurality of common base models: generating a vector representation label of a first data segment of a corresponding data modality using the common base model; and determining a knowledge distillation loss for the common base model based on the vector representation label and the predicted vector representation of the first data segment; A determination module configured to determine an overall loss of the multimodal base model based at least on the knowledge distillation losses of the respective common base models; and An adjustment module is configured to adjust parameters of the multimodal base model based on the overall loss.
25. A device for identifying urban areas, comprising: An acquisition module is configured to acquire city data of a target city area to be identified, wherein the city data includes a plurality of data segments corresponding to a plurality of data modalities respectively; a representation module configured to input the city data into a multimodal basic model to obtain a segment vector representation of each of the plurality of data segments output by the multimodal basic model, wherein the multimodal basic model is trained by the apparatus according to claim 24; a determination module configured to determine a region vector representation of the target city region based on the segment vector representations of each of the plurality of data segments; and The recognition module is configured to use a prediction network to recognize target information of the target urban area based on the area vector representation.
26. An electronic device comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 23.
27. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1-23.
28. A computer program product comprising computer program instructions, wherein: The computer program instructions implement the method of any one of claims 1-23 when executed by a processor.
Citation Information
Patent Citations
Crowd flow prediction method based on urban area multi-modal fusion
CN112396218A
Cross-modal data prediction method based on transfer learning
CN112863182A
Method and device for training model
CN115564059A
Multi-task learning method and device, electronic equipment and medium
CN115690544A
Power equipment operation inspection cognitive large model training method and system
CN117612189A
Cited By
Training method and system for urban rail transit multi-modal model
CN119226493A
Multi-modal generation type pre-training method and system and image question and answer generation method
CN121212375A
