Pre-training method and device for geographic language model

Abstract

The invention discloses a pre-training method and device for a geographic language model, and relates to the technical field of computers. One specific embodiment of the method comprises the following steps: acquiring text information and spatial position information, and determining a feature vector set based on the text information and the spatial position information; performing mask processing on part of the feature vectors in the feature vector set to obtain a mask result; constructing a multi-task language sample set based on the feature vector set and the mask result, and pre-training a geographic language model according to the multi-task language sample set; according to the embodiment, the richness of the pre-training data can be improved, the multi-task training requirement is met, and the accuracy and stability of the prediction result are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a pre-training method and device for a geographic language model. Background Art

[0002] The geographic language model is a natural language processing model used to process geographic information. The model understands and processes text information related to geographic location, geographic features, place names, regional culture, etc.

[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the related art:

[0004] The pre-training data of the geographic language model is relatively single and cannot meet the needs of multi-task training. In addition, the model prediction results have low accuracy and poor stability. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a pre-training method and device for a geographic language model, which can improve the richness of pre-training data, meet multi-task training requirements, and improve the accuracy and stability of prediction results.

[0006] To achieve the above object, according to one aspect of an embodiment of the present invention, a method for pre-training a geographic language model is provided, comprising:

[0007] Acquire text information and spatial position information, and determine a feature vector set based on the text information and spatial position information;

[0008] Performing masking processing on some feature vectors in the feature vector set to obtain a masking result;

[0009] A multi-task language sample set is constructed based on the feature vector set and the mask result, and a geographic language model is pre-trained according to the multi-task language sample set.

[0010] Further, determining a feature vector set based on the text information and the spatial position information includes:

[0011] Encoding the text information and the spatial position information to obtain an encoding result;

[0012] A feature vector set is determined based on the encoding result.

[0013] Further, masking is performed on some feature vectors in the feature vector set, including:

[0014] Mask marks are added to some feature vectors randomly extracted from the feature vector set or extracted according to a preset rule to obtain mask results.

[0015] Furthermore, constructing a multi-task language sample set based on the feature vector set and the mask result includes:

[0016] Based on a predetermined task type, a sample set corresponding to each task type is constructed using the feature vector set and the mask result; wherein the task type includes a mask language model task, a hierarchical mask pre-training task, a geographic prediction AOI task, or an AOI spatial relationship prediction task;

[0017] A multi-task language sample set is determined according to a sample set corresponding to each task type.

[0018] Furthermore, pre-training the geographic language model according to the multi-task language sample set includes:

[0019] The multi-task language sample set is input into a geographic language model for unsupervised training to obtain a geographic information prediction result of the mask result.

[0020] Furthermore, the method further comprises:

[0021] The multi-task language sample set and the geographic information prediction result of the mask result are input into the geographic language model to perform iterative optimization training on the geographic language model.

[0022] Furthermore, the text information and spatial location information are obtained, including:

[0023] In response to the pre-training request, parameter information corresponding to the text information and the spatial position information is parsed from the pre-training request, and the text information and the spatial position information are acquired according to the parameter information.

[0024] According to a second aspect of an embodiment of the present invention, a pre-training device for a geographic language model is provided, comprising:

[0025] An acquisition module, used to acquire text information and spatial position information, and determine a feature vector set based on the text information and spatial position information;

[0026] A masking module, used for performing masking processing on some feature vectors in the feature vector set to obtain a masking result;

[0027] A training module is used to construct a multi-task language sample set based on the feature vector set and the mask result, and pre-train a geographic language model according to the multi-task language sample set.

[0028] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, including:

[0029] one or more processors;

[0030] a memory for storing one or more programs,

[0031] When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method described in any one of the above embodiments.

[0032] According to a fourth aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method described in any one of the above embodiments is implemented.

[0033] An embodiment of the above invention has the following advantages or beneficial effects: by acquiring text information and spatial position information, a feature vector set is determined based on the text information and spatial position information; masking is performed on some feature vectors in the feature vector set to obtain a mask result; a multi-task language sample set is constructed based on the feature vector set and the mask result, and a geographic language model is pre-trained according to the multi-task language sample set; this embodiment can improve the richness of pre-training data, meet multi-task training requirements, and improve the accuracy and stability of prediction results.

[0034] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0036] Figure 1 is a schematic diagram of the main process of the pre-training method of the geographic language model according to an embodiment of the present invention;

[0037] Figure 2 It is a schematic diagram of the main process of a method for pre-training a geographic language model according to a reference embodiment of the present invention;

[0038] Figure 3 is a schematic diagram of the main process of a method for pre-training a geographic language model according to another reference embodiment of the present invention;

[0039] Figure 4 is a schematic diagram of the structure of a geographic language model according to an embodiment of the present invention;

[0040] Figure 5 is a schematic diagram of main modules of a pre-training device for a geographic language model according to an embodiment of the present invention;

[0041] Figure 6 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0042] Figure 7It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0044] It should be noted that the acquisition, storage and application of personal information etc. involved in the embodiments of the present invention are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0045] At present, the pre-training data of geographic language models is relatively simple and cannot meet the needs of multi-task training. In addition, the model prediction results are inaccurate and unstable. In logistics and distribution scenarios, address and trajectory data contain rich text information and spatial location information, which is a key data source for geographic language model training. Large natural language models have been widely used in downstream tasks such as address text entity recognition and geocoding in logistics scenarios. Model pre-training is of great significance for improving the performance of downstream subtasks.

[0046] In view of this, according to one aspect of an embodiment of the present invention, a pre-training method for a geographic language model is provided.

[0047] Figure 1 Schematic diagram of the main process of pre-training a geographic language model according to an embodiment of the present invention. Figure 1 As shown, the pre-training method of the geographic language model according to an embodiment of the present invention includes:

[0048] Step S101, obtaining text information and spatial position information, and determining a feature vector set based on the text information and spatial position information.

[0049] The text information refers to text data describing geographic location information, such as sentences, paragraphs, documents or other forms of address text content. The above text information can be obtained from sources such as web pages, documents, databases, social media, user input information, etc. The text information can be structured, such as table data in a database, or unstructured, such as news articles, social media posts, etc.

[0050] The spatial location information refers to specific coordinates or area information about the geographic spatial location, such as AOI data (Area of ​​Interest), POI data (Point of Interest), trajectory data, longitude and latitude coordinates, etc. The above spatial location information can be obtained through map service API (Application Programming Interface), geographic information database, user input, etc.

[0051] Specifically, the above text information can be preprocessed first, such as word segmentation, stop word removal, stemming or word form restoration, etc., to convert the text information into a form that can be used for feature representation; the spatial location information can be processed and converted, such as converting the longitude and latitude coordinates into geographic area information or other geographic features; the processed text information and spatial location information can be converted into feature vectors to form a feature vector set, each feature vector contains multiple features of the sample, including text features and spatial location features and other related features, which are used to describe various attributes and characteristics of the geographic location.

[0052] Among them, text information features include text content describing the geographic location. For example, if the sample is a descriptive text of a delivery address, the text information features may include the country, province, city, street, etc. in the address. Spatial location information features include the specific coordinates or area information of the geographic location, which can be directly used as part of the feature vector, or converted into numerical or categorical features through feature engineering. For example, if the sample is a coordinate point of a geographic location, the spatial location information features may include longitude, latitude, region, etc. In addition to text information and spatial location information, the feature vector set may also include other related features, such as time information, environmental information, demographic data, etc. For example, if the sample is a delivery address, in addition to text information and spatial location information, it may also include related features such as delivery time, delivery distance, surrounding traffic conditions, etc.

[0053] Step S102, performing mask processing on some feature vectors in the feature vector set to obtain mask results.

[0054] Among them, the partial feature vector refers to a part of the feature vector extracted or determined from the feature vector set. Specifically, extracting or determining the partial feature vector can be to randomly extract a part of the feature vector from the feature vector set; or to cluster the feature vectors and select a part of the feature vector from each cluster; or to select a part of the feature vector according to the label information, for example, for a binary classification task, a part of the positive and negative samples can be selected for mask processing. The above methods can be used alone or in combination to determine or extract the part of the feature vector that needs to be processed according to the specific situation.

[0055] Masking is a data processing technology that can hide, replace or blur a specific part of the data to enable model training or data protection. Specifically, this embodiment masks part of the feature vector, and the masking methods include random masking, fixed masking and fuzzy masking; among them, random masking is to randomly select part or all of the data in the feature vector for masking, using a specific mask mark or replacing it with other data; fixed masking is to perform fixed masking on a specific position or feature, such as masking the street name in the address to "[MASK]"; fuzzy masking is to blur the data, such as blurring a part of the text in the address information; the masked feature vector is saved or output to form a mask result.

[0056] The mask result refers to the data result obtained after mask processing, which can be a processed feature vector set, or processed text data or other data types.

[0057] Step S103: construct a multi-task language sample set based on the feature vector set and the mask result, and pre-train a geographic language model according to the multi-task language sample set.

[0058] Among them, the multi-task language sample set is a language dataset containing multiple tasks, which is used to train an address language model with multi-task learning capabilities in the pre-training stage.

[0059] Specifically, the masked results and unmasked feature vectors in the feature vector set are used to construct a multi-task language sample set. Samples for each task are generated and labeled according to the task type and sample set requirements. Data processing and preprocessing are performed on the constructed multi-task language sample set, including operations such as word segmentation, encoding, and tokenization, and necessary data cleaning and standardization are performed. The constructed multi-task language sample set is used to pre-train the geographic language model. During the pre-training process, the geographic language model learns the feature representations and language rules of multiple task types, thereby improving the accuracy and stability of the model on various geographic tasks. Self-supervised learning methods can be used in pre-training to enable the model to learn through a large amount of unlabeled data.

[0060] The embodiment of the present invention obtains text information and spatial position information, determines a feature vector set based on the text information and the spatial position information; performs masking on some feature vectors in the feature vector set to obtain a mask result; constructs a multi-task language sample set based on the feature vector set and the mask result, and pre-trains a geographic language model based on the multi-task language sample set; thereby, the richness of pre-training data can be improved, the requirements of multi-task training can be met, and the accuracy and stability of prediction results can be improved.

[0061] Optionally, optionally, Figure 2 FIG. 1 is a schematic diagram of the main process of a method for pre-training a geographic language model according to a reference embodiment of the present invention. Figure 2 As shown, step S101 determines a feature vector set based on the text information and the spatial position information, including:

[0062] Step S201, encoding the text information and spatial position information to obtain an encoding result.

[0063] Step S202: determining a feature vector set based on the encoding result.

[0064] Step S201 encodes the above text information and spatial location information and converts them into numerical representations that can be processed by a computer. The encoding process can be carried out in different ways, such as using a bag-of-words model or word embedding technology to vectorize the text; for example, using one-hot encoding, numerical encoding, etc. to decompose and encode the address structure and convert the longitude and latitude coordinates into vector representations; after the encoding process, the encoding results of the text information and spatial location information are obtained; the encoding results are vector or numerical representations, which are used to represent the characteristics of the text and the location.

[0065] After obtaining the encoding results of the text information and the spatial position information, step S202 combines them into a feature vector set, which includes a word embedding vector of the text information, a vector representation of the spatial position information, and a vector representation of other related features. The feature vector set is constructed by operations such as feature selection, feature combination, or feature dimensionality reduction to make the feature vector set have a suitable dimension and expression ability; pre-processing operations such as data normalization and standardization can also be performed to improve the stability and trainability of the feature vector set.

[0066] The embodiments of the present invention are conducive to the model to better understand and analyze geographic information data, and improve the performance and generalization ability of the model in geographic field tasks.

[0067] Optionally, step S102 performs mask processing on some feature vectors in the feature vector set, including:

[0068] Mask marks are added to some feature vectors randomly extracted from the feature vector set or extracted according to a preset rule to obtain mask results.

[0069] Among them, to randomly extract some feature vectors, first determine the proportion or number of some feature vectors to be randomly extracted from the feature vector set, for example, the extraction proportion can be set to 10% or the extraction number can be set to 100 feature vectors; according to the set proportion or number, use a random number generator to randomly select a corresponding number of feature vectors in the feature vector set.

[0070] Extract some feature vectors according to preset rules. First, define the extraction rules, that is, according to which conditions or standards to extract feature vectors. For example, the rules can be set according to the index range of the feature vector, certain attributes of the feature vector, the label of the feature vector, etc.; extract the feature vectors that meet the rules from the feature vector set according to the preset extraction rules. For example, use programming to filter the feature vectors according to the rules and add them to the extraction results.

[0071] The extracted part of the feature vectors can be masked by inserting a mask marker in the feature vector. The mask marker is a special token, such as "[MASK]", which is used to indicate that the information at that position is masked. The mask marker can also be added to a certain position or multiple positions of the feature vector, which can be a random position or a position determined according to a preset rule. After adding the mask marker, the new feature vector set obtained is the mask result, which includes the representation of some feature vectors in the original feature vector set after being masked.

[0072] In addition, in addition to adding mask marks, some feature vectors in the feature vector set can also be masked in other ways; for example, a certain degree of random noise can be added to the feature vector to make the information of the feature vector more difficult to predict; or some dimensions in the feature vector can be truncated or cropped to retain only the information of interest, etc.

[0073] The embodiments of the present invention extract feature vectors randomly or regularly and add mask marks. The model needs to fill in the mask positions with context information when making predictions, thereby improving the geographic language model's understanding of context and implicit information, making the model more adaptable to diverse data and tasks.

[0074] Optionally, Figure 3 FIG. 1 is a schematic diagram of the main process of a method for pre-training a geographic language model according to another reference embodiment of the present invention. Figure 3 As shown, step S103 constructs a multi-task language sample set based on the feature vector set and the mask result, including:

[0075] Step S301, based on a predetermined task type, using the feature vector set and the mask result to construct a sample set corresponding to each task type; wherein the task type includes a mask language model task, a hierarchical mask pre-training task, a geographic prediction AOI task or an AOI spatial relationship prediction task.

[0076] Step S302: determining a multi-task language sample set according to the sample set corresponding to each task type.

[0077] The multi-task language sample set is a language dataset that contains multiple tasks and is used to train a language model with multi-task learning capabilities in the pre-training stage. Multi-tasks include but are not limited to masked language modeling (MLM), hierarchical mask pre-training tasks, geographic prediction AOI tasks, or AOI spatial relationship prediction tasks.

[0078] In step S301, first, according to the predetermined task type, the feature vector set and the mask result are used to construct a sample set corresponding to each task type, and the corresponding sample construction method is determined for different task types. For the masked language model task, an input-output sample pair can be constructed based on the mask result, wherein the input sample contains the masked feature vector, and the output sample is the label or content of the mask position predicted by the model. For the hierarchical mask pre-training task, hierarchical mask processing can be performed on some feature vectors in the feature vector set according to the task requirements, and the corresponding input-output sample pairs can be constructed. For the geographic prediction AOI task or the AOI spatial relationship prediction task, an input sample containing geographic location information and mask results can be constructed according to the task requirements, and the format and label of the output sample can be determined according to the task type.

[0079] In step S302, according to the sample sets corresponding to each task type, the sample sets are merged or combined together to determine a multi-task language sample set, where the multi-task language sample set includes sample data of different task types.

[0080] The embodiment of the present invention merges or combines the sample sets corresponding to each task type to form a multi-task language sample set, which can include sample data of different task types, helps the geographic language model to learn and optimize multiple tasks at the same time, and improves the comprehensive performance and generalization ability of the model.

[0081] Optionally, step S103 pre-trains the geographic language model according to the multi-task language sample set, including:

[0082] The multi-task language sample set is input into a geographic language model for unsupervised training to obtain a geographic information prediction result of the mask result.

[0083] The constructed multi-task language sample set is input into the geographic language model for unsupervised training. Unsupervised training means that the model extracts features and predicts results by learning the inherent rules and structure of language data without explicit labels. During the unsupervised training process, the geographic language model can use the information in the input multi-task language sample set, such as the contextual connection between the mask result and other feature vectors, to predict the geographic information of the masked position.

[0084] The constructed multi-task language sample set is input into the geographic language model for unsupervised training, allowing the model to automatically discover patterns and regularities in the input data without manual labeling. The geographic language model can learn the intrinsic structure and associations of geographic information from diverse language samples, thereby improving the model's understanding and prediction capabilities of geographic information. After training, the model can accurately predict the geographic information of the masked results.

[0085] Optionally, the method further comprises:

[0086] The multi-task language sample set and the geographic information prediction result of the mask result are input into the geographic language model to perform iterative optimization training on the geographic language model.

[0087] In iterative optimization training, the geographic language model optimizes itself by constantly trying different parameter settings, model structures, and learning strategies. The model is trained and evaluated based on the input multi-task language sample set and geographic information prediction results, and then the model parameters and weights are updated based on the evaluation results. The iterative process continues for multiple cycles until the model reaches the preset performance indicators or converges to a stable state.

[0088] Through iterative optimization training, the geographic language model of the embodiment of the present invention will continuously improve its geographic information understanding and prediction capabilities, and enhance its sensitivity and accuracy to geographic context.

[0089] Optionally, step S101 of acquiring text information and spatial location information includes:

[0090] In response to the pre-training request, parameter information corresponding to the text information and the spatial position information is parsed from the pre-training request, and the text information and the spatial position information are acquired according to the parameter information.

[0091] When the system receives a pre-training request, it will first parse the pre-training request to extract parameter information related to text information and spatial location information. For example, text parameter information includes text keywords to be processed (such as address, place name, area description, etc.), data type (such as address description, geographic location description, commercial district information, etc.), language type (Chinese, English or other languages); spatial location parameter information includes geographic coordinates (latitude and longitude, administrative division code, etc.), area name, task type (geographic information prediction, spatial relationship analysis, address recognition, etc.), etc.

[0092] Based on the keywords, data types, and language types in the request, the system retrieves or generates corresponding text information from relevant data sources, for example, extracting information from databases, network resources, API interfaces, or other data sources, or generating descriptive text through text generation models. Based on the geographic coordinates or region name information in the request, the geographic information database is queried or the map service API is called to obtain the corresponding spatial location information, which may include map coordinates, administrative divisions, place names, surrounding facilities, etc.

[0093] Figure 4 It is a structural diagram of a geographic language model according to an embodiment of the present invention. In logistics and distribution scenarios, large-scale natural language models have been widely used in downstream tasks such as address text entity recognition and geocoding in logistics scenarios. Model pre-training is of great significance for improving the performance of downstream subtasks. The classic masked language model pre-training task is used for address text, and the spatial position is grid discretized to generate spatial position encoding, and then the spatial position pre-training task is performed, which is conducive to improving the model's recognition ability for address text, and more accurately mining the semantic information and spatial position information therein, thereby optimizing logistics distribution path planning and other functions. In addition, this method can also be extended to other fields, such as intelligent customer service, intelligent recommendation, etc., to improve the performance of related systems.

[0094] For example, geographic language model pre-training is performed for the following training tasks: mask language model task, hierarchical mask pre-training task, geographic prediction AOI task or AOI spatial relationship prediction task.

[0095] (1) Masked language model training

[0096] In the logistics and transportation scenario, the delivery address is encoded at the word granularity, and the randomly masked part is encoded as "[MASK]" or replaced with other tokens. The masked part does not exceed 15%, and the replaced part does not exceed 10%. The training model predicts the masked part, as shown in Table 1 below.

[0097]

[0098] (2) Address-level mask pre-training task

[0099] Different from daily conversation text, address text has a highly hierarchical structured feature. The classic MLM task cannot meet the pre-training learning of address text, so the Hierarchy Masked Language Modeling (HMLM) pre-training task is used. For text addresses, the named entity recognition (NER) task can be used to identify different components in the text and split them into different components: administrative districts (first level, second level, third level, fourth level), roads, Where, and What. Where refers to a larger limited area, such as a community, a commercial district, an industrial area, a certain road, etc.; What refers to a specific POI within the limited range of Where, such as the building number of the XX community, a commercial street basement store, or a chain store. In the hierarchical mask pre-training task, the administrative district, road, and Where information are randomly masked, and the token is modified to "[MASK]". The training model predicts the masked part, so that the model has the ability to predict the address hierarchical information through contextual information. The What information is not masked because What often refers to lower-level POI information, such as Cainiao Station, 7 Days Inn, McDonald's and other chain stores. Masking will lead to model overfitting, as described in Table 2.

[0100]

[0101] (3) AOI pre-training task

[0102] AOI summary generation: For POI data within the AOI fence range, the POI and Road information are identified through NER technology, and the AOI summary can be generated through entity word frequency statistics and word frequency sorting.

[0103] The Address AOI Prediction (AAP) task counts the length of POIs and finds that 99% of POIs are within 10 characters, so the cutoff for POIs is 10. The top 100 entities ranked by frequency (F) are used as the final AOI summary. The AOI summary is encoded as: "[POI1][F1][POI2][F2]…[POI100][F100]", combined with the administrative district code and other AOI inherent attributes (such as area, perimeter, category: food, educational institution, industrial area, etc.) to get the AOI encoding result "[Province][City][County][Town][Area][Perimeter][Catogory][POI1][F1][POI2][F2]…[POI100][F100]", the insufficient length part is completed by token [PAD], the length is 1107-dimensional vector T, the [1107, 1107]-dimensional learning matrix W1 combined with Relu activation function and [1107, 256]-dimensional conversion matrix W2 is used to convert the AOI encoding result into a 256-dimensional feature vector V, expressed as "[AOI encoding]", as shown below:

[0104]

[0105] Randomly select some delivery addresses with delivery coordinates, encode the addresses with word granularity, set the address truncation length to 50, and the encoding result is "[word1][word2]…[word50]", expressed as "[Address encoding]", and use "[PAD]" to complete the token for the insufficient length. Concatenate the address encoding result and the AOI encoding result to get "[Address encoding][SEP][AOI encoding]", where "[SEP]" represents the separator token. There are three relationships between AOI and delivery address: 0: AOI is consistent; 1: adjacent AOI; 2: unrelated. The final encoding feature vector of AOI can be obtained by pre-training the model for relationship prediction tasks.

[0106] AOI Spatial Relationship Prediction (ASRP). There are multiple spatial relationships between two AOIs. The first is that there are two spatial relationships between AOIs: adjacent AOIs and non-adjacent AOIs. Combining the AOI feature vectors obtained in the above steps, the two AOIs are encoded as "[AOI1][SEP][AOI2]". The short-range AOI relationship closeness is obtained through the spatial relationship prediction task. For the encoded result [AOI1][SEP][AOI2], closeness training is performed by predicting whether AOI1 and AOI2 are adjacent. The adjacent AOIs have a higher closeness. The second is whether the AOIs exist in the same third-level or fourth-level administrative region. Taking the undirected graph represented as the same administrative district, two AOIs are randomly selected for encoding "[AOI1][SEP][AOI2]". Through pre-training, the closeness of AOI relationships in different administrative districts can be obtained. The third method is to use the courier delivery trajectory information in the logistics transportation scenario, map the point-to-point trajectory information to the AOI-to-AOI dimension, and obtain the courier delivery AOI trajectory graph, which is represented as a spatial directed graph model. A series of trajectory information is encoded into AOI trajectory to obtain "[AOI1][AOI2]...[AOIn]". Through mask pre-training, the closeness of long-range AOI relationships can be obtained.

[0107] According to a second aspect of the embodiment of the present invention, Figure 5 As shown, a pre-training device 500 for a geographic language model is provided, comprising:

[0108] An acquisition module 501 is used to acquire text information and spatial position information, and determine a feature vector set based on the text information and spatial position information;

[0109] A masking module 502 is used to perform masking processing on some feature vectors in the feature vector set to obtain a masking result;

[0110] The training module 503 is used to construct a multi-task language sample set based on the feature vector set and the mask result, and pre-train the geographic language model according to the multi-task language sample set.

[0111] Optionally, the acquisition module 501 is further used for:

[0112] Encoding the text information and the spatial position information to obtain an encoding result;

[0113] A feature vector set is determined based on the encoding result.

[0114] Optionally, the mask module 502 is further configured to:

[0115] Mask marks are added to some feature vectors randomly extracted from the feature vector set or extracted according to a preset rule to obtain mask results.

[0116] Optionally, the training module 503 is further used for:

[0117] Based on a predetermined task type, a sample set corresponding to each task type is constructed using the feature vector set and the mask result; wherein the task type includes a mask language model task, a hierarchical mask pre-training task, a geographic prediction AOI task, or an AOI spatial relationship prediction task;

[0118] A multi-task language sample set is determined according to a sample set corresponding to each task type.

[0119] Optionally, the training module 503 is further used for:

[0120] The multi-task language sample set is input into a geographic language model for unsupervised training to obtain a geographic information prediction result of the mask result.

[0121] Optionally, the geographic language model pre-training device 500 further includes an optimization module, and the optimization module is used to:

[0122] The multi-task language sample set and the geographic information prediction result of the mask result are input into the geographic language model to perform iterative optimization training on the geographic language model.

[0123] Optionally, the acquisition module 501 is further used for:

[0124] In response to the pre-training request, parameter information corresponding to the text information and the spatial position information is parsed from the pre-training request, and the text information and the spatial position information are acquired according to the parameter information.

[0125] It should be noted that the specific implementation content of the pre-training device for the geographic language model of the present invention has been described in detail in the pre-training method for the geographic language model described above, so the repeated content will not be described here.

[0126] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided by the first aspect of the embodiment of the present invention.

[0127] According to a fourth aspect of an embodiment of the present invention, there is provided a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method provided by the first aspect of the embodiment of the present invention is implemented.

[0128] Figure 6 An exemplary system architecture 600 is shown to which the geographic language model pre-training method or geographic language model pre-training apparatus according to an embodiment of the present invention can be applied.

[0129] like Figure 6 As shown, system architecture 600 may include terminal devices 601, 602, 603, network 604 and server 605. Network 604 is used to provide a medium for communication links between terminal devices 601, 602, 603 and server 605. Network 604 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0130] The user can use the terminal devices 601, 602, 603 to interact with the server 605 through the network 604 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 601, 602, 603, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0131] The terminal devices 601 , 602 , and 603 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.

[0132] The server 605 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 601, 602, and 603. The backend management server may analyze and process the received data such as the model training request, and feed back the processing results (such as the target geographic language model - only an example) to the terminal device.

[0133] It should be noted that the geographic language model pre-training method provided in the embodiment of the present invention is generally run by the server 605 , and accordingly, the geographic language model pre-training device is generally set in the server 605 .

[0134] It should be understood that Figure 6 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0135] Reference below Figure 7 , which shows a schematic diagram of the structure of a computer system 700 of a terminal device suitable for implementing an embodiment of the present invention. Figure 7 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0136] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can execute various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0137] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 runs a communication process via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0138] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for running the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present invention are executed.

[0139] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0140] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually run basically in parallel, and they can sometimes run in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that runs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0141] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be arranged in a processor, for example, they may be described as: a processor including a training module, a mask module and an acquisition module. The names of these modules do not constitute a limitation on the modules themselves in certain cases, for example, the acquisition module may also be described as "a module for acquiring text information and spatial position information, and determining a feature vector set based on the text information and spatial position information".

[0142] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: obtaining text information and spatial position information, determining a feature vector set based on the text information and spatial position information; performing masking on some feature vectors in the feature vector set to obtain a mask result; constructing a multi-task language sample set based on the feature vector set and the mask result, and pre-training a geographic language model based on the multi-task language sample set.

[0143] The technical solution according to the embodiment of the present invention has the following advantages or beneficial effects: by acquiring text information and spatial position information, a feature vector set is determined based on the text information and spatial position information; masking is performed on some feature vectors in the feature vector set to obtain a mask result; a multi-task language sample set is constructed based on the feature vector set and the mask result, and a geographic language model is pre-trained according to the multi-task language sample set; the richness of pre-training data can be improved to meet multi-task training requirements, and the accuracy and stability of prediction results are improved.

[0144] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0145] It should be noted that the acquisition, storage and application of user personal information involved in the technical solution of the present disclosure are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

Claims

1. A pre-training method for a geographic language model, characterized in that: include: Acquire text information and spatial position information, and determine a feature vector set based on the text information and spatial position information; Performing masking processing on some feature vectors in the feature vector set to obtain a masking result; A multi-task language sample set is constructed based on the feature vector set and the mask result, and a geographic language model is pre-trained according to the multi-task language sample set.

2. The method according to claim 1, characterized in that Determining a feature vector set based on the text information and the spatial position information includes: Encoding the text information and the spatial position information to obtain an encoding result; A feature vector set is determined based on the encoding result.

3. The method according to claim 2, characterized in that Masking some of the feature vectors in the feature vector set includes: Mask marks are added to some feature vectors randomly extracted from the feature vector set or extracted according to a preset rule to obtain mask results.

4. The method according to claim 1, characterized in that: Constructing a multi-task language sample set based on the feature vector set and the mask result, including: Based on a predetermined task type, a sample set corresponding to each task type is constructed using the feature vector set and the mask result; wherein the task type includes a mask language model task, a hierarchical mask pre-training task, a geographic prediction AOI task, or an AOI spatial relationship prediction task; A multi-task language sample set is determined according to a sample set corresponding to each task type.

5. The method according to claim 1, characterized in that Pre-training a geographic language model according to the multi-task language sample set includes: The multi-task language sample set is input into a geographic language model for unsupervised training to obtain a geographic information prediction result of the mask result.

6. The method according to claim 5, characterized in that The method further comprises: The multi-task language sample set and the geographic information prediction result of the mask result are input into the geographic language model to perform iterative optimization training on the geographic language model.

7. The method according to claim 1, characterized in that Get text information and spatial location information, including: In response to the pre-training request, parameter information corresponding to the text information and the spatial position information is parsed from the pre-training request, and the text information and the spatial position information are acquired according to the parameter information.

8. A pre-training device for a geographic language model, characterized in that: include: An acquisition module, used to acquire text information and spatial position information, and determine a feature vector set based on the text information and spatial position information; A masking module, used for performing masking processing on some feature vectors in the feature vector set to obtain a masking result; A training module is used to construct a multi-task language sample set based on the feature vector set and the mask result, and pre-train a geographic language model according to the multi-task language sample set.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images