A high-dimensional data retrieval method, device and medium for digital earth

Abstract

The present application provides a high-dimensional data retrieval method, device and medium for digital earth, belonging to the field of digital earth technology. The method determines dynamic model data based on a pre-built digital earth model and preset data partitioning attributes; inputs the dynamic model data into a pre-trained data conversion model to convert it into a number of first high-dimensional bit vector data sets; performs dimensionality reduction processing on each first high-dimensional bit vector data set according to a preset dimensionality reduction algorithm to obtain a corresponding low-dimensional data set; after receiving the data to be retrieved from the user terminal, performs dimensionality reduction processing on the data to be retrieved to determine one or more query tags corresponding to the data to be retrieved after dimensionality reduction processing; based on the matching operation of the query tag and each low-dimensional data set, determines one or more matching low-dimensional data sets, and determines the target original tag corresponding to the query tag according to the original tag associated with the matching low-dimensional data set, and sends the target original tag to the user terminal.

Description

Technical Field

[0001] The present application relates to the field of digital earth technology, and in particular to a high-dimensional data retrieval method, device and medium for digital earth. Background Art

[0002] In the field of digital earth, it is usually involved in modeling and simulating complex systems in the physical world, and generating a large amount of high-dimensional data. High-dimensional data sets usually require more storage space in digital earth, and more complex algorithms and higher-complexity calculations are required when they are processed; high-dimensional data sets are difficult to display and understand intuitively in high-dimensional space, and are difficult to visualize as images and graphics; and due to the increase in dimensionality, the data sparsity decreases, making it particularly difficult to extract and retrieve data from high-dimensional data sets.

[0003] Today, data sets are growing exponentially, but Internet bandwidth and storage device resources are still limited. In order to use limited resources more efficiently, data needs to be compressed to reduce transmission and storage costs. At this time, how to retrieve efficiently and quickly becomes particularly important. At present, traditional similarity search methods need to compare the contents of the data set one by one during the query time. When applied to such large-scale data sets, this undoubtedly brings high computing costs and requires a huge amount of storage space, which will also affect the user's query experience.

[0004] Based on this, there is an urgent need for a technical solution that can utilize effective resources and efficiently realize high-dimensional data retrieval. Summary of the invention

[0005] The embodiments of the present application provide a high-dimensional data retrieval method, device and medium for digital earth, which are used to solve the technical problems that the current high-dimensional data retrieval occupies too much resources, has high hardware costs, and has low retrieval efficiency, which affects the user experience.

[0006] On the one hand, an embodiment of the present application provides a high-dimensional data retrieval method for digital earth, the method comprising:

[0007] Determine the corresponding dynamic model data based on the pre-built digital earth model and the preset data partitioning attributes; wherein the preset data partitioning attributes are used to divide the dynamic part model data and the static part model data in the digital earth model;

[0008] Inputting the dynamic model data into a pre-trained data conversion model to convert the dynamic model data into a plurality of first high-dimensional bit vector data sets; wherein each of the first high-dimensional bit vector data sets corresponds one-to-one to an original label in an original label set; and each of the first high-dimensional bit vector data sets has a dependency relationship with a digital earth spatial ID of the digital earth model;

[0009] According to a preset dimensionality reduction algorithm, dimensionality reduction processing is performed on each of the first high-dimensional bit vector data sets to obtain low-dimensional data sets corresponding to each of the first high-dimensional bit vector data sets;

[0010] After receiving the data to be retrieved from the user terminal, performing the dimensionality reduction process on the data to be retrieved to determine one or more query tags corresponding to the data to be retrieved after the dimensionality reduction process;

[0011] Based on the matching operation between the query tag and each of the low-dimensional data sets, one or more matching low-dimensional data sets are determined, so as to determine the target original tag corresponding to the query tag according to the original tag associated with the matching low-dimensional data sets, and send the target original tag to the user terminal.

[0012] In one implementation of the present application, before determining the corresponding dynamic model data based on the pre-built digital earth model and the preset data division attributes, the method further includes:

[0013] The digital earth model is constructed based on a second high-dimensional bit vector data set and the original label set stored in the cloud; wherein the second high-dimensional bit vector data set is composed of multiple different byte partitions; the byte partition is one of the following: destination address, data byte or application part; the original label of the original label set is composed of multiple bytes of marked location information.

[0014] In one implementation of the present application, before inputting the dynamic model data into a pre-trained data conversion model to convert the dynamic model data into a plurality of first high-dimensional bit vector data sets, the method further includes:

[0015] Acquire a number of dynamic model data samples corresponding to the digital earth model and a number of corresponding historical high-dimensional bit vector data sets stored in the cloud;

[0016] By executing a selected model conversion algorithm in a specific conversion environment, each of the dynamic model data samples is converted to obtain a third high-dimensional bit vector data set; wherein the selected model conversion algorithm is a regression fitting algorithm or a generative model fitting algorithm;

[0017] The third high-dimensional bit vector dataset is compared with the historical high-dimensional bit vector dataset, and if the comparison result meets the preset conditions, the selected model conversion algorithm executed in the specific conversion environment is used as the pre-trained data conversion model; otherwise, the weight parameters corresponding to the selected model conversion algorithm are updated to re-convert each of the dynamic model data samples.

[0018] In one implementation of the present application, after inputting the dynamic model data into a pre-trained data conversion model to convert the dynamic model data into a plurality of first high-dimensional bit vector data sets, the method further includes:

[0019] Preprocessing each of the first high-dimensional bit vector data sets; the preprocessing includes removing outliers and supplementing missing values; the supplementing of missing values ​​is performed based on one of a selection interpolation method, a K-nearest neighbor interpolation method, a regression model interpolation method, or a mean interpolation method;

[0020] Taking the preprocessed first high-dimensional bit vector data sets as samples and the first high-dimensional bit vector data as features, the first high-dimensional bit vector data sets are sorted into a matrix form.

[0021] In one implementation of the present application, according to a preset dimensionality reduction algorithm, dimensionality reduction processing is performed on each of the first high-dimensional bit vector data sets to obtain low-dimensional data sets corresponding to each of the first high-dimensional bit vector data sets, specifically including:

[0022] After performing dimensionality reduction processing according to the preset dimensionality reduction algorithm, calculating the first probability distributions corresponding to the first high-dimensional bit vector data sets and the second probability distributions corresponding to the corresponding pending low-dimensional data sets; wherein the preset dimensionality reduction algorithm is a t-distribution-random neighbor embedding algorithm;

[0023] Calculate a corresponding KL divergence value according to each of the first probability distributions and each of the second probability distributions;

[0024] When the KL divergence value is greater than a preset threshold, performing iterative dimensionality reduction processing on each of the first high-dimensional bit vector data sets by using the preset dimensionality reduction algorithm to obtain each pending low-dimensional data set after the iterative dimensionality reduction processing, until the KL divergence value is less than or equal to the preset threshold;

[0025] When the KL divergence value is less than or equal to the preset threshold, the corresponding pending low-dimensional data sets are used as the low-dimensional data sets.

[0026] In one implementation of the present application, the dimensionality reduction process is performed on the data to be retrieved to determine one or more query tags corresponding to the data to be retrieved after the dimensionality reduction process, specifically including:

[0027] According to the preset dimensionality reduction algorithm, the data to be retrieved is subjected to the dimensionality reduction processing to obtain one or more dimensionality reduction features corresponding to the data to be retrieved;

[0028] In the case where the data to be retrieved has one dimension reduction feature, using the dimension reduction feature as the query label;

[0029] In the case that there are multiple dimensionality reduction features in the data to be retrieved, the feature priorities corresponding to each dimensionality reduction feature are determined according to a preset feature priority comparison list, so that a predetermined number of the dimensionality reduction features are used as the query labels in descending order of the feature priorities.

[0030] In one implementation of the present application, based on the matching operation between the query tag and each of the low-dimensional data sets, determining one or more matching low-dimensional data sets specifically includes:

[0031] Encoding the query tag into a first vector, and encoding each of the low-dimensional data sets into a corresponding number of second vectors;

[0032] matching the first vector with each of the second vectors to determine a plurality of matching degrees corresponding to the query tag;

[0033] The matching low-dimensional data set is determined based on the comparison result between each matching degree and a preset matching degree threshold or a matching degree selection rule; wherein the matching degree selection rule is to select one or more matching degrees in descending order of the matching degrees.

[0034] In one implementation of the present application, before determining the matching low-dimensional data set based on the comparison result between each of the matching degrees and a preset matching degree threshold or a matching degree selection rule, the method further includes:

[0035] Determine a preset data type corresponding to the data to be retrieved; wherein there is a dependency relationship between the preset data type and each matching condition; the matching condition includes a comparison result of each matching degree with a preset matching degree threshold, and the matching degree selection rule; the dependency relationship is obtained based on each matching degree; when the preset data type is that the matching degree fluctuation amplitude corresponding to each matching degree is greater than a predetermined amplitude value and / or each matching degree is less than a predetermined value, the preset data type and the matching degree selection rule have the dependency relationship;

[0036] The corresponding matching condition is determined according to the preset data type.

[0037] On the other hand, an embodiment of the present application further provides a high-dimensional data retrieval device for digital earth, the device comprising:

[0038] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a high-dimensional data retrieval method for digital earth as described above.

[0039] On the other hand, an embodiment of the present application further provides a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions can execute a high-dimensional data retrieval method for digital earth as described above.

[0040] Compared with the prior art, the present invention has the following significant effects:

[0041] 1) Through the above scheme, this application realizes a technical solution for high-dimensional data retrieval for digital earth, reduces the dimensionality of high-dimensional data, and relies on the low-dimensional data after dimensionality reduction for retrieval, which effectively reduces the system storage space occupied, improves system performance and retrieval efficiency, and also reduces hardware costs and maintenance difficulties to a certain extent, providing better storage management and retrieval performance under limited resources.

[0042] 2) This application can intelligently and flexibly adjust the search strategy according to the data to be searched, and send the target original label to the user terminal, providing users with intuitive and easy-to-understand search results. Users can further explore or take other actions based on the search results, thereby improving the usability and user satisfaction of the entire system. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0044] Figure 1 A schematic diagram of a flow chart of a high-dimensional data retrieval method for digital earth in an embodiment of the present application;

[0045] Figure 2 Another schematic diagram of a high-dimensional data retrieval method for digital earth in an embodiment of the present application;

[0046] Figure 3 A schematic diagram of a high-dimensional bit vector data set in a high-dimensional data retrieval method for digital earth in an embodiment of the present application;

[0047] Figure 4 A schematic diagram of a data conversion model training process in a high-dimensional data retrieval method for digital earth in an embodiment of the present application;

[0048] Figure 5 A schematic diagram of a dimensionality reduction process in a high-dimensional data retrieval method for digital earth in an embodiment of the present application;

[0049] Figure 6 A schematic diagram of a process of selecting a target original label in a high-dimensional data retrieval method for digital earth in an embodiment of the present application;

[0050] Figure 7 It is a structural schematic diagram of a high-dimensional data retrieval device for digital earth in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0052] The embodiments of the present application provide a high-dimensional data retrieval method, device and medium for digital earth, which are used to solve the technical problems that the current high-dimensional data retrieval occupies too much resources, has high hardware costs, and has low retrieval efficiency, which affects the user experience.

[0053] The following describes in detail various embodiments of the present application in conjunction with the accompanying drawings.

[0054] The present application embodiment provides a high-dimensional data retrieval method for digital earth, such as Figure 1 As shown, the method may include steps S101-S105:

[0055] S101, the server determines corresponding dynamic model data based on a pre-built digital earth model and preset data division attributes.

[0056] The preset data partitioning attribute is used to divide the dynamic part model data and the static part model data in the digital earth model.

[0057] It should be noted that the server, as the executor of the high-dimensional data retrieval method for digital earth, is only an example, and the executor is not limited to the server, and this application does not make any specific limitation on this.

[0058] In the embodiment of the present application, before determining the corresponding dynamic model data based on the pre-built digital earth model and the preset data division attributes, the method further includes:

[0059] The digital earth model is constructed based on the second high-dimensional bit vector data set and the original label set stored in the cloud. The second high-dimensional bit vector data set is composed of multiple different byte partitions. The byte partition is one of the following: destination address, data byte or application part. The original label of the original label set is composed of multiple bytes of marked location information.

[0060] The pre-built digital earth model is composed of several second-highest-dimensional bit vector data sets. After the digital earth model is built, the digital earth model has a unique digital earth space identifier (Identity Document, ID). The digital earth space ID is designed based on the digital surrogate encoding rules and is used to identify the unique code of the user or entity. The digital earth space ID is usually a combination of numbers, letters or symbols. The digital surrogate encoding rules organize and integrate the user's identity information with information such as country, region, industry, and user identification. Through a hierarchical encoding structure, it ensures that users have unique identification in different regions, platforms, and systems, thereby achieving cross-domain connections. The Digital Surrogate Encoding Rules (DSER) rules have a hierarchical encoding structure to achieve connections across regions, platforms, systems, and carriers.

[0061] In other words, this application can pre-process the data source, that is, build a preliminary digital earth model based on all high-dimensional bit vector data sets and corresponding labels stored in the cloud. Figure 2 Take the application of digital earth model in the construction and use of digital twin model as an example. Figure 2 In S201, a preliminary digital twin model is built based on all high-dimensional bit vector data sets and corresponding labels stored in the cloud. The composition of the high-dimensional bit vector data set corresponding to the digital earth model of this application is as follows: Figure 3 As shown, it includes the destination address: 6 bytes for longitude and latitude data to determine the physical address of the sender; data bytes: 46~1500 bytes of payload, that is, other indicators of the physical entity to be transmitted; application part: 10 bytes, the application part is to meet the needs of higher-dimensional network space interaction; and position code, which is the original label. The original label set consists of original labels, which are composed of multiple bytes of marked location information. For example, the first 6 bytes store the country number information: the sender's international location is determined using the International Organization for Standardization (ISO) country code; the last 8 bytes store detailed information of the province, prefecture-level city: the specific street information of the specific sender is determined using the postal code method. The above digital earth model is constructed based on the above content. The cloud can be connected to the server, and the cloud stores a large number of high-dimensional bit vector data sets.

[0062] Next, the digital earth model is divided into a dynamic part and a static part using the preset data partitioning attributes. Specifically, the static part model data is the immutable fixed part of the digital earth model, including at least the basic framework and physical constraints of the digital earth model; the dynamic part model data is the variable operation part of the digital earth model, including at least training data, hyperparameters, and tuning strategies.

[0063] S102, the server inputs the dynamic model data into a pre-trained data conversion model to convert the dynamic model data into a plurality of first high-dimensional bit vector data sets.

[0064] Each of the first high-dimensional bit vector data sets corresponds to an original label in the original label set one by one, and each of the first high-dimensional bit vector data sets has a dependency relationship with the digital earth space ID of the digital earth model.

[0065] Since the digital earth model has a digital earth space ID, each first high-dimensional bit vector data set obtained based on the digital earth model has a dependency relationship with the digital earth space ID, and each first high-dimensional bit vector data set anchors the digital earth space ID.

[0066] In other words, the application can divide the constructed digital earth model into a static part and a dynamic part, and then further convert the dynamic part model data back into a high-dimensional bit vector data set. Figure 2 In S201, the constructed preliminary digital twin model is divided into a static part and a dynamic part. Based on the historical data stored in the cloud, a specific conversion environment is used to convert the dynamic part of the model back to a high-dimensional bit vector data set.

[0067] In the embodiment of the present application, before inputting the dynamic model data into the pre-trained data conversion model to convert the dynamic model data into a plurality of first high-dimensional bit vector data sets, the method further includes:

[0068] Obtain several dynamic model data samples and several corresponding historical high-dimensional bit vector data sets corresponding to the digital earth model stored in the cloud. By executing the selected model conversion algorithm in a specific conversion environment, each dynamic model data sample is converted to obtain a third high-dimensional bit vector data set. The selected model conversion algorithm is a regression fitting algorithm or a generative model fitting algorithm. The third high-dimensional bit vector data set is compared with the historical high-dimensional bit vector data set. If the comparison result meets the preset conditions, the selected model conversion algorithm executed in the specific conversion environment is used as the pre-trained data conversion model. Otherwise, the weight parameters corresponding to the selected model conversion algorithm are updated to re-convert each dynamic model data sample.

[0069] In other words, this application will build a data conversion model to convert dynamic model data into a high-dimensional bit vector data set. The training process of the data conversion model is as follows Figure 4 As shown, the details are as follows:

[0070] S401, prepare a specific conversion environment: use a model conversion library or deep learning framework to adapt to the model conversion environment; S402, select a model conversion algorithm: regression fitting or generative model fitting can be used to approximate the high-dimensional bit vector data set; S403, collect and prepare data sets: the dynamic part of the model is converted back to a high-dimensional bit vector data set. First, it is necessary to collect the corresponding historical data stored in the cloud, that is, the historical high-dimensional bit vector data set, and then extract specific weight parameters and network structure parameters from the model to obtain specific information of the model for conversion; S404, perform model conversion: use the selected model conversion algorithm to convert the trained dynamic model data samples. Map the weight parameters in the model back to the high-dimensional bit vector or generate samples that approximate the high-dimensional bit vector data set according to the model generation capability; S405, verify and adjust: compare the historical high-dimensional bit vector data set stored in the cloud with the third high-dimensional bit vector data set obtained after conversion, evaluate the quality of the conversion and make necessary adjustments. Among them, the preset condition can be understood as that the similarity between the third high-dimensional bit vector data set and the historical high-dimensional bit vector data corresponding to the digital earth model before the construction is performed is greater than a similarity threshold. The similarity can be cosine similarity, etc. The similarity is used to evaluate the quality of the conversion. The similarity threshold can be set by the user according to the actual usage scenario. This application does not specifically limit the similarity calculation method and the similarity threshold.

[0071] The present application can complete the conversion of dynamic model data through a pre-trained data conversion model, thereby obtaining a plurality of first high-dimensional bit vector data sets corresponding to the dynamic model data.

[0072] Through the above scheme, the dynamic model data in the digital earth model can be flexibly converted back to the original high-dimensional bit vector data set. On the one hand, this conversion operation can ensure the accuracy of building the digital earth model. On the other hand, it lays the foundation for retrieving high-dimensional data in the digital earth model and efficiently and accurately converts the dynamic model data back to the high-dimensional bit vector data set.

[0073] In the embodiment of the present application, after the dynamic model data is input into the pre-trained data conversion model to convert the dynamic model data into a plurality of first high-dimensional bit vector data sets, the method further includes:

[0074] Each first high-dimensional bit vector data set is preprocessed. The preprocessing includes removing outliers and supplementing missing values. The supplementing of missing values ​​is performed based on one of the selection interpolation method, the K nearest neighbor interpolation method, the regression model interpolation method or the mean interpolation method. Using each preprocessed first high-dimensional bit vector data set as a sample and each first high-dimensional bit vector data as a feature, each first high-dimensional bit vector data set is organized into a matrix form.

[0075] In other words, the present application will further process each first high-dimensional bit vector data set obtained by the conversion to meet the needs of subsequent dimensionality reduction processing. Figure 2 In step S203, the high-dimensional bit vector data set obtained after conversion is further processed: missing values ​​and outliers are processed and the data is organized into a "sample-feature" form. The "sample-feature" form is a matrix form. The present application can delete samples or features containing missing values; for missing samples, interpolation, K-nearest neighbor interpolation, regression model interpolation or mean interpolation can be selected to fill in missing values ​​based on comprehensive considerations such as data type, data distribution, causes of missing values ​​and size of the data set; the data is organized into a "sample-feature" form, that is, each row of the data represents a sample, and each column represents a feature. This form facilitates the application of dimensionality reduction processing algorithms. At this point, the preparations required before dimensionality reduction processing of the high-dimensional bit vector data set are completed.

[0076] S103: The server performs dimensionality reduction processing on each first high-dimensional bit vector data set according to a preset dimensionality reduction algorithm to obtain low-dimensional data sets corresponding to each first high-dimensional bit vector data set.

[0077] This step corresponds to Figure 2 In S204, a dimensionality reduction algorithm is used to compress the sorted high-dimensional bit vector data set into a low-dimensional space, that is, the high-dimensional bit vector data set is subjected to labeled dimensionality reduction processing.

[0078] In the embodiment of the present application, according to a preset dimensionality reduction algorithm, dimensionality reduction processing is performed on each first high-dimensional bit vector data set to obtain a low-dimensional data set corresponding to each first high-dimensional bit vector data set, specifically including:

[0079] After performing dimensionality reduction processing according to a preset dimensionality reduction algorithm, the first probability distributions corresponding to each first high-dimensional bit vector data set and the second probability distributions corresponding to each corresponding pending low-dimensional data set are calculated. Among them, the preset dimensionality reduction algorithm is the t distribution-random neighbor embedding algorithm. According to each first probability distribution and each second probability distribution, the corresponding KL divergence value is calculated. When the KL divergence value is greater than the preset threshold, the first high-dimensional bit vector data set is iteratively reduced in dimension by the preset dimensionality reduction algorithm to obtain each pending low-dimensional data set after iterative dimensionality reduction processing, until the KL divergence value is less than or equal to the preset threshold. When the KL divergence value is less than or equal to the preset threshold, the corresponding pending low-dimensional data sets are used as each low-dimensional data set.

[0080] That is to say, this application uses the t-Distributed Stochastic Neighbor Embedding (t-SNE) algorithm as the preset dimensionality reduction algorithm to perform dimensionality reduction processing on each first high-dimensional bit vector data set to achieve labeled dimensionality reduction processing, such as Figure 5 As shown, the position of the data points in the low-dimensional space is adjusted by t-SNE so that the similarity distribution of each high-dimensional data point (such as X1, X2, X3...) and the low-dimensional space is as close as possible. When the data points corresponding to each first high-dimensional bit vector set are reduced to data points in the low-dimensional space, the application will continue to iterate and optimize to obtain the data points corresponding to the final low-dimensional data set.

[0081] Among them, the calculation formulas of each first probability distribution are as follows:

[0082]

[0083] in, For the A high-dimensional bit vector dataset and The first probability distribution between high-dimensional bit vector data sets, For the A high-dimensional bit vector dataset corresponds to a data point, For the A high-dimensional bit vector dataset corresponds to a data point, The value is based on the data point The Gaussian mean square error centered at is not equal to A natural number, in addition, set , in order to realize the first probability distribution focusing only on the first high-dimensional bit vector data sets between each pair.

[0084] The second probability distribution calculation formula is as follows:

[0085]

[0086] in, No. The pending low-dimensional dataset and The second probability distribution between the undetermined low-dimensional data sets, No. The data points corresponding to the undetermined low-dimensional data set are For the The corresponding data points of the pending low-dimensional data set, where for the corresponding data points of the pending low-dimensional data set, the mean square error of the Gaussian distribution specified in this application is Among them, if the dimensionality reduction effect is better, the local features will be kept intact. .

[0087] In order to minimize the difference between the probability distributions in high-dimensional and low-dimensional spaces, this application uses KL divergence to characterize the difference between the two distributions. The formula is as follows:

[0088]

[0089] in, is the KL divergence value. If the KL divergence value is greater than the preset threshold, it means that the difference between the two probability distributions is too large, and it is necessary to continue to iteratively optimize each first high-dimensional bit vector data set, thereby performing iterative dimensionality reduction processing, updating each pending low-dimensional data set, and recalculating the above second probability distribution and KL divergence value until the KL divergence value is less than or equal to the preset threshold, and a low-dimensional data set is obtained. The preset threshold can be set by the user during actual use, and this application does not make specific restrictions on this.

[0090] S104, after receiving the data to be retrieved from the user terminal, the server performs dimensionality reduction processing on the data to be retrieved to determine one or more query tags corresponding to the data to be retrieved after the dimensionality reduction processing.

[0091] In the embodiment of the present application, the data to be retrieved is subjected to dimensionality reduction processing to determine one or more query tags corresponding to the data to be retrieved after the dimensionality reduction processing, specifically including:

[0092] According to the preset dimensionality reduction algorithm, the data to be retrieved is subjected to dimensionality reduction processing to obtain one or more dimensionality reduction features corresponding to the data to be retrieved. In the case where there is one dimensionality reduction feature in the data to be retrieved, the dimensionality reduction feature is used as the query label. In the case where there are multiple dimensionality reduction features in the data to be retrieved, the feature priorities corresponding to each dimensionality reduction feature are determined according to the preset feature priority comparison list, so as to use a predetermined number of dimensionality reduction features as query labels in descending order of feature priority.

[0093] In addition, before searching, this application can also perform preprocessing such as data normalization and feature selection on the low-dimensional data set after dimensionality reduction to ensure the consistency and accuracy of the data.

[0094] After receiving the data to be retrieved sent by the user terminal, the present application first performs dimensionality reduction processing on the data to be retrieved according to the preset dimensionality reduction algorithm of the dimensionality reduction processing in the above step S103. The user terminal can be understood as a terminal used by a user to retrieve high-dimensional data, such as a mobile phone, a computer and other devices, and the present application does not make specific restrictions on this. After the dimensionality reduction processing of the data to be retrieved, the dimensionality reduction features of the data to be retrieved can be obtained. The present application can select different dimensionality reduction feature selection methods according to the number of dimensionality reduction features to obtain a query label.

[0095] Among them, when there are multiple dimensionality reduction features, for example, 10 dimensionality reduction features, this application can determine the feature priorities of each dimensionality reduction feature through a pre-set preset feature priority comparison list, and select a predetermined number of dimensionality reduction features with larger feature priorities as query labels, the predetermined number is 3, 4, for example, and the predetermined number is positively correlated with the number of multiple dimensionality reduction features, such as 10 dimensionality reduction features, the predetermined number is 3, the dimensionality reduction number is 20, and the predetermined number is 6. The specific predetermined number is set by the user according to the actual usage scenario, and this application does not make specific restrictions on this. Thereby maintaining the amount of information while reducing the computational complexity.

[0096] In addition, the present application can also exclude some original tags that are not related to the original tags according to the query tags, reduce the search domain to a certain extent, and improve the retrieval efficiency. The specific exclusion conditions can be set by the user according to the actual needs of the original tag keywords, and the query tags and keywords are matched to obtain the keyword similarity, so as to exclude the original tags according to the comparison results of the keyword similarity and the similarity threshold. If the keyword similarity is less than the similarity threshold, the corresponding original tags are excluded. The keyword similarity can be obtained by calculating the cosine similarity, the inverse of the Euclidean distance, etc.

[0097] S105, the server determines one or more matching low-dimensional data sets based on the matching operation between the query tag and each low-dimensional data set, determines the target original tag corresponding to the query tag according to the original tag associated with the matching low-dimensional data set, and sends the target original tag to the user terminal.

[0098] In an embodiment of the present application, based on the matching operation between the query tag and each low-dimensional data set, one or more matching low-dimensional data sets are determined, specifically including:

[0099] The query tag is encoded as a first vector, and each low-dimensional data set is encoded as a corresponding number of second vectors. The first vector is matched with each second vector to determine multiple matching degrees corresponding to the query tag. Based on the comparison result of each matching degree with a preset matching degree threshold or a matching degree selection rule, a matching low-dimensional data set is determined. Among them, the matching degree selection rule is to select one or more matching degrees in descending order of each matching degree.

[0100] That is, the present application can encode the query tag and each low-dimensional data set respectively to generate a first vector and multiple second vectors, and the server further calculates the matching degree between the first vector and each second vector. The matching degree can be obtained by calculating the cosine similarity, such as the matching degree S between the first vector and the mth second vector. m =(A·B m ) / (||A||||B m ||), A is the first vector, B m is the mth second vector. In addition, the matching degree can also be obtained by calculating the Jaccard similarity coefficient, the inverse of the Hamming distance, etc., which is not specifically limited in this application.

[0101] After obtaining each matching degree, each matching degree can be compared with a preset matching degree threshold, or partial matching degrees can be selected according to matching degree selection rules, and the low-dimensional data set of the second vector corresponding to the selected partial matching degrees can be used as the matching low-dimensional data set.

[0102] In the embodiment of the present application, before determining the matching low-dimensional data set based on the comparison result of each matching degree with the preset matching degree threshold or the matching degree selection rule, the method further includes:

[0103] Determine the preset data type corresponding to the data to be retrieved. There is a dependency relationship between the preset data type and each matching condition. The matching condition includes the comparison result of each matching degree with the preset matching degree threshold and the matching degree selection rule. The dependency relationship is obtained based on each matching degree. When the preset data type is that the matching degree fluctuation amplitude corresponding to each matching degree is greater than the predetermined amplitude value and / or each matching degree is less than the predetermined value, there is a dependency relationship between the preset data type and the matching degree selection rule. Determine the corresponding matching condition according to the preset data type.

[0104] In other words, the present application can determine the dependency relationship according to the size change of each matching degree, that is, the matching degree fluctuation amplitude and the size relationship of each matching degree. The matching degree fluctuation amplitude can be understood as the curve value standard deviation of the fluctuation curve formed by the difference between each matching degree. The curve value standard deviation, that is, the matching degree fluctuation amplitude is greater than the predetermined amplitude value and / or all matching degrees are less than the predetermined value. At this time, the preset data type is judged as a preset data type with a dependency relationship with the matching degree selection rule, otherwise the matching condition is set to compare each matching degree with the preset matching degree threshold. The matching degree fluctuation amplitude is greater than the predetermined amplitude value, which means that there is a significant inconsistency or anomaly between the data. All matching degrees are less than the predetermined value, indicating that the data quality is not high or there is some anomaly. Both of these situations may be data anomalies or special cases. In these two cases, the matching low-dimensional data set is determined by the matching degree selection rule, that is, one or more larger matching degrees are selected, and the matching low-dimensional data set is determined according to the selected matching degrees, which can avoid setting an unreasonable preset matching degree threshold to obtain a poor matching low-dimensional data set or unable to determine the matching low-dimensional data set.

[0105] Through the above solution, data anomalies can be better identified, retrieval strategies can be optimized, and retrieval efficiency can be improved.

[0106] After obtaining the matching low-dimensional data set, you can associate the labels. Through the association operation, you can find the original labels associated with the matching low-dimensional data set. You can associate the labels based on the correspondence between the reduced-dimensional data and the original data, or use the neighbor relationship around the reduced-dimensional data. This application does not make specific restrictions on this. In addition, this application can also further filter and sort the associated labels. You can sort the labels based on the similarity, confidence or other relevance indicators between the labels and the query labels, and select the most appropriate label as the target original label to return. The above steps correspond to Figure 2 In step S205, based on the label data set after the dimension reduction process, a many-to-one label search is performed on the data to be searched to narrow the scope, and the most appropriate label is selected and returned as the result.

[0107] like Figure 6As shown, step S205 may specifically include: S601, preprocessing the data set. Perform data normalization, feature selection and other preprocessing on the reduced-dimensionality data set to ensure data consistency and accuracy; S602, process query tags. When there is new retrieval data, perform the same dimensionality reduction processing on it to obtain corresponding dimensionality reduction features, and select the most characteristic 3-4 dimensionality reduction features as query tags; S603, narrow the search range domain. According to the query tag value, exclude some unrelated original tag sets to reduce the search domain to a certain extent; S604, search matching. Perform search matching operations in the narrowed search domain to find the most similar dimensionality reduction data to the query tag. If the matching threshold fluctuates greatly and the matching degree is generally low, several groups of data with the highest relevant matching degree can be selected; S605, associate tags. According to the retrieval matching reduced-dimensionality data, find multiple original tags associated with it through association operations. Tags can be associated based on the corresponding relationship between the reduced-dimensionality data and the original data, or by using the neighbor relationship around the reduced-dimensionality data. S606, filter tags. According to task requirements and specific rules, the associated tags are further filtered and sorted. Tags can be sorted based on their similarity, confidence, or other relevance metrics with the query tag, and the most appropriate tag can be selected and returned as the result.

[0108] Through the above scheme, this application realizes a technical solution for high-dimensional data retrieval for digital earth, reduces the dimensionality of high-dimensional data, and relies on the low-dimensional data after dimensionality reduction for retrieval, which effectively reduces the system storage space occupied, improves system performance and retrieval efficiency, and also reduces hardware costs and maintenance difficulties to a certain extent, providing better storage management and retrieval performance under limited resources.

[0109] In addition, the application can intelligently and flexibly adjust the search strategy according to the data to be searched, and send the target original label to the user terminal, providing users with intuitive and easy-to-understand search results. Users can further explore or take other actions based on the search results, thereby improving the usability and user satisfaction of the entire system.

[0110] Figure 7 A schematic diagram of the structure of a high-dimensional data retrieval device for digital earth provided in an embodiment of the present application, such as Figure 7 As shown, the device includes:

[0111] At least one processor. And a memory in communication with the at least one processor. The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to:

[0112] Based on the pre-built digital earth model and the preset data partitioning attributes, the corresponding dynamic model data is determined. The preset data partitioning attributes are used to divide the dynamic part model data and the static part model data in the digital earth model. The dynamic model data is input into the pre-trained data conversion model to convert the dynamic model data into a number of first high-dimensional bit vector data sets. Each first high-dimensional bit vector data set corresponds to an original label in the original label set. Each first high-dimensional bit vector data set has a dependency relationship with the digital earth space ID of the digital earth model. According to the preset dimensionality reduction algorithm, each first high-dimensional bit vector data set is subjected to dimensionality reduction processing to obtain a low-dimensional data set corresponding to each first high-dimensional bit vector data set. After receiving the data to be retrieved from the user terminal, the data to be retrieved is subjected to dimensionality reduction processing to determine one or more query tags corresponding to the data to be retrieved after the dimensionality reduction processing. Based on the matching operation between the query tag and each low-dimensional data set, one or more matching low-dimensional data sets are determined to determine the target original tag corresponding to the query tag according to the original tag associated with the matching low-dimensional data set, and the target original tag is sent to the user terminal.

[0113] The embodiment of the present application further provides a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured as follows:

[0114] Based on the pre-built digital earth model and the preset data partitioning attributes, the corresponding dynamic model data is determined. The preset data partitioning attributes are used to divide the dynamic part model data and the static part model data in the digital earth model. The dynamic model data is input into the pre-trained data conversion model to convert the dynamic model data into a number of first high-dimensional bit vector data sets. Each first high-dimensional bit vector data set corresponds to an original label in the original label set. Each first high-dimensional bit vector data set has a dependency relationship with the digital earth space ID of the digital earth model. According to the preset dimensionality reduction algorithm, each first high-dimensional bit vector data set is subjected to dimensionality reduction processing to obtain a low-dimensional data set corresponding to each first high-dimensional bit vector data set. After receiving the data to be retrieved from the user terminal, the data to be retrieved is subjected to dimensionality reduction processing to determine one or more query tags corresponding to the data to be retrieved after the dimensionality reduction processing. Based on the matching operation between the query tag and each low-dimensional data set, one or more matching low-dimensional data sets are determined to determine the target original tag corresponding to the query tag according to the original tag associated with the matching low-dimensional data set, and the target original tag is sent to the user terminal.

[0115] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0116] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0117] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0118] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A high-dimensional data retrieval method for digital earth, characterized in that: The method comprises: Determine the corresponding dynamic model data based on the pre-built digital earth model and the preset data partitioning attributes; wherein the preset data partitioning attributes are used to divide the dynamic part model data and the static part model data in the digital earth model; Inputting the dynamic model data into a pre-trained data conversion model to convert the dynamic model data into a plurality of first high-dimensional bit vector data sets; wherein each of the first high-dimensional bit vector data sets corresponds one-to-one to an original label in an original label set; and each of the first high-dimensional bit vector data sets has a dependency relationship with a digital earth spatial ID of the digital earth model; According to a preset dimensionality reduction algorithm, dimensionality reduction processing is performed on each of the first high-dimensional bit vector data sets to obtain low-dimensional data sets corresponding to each of the first high-dimensional bit vector data sets; After receiving the data to be retrieved from the user terminal, performing the dimensionality reduction process on the data to be retrieved to determine one or more query tags corresponding to the data to be retrieved after the dimensionality reduction process; Based on the matching operation between the query tag and each of the low-dimensional data sets, one or more matching low-dimensional data sets are determined, so as to determine a target original tag corresponding to the query tag according to the original tag associated with the matching low-dimensional data sets, and send the target original tag to the user terminal; Before inputting the dynamic model data into a pre-trained data conversion model to convert the dynamic model data into a plurality of first high-dimensional bit vector data sets, the method further includes: Acquire a number of dynamic model data samples corresponding to the digital earth model and a number of corresponding historical high-dimensional bit vector data sets stored in the cloud; By executing a selected model conversion algorithm in a specific conversion environment, each of the dynamic model data samples is converted to obtain a third high-dimensional bit vector data set; wherein the selected model conversion algorithm is a regression fitting algorithm or a generative model fitting algorithm; The third high-dimensional bit vector dataset is compared with the historical high-dimensional bit vector dataset, and if the comparison result meets the preset conditions, the selected model conversion algorithm executed in the specific conversion environment is used as the pre-trained data conversion model; otherwise, the weight parameters corresponding to the selected model conversion algorithm are updated to re-convert each of the dynamic model data samples.

2. A high-dimensional data retrieval method for digital earth according to claim 1, characterized in that: Before determining the corresponding dynamic model data based on the pre-built digital earth model and the preset data division attributes, the method further includes: The digital earth model is constructed based on a second high-dimensional bit vector data set and the original label set stored in the cloud; wherein the second high-dimensional bit vector data set is composed of multiple different byte partitions; the byte partition is one of the following: destination address, data byte or application part; the original label of the original label set is composed of multiple bytes of marked location information.

3. A high-dimensional data retrieval method for digital earth according to claim 1, characterized in that: After inputting the dynamic model data into a pre-trained data conversion model to convert the dynamic model data into a plurality of first high-dimensional bit vector data sets, the method further includes: Preprocessing each of the first high-dimensional bit vector data sets; the preprocessing includes removing outliers and supplementing missing values; the supplementing of missing values ​​is performed based on one of a selection interpolation method, a K-nearest neighbor interpolation method, a regression model interpolation method, or a mean interpolation method; Taking the preprocessed first high-dimensional bit vector data sets as samples and the first high-dimensional bit vector data as features, the first high-dimensional bit vector data sets are sorted into a matrix form.

4. A high-dimensional data retrieval method for digital earth according to claim 1, characterized in that: According to a preset dimensionality reduction algorithm, dimensionality reduction processing is performed on each of the first high-dimensional bit vector data sets to obtain low-dimensional data sets corresponding to each of the first high-dimensional bit vector data sets, specifically including: After performing dimensionality reduction processing according to the preset dimensionality reduction algorithm, calculating the first probability distributions corresponding to the first high-dimensional bit vector data sets and the second probability distributions corresponding to the corresponding pending low-dimensional data sets; wherein the preset dimensionality reduction algorithm is a t-distribution-random neighbor embedding algorithm; Calculate a corresponding KL divergence value according to each of the first probability distributions and each of the second probability distributions; When the KL divergence value is greater than a preset threshold, performing iterative dimensionality reduction processing on each of the first high-dimensional bit vector data sets by using the preset dimensionality reduction algorithm to obtain each pending low-dimensional data set after the iterative dimensionality reduction processing, until the KL divergence value is less than or equal to the preset threshold; When the KL divergence value is less than or equal to the preset threshold, the corresponding pending low-dimensional data sets are used as the low-dimensional data sets.

5. A high-dimensional data retrieval method for digital earth according to claim 1, characterized in that: The step of performing the dimensionality reduction process on the data to be retrieved to determine one or more query tags corresponding to the data to be retrieved after the dimensionality reduction process specifically includes: According to the preset dimensionality reduction algorithm, the data to be retrieved is subjected to the dimensionality reduction processing to obtain one or more dimensionality reduction features corresponding to the data to be retrieved; In the case where the data to be retrieved has one dimension reduction feature, using the dimension reduction feature as the query label; In the case that there are multiple dimensionality reduction features in the data to be retrieved, the feature priorities corresponding to each dimensionality reduction feature are determined according to a preset feature priority comparison list, so that a predetermined number of the dimensionality reduction features are used as the query labels in descending order of the feature priorities.

6. A high-dimensional data retrieval method for digital earth according to claim 1, characterized in that: Based on the matching operation between the query tag and each of the low-dimensional data sets, determining one or more matching low-dimensional data sets specifically includes: Encoding the query tag into a first vector, and encoding each of the low-dimensional data sets into a corresponding number of second vectors; matching the first vector with each of the second vectors to determine a plurality of matching degrees corresponding to the query tag; The matching low-dimensional data set is determined based on the comparison result between each matching degree and a preset matching degree threshold or a matching degree selection rule; wherein the matching degree selection rule is to select one or more matching degrees in descending order of the matching degrees.

7. A high-dimensional data retrieval method for digital earth according to claim 6, characterized in that: Based on the comparison result of each of the matching degrees with a preset matching degree threshold or a matching degree selection rule, before determining the matching low-dimensional data set, the method further includes: Determine a preset data type corresponding to the data to be retrieved; wherein there is a dependency relationship between the preset data type and each matching condition; the matching condition includes a comparison result of each matching degree with a preset matching degree threshold, and the matching degree selection rule; the dependency relationship is obtained based on each matching degree; when the preset data type is that the matching degree fluctuation amplitude corresponding to each matching degree is greater than a predetermined amplitude value and / or each matching degree is less than a predetermined value, the preset data type and the matching degree selection rule have the dependency relationship; The corresponding matching condition is determined according to the preset data type.

8. A high-dimensional data retrieval device for digital earth, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a high-dimensional data retrieval method for digital earth as described in any one of claims 1 to 7.

9. A non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions can execute a high-dimensional data retrieval method for digital earth as described in any one of claims 1 to 7 above.

Images