High-dimensional vector compression method and device based on data distribution characteristics

By analyzing the data distribution characteristics and covariance characteristics of high-dimensional text vectors, dimensionality reduction compression of high-dimensional data is achieved, the problems of low information density and large redundancy are solved, and the efficiency and accuracy of similarity retrieval are improved.

CN119088772APending Publication Date: 2024-12-06INST OF COMPUTING TECH CHINESE ACAD OF SCI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410995164.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-24
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The information density of high-dimensional text vectors is low and has high redundancy, which leads to the limitation of the efficiency and accuracy of subsequent similarity retrieval tasks. The existing dimensionality reduction algorithms have limitations in dimension selection and information retention.

Method used

By analyzing the data distribution characteristics of high-dimensional vectors, the dimensions with low data density are removed, and the covariance characteristics are used to merge the dimensions with high correlations to build a linear map, thereby achieving dimensionality reduction compression of high-dimensional data.

Benefits of technology

It effectively reduces the dimension of text vectors, improves the efficiency and accuracy of similarity retrieval, reduces the demand for computing resources, and significantly reduces the query time while maintaining information characteristics.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119088772A_ABST
    Figure CN119088772A_ABST
Patent Text Reader

Abstract

The invention provides a high-dimensional vector compression method based on data distribution features, which comprises the following steps: acquiring a to-be-compressed high-dimensional feature vector which is a text feature vector, an image feature vector or a voice feature vector; counting data distribution of a specified dimension in the high-dimensional feature vector, and removing dimensions of which the data distribution is smaller than a preset value in the high-dimensional feature vector to obtain a compressed intermediate vector; and extracting a covariance feature of the compressed intermediate vector to serve as correlation among specified dimensions in the compressed intermediate vector, and combining the dimensions of which the correlation is greater than a preset value in the compressed intermediate vector to obtain a compressed result vector.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of text vector dimensionality reduction and compression in natural language processing technology. Background Art

[0002] With the increase of data content (text, images, voice, etc.) on the Internet, massive data plays an increasingly important role in computer vision, natural language processing and other fields, and is an important basis for building algorithms. The practical significance of massive data lies not only in its acquisition and storage, but also in its processing methods. However, its huge amount and variety pose a huge challenge to traditional data processing methods. One of the current mainstream solutions is to use encoding vectors to convert various data original content entities (text, images and voice data) into high-dimensional vector representations, which can not only compress the storage of various types of data, but also enable massive multimodal data to be represented in a unified form, while retaining the original content information to the greatest extent. The multi-directional continuous features of high-dimensional vector space are used to represent the rich many-to-many relationships between massive multimodal data. This method plays a particularly important role in natural language processing. The current mainstream methods include encoding text into vectors through bag-of-words models, TF-IDF, word2vec and other schemes, and then applying them to natural language processing tasks such as text classification, sentiment analysis, and topic modeling. The goal of generating text vectors is to map the semantics of massive text content to a vector space of a certain dimension (or "column", no distinction will be made below), and to use the relative position or distance between vectors to map the many-to-many complex semantic relationship between the original texts. However, as the scale of text data continues to increase and the requirements for processing accuracy continue to increase, the dimension and number of text vectors are also expanding rapidly. At the same time, the information density of high-dimensional text vectors decreases exponentially with the increase in dimension. Although more text features can be captured by increasing the dimension of the text vector, the text information carried in each vector dimension is rapidly decreasing. As the dimension increases, the information contained in each dimension becomes limited, and the information content contained in a large number of dimensions is highly similar. Therefore, the information redundancy caused by the increase in dimension increases exponentially, which poses a severe challenge to the processing accuracy and efficiency of the continuously growing massive text vectors, and requires huge computing resources.

[0003] Based on this, dimensionality reduction of high-dimensional vector data is one of the current research hotspots in natural language processing technology. Its basic goal is to reduce the horizontal or vertical dimensions of the vector as much as possible while ensuring the relative position relationship and distance relationship of the original vector group. To a certain extent, it solves the contradiction between accuracy and efficiency of the similarity retrieval process in the processing of massive text vectors and effectively reduces the demand for computing resources.

[0004] The linear dimensionality reduction algorithm maps the original vector data x from a high-dimensional space to a low-dimensional space through a linear transformation, and obtains the projection matrix W by constructing the basis vectors of the new low-dimensional space, thereby obtaining a new low-dimensional vector group y=Wx.

[0005] Principal Component Analysis (PCA) Since there may be correlation within the data itself, it is necessary to remove linearly correlated columns when reducing the dimension. PCA projects the original vector data into a new coordinate system through linear transformation, so that the data columns with large variance are arranged in front. For an n-dimensional vector x = (x0, x1, ..., x n-1 ), PCA constructs a projection matrix W from n dimensions to m dimensions (m<n), so that the original vector x is projected to y by W:

[0006]

[0007] in Satisfy i With y j are independent of each other, and y0 is all x i , i∈[0, n-1], the one with the largest variance among all linear combinations, i∈[0, n-1], is the first principal component; y1 is all x that are unrelated to y0 i The one with the largest variance among all linear combinations between y and y is the second principal component; k , k∈[2,m-1] is the same as y0,y1,…,y k-1 All irrelevant x i The one with the largest variance among all linear combinations is the k+1th principal component. The linear transformation W that meets the above requirements is what we are looking for, and the low-dimensional vector y=Wx obtained using W is what we are looking for. For m n-dimensional data, they are arranged in columns as A, and their rows are zero-meaned (each row element is subtracted from the mean of that row). After calculating the covariance matrix, its eigenvalues ​​and eigenvectors are examined, and the eigenvectors are arranged in descending order of eigenvalues. The first d eigenvectors are selected to form a projection vector P in columns, and the projection PA of the original vector on P is the result of dimensionality reduction to d dimensions. PCA has good interpretability, but it does not have a strong ability to distinguish multi-category data, and it is not easy to determine the dimensionality reduction amplitude, that is, the number of principal components.

[0008] Independent Component Analysis (ICA) is used for multidimensional signal separation tasks to separate mixed signals composed of multiple signals into independent components. Unlike PCA, ICA assumes that the data of each part of the original information are independent of each other, and the original data is formed by linear combination. Therefore, the mixed data can be decomposed into independent data by constructing linear transformation. Its main purpose is to maximize the independence of components, such as Gaussianity, non-Gaussianity (FastICA) or entropy, thereby finding a linear mapping that makes the components more independent. It is widely used in blind source separation (separating different signal sources from mixed signals), information denoising (treating some components in mixed information as noise and separating them), and feature extraction (using ICA to extract basic components from mixed signals for analysis and representation of features). Based on the characteristics of the ICA algorithm that uses linear transformation to decompose the original matrix data into multiple independent components, it is also widely used in matrix decomposition and data dimensionality reduction. Compared with PCA, ICA assumes that different components of the data are independent of each other, while PCA does not consider the independence between components. Its goal is to maximize the dimensional variance. Both methods can be used for matrix decomposition, feature extraction and data compression tasks. However, ICA's strict requirement for independence between different parts of the input data also constitutes its limitation.

[0009] Linear Discriminant Analysis (LDA) is a dimensionality reduction technology for supervised learning. Its main idea is to project the original data into a lower dimensional space and require that the distance between data of different categories is as far as possible and the distance between data of the same category is as close as possible. It uses the inter-class scatter matrix to represent the degree of discreteness between different classes, and the intra-class scatter matrix to represent the degree of aggregation within each class. The projection direction of the data is determined by maximizing the ratio of the two. Unlike PCA, LDA selects the projection direction according to the difference in data categories and can distinguish the differences between different categories. It can be used for both dimensionality reduction and classification, while PCA selects the projection direction with the largest variance of sample data. It should be noted that the strong classification effect of LDA requires that the data obeys the Gaussian distribution, otherwise it is difficult to achieve good results.

[0010] Multidimensional Scaling (MDS) reduces the dimensionality of high-dimensional data while preserving the original data spacing. It first calculates the distance between the original data, such as the Euclidean distance or the Minkowski distance under other norms, and constructs the distance matrix D based on the distance between each pair of data; then the distance matrix is ​​centered to make the data points symmetrical relative to the origin. Construct the centered matrix H, where I is the unit matrix, n is the data size, and i is an n-dimensional vector of all 1s:

[0011]

[0012] Then, the inner product matrix B is calculated by the distance matrix D and the centralization matrix H, which contains the inner product relationship between the data and is used for data structure analysis:

[0013]

[0014] Then, the eigenvalues ​​and eigenvectors of the inner product matrix B are calculated, and the first d eigenvectors are selected as the basis of the vector space after dimensionality reduction according to the descending order of the eigenvalues, and the original matrix is ​​projected onto the new low-dimensional space. The MDS method can effectively preserve the relative relationship between high-dimensional data points and support deeper information mining of data. However, its computational complexity is large, and it is assumed that each dimension contributes equally to the representation of information, making it difficult to distinguish the importance of different dimensions. Summary of the invention

[0015] In this invention, a fusion dimension reduction compression algorithm for high-dimensional vector data is proposed to solve the problem that high-dimensional text vectors have low information density and high redundancy, which in turn brings inconvenience to subsequent similarity retrieval tasks. By examining the data distribution on a specific dimension of the high-dimensional vector, the part with higher information density is extracted, and the remaining part is discarded to achieve screening for each column of the high-dimensional data, and complete the initial dimension reduction of the high-dimensional data; then, the covariance characteristics of the intermediate result data in the previous step are used to construct a projection of the data of each dimension to the direction with the most obvious characteristics, that is, the largest variance, and then a linear mapping to a low-dimensional space is constructed to achieve compression of all columns with strong correlation on a specific dimension, and complete the dimension reduction by merging the parts with higher similarity.

[0016] Specifically, the present invention is as follows Figure 5 As shown in the figure, a high-dimensional vector compression method based on data distribution characteristics is proposed, which includes:

[0017] Initial step S1, obtaining a high-dimensional feature vector to be compressed, the high-dimensional feature vector being a text feature vector, an image feature vector, or a speech feature vector;

[0018] A first compression step S2 is to count the data distribution of a specified dimension in the high-dimensional feature vector, and remove the dimension in which the data distribution of the high-dimensional feature vector is less than a preset value, to obtain a compressed intermediate vector;

[0019] The second compression step S3 extracts the covariance feature of the compressed intermediate vector as the correlation between the specified dimensions in the compressed intermediate vector, and merges the dimensions in the compressed intermediate vector whose correlation is greater than a preset value to obtain a compressed result vector;

[0020] The task executes step S4, inputting the compression result vector into an artificial intelligence-based recognition classification model or retrieval system to obtain a recognition classification result or a retrieval classification result.

[0021] The high-dimensional vector compression method based on data distribution characteristics, wherein the first compression step comprises:

[0022] The size of this high-dimensional feature vector X is N and the dimension is D:

[0023] X j ={X ij |i=0,1,…,N}, j=0,1,…,D

[0024] According to the expected μ of the data on the specified dimension j Calculate the standard deviation σ of the data on the specified dimension j , with the standard deviation σ j As the data distribution of the high-dimensional feature vector X on the specified dimension:

[0025]

[0026] The high-dimensional vector compression method based on data distribution characteristics, wherein the second compression step comprises:

[0027] The specified dimension of the compressed intermediate vector X′ with dimension D′ is zero-meaned to obtain

[0028]

[0029] Afterwards The covariance matrix of is decomposed to obtain the eigenvalue λ k and the corresponding eigenvector v k :

[0030]

[0031] According to the eigenvalue λ k The order of dimensionality reduction is the feature vector v k , determine the principal components projected onto the low-dimensional space in turn, and form the projection matrix W:

[0032] W=[v′1, v′2,…,v′ k ], λ1′>λ2′>…>λ k '

[0033] Use W to project the compressed intermediate vector X′ onto the k-dimensional space to obtain the compressed result vector X″:

[0034] X″=X′W.

[0035] The high-dimensional vector compression method based on data distribution characteristics, wherein the high-dimensional feature vector is a feature vector of image, text, video or audio data; the recognition classification model is used to classify the compression result vector; the retrieval system is used to retrieve the data with the highest similarity to the compression result vector in the database as the retrieval classification result.

[0036] The present invention also proposes a Figure 6 The high-dimensional vector compression device B based on data distribution characteristics shown includes:

[0037] An initial module M1, obtaining a high-dimensional feature vector to be compressed, where the high-dimensional feature vector is a text feature vector, an image feature vector, or a speech feature vector;

[0038] A first compression module M2 counts data distribution of a specified dimension in the high-dimensional feature vector, removes dimensions in the high-dimensional feature vector where data distribution is less than a preset value, and obtains a compressed intermediate vector;

[0039] The second compression module M3 extracts the covariance feature of the compressed intermediate vector as the correlation between the specified dimensions in the compressed intermediate vector, merges the dimensions in the compressed intermediate vector whose correlation is greater than a preset value, and obtains a compressed result vector;

[0040] The task execution module M4 inputs the compression result vector into an artificial intelligence-based recognition classification model or retrieval system to obtain a recognition classification result or a retrieval classification result.

[0041] The high-dimensional vector compression device based on data distribution characteristics, wherein the first compression module comprises:

[0042] The size of this high-dimensional feature vector X is N and the dimension is D:

[0043] X j ={X ij |i=0,1,…,N}, j=0,1,…,D

[0044] According to the expected μ of the data on the specified dimension j Calculate the standard deviation σ of the data on the specified dimension j , with the standard deviation σ j As the data distribution of the high-dimensional feature vector X on the specified dimension:

[0045]

[0046] The high-dimensional vector compression device based on data distribution characteristics, wherein the second compression module comprises:

[0047] The specified dimension of the compressed intermediate vector X′ with dimension D′ is zero-meaned to obtain

[0048]

[0049] Afterwards The covariance matrix of is decomposed to obtain the eigenvalue λ k and the corresponding eigenvector v k :

[0050]

[0051] According to the eigenvalue λ k The order of dimensionality reduction is the feature vector v k , determine the principal components projected onto the low-dimensional space in turn, and form the projection matrix W:

[0052] W=[v′1, v′2,…,v′ k ], λ1′>λ2′>…>λ k '

[0053] Use W to project the compressed intermediate vector X′ onto the k-dimensional space to obtain the compressed result vector X″:

[0054] X″=X′W.

[0055] The high-dimensional vector compression device based on data distribution characteristics, wherein the high-dimensional feature vector is a feature vector of image, text, video or audio data; the recognition classification model is used to classify the compression result vector; the retrieval system is used to retrieve the data with the highest similarity to the compression result vector in the database as the retrieval classification result.

[0056] The present invention also proposes an electronic device, including the high-dimensional vector compression device based on data distribution characteristics, and the electronic device is connected to an information display device, which is used to display the recognition classification results or retrieve the classification results according to the display parameters and attributes set by the user or through an artificial intelligence model.

[0057] The present invention also proposes a storage medium for storing a computer program for executing the high-dimensional vector compression method based on data distribution characteristics.

[0058] It can be seen from the above scheme that the advantages of the present invention are:

[0059] The algorithm was tested on a dataset that was a subset of word vectors trained by ngram2vec (a word vector generation tool that can better capture contextual information than word2vec) based on text corpora from multiple fields on the Internet. The word vector part was extracted when used, with a dimension of 1536 and a scale of 77199. To distinguish it from other datasets used in subsequent experiments, it was recorded as dataset No. 1. The NDCG indicator introduced previously is an evaluation for a single query target. In this study, the average NDCG indicator (AverageNDCG, ANDCG) is used when facing a large number of queries, that is, the NDCG indicators of all results in a batch query are averaged, so as to better reflect the accuracy of the query results. In addition, this study also uses the query result first hit (FirstHit, FH) as an evaluation indicator for judging the results of a single query item. It is a Boolean value representing the result that is considered to be closest to the query vector in the query result, that is, whether the top1 is the query vector itself. If yes, FH is 1, and it can be considered that the query index can find the vector closest to the query vector; if not, FH is 0, indicating that the nearest neighbor given by the index is not the actual nearest neighbor. When facing a large number of queries, the average first hit index (AverageFirstHit, AFH) is used, that is, the FH index of all results in a batch query is averaged, so as to better reflect the accuracy of the query results. Since the vector dimensionality reduction algorithm is implemented in this chapter, if only the NDCG index is examined, the best result must come from the original data without any compression. Therefore, it is also necessary to start with the compression result and examine the time consumed by querying the data set before and after the experiment using the same query scheme to reflect the compression effect. In order to exclude the influence of other factors on the experimental results, this chapter only uses ordinary distance indicators to retrieve similar vectors, that is, calculate the Euclidean distance between all vectors in the vector group and the query vector, and select the smallest part. In order to more intuitively reflect the query effect, this experiment adopts the test method of directly extracting the query data from the original data. In the experiment, 15 vector indexes for query are randomly generated:

[0060] [10465,56611,76541,58340,65088,50588,49115,70965,54827,22537,51529,49456,56663,35839,49328], perform similarity search on the query vector in the original data set (data_array) and the data set (data_pca_cps) after dimensionality reduction using the fusion compression method, and get the top 10 results (top10) closest to the query vector, and record the time consumption of the query process.

[0061] Directly perform an accurate search on the original vector data set, and the result is the real top10 nearest neighbors of each query item, the corresponding NDCG is IDCG, the average NDCG is 1, the FH is 1, and the average FH is 1, as shown in Table 1:

[0062] Table 1. Precise query results of the original dataset

[0063]

[0064] This query took 2928.1 seconds. The vector data set compressed by the fusion algorithm was queried, and PCA and FastICA were used as baseline methods. The query results are shown in Table 2:

[0065] Table 2 Query results of datasets compressed by different methods

[0066]

[0067] This query took 729.4 seconds on the data after dimensionality reduction using the fusion compression method, 733.7 seconds on the data after dimensionality reduction using the baseline method PCA, and 735.2 seconds on the data after dimensionality reduction using the baseline method FastICA, which are similar. The time complexity of the exact query method used depends on the data dimension, so the query time is similar when the original data is reduced to the same dimension. From the above results, we can see that this method can guarantee highly accurate query results while significantly reducing the query time, and its accuracy reaches the baseline level.

[0068] Ablation studies are conducted on the two parts of the fusion dimensionality reduction method in the present invention: a compression method based on high-dimensional vector data distribution and a compression method based on high-dimensional vector covariance features to verify their contributions to the algorithm.

[0069] Using the same original data set, only the compression method based on high-dimensional vector data distribution is implemented on it to obtain the data set data_high_dens. Using the same evaluation index NDCG, the results are shown in Table 3:

[0070] Table 3. Query results of data_high_dens dataset

[0071]

[0072] This query took 1458.1 seconds. It can be seen that when the compression level is small, the ANDCG index is slightly improved compared with this algorithm, but the time consumption is doubled. Using the same original data set, only the compression method based on the high-dimensional vector covariance feature is implemented on it to obtain the data set data_only_pca. Using the same evaluation index NDCG, the results are shown in Table 4:

[0073] Table 4. Query results of data_only_pca dataset

[0074]

[0075] This query took 1461.7 seconds. It can be seen that the result obtained when only using covariance features to reduce the correlation between columns for dimensionality reduction is not as good as this fusion method, and the time consumption has doubled. Comparing the results of the two incomplete algorithms that removed some steps in the ablation experiment and the query results using the original data set and the vector data set compressed by the fusion algorithm, although the compression degree is small, making the NDCG index in the ablation experiment slightly better than the fusion algorithm, its query performance index is far worse than the fusion algorithm. This proves the effectiveness of the compression method based on the distribution of high-dimensional vector data and the compression method based on the covariance feature of high-dimensional vectors. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 This is the standard deviation distribution graph of the data_array data set;

[0077] Figure 2 This is the standard deviation distribution diagram of the data_high_dens data set;

[0078] Figure 3 This is the standard deviation distribution diagram of the data_pca_cps data set;

[0079] Figure 4 A technical process flow chart for high-dimensional vector compression;

[0080] Figure 5 is a flow chart of the method of the present invention;

[0081] Figure 6 It is a module diagram of the device of the present invention;

[0082] Figure 7 This is a schematic diagram of the structure of a first electronic device of the present invention;

[0083] Figure 8 This is a schematic diagram of the application environment structure of the first electronic device of the present invention;

[0084] Fig. 9 It is a schematic diagram of the structure of a second electronic device of the present invention.

[0085] Reference numerals:

[0086] A-First electronic device;

[0087] B-High-dimensional vector compression device based on data distribution characteristics;

[0088] C-data acquisition equipment;

[0089] D-information display device;

[0090] 1000 - second electronic device;

[0091] Ⅰ-computational unit;

[0092] II-ROM;

[0093] III-RAM;

[0094] IV-bus;

[0095] V-interface;

[0096] VI - input unit;

[0097] VII-output unit;

[0098] VIII- Storage medium;

[0099] Ⅸ-Communication unit. DETAILED DESCRIPTION

[0100] It should be noted that the processor described in the present invention is the control center of the electronic device, which can be a processor or a general term for multiple processing elements. For example, it can be one or more central processing units (CPUs), or application specific integrated circuits (ASICs), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (FPGAs).

[0101] Optionally, the processor can perform various functions of the electronic device by running or executing a software program stored in the memory, and calling data stored in the memory.

[0102] In a specific implementation, as an embodiment, the processor may include one or more CPUs. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions). Electronic devices may include: servers, desktop computers, laptops, smart phones, tablet computers, embedded computers, etc., wherein the embedded computers include vehicles and robots, etc.

[0103] The memory is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0104] It should be noted that the structure of the electronic device shown in the drawings of the present invention does not constitute a limitation thereto, and the actual knowledge structure recognition device may include more or fewer components than shown in the drawings, or a combination of certain components, or a different arrangement of components.

[0105] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0106] It should also be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0107] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0108] It should also be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0109] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0110] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0111] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0112] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0113] When the inventors were conducting research on high-dimensional vector compression, they found that the defect in the prior art was caused by considering the data distribution state between the dimensions of the high-dimensional vector data. The main problems are:

[0114] When using the word vector model to encode text into a vector, in order to retain more text semantic information, the resulting encoded vector has a larger dimension, while the data information density in many dimensions within it is very low. There is a lot of information redundancy in the entire vector set, which brings inconvenience to subsequent processing such as similarity retrieval.

[0115] Therefore, it is necessary to reduce the dimension of complex vectors as much as possible while retaining the original information or features as much as possible. The present invention will introduce a fusion compression method C=C2C1 based on high-dimensional vector data distribution and covariance characteristics, so that for the vector group X, the compressed data X″=C(X)=C2(C1(X)), where C1 is the first part of the fusion compression method, which compresses the high-dimensional vector by removing the low-density part based on the data distribution of the high-dimensional vector; C2 is the second part, which merges the parts with higher correlation based on the covariance characteristics of the high-dimensional vector to achieve dimensionality reduction. It should be noted that the execution order of C1 and C2 will not have a qualitative impact on the compression result. However, if C2 is executed before C1, the computational efficiency will be reduced.

[0116] In order to make the above features and effects of the present invention more clearly and understandably described, embodiments are given below and described in detail with reference to the accompanying drawings. This specification discloses one or more embodiments that include the features of the present invention. The disclosed embodiments are only for illustration. The scope of protection of the present invention is not limited to the disclosed embodiments, and the present invention is defined by the attached claims.

[0117] The scenic spot browsing path planning method based on time and space constraints proposed by the present invention specifically includes the following steps 1 and 2.

[0118] Step 1: Compression method based on high-dimensional vector data distribution:

[0119] High similarity of information in a certain dimension will cause information redundancy, so this method aims to remove the part of information redundancy that is too large in a specific dimension (column dimension in this invention). First, the distribution of the original vector data is examined. For a vector group X with a size of N and a dimension of D, since this method only involves dimensionality reduction, a single vector is not considered in this step, and the data on each column is examined:

[0120] X j ={X ij |i=0,1,…,N}, j=0,1,…,D

[0121] Count the data distribution on each column and calculate the expected μ of the data on each column j and standard deviation (data distribution) σ j :

[0122]

[0123] like Figure 1 As shown, Figure 1 This is the standard deviation distribution of each column of data in a vector set generated based on text. The dataset size is 77199 and contains 1536 dimensions.

[0124] Investigation j , j = 0, 1, ..., D-1, column l on the distribution of σ l A lower value means that the data density on this column is lower. It can be removed during the initial dimensionality reduction to obtain the matrix X′ after the first dimensionality reduction. Its dimension is recorded as D′. At this time, the dimension of the data set can be reduced by half. Its standard deviation distribution is as follows: Figure 2 shown.

[0125] Step 2: Compression method based on high-dimensional vector covariance features:

[0126] Since highly correlated data will cause information redundancy, the correlation between different columns should be considered after eliminating the information redundancy on a single column. Highly correlated columns should be merged to reduce the dimension of the vector group. This method remaps the vector group after the initial dimensionality reduction to a new dimension with lower correlation by constructing a mapping from high-dimensional to low-dimensional space. Zero mean is performed on the column dimension of X′ in the previous step to obtain

[0127]

[0128] Afterwards The covariance matrix of is decomposed to obtain the eigenvalue λ k and the corresponding eigenvector v k :

[0129]

[0130] According to the eigenvalue λ k The order of dimensionality reduction is the feature vector v k , determine the principal components projected onto the low-dimensional space in turn, and form the projection matrix W:

[0131] W=[v′1, v′2,…,v′ k ], λ1′>λ2′>…>λ k '

[0132] Use W to project the vector group X′ onto the k-dimensional space and obtain X″:

[0133] X″=X′W

[0134] Thus, we can obtain the vector cluster data X″ with further dimensionality reduction. At this time, the dimension of the vector set is reduced by half to D″, and its dimensional distribution is as follows: Figure 3 shown.

[0135] Figure 4 The present invention shows the basic technical process of high-dimensional vector processing. For the fusion compression and dimensionality reduction scheme of high-dimensional vectors, first, based on the statistical data distribution status in each dimension, find out some columns with low data density, and remove the columns with the lowest density according to a certain ratio, then calculate the covariance characteristics of the vector data, and construct a mapping from high dimension to low dimension in the direction of retaining the maximum variance.

[0136] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment. In order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiment.

[0137] The present invention also proposes a Figure 6 The high-dimensional vector compression device B based on data distribution characteristics shown includes:

[0138] An initial module M1 is used to obtain a high-dimensional feature vector to be compressed, where the high-dimensional feature vector is a text feature vector, an image feature vector, or a speech feature vector;

[0139] A first compression module M2 counts data distribution of a specified dimension in the high-dimensional feature vector, removes dimensions in the high-dimensional feature vector where data distribution is less than a preset value, and obtains a compressed intermediate vector;

[0140] The second compression module M3 extracts the covariance feature of the compressed intermediate vector as the correlation between the specified dimensions in the compressed intermediate vector, merges the dimensions in the compressed intermediate vector whose correlation is greater than a preset value, and obtains a compressed result vector;

[0141] The task execution module M4 inputs the compression result vector into an artificial intelligence-based recognition classification model or retrieval system to obtain a recognition classification result or a retrieval classification result.

[0142] The high-dimensional vector compression device based on data distribution characteristics, wherein the first compression module comprises:

[0143] The size of this high-dimensional feature vector X is N and the dimension is D:

[0144] X j ={X ij |i=0,1,…,N}, j=0,1,…,D

[0145] According to the expected μ of the data on the specified dimension j Calculate the standard deviation σ of the data on the specified dimension j , with the standard deviation σ jAs the data distribution of the high-dimensional feature vector X on the specified dimension:

[0146]

[0147] The high-dimensional vector compression device based on data distribution characteristics, wherein the second compression module comprises:

[0148] The specified dimension of the compressed intermediate vector X′ with dimension D′ is zero-meaned to obtain

[0149]

[0150] Afterwards The covariance matrix of is decomposed to obtain the eigenvalue λ k and the corresponding eigenvector v k :

[0151]

[0152] According to the eigenvalue λ k The order of dimensionality reduction is the feature vector v k , determine the principal components projected onto the low-dimensional space in turn, and form the projection matrix W:

[0153] W=[v′1, v′2,…,v′ k ], λ1′>λ2′>…>λ k '

[0154] Use W to project the compressed intermediate vector X′ onto the k-dimensional space to obtain the compressed result vector X″:

[0155] X″=X′W.

[0156] The high-dimensional vector compression device based on data distribution characteristics, wherein the high-dimensional feature vector is a feature vector of image, text, video or audio data; the recognition classification model is used to classify the compression result vector; the retrieval system is used to retrieve the data with the highest similarity to the compression result vector in the database as the retrieval classification result.

[0157] like Figure 7 As shown, the present invention further proposes a first electronic device A in another embodiment, including the high-dimensional vector compression device based on data distribution characteristics, and the electronic device may be connected to an information display device.

[0158] like Figure 8As shown, the first electronic device A can also be connected to the data acquisition device C and the information display device D through a wired or wireless information transmission scheme. The data acquisition device C is used to collect and obtain compressed high-dimensional feature vectors, such as audio and video features, and the information display device D is used to display the recognition classification results or retrieval classification results obtained by the analysis of the present invention.

[0159] The information display device D can sort and process the data output by the first electronic device A based on the information display mechanism to improve the readability of the data output by the first electronic device A. The information display mechanism can be manually preset, for example, the data output by the first electronic device A is visually displayed, which can be based on the display parameters and / or attributes set by the user. The display parameters can be, for example, the display data range, and the display attributes can be, for example, the display font, color, whether to scroll and play, etc. The user is presented with the key information specified by the user, and the user can understand this information more promptly without having to access the secondary page or scroll the page, saving the user's operation. Or the information display mechanism can be an artificial intelligence AI display model, which can learn the user's key information based on the user's previous usage habits, such as viewing time, number of clicks, number of edits, etc., and then automatically present rich and necessary key information to the user.

[0160] The present invention also proposes a storage medium VIII in another embodiment for storing a computer program for executing the high-dimensional vector compression method based on data distribution characteristics. It should be understood that the storage medium in the embodiment of the present invention can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0161] Fig. 9 A schematic block diagram of a second electronic device 1000 that can be used to implement an embodiment of the present invention is shown. The second electronic device 1000 electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The second electronic device 1000 can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present invention described and / or required herein. The second electronic device 1000 may be the same or different from the first electronic device A.

[0162] The second electronic device 1000 includes a computing unit I, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory II (ROM) or a computer program loaded from a storage medium VIII into a random access memory (RAM) III. In RAM III, various programs and data required for the operation of the device 1000 can also be stored. The computing unit I, ROM II, and RAM III are connected to each other via a bus IV. An input / output (I / O) interface V is also connected to the bus IV.

[0163] A plurality of components in the second electronic device 1000 are connected to the I / O interface V, including: an input unit VI, such as a keyboard, a mouse, etc.; an output unit VII, such as various types of displays, speakers, etc.; a storage medium VIII, such as a disk, an optical disk, etc.; and a communication unit IX, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit IX allows the second electronic device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0164] The computing unit I may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of computing unit I include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit I performs the various methods and processes described above, such as method steps S1-S4. For example, in some embodiments, the method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage medium VIII. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via ROM II and / or a communication unit IX. When the computer program is loaded into RAM III and executed by the computing unit I, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the computing unit I may be configured to execute the method in any other appropriate manner (e.g., by means of firmware).

[0165] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation modes, and they can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

Claims

1. A high-dimensional vector compression method based on data distribution characteristics, characterized in that: include: An initial step is to obtain a high-dimensional feature vector to be compressed, where the high-dimensional feature vector is a text feature vector, an image feature vector, or a speech feature vector; In a first compression step, data distribution of a specified dimension in the high-dimensional feature vector is counted, and dimensions in which data distribution of the high-dimensional feature vector is less than a preset value are removed to obtain a compressed intermediate vector; In a second compression step, a covariance feature of the compressed intermediate vector is extracted as the correlation between the specified dimensions in the compressed intermediate vector, and the dimensions in the compressed intermediate vector whose correlation is greater than a preset value are merged to obtain a compressed result vector; The task execution step is to input the compressed result vector into an artificial intelligence-based recognition and classification model or a retrieval system to obtain a recognition and classification result or a retrieval and classification result.

2. The high-dimensional vector compression method based on data distribution characteristics according to claim 1, characterized in that: The first compression step comprises: The size of this high-dimensional feature vector X is N and the dimension is D: X j ={X ij |i=0,1,…,N},j=0,1,…,D According to the expected μ of the data on the specified dimension j Calculate the standard deviation σ of the data on the specified dimension j , with the standard deviation σ j As the data distribution of the high-dimensional feature vector X on the specified dimension:

3. The high-dimensional vector compression method based on data distribution characteristics according to claim 1 or 2, characterized in that: The second compression step comprises: The specified dimension of the compressed intermediate vector X′ with dimension D′ is zero-meaned to obtain Afterwards The covariance matrix of is decomposed to obtain the eigenvalue λ k and the corresponding eigenvector v k : According to the eigenvalue λ k The order of dimensionality reduction is the feature vector v k , determine the principal components projected onto the low-dimensional space in turn, and form the projection matrix W: W=[v′1,v′2,…,v′ k ],λ1′>λ2′>…>λ l ′ Use W to project the compressed intermediate vector X′ onto the k-dimensional space to obtain the compressed result vector X″: X″=X′W.

4. The high-dimensional vector compression method based on data distribution characteristics according to claim 1, characterized in that: The high-dimensional feature vector is a feature vector of image, text, video or audio data; the recognition classification model is used to classify the compression result vector; and the retrieval system is used to retrieve data with the highest similarity to the compression result vector in the database as the retrieval classification result.

5. A high-dimensional vector compression device based on data distribution characteristics, characterized in that: include: An initial module, obtaining a high-dimensional feature vector to be compressed, where the high-dimensional feature vector is a text feature vector, an image feature vector, or a speech feature vector; A first compression module, which counts data distribution of a specified dimension in the high-dimensional feature vector, removes dimensions in the high-dimensional feature vector where data distribution is less than a preset value, and obtains a compressed intermediate vector; The second compression module extracts the covariance feature of the compressed intermediate vector as the correlation between the specified dimensions in the compressed intermediate vector, merges the dimensions in the compressed intermediate vector whose correlation is greater than a preset value, and obtains a compressed result vector; The task execution module inputs the compression result vector into an artificial intelligence-based recognition and classification model or a retrieval system to obtain a recognition and classification result or a retrieval and classification result.

6. The high-dimensional vector compression device based on data distribution characteristics as claimed in claim 5, characterized in that: The first compression module comprises: The size of this high-dimensional feature vector X is N and the dimension is D: X j ={X ij |i=0,1,…,N},j=0,1,…,D According to the expected μ of the data on the specified dimension j Calculate the standard deviation σ of the data on the specified dimension j , with the standard deviation σ j As the data distribution of the high-dimensional feature vector X on the specified dimension:

7. The high-dimensional vector compression device based on data distribution characteristics according to claim 5 or 6, characterized in that: The second compression module comprises: The specified dimension of the compressed intermediate vector X′ with dimension D′ is zero-meaned to obtain Afterwards The covariance matrix of is decomposed to obtain the eigenvalue λ k and the corresponding eigenvector V k : According to the eigenvalue λ k The order of dimensionality reduction is the feature vector v k , determine the principal components projected onto the low-dimensional space in turn, and form the projection matrix W: W=[v′1,v′2,…,v′ k ],λ1′>λ2′>…>λ k ′ Use W to project the compressed intermediate vector X′ onto the k-dimensional space to obtain the compressed result vector X″: X″=X′W 8. The high-dimensional vector compression device based on data distribution characteristics as claimed in claim 5, characterized in that: The high-dimensional feature vector is a feature vector of image, text, video or audio data; the recognition classification model is used to classify the compression result vector; and the retrieval system is used to retrieve data with the highest similarity to the compression result vector in the database as the retrieval classification result.

9. An electronic device, characterized in that: It includes a high-dimensional vector compression device based on data distribution characteristics as described in claims 5-8, and the electronic device is connected to an information display device, which is used to display the recognition classification result or retrieve the classification result according to the display parameters and attributes set by the user or through an artificial intelligence model.

10. A storage medium for storing a computer program for executing the high-dimensional vector compression method based on data distribution characteristics as described in claims 1-4.