Data determination method and apparatus, storage medium, and electronic device

Abstract

Embodiments of the present application provide a data determination method and device, a storage medium and an electronic device, wherein the method comprises: generating M clusters of a target data set; performing a sorting operation on the M clusters based on an upper bound between a first target data and each cluster to obtain M sorted clusters; determining a first target cluster according to a comparison result between a first lower bound and a first upper bound; and determining a second target data from the first target cluster. Through the present application, the problem that the result is not accurate enough when finding the data farthest from the target data in the related art is solved, and the effect of improving the data screening accuracy is achieved.

Description

Technical Field

[0001] This application relates to the field of data processing, and more specifically, to a method and apparatus for determining data, a storage medium, and an electronic device. Background Technology

[0002] In big data, unstructured data, such as text, images, audio, and video, accounts for more than 85% of the total collected data. However, due to the massive volume, diversity, complexity, and rapid growth of unstructured big data, it poses certain challenges to information storage, computing, and data processing technologies for various applications.

[0003] Due to the structural complexity of raw unstructured data, directly calculating similarity is difficult. Therefore, the common approach is to extract feature vectors from the data (these vectors are typically high-dimensional, ranging from hundreds to thousands), transforming the problem into a nearest neighbor search and farthest neighbor search on a massive set of high-dimensional features. The worst-case method for solving high-dimensional farthest neighbor search is linear scanning. Although it has linear complexity, due to the massive and high-dimensional nature of unstructured data, this method is extremely time-consuming and also suffers from the curse of dimensionality.

[0004] Currently, two main technical approaches have been proposed to address this problem: hashing and heuristic methods. However, hashing methods suffer from low accuracy, and it's difficult to find hash functions that perform well in high-dimensional spaces. The best hashing method currently is locality-sensitive hashing, but it's essentially a one-dimensional projection, losing a lot of information when projecting into higher dimensions, and thus failing to effectively identify the farthest neighbors. The main problem with heuristic methods is performance instability; their performance depends on the size of the subset extracted from the dataset. For some datasets, the farthest neighbor subset is indeed very small. However, for many datasets, the farthest neighbor subset remains very large, resulting in persistently low search efficiency.

[0005] Furthermore, both of these methods suffer from a loss of accuracy, failing to guarantee 100% precision. Therefore, the farthest neighbor structures found are approximate, rather than the true farthest point (i.e., the exact farthest neighbor). Summary of the Invention

[0006] This application provides a method and apparatus for determining data, which at least solves the problem of low accuracy of the farthest data of the target data found in related technologies.

[0007] According to one embodiment of this application, a method for determining data is provided, comprising: generating M clusters of a target dataset, wherein each cluster includes multiple data points, the data being vector representation data obtained by vector transformation of unstructured data, and M being a positive integer greater than 1; performing a sorting operation on the M clusters based on an upper bound between a first target data point and each cluster, to obtain M sorted clusters, wherein the upper bound represents the maximum distance between the first target data point and the data in the clusters, the first target data being vector representation data obtained by vector transformation of first unstructured data; and sorting the clusters according to the ratio between a first lower bound and a first upper bound. Comparing the results, a first target cluster is determined, wherein the first lower bound is the minimum distance between the first target data and the data in the second target cluster, the data included in the second target cluster are data selected from the (M-1)th cluster, the first upper bound is the maximum distance between the first target data and the data in the Mth cluster, and the Mth cluster and the (M-1)th cluster are adjacent clusters among the M sorted clusters; second target data is determined from the first target cluster, wherein the second target data is the data in the first target cluster with the largest distance to the first target data, and the second target data is the data in the target dataset.

[0008] In an exemplary embodiment, generating M clusters of a target dataset includes: obtaining an original dataset, wherein the original dataset includes multiple unstructured data sets; performing vector transformation operations on the multiple unstructured data sets respectively to obtain the target dataset, wherein the vector transformation operations include converting the data features of each unstructured data set into feature vectors; statistically analyzing the value ranges of the data in the target dataset across multiple preset data dimensions, and dividing the value range of each preset data dimension into 2n... N There are 2 intervals, where N is a positive integer greater than or equal to 1; based on the target dataset, each of the above-mentioned preset data dimensions includes 2 N For each interval, perform clustering operations on the data in the target dataset to obtain M clusters.

[0009] In an exemplary embodiment, before performing a sorting operation on the M clusters based on the upper bound between the first target data and each of the clusters to obtain M sorted clusters, the method further includes: for each of the clusters, performing the following operations to determine the upper bound between the first target data and each of the clusters: calculating the distance between the center data in the cluster and the boundary data in the cluster to obtain the radius of the cluster; calculating the distance between the first target data and the center data to obtain a first distance; and determining the sum of the first distance and the radius as the upper bound between the first target data and the cluster.

[0010] In an exemplary embodiment, determining a first target cluster based on a comparison between a first lower bound and a first upper bound includes: if the first lower bound is determined to be less than the first upper bound from the comparison result, calculating a lower bound between the first target data and the data in the Mth cluster to obtain a second lower bound, wherein the second lower bound represents the minimum distance between the first target data and the data in the Mth cluster; if the second lower bound is determined to be greater than the first lower bound, updating the first lower bound to the second lower bound, and storing the data in the Mth cluster into the second target cluster to obtain the first target cluster.

[0011] In an exemplary embodiment, when it is determined from the comparison result that the first lower bound is less than the first upper bound, calculating the lower bound between the first target data and the data in the Mth cluster to obtain a second lower bound includes: determining the VA codes of the plurality of data included in the Mth cluster, wherein the VA codes are used to represent the number sequence of the intervals that the data in the Mth cluster falls into on each preset data dimension; calculating the lower boundary between the first target data and the VA codes of the plurality of data in each preset data dimension to obtain a plurality of first lower boundaries, wherein the preset data dimension is used to represent the value range of the data in the target dataset, the first lower boundary is used to represent the minimum distance between the lower boundary of the first target data and the VA codes of the first data, and the first data is any one of the plurality of data; calculating the sum of the plurality of the first lower boundaries to obtain the second lower boundary of the first target data in each preset data dimension; comparing the plurality of the second lower boundaries of the first target data in the plurality of preset data dimensions to obtain a target lower boundary, wherein the target lower boundary is used to represent a plurality of The maximum value in the second lower boundary mentioned above; the target lower boundary is determined as the lower bound of the first target data to the data in the Mth cluster; wherein, in each preset data dimension, the lower boundary between the first target data and the VA codes of the multiple data is calculated to obtain multiple first lower boundaries, including: determining the components of the first target data in each preset data dimension to obtain multiple components, wherein the components are used to represent the proportion of the first target data in the preset data dimension; determining the data interval where the VA code of each data is located, wherein the data interval is the interval of the target dataset in each preset data dimension; comparing each component with the lower boundary of the data interval where the VA code of each data is located to obtain a first comparison result; comparing each component with the upper boundary of the data interval where the VA code of each data is located to obtain a second comparison result; determining the lower boundary between the first target data and the VA codes of the multiple data in each preset data dimension based on the first comparison result and the second comparison result to obtain multiple first lower boundaries.

[0012] In one exemplary embodiment, determining the second target data from the first target cluster includes: calculating the distance between the first target data and each data in the first target cluster to obtain a plurality of second distances; deleting data with distances less than the first lower bound from the first target cluster according to the plurality of second distances to obtain a third target cluster; sorting the data in the third target cluster based on the distances between the first target data and N data in the third target cluster, where N is a natural number greater than or equal to 1; and traversing the N data according to the sorting result to determine the data with the largest distance from the first target data among the N data as the second target data.

[0013] In one exemplary embodiment, traversing N data points according to the sorting result and determining the data point with the largest distance from the first target data point as the second target data point includes: traversing N data points according to the sorting result, calculating the distance between the first target data point and the Nth data point to obtain a third distance; and determining the Nth data point as the second target data point if the third distance is greater than or equal to a fourth distance, wherein the fourth distance is the distance between the first target data point and the (N-1)th data point, and the (N-1)th data point is the data point adjacent to the Nth data point.

[0014] According to another embodiment of this application, a data determination apparatus is provided, comprising: a first generation module, configured to generate M clusters of a target dataset, wherein each cluster includes multiple data points, the data being vector representation data obtained by vector transformation of unstructured data, and M being a positive integer greater than 1; a first sorting module, configured to perform a sorting operation on the M clusters based on an upper bound between a first target data and each cluster, to obtain M sorted clusters, wherein the upper bound represents the maximum distance between the first target data and the data in the clusters, and the first target data is vector representation data obtained by vector transformation of first unstructured data; and a first determination module, configured to determine according to a first... The comparison between the lower bound and the upper bound determines the first target cluster, wherein the first lower bound is the minimum distance between the first target data and the data in the second target cluster, the data included in the second target cluster are data selected from the (M-1)th cluster, the first upper bound is the maximum distance between the first target data and the data in the Mth cluster, and the Mth cluster and the (M-1)th cluster are adjacent clusters among the M sorted clusters; the second determining module is used to determine the second target data from the first target cluster, wherein the second target data is the data in the first target cluster with the largest distance to the first target data, and the second target data is the data in the target dataset.

[0015] According to yet another embodiment of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer-readable storage medium, and the computer program is configured to perform the steps in any of the above method embodiments when it is run.

[0016] According to yet another embodiment of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0017] This application generates M clusters of the target dataset. The upper bound between the first target data and each cluster is determined using the distance between the first target data and the cluster center data, and the cluster radius. The clusters are then sorted based on these upper bounds. The first target cluster is determined by iteratively comparing the maximum distance between the first target data and data in the Mth cluster, and the minimum distance between the first target data and data in the first M-1 clusters, completing the first filtering step. Next, for the first target cluster, data whose distance to the first target data is less than the first lower bound are deleted, resulting in the third target cluster, completing the second filtering step. Finally, the data with the largest distance between the first target data and N data in the third target cluster is determined, resulting in the second target data, completing the final data search operation. Therefore, this method solves the problem of insufficient accuracy in finding data farthest from the target data in related technologies, achieving the effect of improving data filtering accuracy. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the hardware environment for a data determination method according to an embodiment of this application;

[0019] Figure 2 This is a flowchart of a data determination method according to an embodiment of this application;

[0020] Figure 3 This is a schematic diagram illustrating the determination of image data according to an embodiment of this application;

[0021] Figure 4 This is a structural block diagram of a data determination device according to an embodiment of this application. Detailed Implementation

[0022] The embodiments of this application will be described in detail below with reference to the accompanying drawings and examples.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0024] The methods and embodiments provided in this application can be executed on a server device or a similar computing device. Taking running on a server device as an example, Figure 1 This is a schematic diagram of the hardware environment for a data determination method according to an embodiment of this application. For example... Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The server device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the server equipment described above. For example, the server equipment may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0025] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the data determination method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to server devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0026] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the server device. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0027] This embodiment provides a method for determining data. Figure 2This is a flowchart of a data determination method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0028] Step S202: Generate M clusters of the target dataset, wherein each cluster includes multiple data points, which are vector representation data obtained by vector transformation of unstructured data, and M is a positive integer greater than 1.

[0029] Step S204: Based on the upper bound between the first target data and each cluster, perform a sorting operation on the M clusters to obtain M sorted clusters. The upper bound is used to represent the maximum distance between the first target data and the data in the clusters. The first target data is vector representation data obtained by vector transformation of the first unstructured data.

[0030] Step S206: Determine the first target cluster according to the comparison result between the first lower bound and the first upper bound, wherein the first lower bound is the minimum distance between the first target data and the data in the second target cluster, the data included in the second target cluster is the data selected from the (M-1)th cluster, the first upper bound is the maximum distance between the first target data and the data in the Mth cluster, and the Mth cluster and the (M-1)th cluster are adjacent clusters among the M sorted clusters;

[0031] Step S208: Determine the second target data from the first target cluster, wherein the second target data is the data in the first target cluster that has the largest distance from the first target data, and the second target data is the data in the target dataset.

[0032] Optionally, the data in the target dataset in this embodiment is vector representation data obtained by performing vector transformation operations on multiple unstructured data. Unstructured data includes, but is not limited to: text data, image data, video data, audio data, web page data, log files, scientific data, geospatial data, sensor data, social media data, etc.

[0033] Optionally, the methods for vectorizing unstructured data to obtain vector representation data in this embodiment include, but are not limited to: text data can be converted into word frequency vectors using a bag-of-words model; image data can be directly converted into vectors using pixel values, or image features can be extracted using algorithms such as SIFT and SURF, or image features can be extracted using a pre-trained convolutional neural network (CNN); audio data can be converted into frequency domain features using Fourier transform, or a pre-trained audio processing model such as WaveNet can be used; video data can be extracted by frame extraction, processing each frame of the video as an image, and then merging the vectors, or a pre-trained video processing model such as 3D CNN can be used; social media data can be converted into sentiment vectors, or topic vectors can be extracted using algorithms such as LDA.

[0034] Optionally, in this embodiment, the upper bound is used to represent the maximum distance between the first target data and the data in the cluster. The distance is the Euclidean distance between vectors, and the upper bound is greater than or equal to the maximum distance among the actual distances between the target data and the data in the cluster.

[0035] Optionally, in this embodiment, the first lower bound is the minimum distance between the first target data and the data in the second target cluster, and the lower bound is less than or equal to the minimum actual distance between the first target data and the data in the target cluster. Optionally, the method in this embodiment can be applied to scenarios where searching is performed based on images. Figure 3 This is a schematic diagram illustrating the determination of image data according to an embodiment of this application. Feature extraction is performed on the original image set to obtain its corresponding feature vector set, which is then used to construct an index structure to obtain the target dataset. When the input object is an image, feature extraction is performed to convert it into feature vectors to obtain the first target data. Then, using the data determination method in this scheme, M clusters of the target dataset are generated. Based on the upper bound between the first target data and each cluster, the M clusters are sorted to obtain M sorted clusters. The minimum distance between the first target data and the data in the (M-1)th cluster is compared with the maximum distance between the first target data and the data in the Mth cluster to determine the first target cluster. Next, the distance between the first target data and each data point in the first target cluster is calculated, and data with a distance less than the first lower bound is deleted to obtain the third target cluster. Finally, the actual distance between the first target data and the data in the third target cluster is calculated, and the actual distance is compared to determine the farthest neighbor data of the first target data from the third target cluster, thus obtaining the second target data.

[0036] This application generates M clusters of the target dataset. The upper bound between the first target data and each cluster is determined using the distance between the first target data and the cluster center data, and the cluster radius. The clusters are then sorted based on these upper bounds. The first target cluster is determined by iteratively comparing the maximum distance between the first target data and data in the Mth cluster, and the minimum distance between the first target data and data in the first M-1 clusters, completing the first filtering step. Next, for the first target cluster, data whose distance to the first target data is less than the first lower bound are deleted, resulting in the third target cluster, completing the second filtering step. Finally, the data with the largest distance between the first target data and N data in the third target cluster is determined, resulting in the second target data, completing the final data search operation. Therefore, this method solves the problem of insufficient accuracy in finding data farthest from the target data in related technologies, achieving the effect of improving data filtering accuracy.

[0037] In an exemplary embodiment, generating M clusters of a target dataset includes: obtaining an original dataset, wherein the original dataset includes multiple unstructured data sets; performing vector transformation operations on the multiple unstructured data sets respectively to obtain the target dataset, wherein the vector transformation operations include converting the data features of each unstructured data set into feature vectors; statistically analyzing the value ranges of the data in the target dataset across multiple preset data dimensions, and dividing the value range of each preset data dimension into 2n... N There are 2 intervals, where N is a positive integer greater than or equal to 1; based on the target dataset, each of the above-mentioned preset data dimensions includes 2 N For each interval, perform clustering operations on the data in the target dataset to obtain M clusters.

[0038] Optionally, the type of unstructured data in the original dataset in this embodiment has been described above and will not be repeated here.

[0039] Optionally, the method for performing vector transformation on unstructured data to obtain vector representation data in this embodiment has been described above and will not be repeated here.

[0040] Optionally, in this embodiment, the clustering operation performed on the data in the target dataset to obtain M clusters uses clustering algorithms including but not limited to K-means algorithm, hierarchical clustering, DBSCAN algorithm, etc.

[0041] Optionally, taking the K-means clustering algorithm as an example, since the target dataset is very large and high-dimensional, it is first divided into 2 sub-databases under each preset data dimension. N Intervals, with 2 NThe data centers corresponding to each interval are used as the initial cluster centers, and K (positive integers greater than 1) clusters are generated from them. Then, the cluster centers are continuously updated and the corresponding clusters are regenerated until the cluster centers no longer change significantly or the preset number of iterations is reached, resulting in M ​​clusters.

[0042] This application transforms unstructured data into feature vectors and uses a clustering algorithm to generate M clusters of the target dataset, facilitating the manipulation of unstructured data and providing a foundation for the subsequent first coarse screening.

[0043] In an exemplary embodiment, before performing a sorting operation on the M clusters based on the upper bound between the first target data and each of the clusters to obtain M sorted clusters, the method further includes: for each of the clusters, performing the following operations to determine the upper bound between the first target data and each of the clusters: calculating the distance between the center data in the cluster and the boundary data in the cluster to obtain the radius of the cluster; calculating the distance between the first target data and the center data to obtain a first distance; and determining the sum of the first distance and the radius as the upper bound between the first target data and the cluster.

[0044] Optionally, in this embodiment, the radius of the cluster is obtained by the distance between the center data and the boundary data of the cluster. The specific operation steps are as follows: determine the center data of the cluster based on the cluster center used in determining the cluster, determine the boundary data of the cluster based on the interval corresponding to this cluster, calculate the distance (Euclidean distance) between the two, and determine the radius of the cluster.

[0045] This application utilizes cluster centers and cluster boundaries to determine the radius of a cluster, and combines the distance between the first target data and the cluster center to determine the upper bound between the first target data and the cluster, providing data support for subsequent comparisons. At the same time, this method replaces directly calculating the distance between the first target data and the data in the cluster and then further determining the maximum distance, reducing the computational overhead and saving time.

[0046] In an exemplary embodiment, determining a first target cluster based on a comparison between a first lower bound and a first upper bound includes: if the first lower bound is determined to be less than the first upper bound from the comparison result, calculating a lower bound between the first target data and the data in the Mth cluster to obtain a second lower bound, wherein the second lower bound represents the minimum distance between the first target data and the data in the Mth cluster; if the second lower bound is determined to be greater than the first lower bound, updating the first lower bound to the second lower bound, and storing the data in the Mth cluster into the second target cluster to obtain the first target cluster.

[0047] Optionally, in this embodiment, the step of calculating the lower bound between the first target data and the data in the Mth cluster to obtain the second lower bound uses the VA encoding of the data in the Mth cluster.

[0048] This application uses a loop to compare the upper bound of the Mth cluster with the maximum value of the lower bound determined from the M-1 clusters to determine whether to add the Mth cluster to the first target cluster. This achieves the first coarse screening, narrows down the cluster range where the second target data exists, and reduces the number of clusters needed to accurately calculate the distance from the first target data to the data in the cluster, thus saving overhead.

[0049] In an exemplary embodiment, when it is determined from the comparison result that the first lower bound is less than the first upper bound, calculating the lower bound between the first target data and the data in the Mth cluster to obtain a second lower bound includes: determining the VA codes of the plurality of data included in the Mth cluster, wherein the VA codes are used to represent the number sequence of the intervals that the data in the Mth cluster falls into on each preset data dimension; calculating the lower boundary between the first target data and the VA codes of the plurality of data in each preset data dimension to obtain a plurality of first lower boundaries, wherein the preset data dimension is used to represent the value range of the data in the target dataset, the first lower boundary is used to represent the minimum distance between the lower boundary of the first target data and the VA codes of the first data, and the first data is any one of the plurality of data; calculating the sum of the plurality of the first lower boundaries to obtain the second lower boundary of the first target data in each preset data dimension; comparing the plurality of the second lower boundaries of the first target data in the plurality of preset data dimensions to obtain a target lower boundary, wherein the target lower boundary is used to represent a plurality of The maximum value in the second lower boundary mentioned above; the target lower boundary is determined as the lower bound of the first target data to the data in the Mth cluster; wherein, in each preset data dimension, the lower boundary between the first target data and the VA codes of the multiple data is calculated to obtain multiple first lower boundaries, including: determining the components of the first target data in each preset data dimension to obtain multiple components, wherein the components are used to represent the proportion of the first target data in the preset data dimension; determining the data interval where the VA code of each data is located, wherein the data interval is the interval of the target dataset in each preset data dimension; comparing each component with the lower boundary of the data interval where the VA code of each data is located to obtain a first comparison result; comparing each component with the upper boundary of the data interval where the VA code of each data is located to obtain a second comparison result; determining the lower boundary between the first target data and the VA codes of the multiple data in each preset data dimension based on the first comparison result and the second comparison result to obtain multiple first lower boundaries.

[0050] Optionally, in this embodiment, the steps for determining the VA codes of multiple data S included in the Mth cluster are as follows: In the above steps, the range of values ​​[L] of the target dataset in each dimension is statistically analyzed. i u i ], and divide the value range of each dimension into 2 equal parts. b There are 10 intervals (b is a positive integer), and the boundaries of all intervals within all dimensions are stored in the boundary table T. BIn the diagram, all intervals in each dimension are sequentially numbered 0, 1, ..., 2. b -1, for a given data point X in the cluster, calculate the interval X falls into on each dimension, and concatenate the numbers of all intervals to form the VA code of X.

[0051]

[0052] Optionally, the step of determining the components of the first target data in each preset data dimension in this embodiment can be as follows: standardizing the first target data so that the features of each dimension have the same scale, which can be achieved by Z-score standardization (subtracting the mean and dividing by the standard deviation) or min-max standardization (scaling the features to between 0 and 1); then using feature extraction techniques, such as principal component analysis (PCA) or linear discriminant analysis (LDA), to determine the principal components or discriminant components of the first target data in each dimension; for PCA, calculating the eigenvectors and eigenvalues ​​of the covariance matrix, where the eigenvectors represent the principal component directions of the first target data in different dimensions, and the eigenvalues ​​represent the variance of each principal component; based on the magnitude of the eigenvalues, selecting the eigenvectors corresponding to the first few largest eigenvalues ​​as the principal components of the data, these components capture the main changes in the dataset; projecting the original first target data onto the principal component space, that is, transforming the original first target data through the eigenvector matrix to obtain the components of the first target data in each dimension.

[0053] Optionally, the method for calculating the lower bound between the first target data and the data in the Mth cluster in this embodiment is illustrated using the lower bound between the first target data and the data X in the Mth cluster as an example:

[0054] For each dimension i, let q i This represents the component of the first target data q in the i-th dimension, and q is calculated in each dimension. i The squared distance to each boundary is stored in the distance table T = {t} i，j In the array}, T is a two-dimensional array containing d×(2 b +1) squared distances, t i，j Indicates i-dimensional q i The squared distance to the j-th boundary; let b i，j b i，j+1 Let C represent the VA encoding of data X in the Mth cluster, respectively. X The VA encoding C of the first target data q in the i-th dimension to the data X in the M-th cluster is calculated based on the distance table T at the lower and upper boundaries of the i-th dimension as follows: X The lower bound distance lb i :

[0055] If bi，j <q i <bi ，j+1 If lbi = 0, then lbi = 0;

[0056] If q i ≤b i，j , then lb i =t i，j ;

[0057] If q i ≥b i，j+1 , then lb i =t i，j+1 ;

[0058] Summing the lower bounds across all dimensions yields the VA encoding C of the first target data q up to the data X in the Mth cluster. X The lower bound,

[0059] This application calculates the lower boundary of the VA code from the first target data to the data in the Mth cluster and obtains the maximum value of the lower boundary of the VA code from the first target data to the data in the Mth cluster by comparison. This determines the lower bound of the first target data to the data in the Mth cluster. Since the lower bound of the first target data to the data in the Mth cluster calculated in this step is not the exact actual distance, it lays the foundation for the subsequent second coarse screening. At the same time, using this method instead of directly calculating the distance between the first target data and the data in the cluster and then further determining the maximum distance reduces the computational overhead and saves time.

[0060] In one exemplary embodiment, determining the second target data from the first target cluster includes: calculating the distance between the first target data and each data in the first target cluster to obtain a plurality of second distances; deleting data with distances less than the first lower bound from the first target cluster according to the plurality of second distances to obtain a third target cluster; sorting the data in the third target cluster based on the distances between the first target data and N data in the third target cluster, where N is a natural number greater than or equal to 1; and traversing the N data according to the sorting result to determine the data with the largest distance from the first target data among the N data as the second target data.

[0061] Optionally, the distance obtained in calculating the distance between the first target data and each data in the first target cluster in this embodiment is not the actual distance between the first target data and each data in the first target cluster.

[0062] Optionally, the method for calculating the distance between the first target data and each data point in the first target cluster in this embodiment is illustrated using the distance between the first target data and data X in the first target cluster as an example:

[0063] For each dimension i, let q i This represents the component of the first target data q in the i-th dimension, and q is calculated in each dimension. i The squared distance to each boundary is stored in the distance table T = {t} i，j In the array}, T is a two-dimensional array containing d×(2 b +1) squared distances, t i，j Indicates i-dimensional q i The squared distance to the j-th boundary; let b i，j b i，j+1 Let C represent the VA encoding of data X in the Mth cluster, respectively. X The VA encoding C of the first target data q in the i-th dimension to the data X in the M-th cluster is calculated based on the distance table T at the lower and upper boundaries of the i-th dimension as follows: X Distance ub i :

[0064] If b i，j <q i <b i，j+1 Then ub i =max(t) i，j t i，j+1 );

[0065] If q i ≤b i，j Then ub i =t i，j+1 ;

[0066] If q i ≥b i，j+1 Then ub i =t i，j ;

[0067] Summing the distances across all dimensions yields the VA encoding C of the first target data q to the data X in the Mth cluster. X distance,

[0068] This application calculates the upper boundary of the VA encoding of the first target data to the data in the first target cluster, thereby obtaining the distance between the first target data and each data in the first target cluster. By comparing and deleting data in the first target cluster whose distance is less than the first lower boundary, a second coarse screening is achieved. Since the distance between the first target data and the data in the first target cluster calculated in this step is not the accurate actual distance, this method replaces directly calculating the distance between the first target data and the data in the cluster and then further determining the maximum distance, reducing the computational overhead. The second coarse screening further narrows the cluster range where the second target data is located, reducing the time required for "calculating the distance between the first target data and the data in the third target cluster" in subsequent steps.

[0069] In one exemplary embodiment, traversing N data points according to the sorting result and determining the data point with the largest distance from the first target data point as the second target data point includes: traversing N data points according to the sorting result, calculating the distance between the first target data point and the Nth data point to obtain a third distance; and determining the Nth data point as the second target data point if the third distance is greater than or equal to a fourth distance, wherein the fourth distance is the distance between the first target data point and the (N-1)th data point, and the (N-1)th data point is the data point adjacent to the Nth data point.

[0070] Optionally, in this embodiment, the distance between the first target data and the Nth data is calculated, and the resulting third distance is the actual distance between the first target data and the Nth data, which is the Euclidean distance.

[0071] For example, the first target data q and the i-th data x i The distance between them is defined as ||q,x i Let FN be the second target data, that is, let FN be the data with the maximum distance between it and the first target data. Calculate the distance between the first target data q and the i-th data x. N The distance between them gives the third distance ||q,x i ||, calculate the first target data q and the (i-1)th data x i-1 The distance between them gives the third distance ||q,x i-1 ||, if ||q, x i ||>||q,x i-1 ||, then let FN = x i The comparison continues until all ||q, x are compared. N-1 || and ||q, x N The size of ||.

[0072] This application uses a cyclical comparison of the actual distance between the first target data and the data in the third cluster to determine the final second target data, thus achieving the effect of determining the data that is furthest from the first target data. At the same time, the amount of data in the third cluster involved in this calculation step is less than the amount of data in the target dataset, resulting in lower computational overhead.

[0073] According to another embodiment of this application, a data determination apparatus is provided, comprising: a first generation module, configured to generate M clusters of a target dataset, wherein each cluster includes multiple data points, the data being vector representation data obtained by vector transformation of unstructured data, and M being a positive integer greater than 1; a first sorting module, configured to perform a sorting operation on the M clusters based on an upper bound between a first target data and each cluster, to obtain M sorted clusters, wherein the upper bound represents the maximum distance between the first target data and the data in the clusters, and the first target data is vector representation data obtained by vector transformation of first unstructured data; and a first determination module, configured to determine according to a first... The comparison between the lower bound and the upper bound determines the first target cluster, wherein the first lower bound is the minimum distance between the first target data and the data in the second target cluster, the data included in the second target cluster are data selected from the (M-1)th cluster, the first upper bound is the maximum distance between the first target data and the data in the Mth cluster, and the Mth cluster and the (M-1)th cluster are adjacent clusters among the M sorted clusters; the second determining module is used to determine the second target data from the first target cluster, wherein the second target data is the data in the first target cluster with the largest distance to the first target data, and the second target data is the data in the target dataset.

[0074] Optionally, the data in the target dataset in this embodiment is vector representation data obtained by performing vector transformation operations on multiple unstructured data. The types of unstructured data have been described above and will not be repeated here.

[0075] Optionally, the method for performing vector transformation on unstructured data to obtain vector representation data in this embodiment has been described above and will not be repeated here.

[0076] This application utilizes a data determination device to generate M clusters of a target dataset, determine an upper bound between the first target data and each cluster, and sort the clusters according to these upper bounds. The first target cluster is determined by iteratively comparing the maximum distance between the first target data and data in the Mth cluster and the minimum distance between the first target data and data in the first M-1 clusters, completing the first coarse screening. Finally, the second target data is determined from the first target cluster, completing the data search operation. Therefore, it solves the problem of insufficient accuracy in finding data farthest from the target data in related technologies.

[0077] It should be noted that, through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0078] This embodiment also provides a data determination device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0079] Figure 4 This is a structural block diagram of a data determination device according to an embodiment of this application, such as... Figure 4 As shown, the device includes:

[0080] The generation module 42 is used to generate M clusters of the target dataset, wherein each of the above clusters includes multiple data, which are vector representation data obtained by vector transformation of unstructured data, and M is a positive integer greater than 1.

[0081] The sorting module 44 is used to perform a sorting operation on M clusters based on the upper bound between the first target data and each of the above clusters, to obtain M sorted clusters. The upper bound is used to represent the maximum distance between the first target data and the data in the above clusters. The first target data is vector representation data obtained by vector transformation of the first unstructured data.

[0082] The first determining module 46 is used to determine the first target cluster according to the comparison result between the first lower bound and the first upper bound, wherein the first lower bound is the minimum distance between the first target data and the data in the second target cluster, the data included in the second target cluster is the data selected from the (M-1)th cluster, the first upper bound is the maximum distance between the first target data and the data in the Mth cluster, and the Mth cluster and the (M-1)th cluster are adjacent clusters among the M sorted clusters;

[0083] The second determining module 48 is used to determine the second target data from the first target cluster, wherein the second target data is the data in the first target cluster that has the largest distance from the first target data, and the second target data is the data in the target dataset.

[0084] In an exemplary embodiment, the generation module 42 includes: a first acquisition unit, configured to acquire an original dataset, wherein the original dataset includes multiple unstructured data; a first transformation module, configured to perform vector transformation operations on the multiple unstructured data respectively to obtain the target dataset, wherein the vector transformation operation includes converting the data features of each unstructured data into feature vectors; and a first partitioning unit, configured to statistically analyze the value range of the data in the target dataset across multiple preset data dimensions, and partition the value range of each preset data dimension into 2... N The interval is defined as follows: N is a positive integer greater than or equal to 1; the first clustering unit is used to perform clustering based on the target dataset, which includes 2 elements in each of the preset data dimensions. N For each interval, perform clustering operations on the data in the target dataset to obtain M clusters.

[0085] In an exemplary embodiment, the sorting module 44 includes: a first calculation unit, configured to calculate the distance between the center data and the boundary data in the cluster to obtain the radius of the cluster; a second calculation unit, configured to calculate the distance between the first target data and the center data to obtain a first distance; and a first determination unit, configured to determine the sum of the first distance and the radius as the upper bound between the first target data and the cluster.

[0086] In an exemplary embodiment, the first determining module 46 includes: a third calculation unit, configured to calculate a lower bound between the first target data and the data in the Mth cluster when the first lower bound is determined to be less than the first upper bound from the comparison result, to obtain a second lower bound, wherein the second lower bound represents the minimum distance between the first target data and the data in the Mth cluster; and a first updating unit, configured to update the first lower bound to the second lower bound when the second lower bound is determined to be greater than the first lower bound, and to store the data in the Mth cluster into the second target cluster to obtain the first target cluster.

[0087] In an exemplary embodiment, the first determining module 46 further includes: a second determining unit, configured to determine the VA codes of the plurality of data included in the Mth cluster, wherein the VA codes represent the number sequence of the intervals into which the data in the Mth cluster falls in each preset data dimension; a fourth calculating unit, configured to calculate the lower boundary between the first target data and the VA codes of the plurality of data in each preset data dimension to obtain a plurality of first lower boundaries, wherein the preset data dimension represents the value range of the data in the target dataset, the first lower boundary represents the minimum distance between the lower boundaries of the first target data and the VA codes of the first data, and the first data is any one of the plurality of data; a fifth calculating unit, configured to calculate the sum of the plurality of the first lower boundaries to obtain the second lower boundary of the first target data in each preset data dimension; a first comparing unit, configured to compare the plurality of the second lower boundaries of the first target data in the plurality of preset data dimensions to obtain a target lower boundary, wherein the target lower boundary represents the maximum value among the plurality of the second lower boundaries; and a third determining unit. The unit is used to determine the lower boundary of the target as the lower bound of the first target data to the data in the Mth cluster; wherein, the fourth calculation unit further includes: a fourth determining subunit, used to determine the components of the first target data in each of the preset data dimensions, to obtain multiple components, wherein the components are used to represent the proportion of the first target data in the preset data dimension; a fifth determining subunit, used to determine the data interval where the VA code of each of the data is located, wherein the data interval is an interval among multiple data intervals of the target dataset in each of the preset data dimensions; a second comparison subunit, used to compare each of the components with the lower boundary of the data interval where the VA code of each of the data is located, to obtain a first comparison result; a third comparison subunit, used to compare each of the components with the upper boundary of the data interval where the VA code of each of the data is located, to obtain a second comparison result; and a fifth determining subunit, used to determine the lower boundary between the first target data and the VA codes of multiple of the data in each preset data dimension based on the first comparison result and the second comparison result, to obtain multiple first lower boundaries.

[0088] In an exemplary embodiment, the second determining module 48 includes: a sixth calculation unit, configured to calculate the distance between the first target data and each data in the first target cluster to obtain a plurality of second distances; a first deletion unit, configured to delete data with a distance less than the first lower bound from the first target cluster according to the plurality of the second distances to obtain a third target cluster; a first sorting unit, configured to sort the data in the third target cluster based on the distance between the first target data and N data in the third target cluster, wherein N is a natural number greater than or equal to 1; and a fifth determining unit, configured to traverse the N data according to the sorting result and determine the data with the largest distance from the first target data among the N data as the second target data.

[0089] In an exemplary embodiment, the second determining module 48 further includes: a seventh calculation unit, configured to traverse N of the data according to the sorting result, calculate the distance between the first target data and the Nth data, and obtain a third distance; and a sixth determining unit, configured to determine the Nth data as the second target data when the third distance is greater than or equal to the fourth distance, wherein the fourth distance is the distance between the first target data and the (N-1)th data, and the (N-1)th data is the data adjacent to the Nth data.

[0090] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0091] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when run. In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0092] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0093] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0094] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0095] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0096] The embodiments described herein also provide a computer program that includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in any of the above method embodiments.

[0097] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0098] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0099] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method of determining data, characterized by, The method comprises the following steps: generating M clusters of a target data set, wherein each of the clusters comprises a plurality of data, the data being vector representation data obtained by vector conversion of unstructured data, M being a positive integer greater than 1, and the unstructured data comprising text data, image data, video data and audio data; performing a sorting operation on the M clusters based on an upper bound between a first target data and each of the clusters, to obtain M sorted clusters, wherein the upper bound represents a maximum distance between the first target data and data in the cluster, and the first target data is vector representation data obtained by vector conversion of first unstructured data; determining a first target cluster according to a comparison result between a first lower bound and a first upper bound, wherein the first lower bound is a minimum distance between the first target data and data in a second target cluster, the data in the second target cluster being data selected from an M-1th cluster, the first upper bound is a maximum distance between the first target data and data in an Mth cluster, and the Mth cluster and the M-1th cluster are adjacent clusters in the M sorted clusters; determining a second target data from the first target cluster, wherein the second target data is data in the first target cluster having a maximum distance from the first target data, and the second target data is data in the target data set.

2. The method of claim 1, wherein, Generating M clusters of a target data set comprises the following steps: obtaining an original data set, wherein the original data set comprises a plurality of unstructured data; performing a vector conversion operation on each of the unstructured data to obtain the target data set, wherein the vector conversion operation comprises converting data features of each of the unstructured data into a feature vector; statistically determine value ranges of data in the target data set on a plurality of preset data dimensions, and divide each value range on the preset data dimensions into 2 N N intervals, where N is a positive integer greater than or equal to 1. perform a clustering operation on data in the target data set based on 2 N intervals included in each of the preset data dimensions of the target data set, to obtain M clusters.

3. The method of claim 1, wherein, Before performing a sorting operation on the M clusters based on an upper bound between a first target data and each of the clusters to obtain M sorted clusters, the method further comprises the following steps: For each of the clusters, the following operations are performed to determine the upper bound between the first target data and each of the clusters: calculating a distance between center data in the cluster and boundary data in the cluster to obtain a radius of the cluster; calculating a distance between the first target data and the center data to obtain a first distance; determining a sum value between the first distance and the radius as the upper bound between the first target data and the cluster.

4. The method of claim 1, wherein, Determining a first target cluster according to a comparison result between a first lower bound and a first upper bound comprises the following steps: in a case where the first lower bound is determined to be smaller than the first upper bound from the comparison result, calculating a lower bound between the first target data and data in the Mth cluster to obtain a second lower bound, wherein the second lower bound represents a minimum distance between the first target data and the data in the Mth cluster. In a case where it is determined that the second lower bound is greater than the first lower bound, the first lower bound is updated to the second lower bound, and data in the Mth cluster is stored into the second target cluster, obtaining the first target cluster.

5. The method of claim 4, wherein, In a case where it is determined from the comparison result that the first lower bound is less than the first upper bound, a lower bound between the first target data and data in the Mth cluster is calculated, obtaining a second lower bound, including: determining VA encodings of a plurality of the data included in the Mth cluster, wherein the VA encodings are used to represent a number sequence of intervals into which data in the Mth cluster falls in each preset data dimension; In each preset data dimension, a lower boundary between the first target data and the VA encodings of a plurality of the data is calculated, obtaining a plurality of first lower boundaries, wherein the preset data dimension is used to represent a value range of data in the target data set, and the first lower boundary is used to represent a minimum distance of a lower boundary between the first target data and a first data, the first data being any one of a plurality of the data; a sum value of a plurality of the first lower boundaries is calculated, obtaining a second lower boundary of the first target data in each of the preset data dimensions; a target lower boundary is obtained by comparing a plurality of the second lower boundaries of the first target data in a plurality of the preset data dimensions, wherein the target lower boundary is used to represent a maximum value in a plurality of the second lower boundaries; the target lower boundary is determined as a lower bound of the first target data to data in the Mth cluster; wherein, in each preset data dimension, a lower boundary between the first target data and the VA encodings of a plurality of the data is calculated, obtaining a plurality of first lower boundaries, including: determining components of the first target data in each of the preset data dimensions, obtaining a plurality of the components, wherein the component is used to represent a proportion of the first target data in the preset data dimension; determining a data interval in which the VA encoding of each of the data is located, wherein the data interval is an interval in a plurality of data intervals of the target data set in each of the preset data dimensions; comparing each of the components with a lower boundary of the data interval in which the VA encoding of each of the data is located, obtaining a first comparison result; comparing each of the components with an upper boundary of the data interval in which the VA encoding of each of the data is located, obtaining a second comparison result; determining a lower boundary between the first target data and the VA encodings of a plurality of the data in each preset data dimension based on the first comparison result and the second comparison result, obtaining a plurality of the first lower boundaries.

6. The method of claim 1, wherein, determining a second target data from the first target cluster, including: calculating a distance between the first target data and each data in the first target cluster, obtaining a plurality of second distances; in accordance with a plurality of the second distances, deleting data with a distance less than the first lower bound from the first target cluster, obtaining a third target cluster; sort the data in the third target cluster based on distances between the first target data and N data in the third target cluster, where N is a natural number greater than or equal to 1; determine, as the second target data, data with the largest distance to the first target data among N data traversed according to the sorting result.

7. The method of claim 6, wherein, determine, as the second target data, data with the largest distance to the first target data among N data traversed according to the sorting result, including: traverse N data according to the sorting result, and calculate a third distance between the first target data and the Nth data; determine, as the second target data, the Nth data in the case that the third distance is greater than or equal to a fourth distance, where the fourth distance is a distance between the first target data and an (N-1)th data adjacent to the Nth data.

8. A data determining apparatus characterized by comprising: including: a first generation module configured to generate M clusters of a target data set, where each cluster includes a plurality of data, the data being vector representation data obtained by vector conversion of unstructured data, M being a positive integer greater than 1, and the unstructured data including text data, image data, video data, and audio data; a first sorting module configured to perform a sorting operation on the M clusters based on an upper bound between a first target data and each cluster, to obtain M sorted clusters, where the upper bound is used to represent a maximum distance between the first target data and data in the cluster, and the first target data is vector representation data obtained by vector conversion of first unstructured data; a first determination module configured to determine a first target cluster according to a comparison result between a first lower bound and a first upper bound, where the first lower bound is a minimum distance between the first target data and data in a second target cluster, the data in the second target cluster being data selected from an (M-1)th cluster, the first upper bound is a maximum distance between the first target data and data in an Mth cluster, and the Mth cluster and the (M-1)th cluster are adjacent clusters in the M sorted clusters; a second determination module configured to determine a second target data from the first target cluster, where the second target data is data with the largest distance to the first target data in the first target cluster, and the second target data is data in the target data set.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, where the computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 7.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 7.

Images