Data index construction method and device, equipment, storage medium and program product
By using the index generation model in data index construction to extract and classify the data sets and determine the target index number, the problem of inefficient manual index construction is solved, and more efficient and accurate data index construction and retrieval is achieved.
Patent Information
- Application Number
- CN202411658887.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-19
- Publication Date
- 2025-03-11
AI Technical Summary
Manually constructing data indexing and classification systems is inefficient, resulting in low data retrieval efficiency.
By obtaining a preset number of data sets from the pending data warehouse and inputting it into the index to generate a model, using data characteristics and label characteristics for classification, the target index number with the most matching times is determined as the index of the pending data warehouse.
It improves the construction efficiency and accuracy of data indexes, reduces the time to build a complete tree index, and improves the efficiency of data retrieval.
Smart Images
Figure CN119597860A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of text recognition, and in particular, relates to a method, device, equipment, storage medium and program product for constructing a data index. Background Art
[0002] In the modern information society, data is growing explosively. In order to obtain effective statistical data, data needs to be classified, and then indexes are manually constructed for the classified data according to the specific content and data structure of the classified data. However, the method of manually constructing indexes and classification systems will lead to low overall work efficiency. Summary of the invention
[0003] The embodiments of the present application provide a method, device, equipment, storage medium and program product for constructing a data index, which can improve the efficiency of constructing a data index.
[0004] In a first aspect, an embodiment of the present application provides a method for constructing a data index, including:
[0005] Obtain a preset number of data sets from the data warehouse to be processed;
[0006] Inputting the preset number of data sets into an index generation model to obtain an index number corresponding to each data set output by the index generation model, wherein the index generation model is used to search for an index matching the data set in a preset index based on data features of the data set;
[0007] Determine the target index number with the largest number among the index numbers corresponding to each data set;
[0008] Obtain the target index corresponding to the target index number, and use the target index as the index of the data warehouse to be processed.
[0009] In a possible implementation, the data set includes a data element and a preset label corresponding to the data element; and inputting the preset number of data sets into an index generation model to obtain an index number corresponding to each data set output by the index generation model includes:
[0010] Inputting the preset number of data sets into the index generation model;
[0011] For each data set, the index generation model is used to perform feature extraction on the data element and the preset label corresponding to the data element to obtain data features and label features;
[0012] The data set is classified by the index generation model using the data features and the label features, a preset index corresponding to the data set is determined, and an index number output by the index generation model is obtained.
[0013] In a possible implementation, the classifying the data set by using the data feature and the label feature through the index generation model to determine the preset index corresponding to the data set includes:
[0014] Mosaicing the data features and the label features through the index generation model to obtain mosaic features;
[0015] Performing a reshaping operation on the mosaic feature by using the index generation model to obtain a reshaping feature;
[0016] The data set is classified based on the reshaping feature by using the index generation model to determine the preset index corresponding to the data set.
[0017] In a possible implementation, before inputting the preset number of data sets into the index generation model, the method further includes:
[0018] Get the sample data set in the sample data warehouse;
[0019] Obtaining a preset label corresponding to each data element in the sample data set;
[0020] The sample data set and the preset label are used as inputs of a generative adversarial network, and the index corresponding to the sample data set is used as label data to train the generative adversarial network to obtain the index generation model.
[0021] In a possible implementation manner, after obtaining the preset label corresponding to each data element in the sample data set, the method further includes:
[0022] Performing a vectorization operation on the preset label to obtain a label vector;
[0023] Performing a clustering operation on the label vector to obtain multiple clusters;
[0024] For each cluster, obtaining the center vector of the cluster;
[0025] Using the center vector as label class information of the cluster;
[0026] The sample data set and the preset label are used as inputs of a generative adversarial network, the index corresponding to the sample data set is used as label data, and the generative adversarial network is trained to obtain the index generation model, including:
[0027] The sample data set and the label class information are used as inputs of the generative adversarial network, and the index corresponding to the sample data set is used as label data to train the generative adversarial network to obtain the index generation model.
[0028] In a possible implementation, the generative adversarial network includes a generator model and a discriminator model; the sample data set and the label class information are used as inputs of the generative adversarial network, the index corresponding to the sample data set is used as label data, and the generative adversarial network is trained to obtain the index generation model, including:
[0029] Input the sample data set and the preset label into the generator model to obtain the sample index number output by the generator model;
[0030] Obtaining the sample index corresponding to the sample index number;
[0031] Inputting the sample index and the preset label into the discriminator model to obtain an output result of the discriminator model, wherein the output result includes a probability that the sample index is a true index corresponding to the sample data set;
[0032] The generator model is adjusted using the output result to obtain the index generation model.
[0033] In a second aspect, an embodiment of the present application provides a device for constructing a data index, including:
[0034] An acquisition module is used to acquire a preset number of data sets from the data warehouse to be processed;
[0035] A generation module, used to input the preset number of data sets into an index generation model to obtain an index number corresponding to each data set output by the index generation model, wherein the index generation model is used to search for an index matching the data set in a preset index based on data features of the data set;
[0036] A determination module, used for determining the target index number with the largest number among the index numbers corresponding to each data set;
[0037] The acquisition module is further used to acquire the target index corresponding to the target index number, and use the target index as the index of the data warehouse to be processed.
[0038] In a third aspect, an embodiment of the present application provides an electronic device, the device comprising: a processor and a memory storing computer program instructions;
[0039] When the processor executes the computer program instructions, the method for constructing a data index as described in any one of the first aspects is implemented.
[0040] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon, and when the computer program instructions are executed by a processor, the method for constructing a data index as described in any one of the first aspects is implemented.
[0041] In a fifth aspect, an embodiment of the present application provides a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the method for constructing a data index as described in any one of the first aspects.
[0042] A method, device, equipment, storage medium and program product for constructing a data index according to an embodiment of the present application obtains a preset number of data sets from a data warehouse to be processed, and inputs the preset number of data sets into an index generation model, wherein the index generation model can search for the index that matches each data set in a preset index based on the data features of each data set, thereby outputting the index number corresponding to each data set. Then, based on the number of each index number, the target index number with the largest number is determined, thereby using the target index corresponding to the target index number as the index of the data warehouse to be processed. Among the matched indexes, the target index with the largest number of matches is used as the index of the data warehouse to be processed, thereby improving the accuracy of the index. In this way, the index generation model can be used to quickly match the index corresponding to each data set, thereby improving the efficiency of index construction. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0044] Figure 1 It is a flowchart of a method for constructing a data index provided in an embodiment of the present application;
[0045] Figure 2 It is a flowchart of an index generation model training method provided in an embodiment of the present application;
[0046] Figure 3 It is a flowchart of another index generation model training method provided in an embodiment of the present application;
[0047] Figure 4 is an exemplary schematic diagram of a generated network structure provided in an embodiment of the present application;
[0048] Figure 5 is an exemplary schematic diagram of a discriminant network structure provided in an embodiment of the present application;
[0049] Figure 6It is a flowchart of another index generation model training method provided in an embodiment of the present application;
[0050] Figure 7 It is a structural schematic diagram of a device for constructing a data index provided in an embodiment of the present application;
[0051] Figure 8 It is a structural schematic diagram of an electronic device provided in yet another embodiment of the present application. DETAILED DESCRIPTION
[0052] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.
[0053] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0054] With the development of modern information society, data is growing explosively. In order to effectively process data statistically, it is necessary to classify the data and then build indexes for the classified data, thereby improving data retrieval efficiency.
[0055] To improve the efficiency of data retrieval, there are currently three solutions:
[0056] The first solution is to build an index table by default. After receiving a search task, the electronic device can search for data according to the index table.
[0057] The second solution is to receive the search statement through a pre-trained document search model, use the document search model to parse the semantics of the search statement to match keywords with the same semantics, and then use the preset document index to find the index information of the keyword.
[0058] The third solution is to maintain the database through the model to ensure that the database can run normal data retrieval tasks.
[0059] The above three solutions do not involve the method of building a tree index. Since the classified data has different data content and data structure, it is necessary to manually build a tree index for the classified data. In addition, the classified data still has some commonalities, so manually building a tree index will result in duplication of work, thereby reducing overall work efficiency.
[0060] In order to solve the problems existing in the prior art, the embodiments of the present application provide a method, device, equipment, storage medium and program product for constructing a data index. Figure 1 A method for constructing a data index provided in an embodiment of the present application is introduced, and the method is applied to an electronic device, such as Figure 1 As shown, the method includes:
[0061] S101. Obtain a preset number of data sets from a data warehouse to be processed.
[0062] The data warehouse to be processed refers to a data warehouse for which a tree index needs to be constructed. Each data warehouse to be processed includes multiple data sets. The category distribution of data elements in each data set in the same data warehouse to be processed is similar.
[0063] Specifically, when the electronic device labels and classifies the data elements of the data set in the same data warehouse to be processed according to the preset classification rules, the data categories corresponding to each data set are similar, and the number of data elements corresponding to each data category is similar.
[0064] In one example, the warehouse for data to be processed is used to store the communication service data of a province. The communication service data of a province may include data categories such as node data, content data, and descriptive name data, and the number of data elements corresponding to each data category is similar.
[0065] S102: Input a preset number of data sets into an index generation model to obtain an index number corresponding to each data set output by the index generation model.
[0066] The index generation model is used to search for an index matching the data set in a preset index based on the data features of the data set. The index generation model may be a Generative Adversarial Networks (GAN) model.
[0067] In order to improve the accuracy of the prediction results of the index generation model, the data set can be preprocessed. On the one hand, the data set also includes a preset label corresponding to each data element. The electronic device can use a preset clustering algorithm to cluster the preset labels before inputting a preset number of data sets into the index generation model in the above S102 and obtaining the index number corresponding to each data set output by the index generation model, and obtain the label class information corresponding to the data set, use the label class information as the label of the data set, and input the label class information and the data set into the index generation model. The specific method of clustering the preset labels is described in detail in the subsequent embodiments.
[0068] On the other hand, the electronic device can obtain statistical information of the data set according to a preset statistical method and use the statistical information as input to the index generation model. In this way, the amount of data processed by the index generation model can be reduced, and the statistical information can directly reflect the data characteristics of the data set, thereby improving the processing efficiency of the index generation model.
[0069] S103: Determine the target index number with the largest number among the index numbers corresponding to each data set.
[0070] It can be understood that the index generation model outputs the index number corresponding to each data set, and the electronic device determines the target index number with the most occurrences in the prediction results output by the index generation model. Among them, the tree index corresponding to the target index number can be applicable to most data sets in the data warehouse to be processed.
[0071] S104. Obtain a target index corresponding to the target index number, and use the target index as an index of the data warehouse to be processed.
[0072] The preset index is a preset initial tree index, and the subsequent electronic device can perform operations such as searching, inserting and deleting on the initial tree index based on the initial tree index and the data in the data warehouse to construct a complete tree index corresponding to the data warehouse. In this way, based on the initial tree index determined by the index generation model, the initial tree index and the complete tree index are relatively close, thereby reducing the time for constructing the complete tree index and improving the speed of constructing the tree index.
[0073] By adopting the above method, a preset number of data sets are obtained from the data warehouse to be processed, and the preset number of data sets are input into the index generation model, wherein the index generation model can search for the index matching each data set in the preset index based on the data characteristics of each data set, thereby outputting the index number corresponding to each data set. Then, according to the number of each index number, the target index number with the largest number is determined, so that the target index corresponding to the target index number is used as the index of the data warehouse to be processed. Among the matched indexes, the target index with the largest number of matches is used as the index of the data warehouse to be processed, thereby improving the accuracy of the index. In this way, the index generation model can be used to quickly match the index corresponding to each data set, thereby improving the efficiency of index construction.
[0074] The following combination Figure 2 Introduce the training process of the above index generation model, such as Figure 2 As shown, the method includes:
[0075] S201: Obtain a sample data set in a sample data warehouse.
[0076] The sample data warehouse may be a data warehouse for which a tree index has been constructed.
[0077] S202: Obtain a preset label corresponding to each data element in the sample data set.
[0078] The preset label is set based on experience. For example, the preset label can be node data, content data, or descriptive name data.
[0079] S203: Using the sample data set and the preset label as input of the generative adversarial network, using the index corresponding to the sample data set as label data, training the generative adversarial network, and obtaining an index generation model.
[0080] The method provided in the embodiment of the present application is adopted, the sample data set and the preset label are used as the input of the generative adversarial network, and the index corresponding to the sample data set is used as the label data to train the generative adversarial network. Among them, the index corresponding to the sample data set is the above-mentioned preset index, so the trained index generation model can be used to determine the corresponding relationship between the data set and the preset index, thereby improving the accuracy of the prediction results of the index generation model.
[0081] After obtaining the preset label corresponding to each data element in the sample data set in S202, the electronic device may further process the preset label to ensure that the processed label is in a form that better meets the requirements of the preset index. Specifically, the method further includes:
[0082] Perform vectorization operation on the preset labels to obtain label vectors. Perform clustering operation on the label vectors to obtain multiple clusters. For each cluster, obtain the center vector of the cluster. Use the center vector as the label class information of the cluster.
[0083] Among them, the electronic device can use the K-means clustering algorithm to perform clustering operations on the label vectors, calculate the Euclidean distance between the label vectors, and the electronic device determines the cluster according to the calculated Euclidean distance and the preset K value. The K value is related to the parameter t of the tree index, and the parameter t is used to indicate the maximum number of child nodes that all other nodes can have except the root node in the tree index. As an example, the K value can be 10 times the value of the parameter t.
[0084] It can be understood that after the label vectors are clustered, the label vectors in a cluster belong to the same category label. Therefore, the center vector of the cluster is used as the label class information of the preset label included in the cluster. The index generated by the subsequent index generation model based on the label class information includes the label class information, and the data of each preset label in the cluster can be retrieved based on the label class information, thereby shortening the retrieval path.
[0085] Accordingly, the sample data set and the preset label are used as the input of the generative adversarial network, and the index corresponding to the sample data set is used as the label data. The generative adversarial network is trained to obtain the index generation model, including:
[0086] The sample data set and label class information are used as the input of the generative adversarial network, and the index corresponding to the sample data set is used as the label data. The generative adversarial network is trained to obtain an index generation model.
[0087] By using the method provided in the embodiment of the present application and clustering the label vectors using a clustering algorithm, a cluster corresponding to the sample data set can be obtained. The center vector is used as the label class information corresponding to the cluster, and the label class information can cover more preset labels. The subsequent index generation model predicts the index based on the label class information, and when the index is used for retrieval, the retrieval path can be shortened.
[0088] It should be noted that the generative adversarial network includes a generator model and a discriminator model. The parameters of the generator model can be adjusted through the discriminator model to improve the accuracy of the output results of the generator model. Figure 3 As shown, S203, taking the sample data set and the preset label as the input of the generative adversarial network, taking the index corresponding to the sample data set as the label data, training the generative adversarial network, and obtaining the index generation model, can be specifically implemented as follows:
[0089] S2031. Input the sample data set and preset labels into the generator model to obtain the sample index number output by the generator model.
[0090] S2032. Obtain the sample index corresponding to the sample index number.
[0091] The electronic device stores a correspondence between the index number and the tree index, and the electronic device can search for the corresponding tree index based on the sample index number and use the tree index as the sample index.
[0092] S2033. Input the sample index and preset label into the discriminator model to obtain the output result of the discriminator model.
[0093] The output result includes the probability that the sample index is the real index corresponding to the sample data set.
[0094] S2034. Use the output results to adjust the generator model to obtain an index generation model.
[0095] Specifically, when the probability that the sample index is the real index corresponding to the sample data set is lower than the preset threshold, it means that the discriminator model can distinguish the real index from the tree index output by the generator model. That is, the output result of the generator model is not accurate enough, and the parameters of the generator model need to be adjusted until the probability that the sample index in the output result of the discriminator model is the real index corresponding to the sample data set is greater than or equal to the preset threshold, indicating that the discriminator model cannot distinguish the sample index output by the generator model from the real index, that is, the generator can accurately generate the index corresponding to the data set, so that a more accurate generator model can be obtained.
[0096] Using the method provided in the embodiment of the present application, based on the sample index number output by the generator model, the sample index corresponding to the sample index number can be input into the discriminator model, and the discriminator model can determine the probability that the input index is a true index. When the discriminator can identify that the input sample index is generated by the generator model, the generator model is adjusted until the discriminator cannot identify the difference between the sample index corresponding to the sample index number output by the generator model and the true index. In this way, the accuracy of the prediction results of the index generation model can be improved by adjusting the generator model using the discriminator model.
[0097] It should be noted that the above-mentioned generative adversarial network includes a generative network and a discriminative network. The electronic device needs to be trained separately for the generative network and the discriminative network to obtain a generator model and a discriminative model. S2031, inputting the sample data set and the preset label into the generator model, before obtaining the sample index number output by the generator model, the method also includes:
[0098] Step 1: Input the historical sample data set and the preset label corresponding to the historical sample data set into the generation network to obtain the first sample index number output by the generation network.
[0099] The historical sample data set is a data set for which a tree index has been constructed. Specifically, the method for obtaining the preset label is referred to the relevant description in the above embodiment, which will not be repeated here.
[0100] Step 2: Calculate the first loss function of the generated network using the index corresponding to the historical sample data set and the first sample index number.
[0101] Specifically, the first loss function of the generated network can be calculated according to the following formula:
[0102] L(y,p)=[(y)*log(p)+(1-y)*log(1-p)]
[0103] Among them, y is used to indicate whether the output of the generated network is a real index. If the first sample index number of the generated network output is consistent with the real index number, the value of y is 1. If the first sample index number of the generated network output is inconsistent with the real index number, the value of y is 0. P represents the probability that y takes the value of 1.
[0104] Step 3: Adjust the generation network based on the first loss function to obtain the generator model.
[0105] Specifically, when the electronic device adjusts the generated network, the parameters of the generated network may be adjusted using a gradient descent function. For example, the gradient descent function may be an adam function or an RMSprop function. In the embodiment of the present application, the gradient descent function is an adam function.
[0106] The following combination Figure 4 Introduce the structure of the generated network, such as Figure 4 As shown in the figure, the generated network includes a fully connected layer (Dense), a cascade layer (Concatenate), an embedding layer (Embedding), a reshaping layer (Reshape), a long short-term memory layer (Long Short-Term Memory, LSTM), a time distribution layer (TimeDistributed) and an activation function layer (Softmax).
[0107] The input of the generating network is random noise and label. The random noise can be the above data set, and the label can be the preset label or label class information in the above embodiment. The output of the Softmax layer is the index number.
[0108] Specifically, the Dense layer is used to extract features from the input random noise and labels, the Embedding layer is used to embed the label features into the space of the noise features, and then the label features and the noise features are cascaded through the Concatenate layer to obtain cascade features. The Dense layer then extracts features from the cascade features, and reshapes the extracted features through the Reshape layer to obtain reshaped features, where the reshaped features are in a data format that can be processed by the LSTM layer. The LSTM layer processes the reshaped features into low-dimensional vectors, and then adjusts the number of steps of the low-dimensional vectors through the TimeDistributed layer. The Softmax layer can be used to determine the degree of match between the low-dimensional vector after the number of steps is adjusted and the preset index, thereby outputting the index number of the matching preset index.
[0109] Before the above S2033, inputting the sample index and the preset label into the discriminator model to obtain the output result of the discriminator model, the method further includes:
[0110] Step A: Obtain a first sample index corresponding to a first sample index number.
[0111] Step B: input the first sample index and the preset label corresponding to the historical sample data set into the discriminant network to obtain the output result of the discriminant network, and the output result includes the probability that the first sample index is the index corresponding to the historical sample data set.
[0112] Step C: Calculate the second loss function of the discriminant network using the output result and the index corresponding to the historical sample data set.
[0113] Step D: Adjust the discriminant network based on the second loss function to obtain a discriminator model.
[0114] The following combination Figure 5 Introduce the structure of the discriminant network, such as Figure 5 As shown in the figure, the discriminant network includes a fully connected layer (Dense), a cascade layer (Concatenate), an embedding layer (Embedding) and a dimensionality reduction layer (Dropout).
[0115] The input of the discriminant network is a tree index and a label. The tree index can be a real tree index or a preset index corresponding to the index number output by the above generation network. The label can be a preset label or label type information in the above embodiment.
[0116] Specifically, the Dense layer is used to extract features from the input tree index and label, the Embedding layer is used to embed the label feature into the noise feature space, and then the label feature and index feature are cascaded through the Concatenate layer to obtain the cascade feature. Then the Dense layer extracts the cascade feature, and the Dropout layer performs dimensionality reduction on the extracted features to obtain the reduced dimensionality feature. Finally, the Dense layer classifies the reduced dimensionality feature to obtain the judgment result.
[0117] Based on the index generation model obtained through the above training, the index generation model is used to construct the index of the data warehouse to be processed. Figure 6 As shown, for the above S102, a preset number of data sets are input into the index generation model to obtain the index number corresponding to each data set output by the index generation model, which can be specifically implemented as follows:
[0118] S1021. Input a preset number of data sets into an index generation model.
[0119] It should be noted that, in actual use, the index generation model includes the generator model described above, and the index number corresponding to each data set is obtained through the generator model.
[0120] S1022. For each data set, use the index generation model to perform feature extraction on the data element and the preset label corresponding to the data element to obtain data features and label features.
[0121] S1023. Classify the data set using data features and label features through the index generation model, determine the preset index corresponding to the data set, and obtain the index number output by the index generation model.
[0122] Specifically, the index generation model embeds the label feature into the space of the data feature, and cascades the label feature and the data feature to obtain the cascade feature. By reshaping and classifying the cascade feature, the preset index corresponding to the data set is determined, thereby determining the index number corresponding to the preset index. The method for predicting the index number by the specific generator model refers to the relevant description in the above embodiment, which will not be repeated here.
[0123] By adopting the method provided in the embodiment of the present application, the data features corresponding to the data elements and the label features corresponding to the preset labels can be obtained by extracting features from the data elements in the data set and the preset labels. In this way, based on the data features and label features, the preset index corresponding to the data set can be determined based on the data features and label features through the index generation model, and the accuracy of the prediction results can be improved by combining the label features and data features.
[0124] Specifically, for the above S1023, the data set is classified by using the data features and label features through the index generation model, the preset index corresponding to the data set is determined, and the index number output by the index generation model is obtained, which can be implemented as follows:
[0125] The index generation model is used to mosaic the data features and the label features to obtain mosaic features. The index generation model is used to reshape the mosaic features to obtain reshaped features. The index generation model is used to classify the data set based on the reshaped features to determine the preset index corresponding to the data set.
[0126] Specifically, the process of the above index generation model processing the data set and labels refers to the process of the generation network processing random noise and labels, which will not be repeated here.
[0127] In this way, by mosaicking the data features and the label features, the label features corresponding to each data feature can be obtained. Then, through the reshaping operation, the index generation model can use the reshaping features to classify the mosaic features, determine the preset index applicable to the reshaped mosaic features, and thus obtain the index number of the preset index.
[0128] Based on the same concept, the present application embodiment provides a device for constructing a data index, such as Figure 7 As shown, the device comprises:
[0129] The acquisition module 701 is used to acquire a preset number of data sets from the data warehouse to be processed;
[0130] A generation module 702 is used to input the preset number of data sets into an index generation model to obtain an index number corresponding to each data set output by the index generation model, wherein the index generation model is used to search for an index matching the data set in a preset index based on data features of the data set;
[0131] A determination module 703 is used to determine the target index number with the largest number among the index numbers corresponding to each data set;
[0132] The acquisition module 701 is further used to acquire the target index corresponding to the target index number, and use the target index as the index of the data warehouse to be processed.
[0133] In a possible implementation, the data set includes data elements and preset labels corresponding to the data elements; the generating module 702 is specifically configured to:
[0134] Inputting the preset number of data sets into the index generation model;
[0135] For each data set, the index generation model is used to perform feature extraction on the data element and the preset label corresponding to the data element to obtain data features and label features;
[0136] The data set is classified by the index generation model using the data features and the label features, a preset index corresponding to the data set is determined, and an index number output by the index generation model is obtained.
[0137] In a possible implementation, the generating module 702 is specifically configured to:
[0138] Mosaicing the data features and the label features through the index generation model to obtain mosaic features;
[0139] Performing a reshaping operation on the mosaic feature by using the index generation model to obtain a reshaping feature;
[0140] The data set is classified based on the reshaping feature by using the index generation model to determine the preset index corresponding to the data set.
[0141] In a possible implementation, the device further includes:
[0142] The acquisition module 701 is also used to acquire a sample data set in a sample data warehouse;
[0143] The acquisition module 701 is further used to acquire a preset label corresponding to each data element in the sample data set;
[0144] The training module is used to use the sample data set and the preset label as input of the generative adversarial network, use the index corresponding to the sample data set as label data, train the generative adversarial network, and obtain the index generation model.
[0145] In a possible implementation, the device further includes:
[0146] A vector module, used for performing a vectorization operation on the preset label to obtain a label vector;
[0147] A clustering module, used for performing a clustering operation on the label vector to obtain multiple clusters;
[0148] The acquisition module 701 is further used to acquire the center vector of each cluster;
[0149] The determination module 703 is further configured to use the center vector as label class information of the cluster;
[0150] Training modules are specifically used for:
[0151] The sample data set and the label class information are used as inputs of the generative adversarial network, and the index corresponding to the sample data set is used as label data to train the generative adversarial network to obtain the index generation model.
[0152] In a possible implementation, the generative adversarial network includes a generator model and a discriminator model; a training module, specifically used for:
[0153] Input the sample data set and the preset label into the generator model to obtain the sample index number output by the generator model;
[0154] Obtaining the sample index corresponding to the sample index number;
[0155] Inputting the sample index and the preset label into the discriminator model to obtain an output result of the discriminator model, wherein the output result includes a probability that the sample index is a true index corresponding to the sample data set;
[0156] The generator model is adjusted using the output result to obtain the index generation model.
[0157] It should be noted that the device for constructing the data index is a device corresponding to the method for constructing the data index mentioned above, and all implementation methods in the above method embodiments are applicable to the embodiments of the device and can achieve the same technical effects.
[0158] Figure 8 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.
[0159] The electronic device may include a processor 801 and a memory 802 storing computer program instructions.
[0160] Specifically, the processor 801 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.
[0161] The memory 802 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 802 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 802 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 802 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 802 is a non-volatile solid-state memory.
[0162] The memory may include read-only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical or other physical / tangible memory storage devices. Thus, typically, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present disclosure.
[0163] The processor 801 implements any one of the file access methods in the above embodiments by reading and executing computer program instructions stored in the memory 802 .
[0164] In one example, the electronic device may further include a communication interface 803 and a bus 804. Figure 8 As shown, the processor 801, the memory 802, and the communication interface 803 are connected via a bus 804 and communicate with each other.
[0165] The communication interface 803 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.
[0166] Bus 804 includes hardware, software or both, and the parts of electronic equipment are coupled to each other.For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 804 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.
[0167] In addition, in combination with the data index construction method in the above embodiment, the embodiment of the present application can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the data index construction methods in the above embodiment is implemented.
[0168] In combination with the data index construction method in the above embodiments, an embodiment of the present application may provide a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes any data index construction method in the above embodiments.
[0169] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.
[0170] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.
[0171] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.
[0172] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.
[0173] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.
Claims
1. A method for constructing a data index, characterized in that: include: Obtain a preset number of data sets from the data warehouse to be processed; Inputting the preset number of data sets into an index generation model to obtain an index number corresponding to each data set output by the index generation model, wherein the index generation model is used to search for an index matching the data set in a preset index based on data features of the data set; Determine the target index number with the largest number among the index numbers corresponding to each data set; Obtain the target index corresponding to the target index number, and use the target index as the index of the data warehouse to be processed.
2. The method according to claim 1, characterized in that The data set includes a data element and a preset label corresponding to the data element; the step of inputting the preset number of data sets into an index generation model to obtain an index number corresponding to each data set output by the index generation model includes: Inputting the preset number of data sets into the index generation model; For each data set, the index generation model is used to perform feature extraction on the data element and the preset label corresponding to the data element to obtain data features and label features; The data set is classified by the index generation model using the data features and the label features, a preset index corresponding to the data set is determined, and an index number output by the index generation model is obtained.
3. The method according to claim 2, characterized in that The step of classifying the data set by using the data feature and the label feature through the index generation model to determine the preset index corresponding to the data set includes: Mosaicing the data features and the label features through the index generation model to obtain mosaic features; Performing a reshaping operation on the mosaic feature by using the index generation model to obtain a reshaping feature; The data set is classified based on the reshaping feature by using the index generation model to determine the preset index corresponding to the data set.
4. The method according to claim 1, characterized in that Before inputting the preset number of data sets into the index generation model, the method further includes: Get the sample data set in the sample data warehouse; Obtaining a preset label corresponding to each data element in the sample data set; The sample data set and the preset label are used as inputs of a generative adversarial network, and the index corresponding to the sample data set is used as label data to train the generative adversarial network to obtain the index generation model.
5. The method according to claim 4, characterized in that After obtaining the preset label corresponding to each data element in the sample data set, the method further includes: Performing a vectorization operation on the preset label to obtain a label vector; Performing a clustering operation on the label vector to obtain multiple clusters; For each cluster, obtaining the center vector of the cluster; Using the center vector as label class information of the cluster; The sample data set and the preset label are used as inputs of a generative adversarial network, the index corresponding to the sample data set is used as label data, and the generative adversarial network is trained to obtain the index generation model, including: The sample data set and the label class information are used as inputs of the generative adversarial network, and the index corresponding to the sample data set is used as label data to train the generative adversarial network to obtain the index generation model.
6. The method according to claim 4, characterized in that The generative adversarial network includes a generator model and a discriminator model; the sample data set and the preset label are used as inputs of the generative adversarial network, the index corresponding to the sample data set is used as label data, the generative adversarial network is trained, and the index generation model is obtained, including: Input the sample data set and the preset label into the generator model to obtain the sample index number output by the generator model; Obtaining the sample index corresponding to the sample index number; Inputting the sample index and the preset label into the discriminator model to obtain an output result of the discriminator model, wherein the output result includes a probability that the sample index is a true index corresponding to the sample data set; The generator model is adjusted using the output result to obtain the index generation model.
7. A device for constructing a data index, characterized in that: include: An acquisition module is used to acquire a preset number of data sets from the data warehouse to be processed; A generation module, used to input the preset number of data sets into an index generation model to obtain an index number corresponding to each data set output by the index generation model, wherein the index generation model is used to search for an index matching the data set in a preset index based on data features of the data set; A determination module, used for determining the target index number with the largest number among the index numbers corresponding to each data set; The acquisition module is further used to acquire the target index corresponding to the target index number, and use the target index as the index of the data warehouse to be processed.
8. An electronic device, characterized in that: The device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the method for constructing a data index as described in any one of claims 1 to 6 is implemented.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the method for constructing a data index as described in any one of claims 1 to 6 is implemented.
10. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the method for constructing a data index as described in any one of claims 1 to 6.