A learning index construction method and device for string data and a medium

Abstract

The application discloses a learning index construction method and device for string data and a medium, and the method comprises the following steps: acquiring first string data; dividing the first string data to obtain a plurality of first substrings; encoding the plurality of first substrings to obtain corresponding first encoded integers; searching for a corresponding index layer through layer-by-layer index searching according to a sequence number; creating an initial subnode array; inserting the first encoded integers into the initial subnode array to obtain a to-be-trained subnode array; inputting the first encoded integers in the to-be-trained subnode array into a first preset learning index model for training to obtain a first target learning index model corresponding to the to-be-trained subnode array, and the first target learning index model is used for indexing target string data. The application realizes string data indexing, improves the indexing efficiency, and reduces the space cost of the indexing structure. The application can be widely applied to the technical field of data indexing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data indexing, and particularly relates to a learning index construction method and device for string data and a medium. BACKGROUND

[0002] The learning index model refers to regarding the index as a model, and the model provides a mapping of data to its location. In a current numerical value type learning index design scheme, a simple linear model is usually used in each node to calculate the approximate location of a search key. However, strings have a large entity and different lengths compared with numerical values, and are stored in the form of byte arrays in a computer, which increases the complexity of model calculation and the number of model parameters, and the space cost of the index structure is large. At the same time, because the same prefix and substring exist, string data usually has a more irregular distribution than numerical data, and when a simple linear model is used, the obtained location is not accurate enough, which leads to the need to search in a larger range, and the index efficiency is low.

[0003] In summary, the technical problems in the related art need to be improved. SUMMARY

[0004] The learning index construction method and device for string data and the medium provided by the embodiments of the present application effectively improve the index efficiency and reduce the space cost of the index structure.

[0005] In one aspect, the present application provides a learning index construction method for string data, comprising the following steps:

[0006] obtaining first string data;

[0007] dividing the first string data by a preset number of bytes as a unit to obtain a plurality of first substrings;

[0008] encoding the plurality of first substrings to obtain corresponding first encoded integers;

[0009] According to the order sequence number of the first encoded integer in the division of the first string data, the corresponding index layer is found by layer-by-layer indexing;

[0010] creating an initial child node array in the index layer;

[0011] inserting the first encoded integer into the initial child node array to obtain the to-be-trained child node array;

[0012] input the first encoding integer in the to-be-trained sub-node array into a first preset learning index model, so that the first preset learning index model is trained to obtain a first target learning index model corresponding to the to-be-trained sub-node array, and the first target learning index model is used for indexing target string data.

[0013] In some embodiments, the encoding of the plurality of first sub-strings to obtain corresponding first encoding integers comprises:

[0014] According to a preset encoding rule, each character in the plurality of first sub-strings is converted to obtain a corresponding conversion code;

[0015] The plurality of conversion codes are combined to calculate the first encoding integer.

[0016] In some embodiments, the method further comprises:

[0017] Obtaining to-be-searched string data;

[0018] According to the to-be-searched string data, the storage position of a to-be-searched sub-string in a corresponding first sub-node array is calculated;

[0019] If the array element type of the storage position is a same-layer pointer, a second sub-node array pointed to by the first same-layer pointer is searched according to the first same-layer pointer, and the second sub-node array is used to find the value corresponding to the to-be-searched string data through layer-by-layer indexing;

[0020] If the array element type of the storage position is a lower-layer pointer, a third sub-node array pointed to by the first lower-layer pointer is searched according to the first lower-layer pointer, and the third sub-node array is used to find the value corresponding to the to-be-searched string data through layer-by-layer indexing;

[0021] If the array element type of the storage position is a data value, the value corresponding to the to-be-searched string data is obtained.

[0022] In some embodiments, the calculation of the storage position of the to-be-searched sub-string in the corresponding first sub-node array according to the to-be-searched string data comprises:

[0023] The to-be-searched string data is divided into a plurality of to-be-searched sub-strings in units of the preset number of bytes;

[0024] The plurality of to-be-searched sub-strings are encoded to obtain corresponding to-be-searched encoding integers;

[0025] input the to-be-searched encoded integer into a target learning index model of the first sub-node array, and calculate a storage position of the to-be-searched substring in the corresponding first sub-node array.

[0026] In some embodiments, the method further comprises:

[0027] obtaining to-be-inserted string data and a value corresponding to the to-be-inserted string data;

[0028] dividing the to-be-inserted string data in units of bytes of the preset number of bytes to obtain a plurality of to-be-inserted substrings;

[0029] encoding the plurality of to-be-inserted substrings to obtain corresponding to-be-inserted encoded integers;

[0030] inputting the to-be-inserted encoded integers into a target learning index model of a fourth sub-node array, and calculating a storage position of the to-be-inserted substring in the corresponding fourth sub-node array;

[0031] if the array element type of the storage position is a data value, performing conflict processing on the fourth sub-node array to obtain a fifth sub-node array;

[0032] if the array element type of the storage position is a same-layer pointer, searching, according to a second same-layer pointer, a sixth sub-node array pointed to by the second same-layer pointer, the sixth sub-node array being used to find an insertion position of the value corresponding to the to-be-inserted string data through layer-by-layer indexing;

[0033] if the array element type of the storage position is a lower-layer pointer, determining whether the to-be-inserted substring and a pointer substring corresponding to a second lower-layer pointer are the same;

[0034] if the to-be-inserted substring and the pointer substring corresponding to the second lower-layer pointer are the same, searching a seventh sub-node array pointed to by the second lower-layer pointer, the seventh sub-node array being used to find an insertion position of the value corresponding to the to-be-inserted string data through layer-by-layer indexing;

[0035] if the to-be-inserted substring and the pointer substring corresponding to the second lower-layer pointer are not the same, performing new same-layer processing on the fourth sub-node array to obtain an eighth sub-node array;

[0036] if the array element type of the storage position is empty, inserting the to-be-inserted string data and the value corresponding to the to-be-inserted string data into the storage position.

[0037] In some embodiments, the conflict processing on the fourth sub-node array to obtain the fifth sub-node array comprises:

[0038] create a conflict child node array;

[0039] insert the data value of the storage position and the value corresponding to the to-be-inserted string data into the conflict child node array, to obtain a ninth child node array;

[0040] input the code integer corresponding to the to-be-inserted code integer and the data value of the storage position into a second preset learning index model, so that the second preset learning index model is trained, to obtain a second target learning index model;

[0041] add the second target learning index model to the ninth child node array, to obtain a fifth child node array;

[0042] create a third same-layer pointer in the fourth child node array, and the third same-layer pointer is used to point to the fifth child node array.

[0043] In some embodiments, the fourth child node array is subjected to the adding same-layer processing, to obtain an eighth child node array, and the adding same-layer processing includes:

[0044] create a same-layer child node array;

[0045] insert the second lower-layer pointer and the value corresponding to the to-be-inserted string data into the same-layer child node array, to obtain a tenth child node array;

[0046] input the code integer corresponding to the to-be-inserted code integer and the data value of the storage position into a third preset learning index model, so that the third preset learning index model is trained, to obtain a third target learning index model;

[0047] add the third target learning index model to the tenth child node array, to obtain the eighth child node array;

[0048] create a fourth same-layer pointer in the fourth child node array, and the fourth same-layer pointer is used to point to the eighth child node array.

[0049] In another aspect, an embodiment of the present application provides a learning index construction device for string data, including:

[0050] a first module configured to acquire first string data;

[0051] a second module configured to divide the first string data in a unit of a preset number of bytes, to obtain a plurality of first sub-strings;

[0052] a third module configured to encode the plurality of first sub-strings, to obtain corresponding first code integers;

[0053] a fourth module configured to find a corresponding index layer by layer-by-layer index lookup according to a sequence number of the first coding integer in the first string data division;

[0054] a fifth module configured to create an initial child node array in the index layer;

[0055] a sixth module configured to insert the first coding integer into the initial child node array to obtain the to-be-trained child node array;

[0056] a seventh module configured to input the first coding integer in the to-be-trained child node array into a first preset learning index model, so that the first preset learning index model is trained to obtain a first target learning index model corresponding to the to-be-trained child node array, and the first target learning index model is used to index target string data.

[0057] In another aspect, an embodiment of the present application provides a learning index construction device for string data, comprising:

[0058] at least one processor;

[0059] at least one memory configured to store at least one program;

[0060] When the at least one program is executed by the at least one processor, the at least one processor implements the method.

[0061] In another aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method.

[0062] The present application has the following beneficial effects:

[0063] The present application first obtains first string data, divides the first string data to obtain a plurality of first sub-strings, encodes the first sub-strings to obtain corresponding first coding integers, finds a corresponding index layer according to a sequence number of the first coding integers, creates an initial child node array, inserts the first coding integers into the initial child node array to obtain a to-be-trained child node array, and finally inputs the first coding integers into a first preset learning index model for training to obtain a first target learning index model, so that the first target learning index model can be used to index target string data, and the string data indexing is realized, the indexing efficiency is improved, and the indexing structure space cost is reduced.

[0064] Other features and advantages of the present application will be set forth in the descriptions that follow, and in part will be apparent from the description, or can be learned by practice of the application. The purposes and other advantages of the present application will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings. BRIEF DESCRIPTION OF DRAWINGS

[0065] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort based on these drawings.

[0066] Figure 1 A flow chart of a learning index construction method for string data according to an embodiment of the present application;

[0067] Figure 2 A schematic diagram of a hierarchical division index structure according to an embodiment of the present application;

[0068] Figure 3 A schematic diagram of an index layer structure according to an embodiment of the present application;

[0069] Figure 4 A flow chart of a string data searching method according to an embodiment of the present application;

[0070] Figure 5 A schematic diagram of a string data searching process according to an embodiment of the present application;

[0071] Figure 6 A flow chart of a string data insertion method according to an embodiment of the present application;

[0072] Figure 7 A schematic diagram of a string data insertion process according to an embodiment of the present application;

[0073] Figure 8 A schematic diagram of insertion of a second string data in an application example according to an embodiment of the present application;

[0074] Figure 9 A schematic diagram of insertion of a third string data in an application example according to an embodiment of the present application;

[0075] Figure 10 A schematic diagram of insertion of a fourth string data in an application example according to an embodiment of the present application;

[0076] Figure 11 A schematic diagram of insertion of a fifth string data in an application example according to an embodiment of the present application;

[0077] Figure 12A schematic diagram of inserting a sixth string data in an application example of an embodiment of the present application. DETAILED DESCRIPTION

[0078] For the purpose of the present application, technical solutions and advantages, the following will be further described in detail in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not intended to limit the present application. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application, but are only examples of devices and methods consistent with some aspects of the embodiments of the present application as described in the appended claims.

[0079] It can be understood that the terms "first", "second" and the like used in the present application can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "when" or "in response to determining".

[0080] The terms "at least one", "multiple", "each", "any" and the like used in the present application include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0081] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as understood by those skilled in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0082] The learning index construction method for string data provided by the embodiments of the present application can be applied to a terminal, can be applied to a server, and can also be software running in the terminal or the server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, and the like, but is not limited thereto. The server end can be configured as a stand-alone physical server, can be configured as a server cluster or a distributed system formed by multiple physical servers, can be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, big data and artificial intelligence platform, and the server can also be a node server in a blockchain network. The software can be an application that implements a learning index construction method for string data, and the like, but is not limited to the above forms.

[0083] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0084] As shown in Figure 1 The embodiments of the present application provide a learning index construction method for string data, including but not limited to the following steps:

[0085] Step S11, obtaining first string data;

[0086] Step S12, dividing the first string data in units of a preset number of bytes to obtain a plurality of first substrings.

[0087] In the embodiment, the first string data can be acquired first, and then the first string data is divided into a plurality of first substrings in units of a preset number of bytes. For example, for a given string key[0:N] with a length of N+1, the first string data can be divided into (N+1) / 8 first substrings in units of 8 bytes.

[0088] In step S13, the plurality of first substrings are encoded to obtain corresponding first encoding integers.

[0089] In the embodiment, the plurality of first substrings can be encoded to obtain corresponding first encoding integers by first converting each character in the plurality of first substrings according to a preset encoding rule to obtain corresponding conversion codes, and then combining the conversion codes to calculate the first encoding integers.

[0090] In the embodiment, each character in the plurality of first substrings can be converted according to an ASCII code and a binary encoding rule to obtain corresponding conversion codes. For example, for an 8-byte first substring abcdefgh, a can be converted to 97, b can be converted to 98, c can be converted to 99, d can be converted to 100, e can be converted to 101, f can be converted to 102, g can be converted to 103, and h can be converted to 104 according to the ASCII code, and then 97 can be converted to 01100001, 98 can be converted to 01100010, 99 can be converted to 01100011, 100 can be converted to 01100100, 101 can be converted to 01100101, 102 can be converted to 01100110, 103 can be converted to 01100111, and 104 can be converted to 01101000 according to the binary encoding rule. The conversion codes are combined to obtain a 64-bit unsigned first encoding integer 01100001, 01100010, 01100011, 01100100, 01100101, 01100110, 01100111, 01101000. More preferably, the conversion codes can be encoded in a compressed manner to reduce the length of the first encoding integers and reduce the index height.

[0091] In step S14, the corresponding index layer is found by layer-by-layer index searching according to the order number of the first encoding integers in the division of the first string data.

[0092] In the embodiment, after the first substrings are converted into 64-bit unsigned integer types, numerical type learning index can be used to index each segment. For example, Figure 2As shown, the maximum number of index layers in this embodiment is (N+1) / 8, each layer contains several sub-node arrays, and each sub-node array includes a numerical learning index model. Figure 3 As shown, each index layer is used to index the corresponding substring, such as the 0th index layer (index layer sequence number starts from 0) is responsible for indexing the 1st substring of the first string data (i.e., bytes 0-7), and the 1st index layer is responsible for indexing the 2nd substring of the first string data (i.e., bytes 8-15). If there are strings with the same bytes 0-7, the next layer is responsible for indexing the strings with a common prefix, and the numerical learning index models in the same layer are responsible for indexing strings with different common prefixes. This embodiment can search for the corresponding index layer by indexing layer by layer according to the sequential number of the first coded integer in the first string data partition. For example, when the sequence number is 0, the 0th index layer is searched; when the sequence number is 1, the first fragment of the first string data is first taken, and the lower-level pointer pointing to the 1st index layer is found through the 0th index layer, and the position of the 1st index layer is found through the lower-level pointer; when the sequence number is 2, after finding the position of the 1st index layer, the second fragment of the first string data is taken, and the lower-level pointer pointing to the 2nd index layer is found through the 1st index layer, and the position of the 2nd index layer is found through the lower-level pointer.

[0093] Step S15: Create an initial child node array in the index layer;

[0094] Step S16: Insert the first coded integer into the initial child node array to obtain the child node array to be trained.

[0095] In this embodiment, the sub-node array is used to store string data DATA or storage pointers, wherein the pointers include a same-layer pointer NODE and a lower-layer pointer NODE_cp. The same-layer pointer is used to point to the sub-node array in the same index layer, and the lower-layer pointer is used to point to the sub-node array in the next index layer. The length of the sub-node array can be determined at the time of creation. For example, the length of the sub-node array can be determined to be 8, that is, the sub-node array can store 8 string data or pointers. After finding the corresponding index layer by sequential number, this embodiment can first create an initial sub-node array. The initial sub-node array is empty to prepare for the subsequent storage of data or pointers. Then, the encoded first encoded integer can be inserted into the initial sub-node array by random position or specified position to obtain the sub-node array to be trained.

[0096] Step S17: input the first coded integer in the sub-node array to be trained into the first preset learning index model to train the first preset learning index model to obtain the first target learning index model corresponding to the sub-node array to be trained, and the first target learning index model is used to index the target string data.

[0097] In the embodiment, the prediction of the learning index model can calculate the position of the substring in the array by a linear equation. For example, the array index, i.e., the position of the substring in the array, can be obtained by a linear equation position = a * x + b and rounding down, where x is the first encoding integer, and a and b are model parameters. In the embodiment, the first encoding integer in the to-be-trained child node array can be input into the first preset learning index model position = a * x + b, so that the first preset learning index model is trained to obtain the first target learning index model corresponding to the to-be-trained child node array.

[0098] In the embodiment, the string data can be searched, as shown in Figure 4 The method further includes steps S201-S205.

[0099] In step S201, the to-be-searched string data is obtained.

[0100] In the embodiment, the to-be-searched string data stored can be obtained.

[0101] In step S202, the storage position of the to-be-searched substring in the corresponding first child node array is calculated according to the to-be-searched string data.

[0102] In the embodiment, as shown in Figure 5 According to the to-be-searched string data, the storage position of the to-be-searched substring in the corresponding first child node array can be calculated by first dividing the to-be-searched string data into a plurality of to-be-searched substrings in units of a preset number of bytes, then encoding the plurality of to-be-searched substrings to obtain corresponding to-be-searched encoding integers, and finally inputting the to-be-searched encoding integers into the target learning index model of the first child node array to calculate the storage position of the to-be-searched substring in the corresponding first child node array.

[0103] In the embodiment, the to-be-searched string data can be divided into a plurality of to-be-searched substrings in units of 8 bytes, that is, each to-be-searched substring contains 8 bytes, then the plurality of to-be-searched substrings are encoded by using an ASCII encoding or a binary encoding method to obtain corresponding to-be-searched encoded integers, and finally the to-be-searched encoded integers are input into the target learning index model of the first child node array to calculate the storage position of the to-be-searched substring in the corresponding first child node array. For example, the first to-be-searched substring of the to-be-searched string data is taken first, and the corresponding to-be-searched encoded integer is obtained after encoding. The to-be-searched encoded integer is input into the target learning index model of the first child node array of the 0th index layer, and the storage position of the first to-be-searched substring in the first child node array is calculated.

[0104] In step S203, if the array element type of the storage position is the same layer pointer, the second child node array pointed to by the first same layer pointer is searched according to the first same layer pointer, and the second child node array is used to search the value corresponding to the to-be-searched string data by layer-by-layer indexing.

[0105] In step S204, if the array element type of the storage position is the lower layer pointer, the third child node array pointed to by the first lower layer pointer is searched according to the first lower layer pointer, and the third child node array is used to search the value corresponding to the to-be-searched string data by layer-by-layer indexing.

[0106] In step S205, if the array element type of the storage position is the data value, the value corresponding to the to-be-searched string data is obtained.

[0107] In the embodiment, after the storage position of the to-be-searched substring in the first child node array is calculated, if the element type of the storage position is the same layer pointer NODE, the second child node array pointed to by the first same layer pointer is searched according to the first same layer pointer, and then the storage position of the to-be-searched substring in the second child node array is searched by using the learning index model of the second child node array until the element type of the storage position is data. If the element type of the storage position is the lower layer pointer NODE cp, the third child node array pointed to by the first lower layer pointer is searched according to the first lower layer pointer, and then the storage position of the to-be-searched substring in the third child node array is searched by using the learning index model of the third child node array until the element type of the storage position is data. If the element type of the storage position is data DATA, the searching is completed, and the value corresponding to the to-be-searched string data is obtained.

[0108] In the embodiment, the string data can be inserted, as shown in FIG. 3, the method further includes steps S301-S310. Figure 6

[0109] ​Step S301, obtain the to-be-inserted string data and the value corresponding to the to-be-inserted string data;

[0110] Step S302, divide the to-be-inserted string data in units of a preset number of bytes to obtain a plurality of to-be-inserted substrings;

[0111] Step S303, encode the plurality of to-be-inserted substrings to obtain corresponding to-be-inserted encoded integers;

[0112] Step S304, input the to-be-inserted encoded integers into a target learning index model of a fourth child node array to calculate a storage position of the to-be-inserted substrings in the corresponding fourth child node array.

[0113] In this embodiment, as shown in Figure 7 the to-be-inserted string data and the value corresponding to the to-be-inserted string data can be obtained first, then the to-be-inserted string data is divided in units of 8 bytes to obtain a plurality of to-be-inserted substrings, the plurality of to-be-inserted substrings are encoded by using an ASCII encoding or a binary encoding method to obtain corresponding to-be-inserted encoded integers, and finally the to-be-inserted encoded integers are input into a target learning index model of a fourth child node array to calculate a storage position of the to-be-inserted substrings in the corresponding fourth child node array.

[0114] Step S305, if the array element type of the storage position is a data value, perform conflict processing on the fourth child node array to obtain a fifth child node array.

[0115] In this embodiment, the conflict processing on the fourth child node array to obtain the fifth child node array includes:

[0116] creating a conflict child node array;

[0117] inserting the data value of the storage position and the value corresponding to the to-be-inserted string data into the conflict child node array to obtain a ninth child node array;

[0118] inputting the to-be-inserted encoded integers and the encoded integers corresponding to the data value of the storage position into a second preset learning index model to enable the second preset learning index model to be trained to obtain a second target learning index model;

[0119] adding the second target learning index model to the ninth child node array to obtain the fifth child node array;

[0120] creating a third same-layer pointer in the fourth child node array, and the third same-layer pointer is used to point to the fifth child node array.

[0121] In the embodiment, the learning index model uses a unary linear equation position = a * x + b and is rounded down to calculate the storage position, so different x may also calculate the same array index, at this time, the two data positions conflict, and a new child node can be generated to save the two data. In the embodiment, a conflict child node array can be created first, the data value of the storage position when the conflict occurs and the value corresponding to the to-be-inserted string data are inserted into the conflict child node array to obtain a ninth child node array, then the to-be-inserted code integer and the code integer corresponding to the data value of the storage position are input into the second preset learning index model to enable the second preset learning index model to be trained to obtain a second target learning index model, the second target learning index model is added to the ninth child node array to obtain a fifth child node array, and finally a third same-layer pointer is created in the fourth child node array, the third same-layer pointer is used to point to the fifth child node array. Exemplarily, when the first substring (i.e., the 0th-7th byte) of the two data that conflict is different at the 0th index layer, a conflict child node array can be created in the same layer, the layer level of the conflict child node array is still 0, the data value of the storage position and the to-be-inserted value are inserted into the conflict child node array to obtain a ninth child node array, then the second target learning index model is obtained by training and added to the ninth child node array to obtain a fifth child node array, and finally a third same-layer pointer is created in the fourth child node array, the third same-layer pointer points to the fifth child node array. More, as shown in Figure 8 the length of the conflict child node array can be set to 8, the data value of the storage position is inserted into the 2nd element position, and the value corresponding to the to-be-inserted string data is inserted into the 5th element position (the element position is counted from 0).

[0122] In step S306, if the array element type of the storage position is a same-layer pointer, the sixth child node array pointed to by the second same-layer pointer is searched according to the second same-layer pointer, and the sixth child node array is used to find the insertion position of the value corresponding to the to-be-inserted string data by layer-by-layer indexing.

[0123] In step S307, if the array element type of the storage position is a lower-layer pointer, it is determined whether the to-be-inserted substring is the same as the pointer substring corresponding to the second lower-layer pointer.

[0124] In step S308, if the to-be-inserted substring is the same as the pointer substring corresponding to the second lower-layer pointer, the seventh child node array pointed to by the second lower-layer pointer is searched, and the seventh child node array is used to find the insertion position of the value corresponding to the to-be-inserted string data by layer-by-layer indexing.

[0125] In the embodiment, if the array element type of the storage position is the same-layer pointer NODE, the sixth child node array pointed to by the second same-layer pointer can be found according to the second same-layer pointer, and the insertion position of the value corresponding to the to-be-inserted string data can be found by the layer-by-layer indexing. The layer-by-layer indexing is used to represent that when the indexed array element type is a same-layer pointer or a lower-layer pointer, the next child node array is indexed by the pointer until the indexed array element type is a data value. Further, if the array element type of the storage position is the lower-layer pointer NODE cp, it is determined whether the to-be-inserted substring is the same as the pointer substring corresponding to the second lower-layer pointer. If the to-be-inserted substring is the same as the pointer substring corresponding to the second lower-layer pointer, the seventh child node array pointed to by the second lower-layer pointer is found, and the insertion position of the value corresponding to the to-be-inserted string data is found by the layer-by-layer indexing.

[0126] In step S309, if the to-be-inserted substring is not the same as the pointer substring corresponding to the second lower-layer pointer, the fourth child node array is subjected to the new same-layer processing to obtain the eighth child node array.

[0127] In the embodiment, the fourth child node array is subjected to the new same-layer processing to obtain the eighth child node array, including:

[0128] creating a same-layer child node array;

[0129] inserting the second lower-layer pointer and the value corresponding to the to-be-inserted string data into the same-layer child node array to obtain the tenth child node array;

[0130] inputting the to-be-inserted encoding integer and the encoding integer corresponding to the data value of the storage position into the third preset learning index model, so that the third preset learning index model is trained to obtain a third target learning index model;

[0131] adding the third target learning index model to the tenth child node array to obtain the eighth child node array;

[0132] creating a fourth same-layer pointer in the fourth child node array, and the fourth same-layer pointer is used to point to the eighth child node array.

[0133] In the embodiment, when two string data have a common prefix (i.e. the same order of the same substring) at the current index layer, the learning index model of the index layer predicts the two string data to the same position, however, due to the prediction accuracy of the learning index model, i.e. position = floor(a*x+b), floor is the floor function, it is possible to predict different x to the same position, i.e. two string data without a common prefix are predicted to the same position. In the embodiment, if the to-be-inserted substring is not the same as the pointer substring corresponding to the second lower layer pointer, the same layer child node array can be created first, the second lower layer pointer and the value corresponding to the to-be-inserted string data are inserted into the same layer child node array to obtain the tenth child node array, then the encoding integer corresponding to the to-be-inserted encoding integer and the data value of the storage position are input into the third preset learning index model, so that the third preset learning index model is trained to obtain the third target learning index model, and the third target learning index model is added to the tenth child node array to obtain the eighth child node array, and finally the fourth same layer pointer is created in the fourth child node array, and the fourth same layer pointer points to the eighth child node array.

[0134] In step S310, if the array element type of the storage position is empty, the to-be-inserted string data and the value corresponding to the to-be-inserted string data are inserted into the storage position.

[0135] In the embodiment, if the array element type of the storage position is empty, the to-be-inserted string data and the value corresponding to the to-be-inserted string data can be directly inserted into the storage position. More, after inserting the new value, the learning index model of the fourth child node array can be trained to obtain the fourth child node array after training.

[0136] Next, the scheme of the embodiment of the present application is described in detail in combination with a specific application example:

[0137] The embodiment shows the process of inserting data piece by piece to build an index (the specific location of data in each node is determined by the learning index model in the node, and the process of training and predicting data location of the learning index model is omitted in the following examples). It is assumed that the following six string data are inserted in the index in turn: abcdefff01234567stringl, abcdefgh01234567string2, abcdefgh01234567string3, abcdefghstring4, longlonglonglonglongstr5, string6. The empty child node array Model0 can be initialized first, and when the first string data is inserted, it can be directly inserted into the child node array Model0; when the second string data is inserted, since the second string data conflicts with the first string data in location, and they can be distinguished at the 0th index layer (i.e. 0-7 bytes) (“abcdefff”≠“abcdefgh”), as shown in Figure 8 , the child node array Model1 can be generated at the 0th index layer, and the second string data and the first string data are inserted into the child node array Model1; when the third string data is inserted, since the third string data has a common prefix with the second string data at the 0th index layer and the 1st index layer (“abcdefgh01234567”), they can be distinguished at the 2nd index layer, as shown in Figure 9 , the child node array Model2 can be generated at the 2nd index layer, and the third string data and the second string data are inserted into the child node array Model2, and the lower layer pointer pointing to the child node array Model2 is inserted into the conflict position of the child node array Model1; when the fourth string data is inserted, since the fourth string data has a common prefix with the third string data and the second string data at the 0th index layer (“abcdefgh”), they can be distinguished at the 1st index layer, as shown in Figure 10 , the child node array Model3 can be generated at the 1st index layer, and the fourth string data and the lower layer pointer pointing to the child node array Model2 are inserted into the child node array Model3, and the lower layer pointer pointing to the child node array Model3 is inserted into the conflict position of the child node array Model1; when the fifth string data is inserted, since the fifth string data can be distinguished from the inserted string data at the 0th index layer, as shown in Figure 11As shown, the fifth string data can be inserted into the child node array Model1; when the sixth string data is inserted, since the sixth string data and the fifth string data have a position conflict, i.e., the learning index model of the child node array Model1 predicts that the two string data are in the same position, and they can be distinguished at the 0 index layer (i.e., 0-7 bytes) ("longlong"≠"string6"), as shown in Figure 12 As shown, the child node array Model4 can be generated at the 0 index layer, and the sixth string data and the fifth string data are inserted into the child node array Model4, and the same-layer pointer pointing to the child node array Model4 is inserted into the conflict position of the child node array Model1;

[0138] The beneficial effects of implementing the embodiments of the present application include that the embodiments of the present application first acquire first string data, divide the first string data to obtain a plurality of first sub-strings, encode the first sub-strings to obtain corresponding first encoded integers, then find corresponding index layers according to the order serial numbers of the first encoded integers, create initial child node arrays, insert the first encoded integers into the initial child node arrays to obtain to-be-trained child node arrays, and finally input the first encoded integers into first preset learning index models for training to obtain first target learning index models, so that the target string data can be indexed by the first target learning index models, and thus the string data indexing is realized, the indexing efficiency is improved, and the indexing structure space cost is reduced.

[0139] In the embodiments, the string data is divided into multiple segments and converted into integers, the numerical learning index method is used, the string data is indexed in segments, and the efficient query and insertion of the string data can be supported. Meanwhile, the linear model is used in the indexing, the overhead of modeling the distribution of the string data can be reduced while the accuracy is ensured. In addition, when the information density of the string data is large enough, i.e., the string data can be distinguished by a small prefix, the query speed of the embodiments can approach the numerical index.

[0140] The embodiments of the present application also provide a learning index construction device for string data, which comprises:

[0141] The first module is configured to acquire first string data.

[0142] The second module is configured to divide the first string data in a unit of a preset number of bytes to obtain a plurality of first sub-strings.

[0143] The third module is configured to encode the plurality of first sub-strings to obtain corresponding first encoded integers.

[0144] The fourth module is configured to find the corresponding index layer by layer index lookup according to the order of the first coding integer in the first string data division;

[0145] The fifth module is configured to create an initial child node array in the index layer.

[0146] The sixth module is configured to insert the first coding integer into the initial child node array to obtain a to-be-trained child node array.

[0147] The seventh module is configured to input the first coding integer in the to-be-trained child node array into the first preset learning index model, so that the first preset learning index model is trained to obtain a first target learning index model corresponding to the to-be-trained child node array, and the first target learning index model is used for indexing the target string data.

[0148] The content in the above method embodiments is applicable to the device embodiments, the device embodiments specifically implement the same functions as the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0149] The embodiment of the application further provides a learning index construction device for string data, which comprises:

[0150] At least one processor;

[0151] At least one memory is configured to store at least one program.

[0152] When the at least one program is executed by the at least one processor, the at least one processor implements the method shown in the above method embodiments. Figure 1 The method shown in the above method embodiments.

[0153] The content in the above method embodiments is applicable to the device embodiments, the device embodiments specifically implement the same functions as the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0154] The embodiment of the application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method shown in the above method embodiments. Figure 1 The method shown in the above method embodiments.

[0155] The content in the above method embodiments is applicable to the storage medium embodiments, the storage medium embodiments specifically implement the same functions as the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0156] The above is a specific description of the preferred embodiment of the present application, but the present application is not limited to the above-described embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A learning index construction method for string data, characterized by, The method comprises the following steps: obtaining first string data; dividing the first string data into a plurality of first substrings in units of a preset number of bytes; encoding the plurality of first substrings to obtain corresponding first encoding integers; according to the order numbers of the first encoding integers in the division of the first string data, finding corresponding index layers by layer-by-layer indexing; creating an initial child node array in the index layer; inserting the first encoding integers into the initial child node array to obtain a to-be-trained child node array; inputting the first encoding integers in the to-be-trained child node array into a first preset learning index model to train the first preset learning index model, so as to obtain a first target learning index model corresponding to the to-be-trained child node array, and the first target learning index model is used for indexing target string data; wherein the method further comprises: obtaining to-be-searched string data; calculating the storage position of a to-be-searched substring in the corresponding first child node array according to the to-be-searched string data; if the array element type of the storage position is a same-layer pointer, then according to a first same-layer pointer, a second child node array pointed to by the first same-layer pointer is searched, and the second child node array is used for finding the value corresponding to the to-be-searched string data by layer-by-layer indexing; if the array element type of the storage position is a lower-layer pointer, then according to a first lower-layer pointer, a third child node array pointed to by the first lower-layer pointer is searched, and the third child node array is used for finding the value corresponding to the to-be-searched string data by layer-by-layer indexing; if the array element type of the storage position is a data value, then the value corresponding to the to-be-searched string data is obtained; the calculation of the storage position of the to-be-searched substring in the corresponding first child node array according to the to-be-searched string data comprises: dividing the to-be-searched string data into a plurality of to-be-searched substrings in units of the preset number of bytes; encoding the plurality of to-be-searched substrings to obtain corresponding to-be-searched encoding integers; inputting the to-be-searched encoding integers into the target learning index model of the first child node array to calculate the storage position of the to-be-searched substring in the corresponding first child node array.

2. The method of claim 1, wherein, the encoding of the plurality of first substrings to obtain corresponding first encoding integers comprises: according to a preset encoding rule, converting each character in the plurality of first substrings to obtain corresponding conversion codes; combining a plurality of conversion codes to calculate the first encoding integer.

3. The method of claim 1, wherein, the method further comprises: obtaining to-be-inserted string data and the value corresponding to the to-be-inserted string data; dividing the to-be-inserted string data into a plurality of to-be-inserted substrings in units of the preset number of bytes; encoding the plurality of to-be-inserted substrings to obtain corresponding to-be-inserted encoding integers; inputting the to-be-inserted coding integer into a target learning index model of a fourth sub-node array, calculating a storage position of the to-be-inserted substring in a corresponding fourth sub-node array; if an array element type of the storage position is a data value, performing conflict processing on the fourth sub-node array to obtain a fifth sub-node array; if the array element type of the storage position is a same-layer pointer, searching, according to a second same-layer pointer, a sixth sub-node array pointed to by the second same-layer pointer, the sixth sub-node array being used to find an insertion position of the to-be-inserted string data corresponding value through layer-by-layer index searching; if the array element type of the storage position is a lower-layer pointer, judging whether the to-be-inserted substring is same as a pointer substring corresponding to a second lower-layer pointer; if the to-be-inserted substring is same as the pointer substring corresponding to the second lower-layer pointer, searching a seventh sub-node array pointed to by the second lower-layer pointer, the seventh sub-node array being used to find the insertion position of the to-be-inserted string data corresponding value through layer-by-layer index searching; if the to-be-inserted substring is not same as the pointer substring corresponding to the second lower-layer pointer, performing new same-layer processing on the fourth sub-node array to obtain an eighth sub-node array; if the array element type of the storage position is empty, inserting the to-be-inserted string data and the to-be-inserted string data corresponding value into the storage position.

4. The method of claim 3, wherein, The conflict processing on the fourth sub-node array to obtain the fifth sub-node array includes: creating a conflict sub-node array; inserting the data value of the storage position and the to-be-inserted string data corresponding value into the conflict sub-node array to obtain a ninth sub-node array; inputting the to-be-inserted coding integer and a coding integer corresponding to the data value of the storage position into a second preset learning index model, so that the second preset learning index model is trained to obtain a second target learning index model; adding the second target learning index model to the ninth sub-node array to obtain the fifth sub-node array; creating a third same-layer pointer in the fourth sub-node array, the third same-layer pointer being used to point to the fifth sub-node array.

5. The method of claim 3, wherein, The new same-layer processing on the fourth sub-node array to obtain the eighth sub-node array includes: creating a same-layer sub-node array; inserting the second lower-layer pointer and the to-be-inserted string data corresponding value into the same-layer sub-node array to obtain a tenth sub-node array; inputting the to-be-inserted coding integer and a coding integer corresponding to the data value of the storage position into a third preset learning index model, so that the third preset learning index model is trained to obtain a third target learning index model; adding the third target learning index model to the tenth sub-node array to obtain the eighth sub-node array; creating a fourth same-layer pointer in the fourth sub-node array, the fourth same-layer pointer being used to point to the eighth sub-node array.

6. A learning index construction apparatus for string data, characterized by comprising: The method includes: a first module configured to acquire first string data; The second module is configured to divide the first string data in units of bytes of a preset number of bytes to obtain a plurality of first substrings. The third module is configured to encode the plurality of first substrings to obtain corresponding first encoding integers. The fourth module is configured to find a corresponding index layer by layer index lookup according to an order sequence number of the first encoding integers in the division of the first string data. The fifth module is configured to create an initial child node array in the index layer. The sixth module is configured to insert the first encoding integers into the initial child node array to obtain a to-be-trained child node array. The seventh module is configured to input the first encoding integers in the to-be-trained child node array into a first preset learning index model, so that the first preset learning index model is trained to obtain a first target learning index model corresponding to the to-be-trained child node array, and the first target learning index model is used for indexing target string data. The device further comprises: obtaining to-be-searched string data; calculating a storage position of a to-be-searched substring in a corresponding first child node array according to the to-be-searched string data; if an array element type of the storage position is a same-layer pointer, searching a second child node array pointed to by a first same-layer pointer according to the first same-layer pointer, and the second child node array is used to find a value corresponding to the to-be-searched string data by layer index lookup; if the array element type of the storage position is a lower-layer pointer, searching a third child node array pointed to by a first lower-layer pointer according to the first lower-layer pointer, and the third child node array is used to find the value corresponding to the to-be-searched string data by layer index lookup; if the array element type of the storage position is a data value, obtaining the value corresponding to the to-be-searched string data; The calculation of the storage position of the to-be-searched substring in the corresponding first child node array according to the to-be-searched string data comprises: dividing the to-be-searched string data in units of bytes of the preset number of bytes to obtain a plurality of to-be-searched substrings; encoding the plurality of to-be-searched substrings to obtain corresponding to-be-searched encoding integers; inputting the to-be-searched encoding integers into a target learning index model of the first child node array to calculate the storage position of the to-be-searched substring in the corresponding first child node array.

7. A learning index construction apparatus for string data, characterized by comprising: comprise: at least one processor; at least one memory for storing at least one program; when the at least one program is executed by the at least one processor, the at least one processor implements the method of any one of claims 1-5.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: The computer program is executed by the processor to implement the method of any one of claims 1 to 5.

Images