Sparse data index construction method and device, electronic equipment and medium

Abstract

The invention discloses a sparse data index construction method and device, electronic equipment and a medium. The method comprises the steps that a target character string and a position array are acquired; performing multi-layer scanning processing on the target character string to obtain a virtual character string; sorting the target type substrings corresponding to the two-tuples in the virtual character string to obtain a suffix sequence; based on the position array, scanning from the target character string to obtain a target character of the target position; based on a preset association rule, obtaining an associated character of the target character from the virtual character string; according to the position index numerical value corresponding to each target character, the position index of the tail character of the associated character and the suffix sequence, constructing a triple; and sorting all the triads, outputting a position index value of each triad in sequence based on a sorting result, and sorting to obtain a sparse suffix array of the target character string. The sparse data index can be constructed efficiently, and the method can be widely applied to the technical field of data processing.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, electronic device and medium for constructing a sparse data index. Background Art

[0002] The traditional method of constructing a sparse suffix array is as follows: for a string X of length n, randomly select m target positions from it. The suffix starting with the target position in X is called the target suffix. Sort these m suffixes in lexicographic order, and then store the positions of their first characters in X in the sorted order into the array SSA(X). SSA(X) is called the sparse suffix array of X.

[0003] However, when the data volume of the string is large, each target suffix includes a large number of characters. If a simple lexicographical sorting is still performed according to the above process, the amount of data calculated will be huge and the efficiency will be extremely low. Summary of the invention

[0004] The main purpose of the embodiments of the present invention is to propose a method, device, electronic device and medium for constructing a sparse data index, in order to solve at least one problem of the prior art. The present invention can efficiently realize the construction of a sparse data index.

[0005] To achieve the above object, an embodiment of the present invention provides a method for constructing a sparse data index, the method comprising:

[0006] Get the target string to be processed and the position array; the position array includes position index values ​​marking multiple target positions in the target string; the target string includes characters of the target type;

[0007] Based on the preset number of scanning layers, the target character string is subjected to multi-layer scanning processing with the characters of the target type as scanning targets to obtain a virtual character string; the elements in the virtual character string are two-tuples, the virtual character string includes a plurality of adjacent target substrings in the target character string, the head character and the tail character of the target substring are both of the target type, and the two-tuple includes the position index of the head character and the tail character;

[0008] Sort the target substrings corresponding to the two-tuples in the virtual string to obtain the suffix order corresponding to each two-tuple;

[0009] Based on the position array, the target character at the target position is scanned from the target string; based on the preset association rule, the associated character of the target character is obtained from the virtual string; the associated character corresponds to the target substring;

[0010] A triple is constructed according to the position index value corresponding to each target character, the position index of the tail character of the associated character, and the suffix order of the associated character;

[0011] Sort all triples, and then output the position index value of each triple in order based on the sorting result, and organize to obtain the sparse suffix array of the target string.

[0012] In some embodiments, based on a preset number of scanning layers, perform multi-layer scanning processing on the target string with characters of the target type as the scanning target to obtain a virtual string, including the following steps:

[0013] Use the target string as the scanning string;

[0014] Scan the scanning string from right to left, with characters of the target type as the scanning target, and split the scanning string into target-type substrings based on all the characters of the target type obtained by scanning;

[0015] Output the binary tuple corresponding to the target-type substring according to the position indexes of the head character and the tail character of each target-type substring;

[0016] Organize the hierarchical string according to the binary tuples corresponding to all the target-type substrings; increment the scanning layer by 1; the scanning layer is initially 0;

[0017] If the scanning layer is equal to the number of scanning layers, use the hierarchical string as the virtual string; otherwise, re-determine the character type of the target-type substring corresponding to each binary tuple in the hierarchical string, regard the target-type substring corresponding to the binary tuple as a character, use the hierarchical string as the scanning string, and return to execute the step of scanning the scanning string from right to left, with characters of the target type as the scanning target, and splitting the scanning string into multiple target-type substrings based on all the characters of the target type obtained by scanning until the scanning layer is equal to the number of scanning layers.

[0018] In some embodiments, the character types include L type and S type; the method further includes the following steps:

[0019] When X[i]<X[i + 1], or, X[i]=X[i + 1] and X[i + 1] is of S type, determine that X[i] is of S type; otherwise, determine that X[i] is of L type;

[0020] Wherein, X represents the target string or the hierarchical string, and the last character or element of X is of S type; X[i] represents the i-th character or element in X;

[0021] If X[i] is of S type and X[i - 1] is of L type, then determine that X[i] is of LMS type;

[0022] Use the LMS-type character as the character of the target type.

[0023] In some embodiments, sorting the target substrings corresponding to the two-tuples in the virtual string to obtain the suffix sequence corresponding to each two-tuple includes the following steps:

[0024] The suffix array corresponding to the virtual string is constructed by using a preset suffix array construction algorithm;

[0025] Determine the suffix order corresponding to each tuple in the virtual string according to the suffix array.

[0026] In some embodiments, based on the position array, scanning from the target string to obtain the target character at the target position includes the following steps:

[0027] Mark the character at the target position in the target string based on the position index value in the position array, and then determine that the character at the target position is the target character;

[0028] Scan the target string from right to left, and sequentially scan to obtain the target characters at all target positions in the target string.

[0029] In some embodiments, based on a preset association rule, obtaining an associated character of a target character from a virtual character string includes the following steps:

[0030] Determine a target substring according to characters of the target type adjacent to the right of the target character in the target character string;

[0031] Based on the target substring, match the target substring in the target string from left to right to determine the associated index value;

[0032] Add 1 to the number of associations; use the level string of the level corresponding to the number of associations as the association string; the initial number of associations is 0;

[0033] Determine the hierarchical associated character in the associated string according to the associated index value;

[0034] Determine a hierarchical target substring according to a character of the target type adjacent to the right of the hierarchical associated character in the associated string;

[0035] Based on the hierarchical target substring, match the target substring in the associated string from left to right, and re-determine the associated index value;

[0036] Return to the step of increasing the number of associations by 1 until the number of associations is equal to the number of scanned layers, and use the level-associated character determined based on the level character string of the last layer as the associated character.

[0037] In some embodiments, all triples are sorted, and then the position index value of each triple is outputted in order based on the sorting result, so as to obtain a sparse suffix array of the target string, including the following steps:

[0038] Sort all triples based on the preset sorting rules;

[0039] The elements contained in the triplet represent the position index value, the position index of the tail character of the associated character, and the suffix order of the associated character in turn. For any two triples (i, p i ,o i ) and (j,p j ,o j ) are sorted as follows:

[0040] Compare X[i,p from left to right i ] and X[j,p j ] The lexicographic order of the characters at the corresponding positions; where X represents the target string; the order of the triples represented by the characters with smaller lexicographic orders is smaller;

[0041] When comparing X[i,p i ] and X[j,p j ] If the last digit of one of the two substrings is still not compared to get the lexicographic order, then compare X[i,p i ] and X[j,p j ]The lengths of the two substrings; the substring with the larger length represents a smaller order of triples;

[0042] When X[i,p i ] and X[j,p j ] If the lengths of the two substrings are equal, compare the suffix order. i and j ; Among them, the triples with smaller suffix order have smaller order;

[0043] Based on the sorting results of all triples, the position index value of each triple is output in order from small to large, and then the sparse suffix array of the target string is obtained.

[0044] To achieve the above object, another aspect of an embodiment of the present invention provides a device for constructing a sparse data index, the device comprising:

[0045] The first module is used to obtain a target string to be processed and a position array; the position array includes position index values ​​marking multiple target positions in the target string; the target string includes characters of the target type;

[0046] The second module is used to perform multi-layer scanning processing on the target string based on a preset number of scanning layers and taking the characters of the target type as scanning targets to obtain a virtual string; the elements in the virtual string are two-tuples, the virtual string includes multiple adjacent target-type substrings in the target string, the head character and the tail character of the target-type substring are both of the target type, and the two-tuple includes the position index of the head character and the tail character;

[0047] A third module, configured to sort target substrings corresponding to bigrams in a virtual string to obtain a suffix order corresponding to each bigram;

[0048] A fourth module, configured to scan a target character at a target position from a target string based on a position array; obtain an associated character of the target character from the virtual string based on a preset association rule; the associated character corresponds to a target substring;

[0049] A fifth module, configured to construct a triple based on a position index value corresponding to each target character, a position index of a trailing character of the associated character, and the suffix order of the associated character;

[0050] A sixth module, configured to sort all triples, and then output a position index value of each triple in the order of the sorting result, and organize to obtain a sparse suffix array of the target string.

[0051] In some embodiments, the character types include L type and S type; the apparatus further includes a seventh module, and the seventh module is specifically configured to perform the following operations:

[0052] When X[i]<X[i + 1], or, X[i]=X[i + 1] and X[i + 1] is of S type, determine that X[i] is of S type; otherwise, determine that X[i] is of L type;

[0053] Wherein, X represents a target string or a hierarchical string, and the last character or element of X is of S type; X[i] represents the i-th character or element in X;

[0054] If X[i] is of S type and X[i - 1] is of L type, then determine that X[i] is of LMS type;

[0055] Use the character of LMS type as the character of the target type.

[0056] To achieve the above object, on the other hand, an embodiment of the present invention provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the above method is implemented.

[0057] To achieve the above object, on the other hand, an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the above method is implemented.

[0058] The embodiments of the present invention include at least the following beneficial effects: the present invention provides a method, device, electronic device and medium for constructing a sparse data index, the scheme obtains a target string to be processed and a position array; the position array includes position index values ​​marking multiple target positions in the target string; the target string includes characters of the target type; based on a preset number of scanning layers, multi-layer scanning processing is performed on the target string with the characters of the target type as scanning targets to obtain a virtual string; the elements in the virtual string are two-tuples, the virtual string includes multiple adjacent target substrings in the target string, the head character and the tail character of the target substring are both of the target type, and the two-tuple includes Position index of head character and tail character; sorting the target substring corresponding to the binary in the virtual string to obtain the suffix order corresponding to each binary; based on the position array, scanning the target character at the target position from the target string; based on the preset association rule, obtaining the associated character of the target character from the virtual string; the associated character corresponding to the target substring; constructing a triple according to the position index value corresponding to each target character, the position index of the tail character of the associated character and the suffix order of the associated character; sorting all triples, and then outputting the position index value of each triple based on the sorting result order, and arranging the sparse suffix array of the target string. The present invention scans and splits the target string based on the characters of the target type, and then sorts the target substring of the split result to achieve a preliminary suffix order arrangement, and then realizes the sparse suffix array arrangement for the target position in combination with the substring specification. The technical solution of the present invention can effectively avoid the situation of low processing efficiency caused by directly sorting all characters contained in the target suffix corresponding to the character at the target position in the case of a large amount of characters in the string to be processed. The present invention can efficiently realize the construction of sparse data index. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is a flowchart of a method for constructing a sparse data index provided by an embodiment of the present invention;

[0060] Figure 2 is a specific example diagram of constructing a virtual string provided by an embodiment of the present invention;

[0061] Figure 3 Schematic diagram of the principle of triple sorting provided by an embodiment of the present invention;

[0062] Figure 4 It is a structural schematic diagram of a device for constructing a sparse data index provided by an embodiment of the present invention;

[0063] Figure 5 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. When the following description involves the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the embodiments of the present invention. They are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the appended claims.

[0065] It can be understood that the terms "first", "second", etc. used in the present invention can be used in the present invention to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, the first information can also be called the second information, and similarly, the second information can also be called the first information. Depending on the context, the words "if", "when" as used herein can be interpreted as "when...", "while...", or "in response to determining".

[0066] The terms "at least one", "a plurality of", "each", "any one", etc. used in the present invention, at least one includes one, two or more than two, a plurality includes two or more than two, each refers to each of the corresponding plurality, and any one refers to any one of the plurality.

[0067] Unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs. The terms used in the present invention are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0068] For the convenience of understanding the technical solutions of the present invention, the technical means and features that may appear in the technical solutions of the present invention are explained below:

[0069] Now, the symbols to be used in the following text are defined. For a string X of length n, and any two integers i, j (0 < i < j < n). X[i] and X[j] represent the i-th and j-th characters in X from left to right. X[i, j] represents the substring composed of all characters between the i-th and j-th characters in X from left to right (including the i-th and j-th characters). S(X, i) represents the suffix of X starting with X[i].

[0070] The construction method of the sparse data index provided by the embodiment of the present invention relates to the field of data processing technology. The construction method of the sparse data index provided by the embodiment of the present invention can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements the construction method of the sparse data index, etc., but is not limited to the above forms.

[0071] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0072] Figure 1 is an optional flowchart of a method for constructing a sparse data index provided by an embodiment of the present invention. Figure 1 The method may include but is not limited to steps S100 to S600.

[0073] S100, obtaining a target string and position array to be processed;

[0074] The position array includes position index values ​​marking multiple target positions in the target string; the target string includes characters of the target type;

[0075] Exemplarily, in some specific implementations, firstly, a string X to be processed and an array T storing m target positions are read in from the outside.

[0076] S200. Based on the preset number of scanning layers, perform multi-layer scanning on the target string with characters of the target type as the scanning target to obtain a virtual string;

[0077] Among them, the elements in the virtual string are binary tuples. The virtual string includes multiple adjacent target-type substrings in the target string. The head character and the tail character of the target-type substring are both of the target type. The binary tuple includes the position indexes of the head character and the tail character;

[0078] It should be noted that in some embodiments, step S200 may include the following steps: Take the target string as the scanning string; Scan the scanning string from right to left, with characters of the target type as the scanning target, and split the scanning string into target-type substrings based on all the target-type characters obtained by scanning; Output the binary tuple corresponding to the target-type substring according to the position indexes of the head character and the tail character of each target-type substring; Organize the binary tuples corresponding to all the target-type substrings to obtain a hierarchical string; Increment the scanning level by 1; The scanning level is initially 0; If the scanning level is equal to the number of scanning layers, take the hierarchical string as the virtual string; Otherwise, re-determine the character type of the target-type substring corresponding to each binary tuple in the hierarchical string, regard the target-type substring corresponding to the binary tuple as a character, take the hierarchical string as the scanning string, and return to execute the step of scanning the scanning string from right to left, with characters of the target type as the scanning target, and splitting the scanning string into multiple target-type substrings based on all the target-type characters obtained by scanning, until the scanning level is equal to the number of scanning layers.

[0079] Exemplarily, in some specific embodiments, it is possible to scan X from right to left (scanning with the LMS type as the scanning target, and the scanning results are output from right to left) once to obtain the virtual string Z of the l-th layer l . Z l Each element Z l [i] in is represented by a binary tuple <head, tail>. In the following text, the values of head and tail are represented by Z l [i].head and Z l [i].tail respectively.

[0080] It should also be noted that the character types include the L type and the S type; In some embodiments, the method may further include the following steps: When X[i] < X[i + 1], or, X[i] = X[i + 1] and X[i + 1] is of the S type, determine that X[i] is of the S type; Otherwise, determine that X[i] is of the L type; Among them, X represents the target string or the hierarchical string, and the last character or element of X is of the S type; X[i] represents the i-th character or element in X; If X[i] is of the S type and X[i - 1] is of the L type, then determine that X[i] is of the LMS type; Take the characters of the LMS type as the characters of the target type.

[0081] Exemplarily, in some specific embodiments, for X[i] to be an S-type character, the following conditions must be met: (1) X[i] < X[i + 1]. Or, (2) X[i] = X[i + 1], and X[i + 1] is an S-type character. If the above conditions are not met, it is an L-type character. If X[i] is S-type and X[i - 1] is L-type, then X[i] is called LMS-type. Suppose X[i] and X[j] are two adjacent LMS characters, then X[i, j] is called an LMS substring. S(X, i) can also be divided into S-type, L-type, and LMS-type. When X[i] is S-type or L-type or LMS-type, S(X, i) is S-type or L-type or LMS-type.

[0082] In some specific application scenarios, the following takes X = "bacabacabacabacabab$" ($ represents the end flag, which is default LMS-type and is separately divided in substring partitioning), T = {0, 2, 4, 6, 10}, l = 2 as an example to illustrate the specific embodiments of the present invention:

[0083] As Figure 2 shown ( Figure 2 in the following, the target characters in the X row are marked in red font, and the type row represents the character types corresponding to each character in X, L represents L-type, S represents S-type, and S* represents LMS-type). First, scan to obtain the LMS characters in X (the corresponding index values are 1, 3, 5, 7, 9, 11, 13, 15, 17, and 19), and then divide to obtain Z1: (1, 3), (3, 5), (5, 7), (7, 9), (9, 11), (11, 13), (13, 15), (15, 17), (17, 19), (19, 19), where (19, 19) represents the separate partitioning of the end flag. Taking (1, 3) as an example, this binary group represents the substring of X from index position number 1 to 3 (i.e., aca), and the same applies to others; further, determine the character types of the substrings corresponding to each binary group in Z1, and then perform substring partitioning with LMS-type characters as the target again, then Z2 is obtained (the partitioning principle and parameter meanings are the same as those of Z1 and will not be elaborated here).

[0084] S300. Sort the target-type substrings corresponding to the binary groups in the virtual string to obtain the suffix order corresponding to each binary group;

[0085] It should be noted that in some embodiments, step S300 may include the following steps: construct the suffix array corresponding to the virtual string using a preset suffix array construction algorithm; determine the suffix order corresponding to each binary group in the virtual string according to the suffix array.

[0086] For example, in some specific implementations, the existing suffix array construction algorithm can be used to sort the target substrings corresponding to each element (two-tuple) in the virtual string in the target string based on the lexicographic order to achieve suffix order sorting output. Figure 2 As shown, the substrings corresponding to each binary group in Z2 (for example, (3,9) corresponds to abacaba) are sorted in lexicographic order. Specifically, the lexicographic order of the characters at the corresponding positions of each substring is compared from left to right. If different characters are encountered, the triple represented by the character with a smaller lexicographic order has a smaller order (the end mark (19,19) is the smallest by default). If the last bit of one of the two substrings is still not compared, the lengths of the two substrings are compared. The binary group represented by the larger length has a smaller order (for example, the length of (7,13) is less than (11,19)). If the lengths are equal, the binary group represented by the larger index position value has a smaller order (for example, the index position value of (7,13) is greater than (3,9)).

[0087] S400, based on the position array, scanning from the target character string to obtain the target character at the target position; based on the preset association rule, obtaining the associated character of the target character from the virtual character string;

[0088] Among them, the associated character corresponds to the target substring;

[0089] It should be noted that, in some embodiments, scanning from the target string to obtain the target character at the target position based on the position array may include the following steps: marking the character at the target position in the target string based on the position index value in the position array, and then determining that the character at the target position is the target character; scanning the target string from right to left, and sequentially scanning to obtain the target characters at all target positions in the target string.

[0090] In some embodiments, based on a preset association rule, obtaining associated characters of a target character from a virtual string may include the following steps: determining a target substring based on characters of the target type adjacent to the right of the target character in the target string; determining an associated index value based on matching the target type substring in the target string from left to right based on the target substring; adding 1 to the number of associations; using the hierarchical string of the level corresponding to the number of associations as the associated string; the number of associations is initially 0; determining a hierarchical associated character in the associated string based on the associated index value; determining a hierarchical target substring based on characters of the target type adjacent to the right of the hierarchical associated character in the associated string; matching the target type substring in the associated string from left to right based on the hierarchical target substring, and re-determining the associated index value; returning to execute the step of adding 1 to the number of associations until the number of associations is equal to the number of scan layers, and using the hierarchical associated character determined based on the hierarchical string of the last layer as the associated character.

[0091] For example, in some specific implementations, the target character X[i] is in Z l The associated character Z in l [f] is defined as follows: Assume that X[i1] and X[i2] are the first and second LMS characters to the right of X[i], and X[i1,i2] is the hth LMS substring from left to right in X. Then the character in Z1 represented by X[i1,i2] is Z1[j]. Z1[j] is the associated character of X[i] in Z1. Assume that Z1[j1] and Z1[j2] are the first and second LMS characters to the right of Z1[j], and Z1[j1,j2] is the kth LMS substring from left to right in Z1. Then the character in Z2 represented by Z1[j1,j2] is Z2[k]. Z2[k] is the associated character of X[i] in Z2. X[i] in Z l The associated character Z in l [f] is obtained by recursively following the above rules. In some specific application scenarios, such as Figure 2 As shown, the red font in the rows corresponding to Z1 and Z2 marks the associated characters of the target character in Z1 and Z2.

[0092] S500, constructing a triple according to the position index value corresponding to each target character, the position index of the tail character of the associated character, and the suffix sequence of the associated character;

[0093] For example, in some specific implementations, scan X from right to left. If X[i] is the target character, scan from Z l Find the associated character Z of X[i] l [f], and from SA(Z l ) to obtain the Z l [f] is the suffix order of the suffix beginning with o f . Use i,Z l [f].tail,o f Forming a triple <i,Z l [f].tail,o f >, and store it in the queue EQ. Continue to scan the next character in X until X[0] is scanned. Each target character found during this period forms a triplet according to the above rules and is stored in EQ.

[0094] S600, sorting all triples, and then outputting the position index value of each triple in order based on the sorting result, so as to obtain a sparse suffix array of the target string.

[0095] It should be noted that, in some embodiments, step S600 may include the following steps: sorting all triples based on a preset sorting rule; wherein the elements contained in the triples sequentially represent the position index value, the position index of the tail character of the associated character, and the suffix order of the associated character, and for any two triples (i, p i ,o i ) and (j,p j ,o j ) is sorted as follows: Compare X[i,p from left to right i ] and X[j,p j ] The lexicographic order of the characters at the corresponding positions; where X represents the target string; the order of the triples represented by characters with smaller lexicographic orders is smaller; when comparing X[i,p i ] and X[j,p j ] If the last digit of one of the two substrings is still not compared to get the lexicographic order, then compare X[i,p i ] and X[j,p j ] the lengths of the two substrings; the substring with a larger length represents a smaller order of triples; when X[i,p i ] and X[j,p j ] If the lengths of the two substrings are equal, compare the suffix order. i and j ; Among them, the triples with smaller suffix order have smaller order; based on the sorting results of all triples, the position index value of each triple is output in order from small to large, and then the sparse suffix array of the target string is obtained.

[0096] For example, in some specific implementations, all triples in EQ are sorted, and for any two triples (i, p i ,o i ) and (j,p j ,o j ) The sorting rules are as follows: (1) First compare X[i,p from left to right i ] and X[j,p j ] The lexicographic order of the characters at the corresponding positions. If different characters are encountered, the triples represented by the characters with smaller lexicographic order will have smaller order. (2) If the comparison is to X[i,p i ] and X[j,p j ] If the last digit of one of the two substrings is still not compared, then compare the lengths of the two substrings μ i and μ j If μ i and μ j If they are not equal, the larger one represents a smaller triple. i and μ j If they are equal, then compare oi and j The size of the triple represented by the smaller one has a smaller order. Sort all the triples in EQ according to the above rules to get an ordered EQ.

[0097] Pop up the sorted EQ triples (i, p i ,o i ), store the i in each triplet into the array SSA(X) in turn, and the SSA(X) obtained after all elements are popped out is the sorted sparse suffix array.

[0098] In order to explain the principle of the technical solution of the present invention in detail, the overall process of the present invention is described below in combination with some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and cannot be regarded as a limitation of the present invention.

[0099] The specific steps of the present invention are as follows:

[0100] 1. Read the string X to be processed and the array T containing m target positions from the outside.

[0101] 2. Scan X from right to left (taking LMS type as the scanning target, the scanning result is output from right to left) once to get the virtual string Z of the lth layer l . Z l Each element Z in l [i] Each consists of a binary<head,tail> In the following, the values ​​of headd and tail are represented by Z l [i].head and Z l [i].tail is used to represent it.

[0102] 3. Calculate Z using the existing suffix array calculation method l The suffix array SA(Z l ).

[0103] 4. Scan X from right to left. If X[i] is the target character, scan from Z l Find the associated character Z of X[i] l [f], and from SA(Z l ) to obtain the Z l [f] is the suffix order of the suffix beginning with o f . Use i,Z l [f].tail,o f Forming a triple <i,Z l [f].tail,o f>, and store it in the queue EQ. Continue to scan the next character in X until X[0] is scanned. Each target character found during this period forms a triplet according to the above rules and is stored in EQ.

[0104] 5. In step 4, the target character X[i] is in Z l The associated character Z in l [f] is defined as follows: Assume that X[i1] and X[i2] are the first and second LMS characters to the right of X[i], and X[i1,i2] is the jth LMS substring from left to right in X, then the character in Z1 represented by X[i1,i2] is Z1[j]. Z1[j] is the associated character of X[i] in Z1. Assume that Z1[j1] and Z1[j k ] are the first and second LMS characters to the right of Z1[j], and Z1[j1,j2] is the kth LMS substring from left to right in Z1. Then the character in Z2 represented by Z2[j1,j2] is Z2[k]. Z2[k] is the associated character of X[i] in Z2. l The associated character Z in l [f] is obtained by recursively following the above rules layer by layer.

[0105] 6. Sort all the triplets in EQ. For any two triplets (i, p i ,o i ) and (j,p j ,o j ) The sorting rules are as follows: (1) First compare X[i,p from left to right i ] and X[j,p j ] The lexicographic order of the characters at the corresponding positions. If different characters are encountered, the triples represented by the characters with smaller lexicographic order will have smaller order. (2) If the comparison is to X[i,p i ] and X[j,p j ] If the last digit of one of the two substrings is still not compared, then compare the lengths of the two substrings μ i and μ j If μ i and μ j If they are not equal, the larger one represents a smaller triple. i and μ j If they are equal, then compare o i and j The size of the triple represented by the smaller one has a smaller order. Sort all the triples in EQ according to the above rules to get an ordered EQ.

[0106] 7. Pop out the triples (i, p) in the EQ sorted in step 6 one by one i ,oi ), store the i in each triplet into the array SSA(X) in turn, and the SSA(X) obtained after all elements are popped out is the sorted sparse suffix array.

[0107] In some specific embodiments, Figure 2 As shown, the specific implementation mode of the present invention is described below by taking = "bacabacabacabacabab$, = {0, 2, 4, 6, 10}, = 2" as an example:

[0108] 1. Scan X from right to left to obtain the virtual string Z2 and calculate SA(Z2). Figure 2 Z2 and SA(Z2) are given. Figure 2 The red font in X marks the target character, and the red font in Z1 and Z2 marks the associated characters of the target character in Z1 and Z2.

[0109] 2. Scan X from right to left to obtain EQ = {(0,9,3),(2,13,2),(4,13,2),(6,19,1),(10,19,0)}.

[0110] 3. Sort all the triplets in EQ, Figure 3 A schematic diagram of the sorting process is given. The specific sorting process is as follows:

[0111] (1) The initial state is {(0,9,3),(2,13,2),(4,13,2),(6,19,1),(10,19,0)}. The smallest triplet is (4, 13, 2). Pop it out and store 4 in the array SSA(X).

[0112] (2) At this time, EQ = {(0,9,3), (2,13,2), (6,19,1), (10,19,0)}. The smallest triple is (0,9,3). It is popped out and 0 is stored in the array SSA(X).

[0113] (3) At this time, EQ = {(2, 13, 2), (6, 19, 1), (10, 19, 0)}. The smallest triple is (10, 19, 0). It is popped out and 10 is stored in the array SSA(X).

[0114] (4) At this time, EQ = {(2, 13, 2), (6, 19, 1)}. The smallest triple is (6, 19, 1). It is popped out and 6 is stored in the array SSA(X).

[0115] (5) At this time, EQ = {(2, 13, 2)}, the smallest triple is (2, 13, 2), which is popped out and 2 is stored in the array SSA(X).

[0116] 4. Obtain SSA(X) = {4, 0, 10, 6, 2}.

[0117] As Figure 4 shown, an embodiment of the present invention further provides a construction device 900 for sparse data indexing, which may include:

[0118] A first module 901, configured to obtain a target string to be processed and a position array; the position array includes position index values for marking multiple target positions in the target string; the target string includes characters of a target type;

[0119] A second module 902, configured to perform multi-layer scanning processing on the target string with the characters of the target type as the scanning target based on a preset number of scanning layers to obtain a virtual string; the elements in the virtual string are binary tuples, the virtual string includes multiple adjacent target-type substrings in the target string, the head character and the tail character of the target-type substring are both of the target type, and the binary tuple includes the position indexes of the head character and the tail character;

[0120] A third module 903, configured to sort the target-type substrings corresponding to the binary tuples in the virtual string to obtain the suffix order corresponding to each binary tuple;

[0121] A fourth module 904, configured to scan and obtain the target characters at the target positions from the target string based on the position array; obtain the associated characters of the target characters from the virtual string based on a preset association rule; the associated characters correspond to the target-type substrings;

[0122] A fifth module 905, configured to construct a triple according to the position index value corresponding to each target character, the position index of the tail character of the associated character, and the suffix order of the associated character;

[0123] A sixth module 906, configured to sort all the triples, and then output the position index values of each triple in the order of the sorting result, and organize to obtain the sparse suffix array of the target string.

[0124] In some embodiments, the character types include L type and S type; the device may further include a seventh module, and the seventh module is specifically configured to perform the following operations:

[0125] When X[i] < X[i + 1], or X[i] = X[i + 1] and X[i + 1] is of the S type, determine that X[i] is of the S type; otherwise, determine that X[i] is of the L type;

[0126] wherein, X represents the target string or the hierarchical string, and the last character or element of X is of the S type; X[i] represents the i-th character or element in X;

[0127] If X[i] is of type S and X[i-1] is of type L, then determine that X[i] is of type LMS;

[0128] The LMS type character is used as the target type character.

[0129] The contents of the method embodiments of the present invention are all applicable to the device embodiments. The functions specifically implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0130] The embodiment of the present invention further provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned sparse data index construction method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0131] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0132] See also Figure 5 , Figure 5 The hardware structure of an electronic device 1000 of another embodiment is illustrated. The electronic device 1000 includes:

[0133] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0134] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the construction method of the sparse data index of the embodiment of the present invention;

[0135] Input / output interface 1003, used to implement information input and output;

[0136] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0137] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0138] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0139] An embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method for constructing the sparse data index is implemented.

[0140] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0141] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0142] The method for constructing a sparse data index, the device for constructing a sparse data index, the electronic device and the storage medium provided by the embodiments of the present invention obtain a target string to be processed and a position array; the position array includes position index values ​​marking multiple target positions in the target string; the target string includes characters of the target type; based on a preset number of scanning layers, multi-layer scanning processing is performed on the target string with the characters of the target type as scanning targets to obtain a virtual string; the elements in the virtual string are two-tuples, the virtual string includes multiple adjacent target substrings in the target string, the head character and the tail character of the target substring are both of the target type, and the two-tuple includes the head character The present invention scans and splits the target string based on the characters of the target type, and then sorts the target substrings of the split results to achieve a preliminary suffix order arrangement, and then realizes the sparse suffix array arrangement for the target position in combination with the substring specifications. The technical solution of the present invention can effectively avoid the situation where the processing efficiency is low by directly sorting all the characters contained in the target suffix corresponding to the characters at the target position in the case where the amount of characters in the string to be processed is large, and the present invention can efficiently realize the construction of sparse data index.

[0143] The embodiments described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art can appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0144] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present invention and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0145] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0146] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0147] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0148] It should be understood that in the present invention, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0149] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0150] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0151] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0152] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store programs.

[0153] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the embodiments of the present invention is not limited thereby. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present invention shall be within the scope of the rights of the embodiments of the present invention.

Claims

1. A method for constructing a sparse data index, characterized in that: The method comprises the following steps: Obtaining a target string to be processed and a position array; the position array includes position index values ​​marking a plurality of target positions in the target string; the target string includes characters of a target type; Based on a preset number of scanning layers, the target character string is subjected to multi-layer scanning processing with the characters of the target type as scanning targets to obtain a virtual character string; the elements in the virtual character string are two-tuples, the virtual character string includes a plurality of adjacent target-type substrings in the target character string, the head character and the tail character of the target-type substring are both of the target type, and the two-tuple includes the position index of the head character and the tail character; Sorting the target substrings corresponding to the two-tuples in the virtual string to obtain a suffix order corresponding to each of the two-tuples; Based on the position array, a target character at the target position is obtained by scanning from the target character string; based on a preset association rule, an associated character of the target character is obtained from the virtual character string; the associated character corresponds to the target type substring; Constructing a triplet according to the position index value corresponding to each of the target characters, the position index of the tail character of the associated character, and the suffix sequence of the associated character; All the triples are sorted, and then the position index value of each triple is outputted in order based on the sorting result, so as to obtain a sparse suffix array of the target string.

2. The method for constructing a sparse data index according to claim 1, characterized in that: The method of performing multi-layer scanning processing on the target character string based on the preset number of scanning layers and taking the characters of the target type as scanning targets to obtain a virtual character string includes the following steps: Using the target string as a scan string; Scan the scan string from right to left, taking the characters of the target type as the scanning target, and splitting the scan string into the target type substrings based on all the characters of the target type obtained by scanning; Outputting the two-tuple corresponding to the target type substring according to the position index of the head character and the tail character of each target type substring; Arrange the binary groups corresponding to all the target substrings to obtain a hierarchical string; add 1 to the scanning level; the scanning level is initially 0; If the scanning level is equal to the scanning number of levels, the level string is used as the virtual string; otherwise, the character type of the target substring corresponding to each of the two-tuples in the level string is re-determined, the target substring corresponding to the two-tuple is regarded as a character, the level string is used as the scanning string, and the step of scanning the scanning string from right to left is returned to execute, with the characters of the target type as the scanning target, and the scanning string is split into multiple target substrings based on all the characters of the target type obtained by scanning, until the scanning level is equal to the scanning number of levels.

3. The method for constructing a sparse data index according to claim 2, characterized in that: The character types include L type and S type; the method further includes the following steps: When X[i] < X[i + 1], or when X[i] = X[i + 1] and X[i + 1] is the S type, determine that X[i] is the S type; otherwise, determine that X[i] is the L type; Where X represents the target string or the hierarchical string, and the last character or element of X is the S type; X[i] represents the i-th character or element in X; If X[i] is the S type and X[i - 1] is the L type, then determine that X[i] is the LMS type; Take the character of the LMS type as the character of the target type.

4. The method for constructing a sparse data index according to claim 1, characterized in that: The sorting of the target type substrings corresponding to the binary groups in the virtual string to obtain the suffix order corresponding to each binary group includes the following steps: Construct the suffix array corresponding to the virtual string using a preset suffix array construction algorithm; Determine the suffix order corresponding to each binary group in the virtual string according to the suffix array.

5. The method for constructing a sparse data index according to claim 1, characterized in that: The scanning of the target character at the target position from the target string based on the position array includes the following steps: Mark the character at the target position in the target string based on the position index value in the position array, and then determine that the character at the target position is the target character; Scan the target string from right to left, and sequentially scan to obtain the target characters at all the target positions in the target string.

6. The method for constructing a sparse data index according to claim 1, characterized in that: The obtaining of the associated character of the target character from the virtual string based on a preset association rule includes the following steps: Determine the target substring according to the target type character adjacent to the right of the target character in the target string; Match the target type substring in the target string from left to right based on the target substring to determine the associated index value; Increment the association count by 1; take the hierarchical string corresponding to the association count level as the associated string; the association count is initially 0; Determine the hierarchical associated character in the associated string according to the associated index value; Determine the hierarchical target substring according to the target type character adjacent to the right of the hierarchical associated character in the associated string; Match the target type substring in the associated string from left to right based on the hierarchical target substring to re-determine the associated index value; Return to execute the step of incrementing the association count by 1 until the association count is equal to the scanning layer count, and take the hierarchical associated character determined based on the hierarchical string of the last layer as the associated character.

7. The method for constructing a sparse data index according to claim 1, characterized in that: The sorting of all the triples, and then sequentially outputting the position index values of each triple based on the sorting result to organize and obtain the sparse suffix array of the target string includes the following steps: Sort all the triples based on a preset sorting rule; The elements contained in the triplet represent the position index value, the position index of the tail character of the associated character, and the suffix order of the associated character in sequence. For any two of the triplet (i, p i ,o i ) and (j,p j ,o j ) is as follows: Compare X[i,p from left to right i ] and X[j,p j ] The lexicographic order of the characters at the corresponding positions; wherein X represents the target string; the order of the triples represented by the characters with smaller lexicographic orders is smaller; When comparing X[i,p i ] and X[j,p j ] If the last digit of one of the two substrings is still not compared to get the lexicographic order, then compare X[i,p i ] and X[j,p j ] The lengths of the two substrings; wherein the substring with a larger length represents a smaller order of the triples; When X[i,p i ] and X[j,p j ] If the lengths of the two substrings are equal, compare the suffix order o i and j ; Wherein, the triples with smaller suffix order have smaller order; Sequentially output the position index values of each triple from small to large based on the sorting result of all the triples, and then organize and obtain the sparse suffix array of the target string.

8. A device for constructing a sparse data index, characterized in that: The device includes: The first module is used to obtain a target string to be processed and a position array; the position array includes position index values ​​marking multiple target positions in the target string; the target string includes characters of a target type; The second module is used to perform multi-layer scanning processing on the target string based on a preset number of scanning layers, taking the characters of the target type as scanning targets, to obtain a virtual string; the elements in the virtual string are two-tuples, the virtual string includes a plurality of adjacent target-type substrings in the target string, the head character and the tail character of the target-type substring are both of the target type, and the two-tuple includes the position index of the head character and the tail character; The third module is used to sort the target substrings corresponding to the two-tuples in the virtual string to obtain the suffix sequence corresponding to each of the two-tuples; The fourth module is used to scan the target character at the target position from the target character string based on the position array; obtain the associated character of the target character from the virtual character string based on a preset association rule; the associated character corresponds to the target type substring; A fifth module is used to construct a triple according to the position index value corresponding to each of the target characters, the position index of the tail character of the associated character, and the suffix sequence of the associated character; The sixth module is used to sort all the triples, and then output the position index value of each triple in order based on the sorting result, so as to obtain a sparse suffix array of the target string.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images