A method and device for constructing a sparse data index, an electronic device and a medium

Abstract

The application discloses a sparse data index construction method and device, electronic equipment and medium. The method comprises the following steps: obtaining a target string and a position array; performing multi-layer scanning processing on the target string to obtain a virtual string; sorting a target type substring corresponding to a binary tuple in the virtual string to obtain a suffix order; scanning a target character at a target position from the target string based on the position array; obtaining an associated character of the target character from the virtual string based on a preset association rule; constructing a triple based on a position index value corresponding to each target character, a position index of a tail character of the associated character and the suffix order; sorting all the triples, and then sequentially outputting the position index value of each triple based on a sorting result to obtain a sparse suffix array of the target string. The application can efficiently implement the construction of the sparse data index, and can be widely applied to the technical field of data processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a sparse data index construction method and device, electronic equipment and medium. BACKGROUND

[0002] The traditional sparse suffix array construction method is as follows: for a string X with a length of n, m target positions are randomly selected from X. The suffixes in X starting with the target positions are called target suffixes. The m target suffixes are sorted in lexicographical order, and the positions of the first characters of the m target suffixes in X are stored in an array SSA(X) in the sorted order. SSA(X) is called the sparse suffix array of X.

[0003] However, when the data amount of the string is large, each target suffix includes a large amount of characters, and if the simple lexicographical sorting is still performed according to the above process, the amount of calculation data is huge and the efficiency is extremely low. SUMMARY

[0004] The main purpose of the embodiments of the present application is to provide a sparse data index construction method and device, electronic equipment and medium, so as to solve at least one problem in the prior art, and the present application can efficiently implement the construction of the sparse data index.

[0005] To achieve the above object, one aspect of the embodiments of the present application provides a sparse data index construction method, which comprises the following steps:

[0006] obtaining a target string to be processed and a position array; the position array comprises position index values marking a plurality of target positions in the target string; the target string comprises characters of a target type;

[0007] performing multi-layer scanning processing on the target string based on a preset number of scanning layers and taking the characters of the target type as scanning targets to obtain a virtual string; the elements in the virtual string are two-tuples, the virtual string comprises a plurality of target type substrings adjacent in the target string, the head character and the tail character of the target type substring are both of the target type, and the two-tuple comprises the position indexes of the head character and the tail character;

[0008] sorting the target type substrings corresponding to the two-tuples in the virtual string to obtain the suffix order corresponding to each two-tuple;

[0009] scanning the target characters of the target positions from the target string based on the position array; obtaining associated characters of the target characters from the virtual string based on a preset association rule; the associated characters correspond to the target type substrings;

[0010] constructing a three-tuple according to the position index value of each target character, the position index of the tail character of the associated character, and the suffix order of the associated character.

[0011] The three tuples are sorted, and then the position index values of each three tuple are sequentially output based on the sorting result, and a sparse suffix array of the target string is obtained.

[0012] In some embodiments, based on the preset number of scanning layers, the target string is scanned in multiple layers with characters of the target type as the scanning target to obtain a virtual string, including the following steps:

[0013] The target string is taken as a scanning string.

[0014] The scanning string is scanned from right to left, and the characters of the target type are taken as the scanning target, and the scanning string is split into target type substrings based on all the characters of the target type obtained by scanning.

[0015] According to the position indexes of the head character and the tail character of each target type substring, a binary tuple corresponding to the target type substring is output.

[0016] The hierarchical string is obtained according to all the binary tuples corresponding to the target type substrings; the scanning level is increased by 1; and the initial scanning level is 0.

[0017] If the scanning level is equal to the number of scanning layers, the hierarchical string is taken as the virtual string; otherwise, the character type of each binary tuple corresponding to the target type substring of the hierarchical string is determined, the binary tuple corresponding to the target type substring is taken as a character, the hierarchical string is taken as a scanning string, and the step of scanning the scanning string from right to left with the characters of the target type as the scanning target and splitting the scanning string into multiple target type substrings based on all the characters of the target type obtained by scanning is executed until the scanning level is equal to the number of scanning layers.

[0018] In some embodiments, the character type includes L type and S type; the method further includes the following steps:

[0019] When X[i]<X[i+1] or X[i]=X[i+1] and X[i+1] is S type, X[i] is determined to be S type; otherwise, X[i] is determined to be L type.

[0020] Wherein, X represents the target string or the hierarchical string, the last character or element of X is S type; X[i] represents the i-th character or element in X.

[0021] If X[i] is S type and X[i-1] is L type, X[i] is determined to be LMS type.

[0022] The character of LMS type is taken as the character of the target type.

[0023] In some embodiments, the target type substring corresponding to each bigram in the virtual string is sorted to obtain a suffix order corresponding to each bigram, including the following steps:

[0024] The suffix array corresponding to the virtual string is constructed by using a preset suffix array construction algorithm.

[0025] The suffix order corresponding to each bigram in the virtual string is determined according to the suffix array.

[0026] In some embodiments, the target character at the target position in the target string is scanned based on the position array, including the following steps:

[0027] The character at the target position in the target string is marked based on the position index value in the position array, and then the character at the target position is determined as the target character.

[0028] The target character at all target positions in the target string is sequentially scanned from right to left.

[0029] In some embodiments, the associated character of the target character is obtained from the virtual string based on a preset association rule, including the following steps:

[0030] The target substring is determined according to the character of the target type adjacent to the right of the target character in the target string.

[0031] The associated index value is determined by matching the target type substring in the target string from left to right based on the target substring.

[0032] The association frequency is incremented by 1; the level string corresponding to the association frequency is taken as the associated string; and the association frequency is initially 0.

[0033] The level associated character is determined in the associated string according to the associated index value.

[0034] The level target substring is determined according to the character of the target type adjacent to the right of the level associated character in the associated string.

[0035] The associated index value is re-determined by matching the target type substring in the associated string from left to right based on the level target substring.

[0036] The step of incrementing the association frequency by 1 is returned to execute until the association frequency is equal to the number of scanning layers, and the level associated character determined based on the level string of the last layer is taken as the associated character.

[0037] In some embodiments, all triplets are sorted, and then the position index value of each triplet is sequentially output based on the sorting result, and the sparse suffix array of the target string is arranged, including the following steps:

[0038] Sort all triplets based on a preset sorting rule;

[0039] Wherein, the elements contained in the triplet represent the position index value, the position index of the tail character of the associated character and the suffix order of the associated character in turn, and the sorting rule of any two triplets (i, p i ,o i ) and (j, p j ,o j ) is as follows:

[0040] Compare the dictionary order of the characters in the corresponding positions of X[i, p i ] and X[j, p j ] from left to right; wherein, X represents the target string; the triplet represented by the character with smaller dictionary order is smaller in order;

[0041] When the last position of one of the two substrings X[i, p i ] and X[j, p j ] has not been compared in the dictionary order, compare the lengths of X[i, p i ] and X[j, p j ]; wherein, the triplet represented by the substring with larger length is smaller in order;

[0042] When the lengths of X[i, p i ] and X[j, p j ] are equal, compare the suffix orders o i and o j ; wherein, the triplet with smaller suffix order is smaller in order;

[0043] Based on the sorting results of all triplets, output the position index values of each triplet in turn from small to large, and then arrange to obtain the sparse suffix array of the target string.

[0044] To achieve the above object, another aspect of the embodiment of the application proposes a sparse data index construction device, which comprises:

[0045] The first module is configured to obtain a target string to be processed and a position array; the position array comprises position index values marking a plurality of target positions in the target string; the target string comprises characters of a target type;

[0046] The second module is configured to perform multi-layer scanning processing on the target string by taking the characters of the target type as the scanning target based on a preset number of scanning layers, and obtain a virtual string; the elements in the virtual string are binary tuples, the virtual string comprises a plurality of target type substrings adjacent in the target string, the head character and the tail character of the target type substring are both of the target type, and the binary tuple comprises the position indexes of the head character and the tail character;

[0047] The third module is configured to sort the target type substring corresponding to each pair of binary tuples in the virtual string to obtain a suffix order corresponding to each pair of binary tuples.

[0048] The fourth module is configured to scan a target character at a target position from the target string based on the position array, obtain an associated character of the target character from the virtual string based on a preset association rule, and determine that the associated character corresponds to the target type substring.

[0049] The fifth module is configured to construct a triple according to a position index value of each target character, a position index of a tail character of the associated character, and a suffix order of the associated character.

[0050] The sixth module is configured to sort all triples, and then sequentially output the position index value of each triple based on a sorting result to obtain a sparse suffix array of the target string.

[0051] In some embodiments, the character types include L type and S type, and the device further includes a seventh module configured to perform the following operations:

[0052] When X[i]<X[i+1] or X[i]=X[i+1] and X[i+1] is S type, it is determined that X[i] is S type; otherwise, it is determined that X[i] is L type.

[0053] wherein X represents the target string or the hierarchical string, and the last character or element of X is S type; and X[i] represents the ith character or element in X.

[0054] If X[i] is S type and X[i-1] is L type, it is determined that X[i] is LMS type.

[0055] The character of LMS type is taken as the target type character.

[0056] To achieve the above object, another aspect of the embodiment of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the above method when executing the computer program.

[0057] To achieve the above object, another aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the above method.

[0058] The embodiments of the present application at least have the following beneficial effects: the present application provides a sparse data index construction method, device, electronic equipment and medium, which obtains a target string to be processed and a position array; the position array includes position index values marking a plurality of target positions in the target string; the target string includes characters of a target type; based on a preset number of scanning layers, the characters of the target type are taken as scanning targets to perform multi-layer scanning processing on the target string to obtain a virtual string; elements in the virtual string are two-tuples, the virtual string includes a plurality of target type substrings adjacent in the target string, the head character and the tail character of the target type substring are both of the target type, and the two-tuple includes position indexes of the head character and the tail character; the target type substrings corresponding to the two-tuples in the virtual string are sorted to obtain a suffix order corresponding to each two-tuple; based on the position array, target characters of the target positions are scanned from the target string; based on a preset association rule, associated characters of the target characters are obtained from the virtual string; the associated characters correspond to the target type substrings; three-tuples are constructed based on the position index values of each target character, the position indexes of the tail characters of the associated characters, and the suffix orders of the associated characters; all the three-tuples are sorted, and then the position index values of each three-tuple are sequentially output based on the sorting result, and a sparse suffix array of the target string is obtained by sorting. Based on the characters of the target type, the target string is scanned and split, and then the target type substrings of the split results are sorted to realize preliminary suffix order arrangement, and then the sparse suffix array is sorted for the target positions in combination with the specifications of the substrings. In the case that the character quantity in the string to be processed is large, the present application can effectively avoid the situation that the processing efficiency is low due to the lexicographic order sorting of all characters included in the target suffix corresponding to the character of the target position, and the construction of the sparse data index can be efficiently realized. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 is a flowchart of the sparse data index construction method provided by the embodiments of the present application;

[0060] Figure 2 is a specific example diagram of constructing a virtual string provided by the embodiments of the present application;

[0061] Figure 3 is a principle diagram of three-tuple sorting provided by the embodiments of the present application;

[0062] Figure 4 is a structure diagram of the sparse data index construction device provided by the embodiments of the present application;

[0063] Figure 5 is a hardware structure diagram of the electronic equipment provided by the embodiments of the present application. DETAILED DESCRIPTION

[0064] In order to make the objects, technical solutions and advantages of the present application clearer, the following further describes the present application in conjunction with the accompanying drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and not intended to limit the present application. When the following description refers to the accompanying drawings, identical numbers in different drawings represent identical or similar elements unless otherwise indicated. The implementations described in the following example embodiments do not represent all implementations consistent with embodiments of the present application. They are merely examples of apparatuses and methods consistent with some aspects of embodiments of the present application as detailed in the appended claims.

[0065] It can be understood that the terms "first", "second", and the like used in the present application can be used in the present application to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon" or "in response to determining".

[0066] The terms "at least one", "multiple", "each", "any", and the like used in the present application include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0067] Unless otherwise defined, all technical and scientific terms used in the present application have the same meaning as understood by a person skilled in the art to which the present application belongs. The terms used in the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0068] In order to facilitate the understanding of the technical solutions of the present application, the following explains and describes the technical means and features that can appear in the technical solutions of the present application:

[0069] Now the symbols to be used in the following text are defined. For a string X of length n, and any two integers i, j (0 < i < j < n). X[i] and X[j] represent the i and j characters from left to right in X. X[i, j] represents all the characters between the i and j characters from left to right in X (including the i and j characters). S(X, i) represents the suffix of X starting with X[i].

[0070] The sparse data index construction method provided by the embodiments of the present application relates to the technical field of data processing. The sparse data index construction method provided by the embodiments of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in the terminal or the server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, and the like, but is not limited thereto; the server end can be configured as a stand-alone physical server, can also be configured as a server cluster or a distributed system formed by multiple physical servers, can also be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and big data and artificial intelligence platforms, and the server can also be a node server in a blockchain network; and the software can be an application that implements the sparse data index construction method, and the like, but is not limited to the above forms.

[0071] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as a program module. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0072] Figure 1 is an optional flowchart of the sparse data index construction method provided by the embodiments of the present application, Figure 1 The method in the flowchart can include, but is not limited to, steps S100 to S600.

[0073] S100, obtaining a target string to be processed and a position array;

[0074] The position array includes position index values marking a plurality of target positions in the target string; and the target string includes characters of a target type.

[0075] Exemplarily, in some specific embodiments, first, a string X to be processed and an array T containing m target positions are read from the outside.

[0076] S200. Based on a preset number of scanning layers, perform multi-layer scanning on the target string with characters of the target type as the scanning target to obtain a virtual string;

[0077] Among them, the elements in the virtual string are binary tuples. The virtual string includes multiple adjacent target-type substrings in the target string. The head character and the tail character of the target-type substring are both of the target type. The binary tuple includes the position indexes of the head character and the tail character;

[0078] It should be noted that in some embodiments, step S200 may include the following steps: Take the target string as the scanning string; Scan the scanning string from right to left, with characters of the target type as the scanning target, and split the scanning string into target-type substrings based on all the characters of the target type obtained by scanning; Output the binary tuple corresponding to the target-type substring according to the position indexes of the head character and the tail character of each target-type substring; Organize the binary tuples corresponding to all target-type substrings to obtain a hierarchical string; Increment the scanning level by 1; The scanning level is initially 0; If the scanning level is equal to the number of scanning layers, take the hierarchical string as the virtual string; Otherwise, re-determine the character type of the target-type substring corresponding to each binary tuple in the hierarchical string, regard the target-type substring corresponding to the binary tuple as a character, take the hierarchical string as the scanning string, and return to execute the step of scanning the scanning string from right to left, with characters of the target type as the scanning target, and splitting the scanning string into multiple target-type substrings based on all the characters of the target type obtained by scanning, until the scanning level is equal to the number of scanning layers.

[0079] Exemplarily, in some specific embodiments, it is possible to scan X from right to left (scanning with the LMS type as the scanning target, and the scanning results are output from right to left) once to obtain the virtual string Z of the l-th layer l . Z l Each element Z in l [i] is represented by a binary tuple <head, tail>. In the following text, the values of head and tail are represented by Z l [i].head and Z l [i].tail respectively.

[0080] It should also be noted that the character types include L type and S type; In some embodiments, the method may further include the following steps: When X[i] < X[i + 1], or, X[i] = X[i + 1] and X[i + 1] is of the S type, determine that X[i] is of the S type; Otherwise, determine that X[i] is of the L type; Among them, X represents the target string or the hierarchical string, and the last character or element of X is of the S type; X[i] represents the i-th character or element in X; If X[i] is of the S type and X[i - 1] is of the L type, then determine that X[i] is of the LMS type; Take the characters of the LMS type as the characters of the target type.

[0081] Exemplarily, in some specific embodiments, for X[i] to be an S-type character, the following conditions must be met: (1) X[i] < X[i + 1]. Or, (2) X[i] = X[i + 1], and X[i + 1] is an S-type character. If the above conditions are not met, it is an L-type character. If X[i] is S-type and X[i - 1] is L-type, then X[i] is called LMS-type. Suppose X[i] and X[j] are two adjacent LMS characters, then X[i,j] is called an LMS substring. S(X,i) can also be divided into S-type, L-type, and LMS-type. When X[i] is S-type or L-type or LMS-type, S(X,i) is S-type or L-type or LMS-type.

[0082] In some specific application scenarios, the following takes X = "bacabacabacabacabab$ ($ represents the end flag, which is default LMS-type and is separately divided in substring division), T = {0, 2, 4, 6, 10}, l = 2" as an example to illustrate the specific implementation manners of the present invention:

[0083] As Figure 2 shown ( Figure 2 in the following figure, the target characters in the X row are marked in red font, and the type row represents the character types corresponding to each character in X, L represents L-type, S represents S-type, and S* represents LMS-type), first, scan to obtain the LMS characters in X (the corresponding index values are 1, 3, 5, 7, 9, 11, 13, 15, 17, and 19), and then divide to obtain Z1: (1, 3), (3, 5), (5, 7), (7, 9), (9, 11), (11, 13), (13, 15), (15, 17), (17, 19), (19, 19), where (19, 19) represents the separate division of the end flag. Taking (1, 3) as an example, this binary group represents the substring of X from index position number 1 to 3 (i.e., aca), and the same applies to others; further, determine the character types of the substrings corresponding to each binary group in Z1, and then perform substring division targeting LMS-type characters again, then Z2 is obtained (the division principle and parameter meanings are the same as those of Z1 and will not be elaborated here).

[0084] S300. Sort the target-type substrings corresponding to the binary groups in the virtual string to obtain the suffix order corresponding to each binary group;

[0085] It should be noted that in some embodiments, step S300 may include the following steps: construct the suffix array corresponding to the virtual string using a preset suffix array construction algorithm; determine the suffix order corresponding to each binary group in the virtual string according to the suffix array.

[0086] For example, in some specific implementations, the existing suffix array construction algorithm can be used to sort the target substrings corresponding to each element (two-tuple) in the virtual string in the target string based on the lexicographic order to achieve suffix order sorting output. In some specific application scenarios, such as Figure 2 As shown, the substrings corresponding to each binary group in Z2 (for example, (3,9) corresponds to abacaba) are sorted in lexicographic order. Specifically, the lexicographic order of the characters at the corresponding positions of each substring is compared from left to right. If different characters are encountered, the triple represented by the character with the smaller lexicographic order has a smaller order (the end mark (19,19) is the smallest by default). If the last bit of one of the two substrings is still not compared, the lengths of the two substrings are compared. The binary group represented by the one with the larger length has a smaller order (for example, the length of (7,13) is smaller than that of (11,19)). If the lengths are equal, the binary group represented by the one with the larger index position value has a smaller order (for example, the index position value of (7,13) is larger than that of (3,9)).

[0087] S400, based on the position array, scanning from the target character string to obtain the target character at the target position; based on the preset association rule, obtaining the associated character of the target character from the virtual character string;

[0088] Among them, the associated character corresponds to the target substring;

[0089] It should be noted that, in some embodiments, scanning the target character at the target position from the target string based on the position array may include the following steps: marking the character at the target position in the target string based on the position index value in the position array, and then determining that the character at the target position is the target character; scanning the target string from right to left, and sequentially scanning to obtain the target characters at all target positions in the target string.

[0090] In some embodiments, obtaining the associated characters of the target character from the virtual string based on the preset association rules may include the following steps: determining the target substring based on the target type character adjacent to the right of the target character in the target string; matching the target type substring in the target string from left to right based on the target substring to determine the associated index value; adding 1 to the number of associations; using the hierarchical string of the level corresponding to the number of associations as the associated string; the initial number of associations is 0; determining the hierarchical associated character in the associated string based on the associated index value; determining the hierarchical target substring based on the target type character adjacent to the right of the hierarchical associated character in the associated string; matching the target type substring in the associated string from left to right based on the hierarchical target substring to re-determine the associated index value; returning to execute the step of adding 1 to the number of associations until the number of associations is equal to the number of scan layers, and using the hierarchical associated character determined based on the hierarchical string of the last layer as the associated character.

[0091] For example, in some specific implementations, the target character X[i] is in Z l The associated character Z in l [f] is defined as follows: Assume that X[i1] and X[i2] are the first and second LMS characters to the right of X[i], and X[i1,i2] is the hth LMS substring from left to right in X. Then the character in Z1 represented by X[i1,i2] is Z1[j]. Z1[j] is the associated character of X[i] in Z1. Assume that Z1[j1] and Z1[j2] are the first and second LMS characters to the right of Z1[j], and Z1[j1,j2] is the kth LMS substring from left to right in Z1. Then the character in Z2 represented by Z1[j1,j2] is Z2[k]. Z2[k] is the associated character of X[i] in Z2. X[i] in Z l The associated character Z in l [f] is obtained by recursion layer by layer according to the above rules. In some specific application scenarios, such as Figure 2 As shown, the red font in the rows corresponding to Z1 and Z2 marks the associated characters of the target character in Z1 and Z2.

[0092] S500, constructing a triplet according to the position index value corresponding to each target character, the position index of the tail character of the associated character, and the suffix sequence of the associated character;

[0093] For example, in some embodiments, scan X from right to left. If X[i] is the target character, scan from Z l Find the associated character Z of X[i] l [f], and from SA(Z l ) to obtain the Z l [f] is the suffix order of the suffix beginning with o f . Use i, Z l [f].tail,o f Forming triples <i,Z l [f].tail,o f >, and store it in the queue EQ. Continue scanning the next character in X until X[0] is scanned. Each target character found during this period forms a triple according to the above rules and stores it in EQ.

[0094] S600 , sorting all triples, and then outputting the position index value of each triple in order based on the sorting result, to obtain a sparse suffix array of the target string.

[0095] It should be noted that in some embodiments, step S600 can include the following steps: sorting all triplets based on a preset sorting rule; wherein the elements contained in the triplet represent, in turn, the position index value, the position index of the tail character of the associated character, and the suffix order of the associated character, and the sorting rule for any two triplets (i, p i ,o i ) and (j, p j ,o j ) is as follows: compare the characters in the corresponding positions of X[i, p i ] and X[j, p j ] from left to right in lexicographical order; wherein the triplet represented by the character with smaller lexicographical order is in smaller order; when the last position of one of the two substrings X[i, p i ] and X[j, p j ] has not been compared in lexicographical order, compare the lengths of the two substrings X[i, p i ] and X[j, p j ]; wherein the triplet represented by the longer substring is in smaller order; when the lengths of the two substrings X[i, p i ] and X[j, p j ] are equal, compare the suffix orders o i and o j ; wherein the triplet with smaller suffix order is in smaller order; based on the sorting results of all triplets, output the position index values of each triplet in turn from small to large, and then arrange to obtain the sparse suffix array of the target string.

[0096] Exemplarily, in some specific embodiments, all triplets in EQ are sorted, and for any two triplets (i, p i ,o i ) and (j, p j ,o j ) in them, the sorting rule is as follows: (1) first compare the characters in the corresponding positions of X[i, p i ] and X[j, p j ] from left to right in lexicographical order, if different characters are encountered, the triplet represented by the character with smaller lexicographical order is in smaller order. (2) If the last position of one of the two substrings X[i, p i ] and X[j, p j ] has not been compared in size, compare the lengths μ i and μ j of the two substrings. If μ i and μ j are not equal, the triplet represented by the larger one is in smaller order. If μ i and μ j are equal, compare oi and o j The size of the triples represented by the smaller one has a smaller order. Sort all the triples in EQ according to the above rules to get an ordered EQ.

[0097] Pop up the triples (i, p in the sorted EQ in turn i ,o i ), store the i in each triplet into the array SSA(X) in turn, and the SSA(X) obtained after all elements are popped out is the sorted sparse suffix array.

[0098] In order to explain the principle of the technical solution of the present invention in detail, the overall process of the present invention is described below in combination with some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and cannot be regarded as a limitation of the present invention.

[0099] The specific steps of the present invention are as follows:

[0100] 1. Read the string X to be processed and the array T containing m target locations from the outside.

[0101] 2. Scan X from right to left (with LMS type as the scanning target, the scanning results are output from right to left) once to obtain the virtual string Z of the lth layer l . Z l Each element Z in l [i] Each consists of a binary<head,tail> In the following, the values ​​of headd and tail are represented by Z l [i].head and Z l [i].tail to indicate.

[0102] 3. Calculate Z using the existing suffix array calculation method l The suffix array SA(Z l ).

[0103] 4. Scan X from right to left. If X[i] is the target character, scan from Z l Find the associated character Z of X[i] l [f], and from SA(Z l ) to obtain the Z l [f] is the suffix order of the suffix beginning with o f . Use i, Z l [f].tail,o f Forming triples <i,Z l [f].tail,o f>, and store it in the queue EQ. Continue scanning the next character in X until X[0] is scanned. Each target character found during this period forms a triple according to the above rules and stores it in EQ.

[0104] 5. In step 4, the target character X[i] is in Z l The associated character Z in l [f] is defined as follows: Assume that X[i1] and X[i2] are the first and second LMS characters to the right of X[i], and X[i1,i2] is the jth LMS substring from left to right in X, then the character in Z1 represented by X[i1,i2] is Z1[j]. Z1[j] is the associated character of X[i] in Z1. Assume that Z1[j1] and Z1[j k ] are the first and second LMS characters to the right of Z1[j], and Z1[j1,j2] is the kth LMS substring from left to right in Z1. Then the character in Z2 represented by Z2[j1,j2] is Z2[k]. Z2[k] is the associated character of X[i] in Z2. l The associated character Z in l [f] is obtained recursively according to the above rules layer by layer.

[0105] 6. Sort all the triples in EQ. For any two triples (i, p i ,o i ) and (j,p j ,o j ) The sorting rules are as follows: (1) First compare X[i,p from left to right i ] and X[j,p j ] The lexicographic order of the characters at the corresponding positions. If different characters are encountered, the lexicographically smaller character represents a smaller triple. (2) If the comparison is to X[i,p i ] and X[j,p j If the last digit of one of the two substrings is still not compared, then the lengths of the two substrings are compared. i and μ j If μ i and μ j If they are not equal, the larger one represents a smaller triple. i and μ j If they are equal, then compare o i and o j The size of the triples represented by the smaller one has a smaller order. Sort all the triples in EQ according to the above rules to get an ordered EQ.

[0106] 7. Pop out the triples (i, p) in the EQ sorted in step 6 one by one i ,oi ), and the SSA(X) obtained after all elements are popped is the sorted sparse suffix array.

[0107] In some embodiments, as shown in FIG. 2, the following example illustrates the embodiment of the present application: X = "bacabacabacabacabab$", = {0, 2, 4, 6, 10}, = 2. Figure 2

[0108] 1. Scan X from right to left to obtain a virtual string Z2 and calculate SA(Z2). Figure 2 Z2 and SA(Z2) are given. Figure 2 In FIG. 3, the target character in X is marked in red font, and the associated characters of the target character in Z1 and Z2 are marked in red font.

[0109] 2. Scan X from right to left to obtain EQ = {(0, 9, 3), (2, 13, 2), (4, 13, 2), (6, 19, 1), (10, 19, 0)}.

[0110] 3. Sort all triplets in EQ, Figure 3 The schematic diagram of the sorting process is given, and the specific sorting process is as follows:

[0111] (1) The initial state is {(0, 9, 3), (2, 13, 2), (4, 13, 2), (6, 19, 1), (10, 19, 0)}, and the smallest triplet is (4, 13, 2). Pop it out and store 4 in the array SSA(X).

[0112] (2) At this time, EQ = {(0, 9, 3), (2, 13, 2), (6, 19, 1), (10, 19, 0)}, the smallest triplet is (0, 9, 3), pop it out and store 0 in the array SSA(X).

[0113] (3) At this time, EQ = {(2, 13, 2), (6, 19, 1), (10, 19, 0)}, the smallest triplet is (10, 19, 0), pop it out and store 10 in the array SSA(X).

[0114] (4) At this time, EQ = {(2, 13, 2), (6, 19, 1)}, the smallest triplet is (6, 19, 1), pop it out and store 6 in the array SSA(X).

[0115] (5) At this time, EQ = {(2, 13, 2)}, the smallest triplet is (2, 13, 2), pop it out and store 2 in the array SSA(X). ​

[0116] 4, 0, 10, 6, 2}.

[0117] As shown in Figure 4 The embodiment of the application further provides a sparse data index construction device 900, which can comprise:

[0118] A first module 901 is configured to acquire a target string to be processed and a position array; the position array comprises position index values for marking a plurality of target positions in the target string; the target string comprises characters of a target type;

[0119] A second module 902 is configured to perform multi-layer scanning processing on the target string based on a preset number of scanning layers and taking the characters of the target type as scanning targets, to obtain a virtual string; elements in the virtual string are two-tuples, and the virtual string comprises a plurality of target type substrings adjacent in the target string, the head character and the tail character of the target type substring are of the target type, and the two-tuple comprises position indexes of the head character and the tail character;

[0120] A third module 903 is configured to sort the target type substrings corresponding to the two-tuples in the virtual string, to obtain a suffix order corresponding to each two-tuple;

[0121] A fourth module 904 is configured to scan the target characters of the target positions from the target string based on the position array, and acquire associated characters of the target characters from the virtual string based on a preset association rule; the associated characters correspond to the target type substrings;

[0122] A fifth module 905 is configured to construct a three-tuple according to the position index value of each target character, the position index of the tail character of the associated character, and the suffix order of the associated character.

[0123] A sixth module 906 is configured to sort all the three-tuples, and then sequentially output the position index values of each three-tuple based on the sorting result, to obtain a sparse suffix array of the target string.

[0124] In some embodiments, the character type comprises an L type and an S type; the device can further comprise a seventh module, which is specifically configured to perform the following operation:

[0125] When X[i]<X[i+1] or X[i]=X[i+1] and X[i+1] is of the S type, it is determined that X[i] is of the S type; otherwise, it is determined that X[i] is of the L type.

[0126] Wherein, X represents the target string or the hierarchical string, the last character or element of X is of the S type; X[i] represents the i-th character or element in X.

[0127] If X[i] is S type and X[i-1] is L type, then X[i] is determined as LMS type.

[0128] The character of the LMS type is taken as the character of the target type.

[0129] The contents of the method embodiments of the present application are applicable to the device embodiments, the device embodiments specifically implement the same functions as the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0130] The present application also provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the above method for constructing a sparse data index when executing the computer program. The electronic device can be any smart terminal, such as a tablet computer or a vehicle-mounted computer.

[0131] It can be understood that the contents of the above method embodiments are applicable to the device embodiments, the device embodiments specifically implement the same functions as the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0132] Please refer to Figure 5 , Figure 5 The hardware structure of the electronic device 1000 of another embodiment is illustrated, and the electronic device 1000 comprises:

[0133] The processor 1001 can be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is used to execute related programs to implement the technical solutions provided by the embodiments of the present application;

[0134] The memory 1002 can be implemented in the form of a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory). The memory 1002 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 1002 and are called and executed by the processor 1001 to implement the method for constructing a sparse data index according to the embodiments of the present application;

[0135] The input / output interface 1003 is used to realize information input and output.

[0136] The communication interface 1004 is configured to realize the communication interaction between the device and other devices, and can realize the communication through a wired manner (for example, a USB, a network cable and the like) or a wireless manner (for example, a mobile network, WIFI, Bluetooth and the like).

[0137] The bus 1005 is configured to transmit information between various components (for example, the processor 1001, the memory 1002, the input / output interface 1003 and the communication interface 1004) of the device.

[0138] The processor 1001, the memory 1002, the input / output interface 1003 and the communication interface 1004 are connected to each other through the bus 1005 to realize the communication connection between the device.

[0139] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the construction method of the sparse data index.

[0140] It can be understood that the contents in the above method embodiments are all applicable to the present storage medium embodiment, the present storage medium embodiment specifically realizes the functions of the above method embodiments, and achieves the same beneficial effects as the above method embodiments.

[0141] The memory is a non-transitory computer readable storage medium, and can be used to store a non-transitory software program and a non-transitory computer executable program. In addition, the memory can include a high-speed random access memory, and can further include a non-transitory memory, for example, at least one magnetic disk storage device, a flash memory device or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and the remote memory can be connected to the processor through a network. Examples of the network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0142] The sparse data index construction method, sparse data index construction device, electronic equipment and storage medium provided by the embodiment of the application obtain a target string to be processed and a position array; the position array includes position index values marking a plurality of target positions in the target string; the target string includes characters of a target type; based on a preset number of scanning layers, the characters of the target type are taken as scanning targets to perform multi-layer scanning processing on the target string to obtain a virtual string; elements in the virtual string are two-tuples, the virtual string includes a plurality of target type substrings adjacent in the target string, the head character and the tail character of the target type substring are both of the target type, and the two-tuple includes position indexes of the head character and the tail character; the target type substrings corresponding to the two-tuples in the virtual string are sorted to obtain a suffix order corresponding to each two-tuple; based on the position array, target characters of the target positions are scanned from the target string; based on a preset association rule, associated characters of the target characters are obtained from the virtual string; the associated characters correspond to the target type substrings; three-tuples are constructed based on the position index values of each target character, the position indexes of the tail characters of the associated characters, and the suffix orders of the associated characters; all the three-tuples are sorted, and then the position index values of each three-tuple are sequentially output based on the sorting result, and a sparse suffix array of the target string is obtained by sorting. Based on the characters of the target type, the target string is scanned and split, and then the target type substrings of the split results are sorted to realize preliminary suffix order arrangement, and then the sparse suffix array is sorted for the target positions in combination with the specifications of the substrings. In the case that the character quantity of the string to be processed is large, the application can effectively avoid the situation that the processing efficiency is low due to the lexicographic sorting of all characters included in the target suffix corresponding to the character of the target position, and the construction of the sparse data index can be efficiently realized.

[0143] The embodiments described in the embodiments of the application are used to more clearly illustrate the technical solutions of the embodiments of the application, and do not constitute a limitation on the technical solutions provided by the embodiments of the application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the application are also applicable to similar technical problems.

[0144] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the application, and can include more or fewer steps than shown in the figures, or combine certain steps or different steps.

[0145] The system embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, they can be located in one place or distributed on a plurality of network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments.

[0146] Those skilled in the art can understand that all or some of the steps in the method disclosed above, the functional modules / units in the system and the device can be implemented by software, firmware, hardware or a combination thereof.

[0147] The terms "first", "second", "third", "fourth" and the like in the description of the application and in the claims hereof, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so termed is interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of orderly or chronological permutations. Moreover, the terms "comprise", "have" and any variations thereof are intended to cover a non-exclusive inclusion, for example, a process, method, article, or apparatus that comprises a list of steps or units can not necessarily be limited to those steps or units, but can include additional steps or units not expressly listed or inherent to such process, method, article, or apparatus.

[0148] It should be understood that, in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that there are three cases: only A, only B, and A and B at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be singular or plural.

[0149] In several embodiments provided by the present application, it should be understood that the disclosed system and method can be implemented in other ways. For example, the above-described system embodiments are only illustrative, for example, the division of the above-mentioned units is only a logical functional division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be through some interface, indirect coupling or communication connection between the system or unit, which can be electrical, mechanical or other forms.

[0150] The units described as separate components above can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0151] In addition, each functional unit in each embodiment of the application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0152] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the application, the essential part or the whole or part of the contribution to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method of each embodiment of the application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0153] The preferred embodiments of the embodiments of the application are described above with reference to the accompanying drawings, but this does not limit the scope of the embodiments of the application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the application shall be within the scope of the embodiments of the application.

Claims

1. A method of constructing a sparse data index, characterized by, The method comprises the following steps: Obtaining a target string to be processed and a position array; the position array comprises position index values marking a plurality of target positions in the target string; the target string comprises characters of a target type; Based on a preset number of scanning layers, the target string is scanned in multiple layers with the characters of the target type as the scanning target to obtain a virtual string; elements in the virtual string are two-tuples, the virtual string comprises a plurality of target substrings adjacent in the target string, the head character and the tail character of the target substring are both of the target type, and the two-tuple comprises the position indexes of the head character and the tail character; The target substrings corresponding to the two-tuples in the virtual string are sorted to obtain the suffix order corresponding to each two-tuple; Based on the position array, target characters of the target positions are scanned from the target string; based on a preset association rule, associated characters of the target characters are obtained from the virtual string; the associated characters correspond to the target substrings; A three-tuple is constructed according to the position index value corresponding to each target character, the position index of the tail character of the associated character, and the suffix order of the associated character; All the three-tuples are sorted, and then the position index values of each three-tuple are sequentially output based on the sorting result to obtain a sparse suffix array of the target string.

2. The method of claim 1, wherein, The method comprises the following steps: The target string is taken as a scanning string; The scanning string is scanned from right to left, the characters of the target type are taken as the scanning target, and the scanning string is split into the target substrings based on all the characters of the target type scanned; The two-tuple corresponding to each target substring is output according to the position indexes of the head character and the tail character of the target substring; The hierarchical string is obtained according to the two-tuples corresponding to all the target substrings; the scanning level is increased by 1; the initial scanning level is 0; If the scanning level is equal to the number of scanning layers, the hierarchical string is taken as the virtual string; otherwise, the character type of the target substring corresponding to each two-tuple of the hierarchical string is re-determined, the target substring corresponding to the two-tuple is regarded as a character, the hierarchical string is taken as the scanning string, and the step of scanning the scanning string from right to left with the characters of the target type as the scanning target is executed again until the scanning level is equal to the number of scanning layers.

3. The method of claim 2, wherein, The character type comprises L type and S type; the method further comprises the following steps: determining that X[i] is the S type when X[i]<X[i+1] or X[i]=X[i+1] and X[i+1] is the S type; otherwise, determining that X[i] is the L type; wherein X represents the target string or the hierarchical string, and the last character or element of X is the S type; X[i] represents the ith character or element in X; determining that X[i] is the LMS type when X[i] is the S type and X[i-1] is the L type; taking the character of the LMS type as the character of the target type.

4. The method of claim 1, wherein, The step of sorting the target type substring corresponding to the bigram in the virtual string to obtain the suffix order corresponding to each bigram comprises the following steps: constructing a suffix array corresponding to the virtual string by using a preset suffix array construction algorithm; determining the suffix order corresponding to each bigram in the virtual string according to the suffix array.

5. The method of claim 1, wherein, The step of scanning the target character of the target position from the target string based on the position array comprises the following steps: marking the character of the target position in the target string based on the position index value in the position array, and then determining that the character of the target position is the target character; scanning the target character of all the target positions in the target string in sequence from right to left.

6. The method of claim 1, wherein, The step of obtaining the associated character of the target character from the virtual string based on a preset association rule comprises the following steps: determining a target substring according to the character of the target type adjacent to the right of the target character in the target string; determining an association index value by matching the target type substring in the target string from left to right based on the target substring; increasing the association frequency by 1; taking the hierarchical string corresponding to the association frequency as an associated string; the association frequency is initially 0; determining a hierarchical associated character in the associated string according to the association index value; determining a hierarchical target substring according to the character of the target type adjacent to the right of the hierarchical associated character in the associated string; redetermining the association index value by matching the target type substring in the associated string from left to right based on the hierarchical target substring; returning to the step of increasing the association frequency by 1 until the association frequency is equal to the scanning layer number, and taking the hierarchical associated character determined based on the hierarchical string of the last layer as the associated character.

7. The method of claim 1, wherein, The step of sorting all the triplets and then sequentially outputting the position index value of each triplet based on the sorting result to arrange the sparse suffix array of the target string comprises the following steps: sorting all the triplets based on a preset sorting rule; The triplets contain elements representing, in order, the position index value, the position index of the tail character of the associated character, and the suffix order of the associated character. For any two triplets (i, p i ,o i ) and (j, p j ,o j ), the ordering rule is as follows: compare the characters in the corresponding positions of X[i, p i ] and X[j, p j ] from left to right; wherein X represents the target string; the order of the triplets represented by the characters with smaller lexicographic order is smaller; When the last bit of one of the two substrings X[i, p i ] and X[j, p j ] has not been compared, the length of the two substrings X[i, p i ] and X[j, p j ] is compared; wherein the order of the triplet represented by the longer substring is smaller. When X[i, p i ] and X[j, p j ] are two substrings of equal length, then the suffix orders o i and o j are compared; where the order of the triplet is smaller if the suffix order is smaller. sequentially outputting the position index value of each triplet from small to large based on the sorting result of all the triplets, and then arranging the sparse suffix array of the target string.

8. A device for constructing a sparse data index, characterized in that: The device comprises: The first module is configured to obtain a target string to be processed and a position array; the position array includes position index values of a plurality of target positions in the target string; the target string includes characters of a target type; The second module is configured to perform multi-layer scanning processing on the target string based on a preset number of scanning layers and taking the characters of the target type as a scanning target to obtain a virtual string; elements in the virtual string are two-tuples; the virtual string includes a plurality of target type substrings adjacent in the target string; the head character and the tail character of the target type substring are of the target type; and the two-tuple includes position indexes of the head character and the tail character. The third module is configured to sort the target type substring corresponding to the two-tuple in the virtual string to obtain a suffix order corresponding to each two-tuple. The fourth module is configured to scan target characters of the target positions from the target string based on the position array; and obtain associated characters of the target characters from the virtual string based on a preset association rule; the associated characters correspond to the target type substring. The fifth module is configured to construct a three-tuple according to the position index value corresponding to each target character, the position index of the tail character of the associated character, and the suffix order of the associated character. The sixth module is configured to sort all the three-tuples, and then sequentially output the position index value of each three-tuple based on the sorting result to obtain a sparse suffix array of the target string.

9. An electronic device, comprising: The electronic device includes a memory and a processor; the memory stores a computer program; and the processor implements the method of any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: The computer program is executed by the processor to implement the method of any one of claims 1 to 7.

Images