Similar character sequence query and vector generation method, electronic device and storage medium
By generating feature vectors and calculating vector distances, the problem of low efficiency in querying similar characters in character sequences is solved, and efficient determination of similar character sequences is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- QI-ANXIN LEGENDSEC INFORMATION TECH (BEIJING) INC
- Filing Date
- 2022-12-21
- Publication Date
- 2026-05-29
AI Technical Summary
Existing technologies are inefficient in querying similar characters in character sequences, making it difficult to efficiently determine similar character sequences.
By identifying the characteristic character segments of the character sequence to be queried, a feature vector is generated, and the similarity is calculated using vector distance, thus reducing time and space complexity.
It improves the efficiency of querying similar characters in a character sequence and reduces time and space complexity.
Smart Images
Figure CN115840839B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, specifically to methods for querying similar character sequences and generating vectors, electronic devices, and storage media. Background Technology
[0002] In fields such as malware detection, similar text retrieval, and electronic data forensics, it is often necessary to query whether a given character sequence contains similar characters. Therefore, the method of querying similar characters within a character sequence is crucial. Summary of the Invention
[0003] The purpose of this application is to provide a method for querying similar character sequences and generating vectors, an electronic device, and a storage medium to solve the technical problems in the prior art.
[0004] The first aspect of this application provides a method for querying similar character sequences, including:
[0005] Identify the characteristic character segments of the character sequence to be queried;
[0006] Using the characteristic character fragments, a feature vector of the character sequence to be queried is generated;
[0007] Based on the feature vector of the character sequence to be queried and the vector distance between it and the feature vector of each sample character sequence, similar character sequences are determined from the sample character sequences.
[0008] In one embodiment, before determining the characteristic character segments of the character sequence to be queried, the method further includes:
[0009] A query request for obtaining similar character sequences, wherein the query request includes the character sequence to be queried.
[0010] In one embodiment, determining the characteristic character segments of the character sequence to be queried specifically includes:
[0011] Multiple consecutive strings are extracted from the character sequence to be queried, wherein the character length of each consecutive string is within a preset character length range;
[0012] Determine the hash value of each consecutive string;
[0013] Based on whether the hash value of each consecutive string is divisible by a preset filtering ratio, at least one characteristic character segment of the character sequence to be queried is determined from each consecutive string.
[0014] In one embodiment, the feature vector of the query character sequence is generated using the feature character fragment, specifically including:
[0015] Each feature character fragment is converted into a multi-dimensional vector;
[0016] The multidimensional vectors are summed with weights, where the weight of each multidimensional vector is positively correlated with the character length of the corresponding feature character segment.
[0017] The weighted vector obtained by weighted summation is normalized to generate the feature vector of the character sequence to be queried.
[0018] In one embodiment, each feature character fragment is converted into a multi-dimensional vector, specifically including:
[0019] For each feature character segment, multiple different dimension numbers are added to the feature character segment to obtain multiple concatenated strings corresponding to the feature character segment;
[0020] Determine the hash value of each concatenated string corresponding to the feature character fragment;
[0021] Using the hash values of each concatenated string, a multidimensional vector of the feature character fragment is generated.
[0022] In one embodiment, the method further includes: determining the feature vector corresponding to each sample character sequence.
[0023] In one embodiment, the method further includes: determining the feature vector of the character sequence to be queried, and the Euclidean distance or cosine distance between it and the feature vector corresponding to each sample character sequence, respectively, as the vector distance.
[0024] A second aspect of this application provides a method for generating feature vectors, comprising:
[0025] Identify the characteristic character segments of the character sequence to be used to generate the feature vector;
[0026] Using the aforementioned characteristic character fragments, a feature vector for the character sequence is generated:
[0027] A third aspect of this application provides an electronic device, including:
[0028] processor;
[0029] A memory for storing processor-executable instructions; wherein the processor is configured to perform the method described in any of the first aspects above.
[0030] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that can be executed by a processor to perform the method described in any of the first aspects above.
[0031] The data processing method provided in this application includes determining whether a target process is a zero-trust process, and if the target process is a zero-trust process, redirecting the data processing results of the target process on runtime resources to a data isolation zone. Therefore, this method distinguishes whether the target process is a zero-trust process before it processes data on runtime resources to generate data processing results, and further redirects the data processing results to a data isolation zone if it is a zero-trust process. This avoids the need for virus scanning of files. Compared to existing technologies that do not distinguish between processes but perform virus scanning on files after the process downloads them, this method improves efficiency and is therefore suitable for scenarios with a large number of files. Attached Figure Description
[0032] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0033] Figure 1 A schematic diagram of the specific structure of an electronic device provided in an embodiment of this application;
[0034] Figure 2 A schematic diagram illustrating the interaction between an electronic device and a terminal, provided as an embodiment of this application;
[0035] Figure 3 A schematic diagram illustrating the specific process of a similar character sequence query method provided in an embodiment of this application;
[0036] Figure 4 This is a schematic diagram illustrating the specific process for determining the characteristic character segments of a character sequence to be queried, provided as an embodiment of this application.
[0037] Figure 5 This is a schematic diagram illustrating the specific process of generating a feature vector of a character sequence to be queried, provided as an embodiment of this application.
[0038] Figure 6 A schematic diagram of the specific structure of a similar character sequence query device provided in an embodiment of this application;
[0039] Figure 7 This is a schematic diagram of the specific structure of a feature vector generation device provided in an embodiment of this application. Detailed Implementation
[0040] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. In the description of this application, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance or order.
[0041] As mentioned earlier, in fields such as malicious code detection, similar text retrieval, and electronic data forensics, it is often necessary to query whether a certain character sequence contains similar characters. Therefore, how to query similar characters in a character sequence is crucial.
[0042] For example, in the process of malicious code detection, after obtaining suspected malicious code, the suspected malicious code can usually be converted into a character sequence using methods such as fuzzy hashing algorithm. Then, it is checked whether there are similar characters in the sample library of malicious code. If they exist, it means that the suspected malicious code is likely to be malicious code.
[0043] Based on this, embodiments of this application provide a method for querying similar character sequences and generating vectors, an electronic device, and a storage medium, which can be used for querying similar character sequences. For example... Figure 1 The image shows an electronic device 1 provided in this embodiment. The electronic device 1 includes at least one processor 11 and a memory 12. Figure 1 Taking a processor as an example, the processor 11 and the memory 12 can be connected via a bus 10. The memory 12 stores instructions that can be executed by the processor 11. The instructions are executed by the processor 11 to enable the electronic device 1 to perform all or part of the process of the method in the following embodiments.
[0044] In practical applications, the electronic device 1 can be a laptop, a desktop computer, a server composed of them, or a server cluster, etc.
[0045] like Figure 2 As shown, electronic device 1 can also interface with one or more terminals 2, and electronic device 1 can communicate with each terminal 2 separately. The terminal can be a user's mobile phone, laptop, desktop computer, etc. In one embodiment, when a user needs to query similar character sequences for a certain character sequence, they can send a query request for similar character sequences to electronic device 1 through the terminal 2.
[0046] like Figure 3 The diagram shown is a flowchart illustrating a method for querying similar character sequences according to an embodiment of this application. Some or all of the steps of this method can be derived by... Figure 1 The electronic device 1 shown acts as a server to execute the method, which includes the following steps S31 to S34.
[0047] Step S31: Obtain the character sequence to be queried.
[0048] The character sequence to be queried can be a string composed of multiple numbers and / or characters arranged in a certain order. For example, multiple numbers and / or characters can be arranged and combined in a preset order to generate a character sequence, which can then be used as the character sequence to be queried.
[0049] In practical applications, such as in the field of malicious code detection, suspected malicious code can be converted into a character sequence (also known as a digest) using fuzzy hashing algorithms, which can then be used as the character sequence to be queried. In the field of similar text retrieval, the text to be retrieved can be converted into a character sequence using fuzzy hashing algorithms, which can then be used as the character sequence to be queried.
[0050] There are several ways to obtain the character sequence to be queried in step S31 in practical applications. For example, one way is to obtain the character sequence to be queried from a database. This database may store multiple character sequences, and when a query is needed, such as when triggered by a scheduled task, a specific character sequence can be obtained from the database as the character sequence to be queried.
[0051] In step S31, the specific way to obtain the character sequence to be queried can also be to obtain a query request for similar character sequences, which includes the character sequence to be queried.
[0052] For example, in practical applications, when a user needs to detect whether suspected malicious code is indeed malicious, one approach is to convert the suspected malicious code into a character sequence using a fuzzy hash algorithm or similar method, and then send a query request for a similar character sequence to the server via a mobile phone or other terminal. This query request includes the character sequence (as the character sequence to be queried). Another approach is for the user to send the suspected malicious code to a node in the middleware or server via a mobile phone or other terminal. The middleware or node then converts the suspected malicious code into a character sequence using a fuzzy hash algorithm or similar method, and further sends a query request for a similar character sequence to the server. This query request includes the character sequence (as the character sequence to be queried). Therefore, the server can obtain the query request for a similar character sequence sent by the user via the terminal, as well as query requests for similar character sequences sent by other nodes in the middleware or server, which include the character sequence to be queried.
[0053] Step S32: Determine the characteristic character segments of the character sequence to be queried.
[0054] The feature character fragment can be used to characterize the inherent features of the query character sequence, such as the number of characters in the query character sequence and their order.
[0055] Typically, the character sequence to be queried can contain one or more characteristic character segments. In practical applications, these can be identified through methods such as... Figure 4 The method shown is used to determine the characteristic character segments of the character sequence to be queried:
[0056] Step S321: Determine the preset character length range.
[0057] Step S322: Based on the preset character length range, extract multiple consecutive strings from the character sequence to be queried, wherein the character length of each consecutive string is within the preset character length range.
[0058] Step S323: Determine the hash value of each consecutive string.
[0059] Step S324: Based on whether the hash value of each consecutive string is divisible by a preset filtering ratio, determine at least one characteristic character segment of the character sequence to be queried from each consecutive string.
[0060] Here we can provide a unified explanation of steps S321 to S324.
[0061] Since feature character segments can be used to characterize inherent features in the query character sequence, such as the number of characters and their order, when determining the feature character segments of the query character sequence, a preset character length range can be determined first, thereby extracting multiple consecutive strings from the query character sequence whose character length belongs to the preset character length range.
[0062] Depending on actual needs, the preset character length range can be greater than or equal to 1. min And less than or equal to l max In other words, we can first determine two boundary values l. min and l max This determines the preset character length range. For l min and l max The specific size can be determined by combining the character length of the character sequence to be queried. For example, when the character length of the character sequence to be queried is less than 128 characters, the value of l is [not specified]. min It can be 7, 8, 9, or 10. max It can be 15, 16, 17, or 18; for example, when the length of the character sequence to be queried is greater than 128 characters, such as 200 characters, l min It can be and l max It can be increased accordingly, for example, l min It can be 10, 11, or 12, etc. max It can be 20, 21, 22, or 23, etc.
[0063] After determining the preset character length range, multiple consecutive strings can be extracted from the character sequence to be queried. Each consecutive string consists of multiple consecutive characters in the character sequence to be queried, and the character length of each consecutive string is within the preset character length range.
[0064] For example, the specific character sequence to be queried is "NTYUlcL6DKQ9IKPlio", and l min Specifically, it is 8, l max The specific value is 16, so the preset character length range is [8, 16]. At this time, the multiple consecutive strings extracted according to the above step S322 can include the consecutive strings shown in Table 1 below. The character length of these consecutive strings all belongs to [8, 16].
[0065] Table 1
[0066]
[0067]
[0068] After obtaining multiple consecutive strings as shown in Table 1 above, step S323 can further determine the hash value of each consecutive string. For example, a preset hash function can be used to determine the hash value of each consecutive string. The preset hash function can be: A = B % N, where A is the calculated hash value, % represents the modulo operation, B is used to associate the consecutive strings, and N is a preset constant. For example, the hash value of the consecutive strings can be calculated using the following code:
[0069]
[0070]
[0071] In this code, `return ret` returns the calculated hash value; `int(pow(2, 64))` is a preset constant N, specifically 2 to the power of 64.
[0072] for i in range(len(inputString)):
[0073] ret=(ret*16777619)^ord(inputString[i])
[0074] The following statement indicates that the i-th character is retrieved from the continuous string inputString each time:
[0075] ret=(ret*16777619)^ord(inputString[i])
[0076] This updates the original ret value, and the final ret is B, which is used to associate consecutive strings, where ord indicates that the character is converted into an integer.
[0077] After obtaining the hash value of each consecutive string in step S323, step S324 can further divide the hash value of each consecutive string by a preset filtering ratio to determine whether the hash value of each consecutive string is divisible by the preset filtering ratio. If it is divisible, the consecutive string is retained; if it is not divisible, the consecutive string is deleted.
[0078] For example, for the hash values of each consecutive string obtained in step S323 above, the hash value of each consecutive string is divided by a preset filtering ratio, thereby retaining the consecutive strings that can be divided evenly and deleting (or discarding) the consecutive strings that cannot be divided evenly, so as to determine at least one consecutive string from the consecutive strings extracted in step S322 above as the feature character segment of the character sequence to be queried.
[0079] The preset filter ratio can be set according to actual needs, for example, the preset filter ratio (hereinafter referred to as f) r () can be 6, 7, 8, 9 or other values.
[0080] Step S33: Use the feature character fragments to generate the feature vector of the character sequence to be queried.
[0081] After determining the characteristic character segment of the character sequence to be queried in step S32 above, the characteristic character segment can be used to generate the feature vector of the character sequence to be queried. For example, one way to generate the feature vector is to use a vector generation tool to generate the feature vector from the characteristic character segment.
[0082] Of course, this feature character fragment can also be used to generate feature vectors for the query character sequence in other ways, such as through methods like... Figure 5 The method shown generates a feature vector for the character sequence to be queried:
[0083] Step S331: Convert each feature character fragment into a multidimensional vector.
[0084] Step S332: Perform a weighted summation of the various multidimensional vectors.
[0085] Step S333: Normalize the weighted vector obtained by weighted summation to generate the feature vector of the character sequence to be queried.
[0086] Here we can provide a unified explanation of steps S331 to S333.
[0087] In step S331, the feature character fragments can be converted into multi-dimensional vectors, where all multi-dimensional vectors have the same dimension d, such as 128, 64, 32, or 16. The specific method for converting each feature character fragment into a multi-dimensional vector can be as follows:
[0088] First, for each feature character segment, multiple different dimension indices are added to that segment to obtain multiple concatenated strings corresponding to that feature character segment. For example, if the dimension d of the generated multidimensional vector is 64, each integer from 0, 1, 2, 3...63 can be used as a dimension indice, resulting in 64 different dimension indices. Taking the feature character segment "NTYUlcL6" in Table 1 as an example, 64 different dimension indices are added to the feature character segment "NTYUlcL6" to obtain 64 concatenated strings corresponding to the feature character segment "NTYUlcL6", namely ["NTYUlcL6_0", "NTYUlcL6_1", "NTYUlcL6_2", "NTYUlcL6_3"... …"NTYUlcL6_63"]; Of course, for a dimension d of 128, each integer from 0, 1, 2...127 can be used as a dimension number to obtain 128 different dimension numbers. Then these dimension numbers are concatenated with the feature character fragment "NTYUlcL6" to obtain 128 concatenated strings corresponding to the feature character fragment "NTYUlcL6", namely ["NTYUlcL6_0", "NTYUlcL6_1", "NTYUlcL6_2", "NTYUlcL6_3"..."NTYUlcL6_127"].
[0089] For each characteristic character segment, after obtaining the multiple concatenated strings corresponding to that characteristic character segment, the hash value of each concatenated string can be further determined. For example, a preset hash function can be used to determine the hash value of each concatenated string. For example, the 64 concatenated strings corresponding to the characteristic character segment "NTYUlcL6" are ["NTYUlcL6_0", "NTYUlcL6_1", "NTYUlcL6_2", "NTYUlcL6_3"……"NTYUlcL6_63"]. Using a preset hash function, the hash values of each concatenated string are determined as [-8376996883554467746, -8351413898004317181, 8300055146581427339, 3754259992259289243……].
[0090] For each characteristic character segment, after determining the hash value of each concatenated string corresponding to the characteristic character segment, the hash value of each concatenated string can be used to generate a multidimensional vector of the characteristic character segment.
[0091] For example, the hash value of each concatenated string can be divided by a preset constant and the remainder can be taken. This preset constant can be 2. For example, the hash values of the concatenated strings [-8376996883554467746, -8351413898004317181, 8300055146581427339, 3754259992259289243……] can be divided by 2 and the remainder can be taken to obtain a 64-dimensional vector [0, 1, 1, 1……]. This 64-dimensional vector can be directly used as the multidimensional vector of the feature character segment. Of course, to facilitate subsequent calculations, this 64-dimensional vector can be further processed, such as replacing 0 with -1 to obtain a new 64-dimensional vector [-1, 1, 1, 1……], and this new 64-dimensional vector can be used as the multidimensional vector of the feature character segment.
[0092] After obtaining the multidimensional vectors of each feature character segment in step S331, step S332 involves weighted summation of these multidimensional vectors to obtain a weighted vector. The weight of each multidimensional vector is positively correlated with the character length of the corresponding feature character segment; for example, this positive correlation can be proportional, meaning the weight of each multidimensional vector is positively correlated with the character length of the corresponding feature character segment.
[0093] For example, for a feature character segment with a length of 8 characters, the weight of the multidimensional vector of the feature character segment can be 8 or other values that are adjusted proportionally when weighted and summed; for a feature character segment with a length of 9 characters, the weight of the multidimensional vector of the feature character segment can be 9 or other values that are adjusted proportionally when weighted and summed.
[0094] After obtaining the weighted vector in step S332, step S333 can further normalize the weighted vector to generate the feature vector of the character sequence to be queried. The specific method of normalization is to convert the modulus of the weighted vector to 1 to obtain the result of normalization, and use the result as the feature vector of the character sequence to be queried.
[0095] Step S34: Based on the feature vector of the character sequence to be queried and the vector distance between it and the feature vector of each sample character sequence, determine the similar character sequences of the character sequence to be queried from each sample character sequence.
[0096] After obtaining the feature vector of the query character sequence in step S33 above, the vector distance between the feature vector of the query character sequence (hereinafter referred to as the first feature vector) and the feature vectors corresponding to each sample character sequence (hereinafter referred to as the second feature vector) can be further determined. This vector distance reflects the similarity between the first feature vector and each second feature vector. Generally, the smaller the vector distance, the more similar the first and second feature vectors are, and correspondingly, the more similar the query character sequence is to the corresponding sample character sequence. Conversely, the larger the vector distance, the less similar the first and second feature vectors are, and correspondingly, the less similar the query character sequence is to the corresponding sample character sequence. In practical applications, this vector distance can be Euclidean distance or cosine distance.
[0097] Therefore, after determining the vector distances between the first feature vector and each of the second feature vectors, similar character sequences to the query character sequence can be further determined from the sample character sequences based on these distances. For example, vector distances smaller than a distance threshold can be selected from these vector distances, and then the sample character sequences corresponding to the corresponding second feature vectors can be identified as similar character sequences to the query character sequence. In practical applications, this distance threshold can be 0.95, 0.9, 0.85, or other values. When the vector distance between the first feature vector and each of the second feature vectors is less than this distance threshold, it indicates that the first feature vector and the corresponding second feature vector are relatively similar. Consequently, the query character sequence and the corresponding sample character sequence are also relatively similar, and the sample character sequence can be considered as a similar character sequence to the query character sequence.
[0098] The method for querying similar character sequences provided in this application first determines the feature character segments of the character sequence to be queried, then uses the feature character segments to generate the feature vector of the character sequence to be queried, and then, based on the vector distance between the feature vector of the character sequence to be queried and the feature vectors corresponding to each sample character sequence, the similar character sequences of the character sequence to be queried can be determined from each sample character sequence.
[0099] It should be noted that, in the similar character sequence query method provided in this application embodiment, on the one hand, the similarity of the character sequence to be queried is not directly compared with the similarity of each sample character sequence. Instead, the feature vector of the character sequence to be queried is generated first, and then the similar character sequences of the character sequence to be queried are determined based on the vector distance between the feature vector of the character sequence to be queried and the corresponding feature vectors of each sample character sequence. In this process, the similarity comparison of strings is transformed into the calculation of vector distance, which can reduce the time and space complexity.
[0100] On the other hand, when generating the feature vector of the query character sequence, this method does not directly generate the feature vector from the query character sequence. Instead, it first determines the feature character segments of the query character sequence and then uses these feature character segments to generate the feature vector of the query character sequence. In this process, since the character length of the feature character segments is less than the character length of the query character sequence, the time and space complexity can also be reduced during the generation of the feature vector.
[0101] The method for querying similar character sequences provided in this application embodiment may further include determining the feature vector corresponding to each sample character sequence.
[0102] For example, in practical applications, a sample library can be provided, which includes multiple sample character sequences. Depending on the application scenario, this sample library can be a sample library of malicious code; in this case, each sample character sequence in the malicious code sample library is a character sequence of malicious code.
[0103] For the sample character sequences in the sample library, determine the feature vector corresponding to each sample character sequence. Specifically, the same method as step S32 above can be used to first determine the feature character segments of the sample character sequence, and then the same method as step S33 above can be used to generate the feature vector of the sample character sequence using the feature character segments of the sample character sequence.
[0104] In determining the characteristic character segments of a sample character sequence, a preset character length range can be determined first. Then, based on the preset character length range, multiple consecutive strings can be extracted from the sample character sequence, where the character length of each consecutive string is within the preset character length range. Next, the hash value of each consecutive string is determined. Then, based on whether the hash value of each consecutive string is divisible by a preset filtering ratio, at least one characteristic character segment of the sample character sequence can be determined from each consecutive string.
[0105] When generating the feature vector of a sample character sequence using its feature character segments, each feature character segment can be converted into a multi-dimensional vector. Then, the multi-dimensional vectors are weighted and summed. Finally, the weighted vector obtained by the weighted sum is normalized to generate the feature vector of the sample character sequence.
[0106] It should be further explained that the embodiments of this application can also provide a method for generating feature vectors. This method can generate feature vectors for a character sequence for which a feature vector is to be generated. The character sequence for which the feature vector is to be generated can be the above-mentioned query character sequence or the above-mentioned sample character sequence. Therefore, the feature vector generated by this feature vector generation method can be used for querying similar character sequences.
[0107] When generating a feature vector for a character sequence using this feature vector generation method, the feature character segments of the character sequence can be determined first, for example, by determining the feature character segments of the character sequence in the same way as step S32 above. Then, the feature vector of the character sequence can be generated using the feature character segments of the character sequence, for example, by generating the feature vector of the character sequence in the same way as step S33 above.
[0108] Based on the same inventive concept as the similar character sequence query method provided in the embodiments of this application, the embodiments of this application also provide a similar character sequence query device. For any unclear aspects of this device embodiment, please refer to the corresponding content of the method embodiment. Figure 6 The diagram shows the specific structure of the device 60, which includes a feature character fragment determination unit 601, a feature vector generation unit 602, and a similar character sequence determination unit 603, wherein:
[0109] Feature character segment determination unit 601 is used to determine the feature character segments of the character sequence to be queried;
[0110] The feature vector generation unit 602 is used to generate a feature vector of the character sequence to be queried using the feature character fragment;
[0111] The similar character sequence determination unit 603 is used to determine the similar character sequence of the query character sequence from the sample character sequences based on the vector distance between the feature vector of the query character sequence and the feature vector corresponding to each sample character sequence.
[0112] The device 60 provided in the embodiments of this application adopts the same inventive concept as the query method for similar character sequences provided in the embodiments of this application. Under the premise that the method can solve the technical problem, the device 60 can also solve the technical problem, which will not be elaborated here.
[0113] In addition, in practical applications, the technical effects achieved by combining the device 60 with specific hardware devices are also within the scope of protection of this application. For example, different units in the device 60 can be deployed in different nodes of the distributed cluster to further improve processing efficiency; or, cloud technology can be combined to reduce costs.
[0114] The device 60 may further include a query request acquisition unit for acquiring query requests for similar character sequences, wherein the query request includes the character sequence to be queried.
[0115] The feature character fragment determination unit 601 may specifically include: a continuous string extraction subunit, a hash value determination subunit, and a feature character fragment determination subunit, wherein:
[0116] The continuous string extraction subunit is used to extract multiple continuous strings from the character sequence to be queried, wherein the character length of each continuous string is within a preset character length range;
[0117] The hash value determination sub-unit is used to determine the hash value of each consecutive string;
[0118] The feature character fragment determination subunit is used to determine at least one feature character fragment of the query character sequence from the various consecutive strings based on whether the hash value of each consecutive string is divisible by a preset filtering ratio.
[0119] The feature vector generation unit 602 may specifically include a multi-dimensional vector transformation subunit, a weighted summation subunit, and a normalization subunit, wherein:
[0120] The multidimensional vector transformation subunit is used to transform each feature character fragment into a multidimensional vector;
[0121] The weighted summation subunit is used to sum the weights of the various multidimensional vectors, where the weight of each multidimensional vector is positively correlated with the character length of the corresponding feature character segment;
[0122] The normalization subunit is used to normalize the weighted vector obtained by weighted summation to generate the feature vector of the character sequence to be queried.
[0123] The multidimensional vector transformation subunit can further include: a concatenation submodule, a hash value determination submodule, and a multidimensional vector generation submodule, wherein:
[0124] The concatenation submodule is used to add multiple different dimension numbers to each feature character fragment to obtain multiple concatenated strings corresponding to the feature character fragment;
[0125] The hash value determination submodule is used to determine the hash value of each concatenated string corresponding to the feature character fragment;
[0126] The multidimensional vector generation submodule is used to generate a multidimensional vector of the feature character fragment using the hash values of each concatenated string.
[0127] The device 60 may further include a sample character sequence feature vector determination unit, used to determine the feature vector corresponding to each sample character sequence.
[0128] The device 60 may further include: a vector distance determination unit, used to determine the feature vector of the character sequence to be queried, and the Euclidean distance or cosine distance between it and the feature vector corresponding to each sample character sequence, respectively, as the vector distance.
[0129] Based on the same inventive concept as the feature vector generation method provided in the embodiments of this application, the embodiments of this application also provide a feature vector generation apparatus. For any unclear aspects of this apparatus embodiment, please refer to the corresponding content of the method embodiment. Figure 7 The diagram shows the specific structure of the device 70, which includes a second feature character fragment determination unit 701 and a second feature vector generation unit 702, wherein:
[0130] The second feature character segment determination unit 701 is used to determine the feature character segments of the character sequence to be generated as the feature vector;
[0131] The second feature vector generation unit 702 is used to generate the feature vector of the character sequence using the feature character fragment.
[0132] The device 70 can be used to generate feature vectors of the character sequence to be queried, and can also be used to generate feature vectors of sample character sequences. Therefore, it can be used to query similar character sequences, thereby solving the problems in the prior art.
[0133] This invention also provides a computer-readable storage medium storing a computer program that can be executed by a processor to complete all or part of the processes of the methods described in the above embodiments. The storage medium may be a disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc. The storage medium may also include combinations of the above types of memory.
[0134] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.
Claims
1. A method for querying similar character sequences, characterized in that, The method includes: Identify the characteristic character segments of the character sequence to be queried; Using the characteristic character fragments, a feature vector of the character sequence to be queried is generated; Based on the feature vector of the character sequence to be queried and the vector distance between it and the feature vector of each sample character sequence, similar character sequences to the character sequence to be queried are determined from each sample character sequence. Specifically, determining the characteristic character segments of the character sequence to be queried includes: Multiple consecutive strings are extracted from the character sequence to be queried, wherein the character length of each consecutive string is within a preset character length range; Determine the hash value of each consecutive string; Based on whether the hash value of each consecutive string is divisible by a preset filtering ratio, at least one characteristic character segment of the character sequence to be queried is determined from each consecutive string. Using the aforementioned feature character fragments, a feature vector for the query character sequence is generated, specifically including: Each feature character fragment is converted into a multi-dimensional vector; The multidimensional vectors are summed with weights, where the weight of each multidimensional vector is positively correlated with the character length of the corresponding feature character segment. The weighted vector obtained by weighted summation is normalized to generate the feature vector of the character sequence to be queried.
2. The method according to claim 1, characterized in that, Before determining the characteristic character segments of the character sequence to be queried, the method further includes: A query request for obtaining similar character sequences, wherein the query request includes the character sequence to be queried.
3. The method according to claim 1, characterized in that, Each feature character fragment is converted into a multi-dimensional vector, specifically including: For each feature character segment, multiple different dimension numbers are added to the feature character segment to obtain multiple concatenated strings corresponding to the feature character segment; Determine the hash value of each concatenated string corresponding to the feature character fragment; Using the hash values of each concatenated string, a multidimensional vector of the feature character fragment is generated.
4. The method according to claim 1, characterized in that, The method further includes: determining the feature vector corresponding to each sample character sequence.
5. The method according to claim 1, characterized in that, The method further includes: determining the feature vector of the character sequence to be queried, and using the Euclidean distance or cosine distance between it and the feature vector corresponding to each sample character sequence as the vector distance.
6. A method for generating feature vectors, characterized in that, include: Identify the characteristic character segments of the character sequence to be used to generate the feature vector; Using the characteristic character fragments, a feature vector of the character sequence is generated; Specifically, determining the characteristic character segments of the character sequence to be queried includes: Multiple consecutive strings are extracted from the character sequence to be queried, wherein the character length of each consecutive string is within a preset character length range; Determine the hash value of each consecutive string; Based on whether the hash value of each consecutive string is divisible by a preset filtering ratio, at least one characteristic character segment of the character sequence to be queried is determined from each consecutive string. Using the aforementioned feature character fragments, a feature vector for the query character sequence is generated, specifically including: Each feature character fragment is converted into a multi-dimensional vector; The multidimensional vectors are summed with weights, where the weight of each multidimensional vector is positively correlated with the character length of the corresponding feature character segment. The weighted vector obtained by weighted summation is normalized to generate the feature vector of the character sequence to be queried.
7. An electronic device, characterized in that, include: processor; Memory for storing processor-executable instructions; wherein the processor is configured to perform the method as described in any one of claims 1-6.
8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that can be executed by a processor to perform the method described in any one of claims 1-6.