A method, device and equipment for processing text data

Through vectorization processing and similarity analysis, search vocabulary data is generated, which solves the problem of data leakage when ES storing documents, and realizes the protection and efficient retrieval of text data.

CN118673118BActive Publication Date: 2024-11-29BEIJING SUWELL TECHNOLOGY CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410815468.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-24
Publication Date
2024-11-29
Estimated Expiration
2044-06-24

AI Technical Summary

Technical Problem

When ES stores documents, data leakage is easily caused by the index information because the data of the original document is contained.

Method used

By vectorizing the text data to be retrieved, a search text vector is generated, and search vocabulary data is generated based on the similarity between the search text vector and multiple document vectors, and mapping conversion is performed to avoid storing the original document content, and content protection is achieved Efficient search.

Benefits of technology

It realizes protection of text data while maintaining efficient retrieval capabilities, avoiding the risk of storing original document content in the index.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118673118B_ABST
    Figure CN118673118B_ABST
Patent Text Reader

Abstract

The present invention provides a method, device and equipment for processing text data, which belongs to the field of computer information processing technology, and solves the problem that text retrieval based on original distributed search easily leads to leakage of original data. The method includes: obtaining text data to be retrieved; vectorizing characters in the text data to be retrieved to obtain retrieval text vectors; obtaining retrieval vocabulary data according to the similarity between the retrieval text vector and multiple document vectors, wherein the multiple document vectors are obtained by vectorizing each character in multiple text data; mapping and converting the retrieval vocabulary data with the vocabulary file to obtain target retrieval data of the text data to be retrieved, wherein the vocabulary file is obtained by traversing characters in multiple document data. This solution can avoid storing the original document content in the index by converting the text into the form of vectors and vocabulary files, thereby realizing efficient retrieval while protecting the content.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of computer information processing, and in particular to a text data processing method, device and equipment. Background Art

[0002] In the context of the rapid development of the data era and large models, the importance of data content security protection mechanisms cannot be ignored. By implementing effective protection measures, the security and integrity of data can be ensured, providing solid support for the training and application of large models. ES (elastic search) is an open source distributed search and analysis engine that provides a distributed full-text search engine that can store, retrieve and analyze large-scale data in real time. ES stores data in a structure called an "index". Each index can contain multiple documents, each of which has a unique identifier and multiple attribute fields. For text data, ES performs a series of processing and analysis, including word segmentation, stop word removal, stemming, etc. ES uses an inverted index to store data. The inverted index consists of two parts: one is a word dictionary, which is used to record the words of all documents and the list of documents corresponding to each word; the other is an inverted list, which is used to record the specific occurrence position of each word in each document. This structure enables ES to quickly find relevant documents based on query conditions, but because the index information carries the data of the original document, the original text can be restored through the index data, which easily exposes the original data content and leads to data leakage. Summary of the invention

[0003] The present invention provides a method, device and equipment for processing text data, which solves the problem that when storing documents based on ES, since the index information carries the data of the original document, the original text can be restored through the index data, which easily exposes the original data content and causes data leakage.

[0004] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0005] An embodiment of the present invention provides a method for processing text data, comprising:

[0006] Get the text data to be retrieved;

[0007] Vectorizing the characters in the text data to be retrieved to obtain a retrieval text vector;

[0008] Obtaining retrieval vocabulary data according to the similarity between the retrieval text vector and a plurality of document vectors, wherein the plurality of document vectors are obtained by vectorizing each character in the plurality of text data;

[0009] The search word list data and the word list file are mapped and converted to obtain target search data of the text data to be searched, wherein the word list file is obtained by traversing the characters in the plurality of document data.

[0010] Optionally, vectorizing the characters in the text data to be retrieved to obtain a retrieval text vector includes:

[0011] Obtaining the position vector of each character in the text data to be retrieved and performing linearization processing to obtain an intermediate result;

[0012] The intermediate result is input into the text conversion model for processing to obtain the search text vector.

[0013] Optionally, obtaining the position vector of each character in the text data to be retrieved and performing linearization processing to obtain an intermediate result includes:

[0014] according to

[0015]

[0016] Get the position vector of each character, where PE pos is the position vector, p os is the absolute position of the character in the text, i is a positive integer, d model is the vector dimension;

[0017] Perform linear weighting on the position vector of each character to obtain an intermediate result.

[0018] Optionally, the intermediate result is input into a text conversion model for processing to obtain a search text vector, including:

[0019] The text conversion model uses the objective function:

[0020] FFN(x)=max(0,xW1+b1)W2+b2,

[0021] The intermediate result is processed to obtain a search text vector, where x is the intermediate result, and W1, W2, b1, and b2 are preset parameters of the text conversion model.

[0022] Optionally, obtaining search word list data according to the similarity between the search text vector and a plurality of document vectors includes:

[0023] according to

[0024]

[0025] Get the similarity data between the retrieved text vector and multiple document vectors, where a1, a2, ..., a nare the elements in vector a, b1, b2, …, b n is an element in vector b, ||a|| and ||b|| are the moduli of vector a and vector b respectively, vector a is the retrieved text vector, and vector b is a plurality of document vectors;

[0026] According to the similarity data, multiple similar documents are obtained;

[0027] According to the identification data of the plurality of similar documents in the vocabulary file, retrieval vocabulary data is obtained.

[0028] Optionally, mapping and converting the search word table data and the word table file to obtain target search data of the text data to be searched includes:

[0029] Comparing the search word table data with the identification data in the word table file in sequence, and when the search word table data and the identification data are equal, obtaining the characters in the word table file corresponding to the identification data;

[0030] The acquired characters are outputted in sequence to obtain target search data of the text data to be searched.

[0031] Optionally, traversing the characters in the plurality of document data includes:

[0032] Obtain each character in the plurality of document data and construct a character set;

[0033] Setting identification data for each character to construct an identification data set, wherein each character uniquely corresponds to one identification data;

[0034] Traversing all character combinations in the plurality of document data to obtain frequency data of character occurrences;

[0035] Combining the two characters with the largest frequency data to obtain a combined character, and appending the combined character to the character set;

[0036] Setting combined character identification data for the combined character, and adding the combined character identification data to the identification data set;

[0037] The character set is combined with the identification data set to obtain the vocabulary file, wherein each character in the character set corresponds to an identification data in the identification data set.

[0038] The embodiment of the present invention further provides a text data processing device, comprising:

[0039] An acquisition module, used for acquiring text data to be retrieved;

[0040] A processing module, used for vectorizing the characters in the text data to be retrieved to obtain a retrieval text vector;

[0041] A generating module, used to obtain the search word list data according to the similarity between the search text vector and a plurality of document vectors, wherein the plurality of document vectors are obtained by vectorizing each character in the plurality of text data;

[0042] The conversion module is used to map and convert the search word table data with the word table file to obtain the target search data of the text data to be searched, wherein the word table file is obtained by traversing the characters in the multiple document data.

[0043] An embodiment of the present invention further provides a computing device, comprising: a processor and a memory storing a computer program, wherein the computer program executes the above method when executed by the processor.

[0044] An embodiment of the present invention further provides a computer-readable storage medium storing instructions, which, when executed on a computer, enable the computer to execute the above method.

[0045] The technical solution of the present invention includes at least the following effects:

[0046] The above scheme of the present invention obtains the text data to be retrieved; vectorizes the characters in the text data to be retrieved to obtain the retrieval text vector; obtains the retrieval vocabulary data according to the similarity between the retrieval text vector and multiple document vectors, wherein the multiple document vectors are obtained by vectorizing each character in the multiple text data; maps and converts the retrieval vocabulary data with the vocabulary file to obtain the target retrieval data of the text data to be retrieved, wherein the vocabulary file is obtained by traversing the characters in the multiple document data, and the text can be converted into the form of vectors and vocabulary files, which can avoid storing the original document content in the index, and realizes the protection of the content while enabling efficient retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flowchart of a method for processing text data provided by an embodiment of the present invention;

[0048] Figure 2 is a structural diagram of a text data processing device provided by an embodiment of the present invention;

[0049] Figure 3 It is a schematic diagram of the structure of a computing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0051] like Figure 1 As shown, an embodiment of the present invention provides a method for processing text data, comprising:

[0052] Step 11, obtaining the text data to be retrieved;

[0053] Step 12, vectorizing the characters in the text data to be retrieved to obtain a retrieval text vector;

[0054] Step 13, obtaining retrieval vocabulary data according to the similarity between the retrieval text vector and a plurality of document vectors, wherein the plurality of document vectors are obtained by vectorizing each character in the plurality of text data;

[0055] Step 14, mapping and converting the search word list data and the word list file to obtain target search data of the text data to be searched, wherein the word list file is obtained by traversing the characters in the plurality of document data.

[0056] In this example, the text data processing method obtains a retrieval text vector by vectorizing the characters in the text data to be retrieved; wherein the retrieval text vector is irreversible and cannot be restored to the original text content by conversion, but the retrieval text vector can perform semantic retrieval; based on the similarity between the retrieval text vector and multiple document vectors, retrieval vocabulary data is obtained, wherein the multiple document vectors are obtained by vectorizing each character in multiple text data; the retrieval vocabulary data and the vocabulary file are mapped and converted to obtain target retrieval data of the text data to be retrieved; wherein, in order to avoid storing the original document content in the index, the content retrieval function can be realized based on the text vector retrieval, which not only realizes content protection but also enables efficient retrieval.

[0057] In an optional embodiment, the embodiment of the present invention provides that step 12 may include:

[0058] Step 121, obtaining the position vector of each character in the text data to be retrieved and performing linearization processing to obtain an intermediate result;

[0059] Step 122: input the intermediate result into a text conversion model for processing to obtain a search text vector.

[0060] Specifically, for a given text data to be retrieved, each character in the text is vectorized, that is, each character is converted into a vector with a dimension of 512; wherein, special characters [CLS] and [SEP] are appended before and after the text, for example: [CLS] text content [SEP]; the position vector PE (positional encoding) of each character is obtained, and according to

[0061]

[0062] Get the position vector of each character, where PE pos is the position vector, p os is the absolute position of the character in the text, p os =0, 1, 2, ..., i is a positive integer, indicating the dimension in the word vector, d model is the vector dimension; for example, d model =512, then i=0, 1, 2,..., 255;

[0063] After adding the word embedding and the position vector, we get the final vector representation of each word, and multiply the final vector with the linear weight matrix to get the query vector, key vector, and value vector, namely:

[0064] Q=(E Embedding +PE pos )×W Q

[0065] K=(E Embedding +PE pos )×W K

[0066] V=(E Embedding +PE pos )×W V

[0067] Among them, Q is the query vector; K is the key vector; V is the value vector; E Embedding is word embedding; W Q , W K , W V is the linear weight matrix;

[0068] according to:

[0069]

[0070] x=Concat(head1,...,head n )W o

[0071] Get the intermediate result x; where, are parameter matrices, d model =512,d k =d v =64;

[0072] Through the objective function:

[0073] FFN(x)=max(0,xW1+b1)W2+b2,

[0074] The intermediate result x is processed to obtain the search text vector, where x is the intermediate result, W1, W2, b1, and b2 are the preset parameters of the text conversion model; the vector of each character in the final text is calculated, and the vector of [CLS] is taken as the total vector of the text. The final vector of the text is as follows:

[0075] [-0.00849278923124075,0.005225188098847866,-0.008979364298284054,0.05190980061888695,-0.04316289350390434,0.03323003649711609,……,0.0198740903288126,-0.012753183022141457,0.01027618907392025,0.006965011823922396], a total of 768 numbers forming a 768-dimensional vector.

[0076] In an optional embodiment, the embodiment of the present invention provides that step 13 may include:

[0077] Step 131, according to

[0078]

[0079] Get the similarity data between the retrieved text vector and multiple document vectors, where a1, a2, ..., a n are the elements in vector a, b1, b2, …, b n is an element in vector b, ||a|| and ||b|| are the moduli of vector a and vector b respectively, vector a is the retrieved text vector, and vector b is a plurality of document vectors;

[0080] Step 132, obtaining a plurality of similar documents according to the similarity data;

[0081] Step 133: Obtain search vocabulary data according to the identification data of the multiple similar documents in the vocabulary file.

[0082] Specifically, all texts that need to be stored, for example, 10,000, are vectorized and stored in the ES library. The ES will store 10,000 768-dimensional vectors. For the text to be searched, the same vectorization algorithm is used to convert it into a 768-dimensional vector. The semantic retrieval is achieved by calculating the similarity between the vector of the text to be searched and the 10,000 vectors in the library. There are many ways to calculate the similarity of vectors, such as cosine similarity, which is based on

[0083]

[0084] Get the similarity data between the retrieved text vector and multiple document vectors, where a1, a2, ..., a n are the elements in vector a, b1, b2, …, b n is an element in vector b, ||a|| and ||b|| are the moduli of vector a and vector b respectively, vector a is the retrieval text vector, and vector b is a plurality of document vectors; the larger the cosine similarity value, the more similar the two vectors are; the retrieval vocabulary data is obtained according to the identification data of the plurality of similar documents in the vocabulary file.

[0085] In an optional embodiment, the embodiment of the present invention provides that step 14 may include:

[0086] Step 141, comparing the search word table data with the identification data in the word table file in sequence, and when the search word table data and the identification data are equal, obtaining the characters in the word table file corresponding to the identification data;

[0087] Step 142, output the acquired characters in sequence to obtain target search data of the text data to be searched.

[0088] In this example, the search vocabulary data is compared with the identification data in the vocabulary file in turn. When the search vocabulary data is equal to the identification data, the characters corresponding to the identification data in the vocabulary file are obtained; the obtained characters are output in turn to obtain the target search data of the text data to be retrieved; specifically, assuming that the search vocabulary data of a given text is [1,3,4], the characters corresponding to the identification data in the vocabulary file are obtained as follows: {I: 1, we: 2, you: 3, he: 4}, then all the search vocabulary data of the text can be restored to "I, you, and he" through the vocabulary.

[0089] The embodiment of the present invention proposes an optional embodiment, wherein the characters in the plurality of document data are traversed and processed, including:

[0090] Step 151, obtaining each character in the plurality of document data and constructing a character set;

[0091] Step 152, setting identification data for each character to construct an identification data set, wherein each character uniquely corresponds to one identification data;

[0092] Step 153, traversing all character combinations in the plurality of document data to obtain frequency data of character occurrences;

[0093] Step 154, combining the two characters with the largest frequency data to obtain a combined character, and appending the combined character to the character set;

[0094] Step 155, setting combined character identification data for the combined character, and adding the combined character identification data to the identification data set;

[0095] Step 156: Combine the character set with the identification data set to obtain the vocabulary file, wherein each character in the character set corresponds to an identification data in the identification data set.

[0096] In this example, each character in multiple document data is converted into the format of {text:id} and finally saved as a .tiktoken file; this file is a vocabulary (identification data set) that stores each word and the corresponding id (identification data); the text is mapped and converted using the vocabulary to obtain the token_id (identification data) of the text; the specific process includes:

[0097] (1) Count all the characters in the training corpus (such as single Chinese characters);

[0098] (2) Initialize the vocabulary and merge rule list. The vocabulary is the set of all single characters, that is, traverse all possible character sets in the corpus. For example, the initialized vocabulary is: I: 1, we: 2, you: 3, he: 4;

[0099] (3) Traverse all possible combinations, count the frequencies of occurrence, combine the two characters with the highest frequency, for example, "we", add "we" to the vocabulary, and add the merged rules to the merge rule list;

[0100] (4) Repeat step (3) until the vocabulary size reaches a preset size, for example, the vocabulary size of "we" is 151856.

[0101] Assume that the given text: I, you, and he;

[0102] Given the vocabulary: I: 1, we: 2, you: 3, he: 4;

[0103] The token_id of the text can be converted to [1,3,4] through the vocabulary and stored in ES as [1,3,4].

[0104] Based on secure encryption media, the .tiktoken file is encrypted and stored using the ECC non-encrypted algorithm. During use, it is decrypted in memory before use.

[0105] A specific embodiment of the text data processing method provided by the embodiment of the present invention is:

[0106] Step 1: Process multiple document data to generate multiple document vectors.

[0107] For a given set of document data, each character in the text is vectorized, that is, each character is converted into a 512-dimensional vector; in which, special characters [CLS] and [SEP] are appended before and after the text, for example: [CLS] text content [SEP]; the position vector PE (positional encoding) of each character is obtained, according to

[0108]

[0109] Get the position vector of each character, where PE pos is the position vector, p os is the absolute position of the character in the text, p os =0, 1, 2, ..., i is a positive integer, indicating the dimension in the word vector, d model is the vector dimension; for example, d model =512, then i=0, 1, 2,..., 255;

[0110] After adding the word embedding and the position vector, we get the final vector representation of each word, and multiply the final vector with the linear weight matrix to get the query vector, key vector, and value vector, namely:

[0111] Q=(E Embedding +PE pos )×W Q

[0112] K=(E Embedding +PE pos )×W K

[0113] V=(E Embedding +PE pos )×W V

[0114] Among them, Q is the query vector; K is the key vector; V is the value vector; E Embedding is word embedding; W Q , W K , WV is the linear weight matrix;

[0115] according to:

[0116]

[0117] x=Concat(head1,...,head n )W o

[0118] Get the intermediate result x; where, are parameter matrices, d model =512,d k =d v =64;

[0119] Through the objective function:

[0120] FFN(x)=max(0,xW1+b1)W2+b2,

[0121] The intermediate result x is processed to obtain multiple document vectors, where x is the intermediate result, W1, W2, b1, and b2 are the preset parameters of the text conversion model; the vector of each character in the final text is calculated, and the vector of [CLS] is taken as the total vector of the text. The final vector of the text is as follows: [-0.00849278923124075, 0.005225188098847866, -0.00897936429828405 4,0.05190980061888695,-0.04316289350390434,0.03323003649711609,……,0.0198740903288126,-0.012753183022141457,0.01027618907392025,0.006965011823922396], a total of 768 numbers forming a 768-dimensional vector.

[0122] Step 2: Process multiple document data to generate a vocabulary file.

[0123] Convert each character in multiple document data into the format of {text:id} and finally save it as a .tiktoken file; this file is a vocabulary (identification data set) that stores each word and the corresponding id (identification data); use the vocabulary to map and convert the text to get the token_id (identification data) of the text; the specific process includes:

[0124] (1) Count all the characters in the training corpus (such as single Chinese characters);

[0125] (2) Initialize the vocabulary and merge rule list. The vocabulary is the set of all single characters, that is, traverse all possible character sets in the corpus. For example, the initialized vocabulary is: I: 1, we: 2, you: 3, he: 4;

[0126] (3) Traverse all possible combinations, count the frequencies of occurrence, combine the two characters with the highest frequency, for example, "we", add "we" to the vocabulary, and add the merged rules to the merge rule list;

[0127] (4) Repeat step (3) until the vocabulary size reaches a preset size, for example, the vocabulary size of "we" is 151856.

[0128] Assume that the given text: I, you, and he;

[0129] Given the vocabulary: I: 1, we: 2, you: 3, he: 4;

[0130] The token_id of the text can be converted to [1,3,4] through the vocabulary and stored in es as [1,3,4].

[0131] Based on secure encryption media, the .tiktoken file is encrypted and stored using the ECC non-encrypted algorithm. During use, it is decrypted in memory before use.

[0132] Step 3: Process the text to be retrieved and generate a retrieval text vector.

[0133] According to the method of processing multiple document data and generating multiple document vectors in step 1, the text to be retrieved is processed to generate a retrieval text vector.

[0134] Step 4: determine the similarity between the retrieved text vector and multiple document vectors, and obtain multiple similar documents based on the similarity data.

[0135] For all texts that need to be stored in the database (i.e., multiple documents), for example, 10,000, all are vectorized and stored in the es database, which will store 10,000 768-dimensional vectors; for the text to be retrieved, the same vectorization algorithm is used to convert it into a 768-dimensional retrieval text vector, and semantic retrieval is achieved by calculating the similarity between the retrieval text vector and the 10,000 vectors (i.e., multiple document vectors) in the database, that is, calculating the cosine similarity:

[0136]

[0137] Get the similarity data between the retrieved text vector and multiple document vectors, where a1, a2, ..., a nare the elements in vector a, b1, b2, …, b n is an element in vector b, ||a|| and ||b|| are the moduli of vector a and vector b respectively, vector a is the retrieved text vector, and vector b is a plurality of document vectors; the larger the cosine similarity value, the more similar the two vectors are; and multiple similar documents are determined based on the similarity data.

[0138] Step 5, obtaining identification data of multiple similar documents to obtain search word list data;

[0139] Step 6, compare the search vocabulary data with the identification data in the vocabulary file in turn, and when the search vocabulary data is equal to the identification data, obtain the characters corresponding to the identification data in the vocabulary file; output the obtained characters in turn to obtain the target search data of the text data to be retrieved; assuming that the search vocabulary data of a given text is [1,3,4], the characters corresponding to the identification data in the vocabulary file are: {I: 1, we: 2, you: 3, he: 4}, then all the search vocabulary data of the text can be restored to "I, you, he" through the vocabulary.

[0140] The text data processing method proposed in the present invention can avoid storing the original document content in the index by converting the text into the form of vectors and word list files, thereby achieving efficient retrieval while protecting the content.

[0141] like Figure 2 As shown, the embodiment of the present invention further provides a text data processing device 20, the processing device 20 comprising:

[0142] An acquisition module 21 is used to acquire text data to be retrieved;

[0143] A processing module 22, used for vectorizing the characters in the text data to be searched to obtain a search text vector;

[0144] A generating module 23, configured to obtain search word list data according to the similarity between the search text vector and a plurality of document vectors, wherein the plurality of document vectors are obtained by vectorizing each character in the plurality of text data;

[0145] The conversion module 24 is used to perform mapping conversion processing on the search word list data and the word list file to obtain the target search data of the text data to be searched, wherein the word list file is obtained by traversing the characters in the plurality of document data.

[0146] Optionally, the processing module 22 is specifically used for:

[0147] Obtaining the position vector of each character in the text data to be retrieved and performing linearization processing to obtain an intermediate result;

[0148] The intermediate result is input into the text conversion model for processing to obtain the search text vector.

[0149] Optionally, obtaining the position vector of each character in the text data to be retrieved and performing linearization processing to obtain an intermediate result includes:

[0150] according to

[0151]

[0152] Get the position vector of each character, where PE pos is the position vector, p os is the absolute position of the character in the text, i is a positive integer, d model is the vector dimension;

[0153] Perform linear weighting on the position vector of each character to obtain an intermediate result.

[0154] Optionally, the intermediate result is input into a text conversion model for processing to obtain a search text vector, including:

[0155] The text conversion model uses the objective function:

[0156] FFN(x)=max(0,xW1+b1)W2+b2,

[0157] The intermediate result is processed to obtain a search text vector, where x is the intermediate result, and W1, W2, b1, and b2 are preset parameters of the text conversion model.

[0158] Optionally, the generating module 23 is specifically used for:

[0159] according to

[0160]

[0161] Get the similarity data between the retrieved text vector and multiple document vectors, where a1, a2, ..., a n are the elements in vector a, b1, b2, …, b n is an element in vector b, ||a|| and ||b|| are the moduli of vector a and vector b respectively, vector a is the retrieved text vector, and vector b is a plurality of document vectors;

[0162] According to the similarity data, multiple similar documents are obtained;

[0163] According to the identification data of the plurality of similar documents in the vocabulary file, retrieval vocabulary data is obtained.

[0164] Optionally, the conversion module 24 is specifically used for:

[0165] Comparing the search word table data with the identification data in the word table file in sequence, and when the search word table data and the identification data are equal, obtaining the characters in the word table file corresponding to the identification data;

[0166] The acquired characters are outputted in sequence to obtain target search data of the text data to be searched.

[0167] Optionally, traversing the characters in the plurality of document data includes:

[0168] Obtain each character in the plurality of document data and construct a character set;

[0169] Setting identification data for each character to construct an identification data set, wherein each character uniquely corresponds to one identification data;

[0170] Traversing all character combinations in the plurality of document data to obtain frequency data of character occurrences;

[0171] Combining the two characters with the largest frequency data to obtain a combined character, and appending the combined character to the character set;

[0172] Setting combined character identification data for the combined character, and adding the combined character identification data to the identification data set;

[0173] The character set is combined with the identification data set to obtain the vocabulary file, wherein each character in the character set corresponds to an identification data in the identification data set.

[0174] It should be noted that the device is a device corresponding to the above method, and all implementation methods in the above method embodiment are applicable to this embodiment and can achieve the same technical effect.

[0175] like Figure 3 As shown, the embodiment of the present invention further provides a computing device 30, including a processor 31, a memory 32, a program or instruction stored in the memory 32 and executable on the processor 31, and when the program or instruction is executed by the processor 31, each process of the above-mentioned text data processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. It should be noted that the computing device in the embodiment of the present invention includes the above-mentioned mobile electronic device and non-mobile electronic device.

[0176] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0177] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0178] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0179] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0180] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0181] Stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical disks.

[0182] In addition, it should be noted that in the apparatus and method of the present invention, it is obvious that each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent schemes of the present invention. Moreover, the steps of performing the above-mentioned series of processing can naturally be performed in chronological order according to the order of description, but it is not necessary to perform them in chronological order, and some steps can be performed in parallel or independently of each other. For those of ordinary skill in the art, it is understood that all or any steps or components of the method and apparatus of the present invention can be implemented in any computing device (including processors, storage media, etc.) or a network of computing devices in hardware, firmware, software or a combination thereof, which can be achieved by those of ordinary skill in the art using their basic programming skills after reading the description of the present invention.

[0183] Therefore, the purpose of the present invention can also be achieved by running a program or a group of programs on any computing device. The computing device can be a well-known general-purpose device. Therefore, the purpose of the present invention can also be achieved by simply providing a program product containing a program code for implementing a method or device. That is to say, such a program product also constitutes the present invention, and a storage medium storing such a program product also constitutes the present invention. Obviously, the storage medium can be any well-known storage medium or any storage medium developed in the future. It should also be pointed out that in the device and method of the present invention, it is obvious that each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent schemes of the present invention. In addition, the steps of performing the above-mentioned series of processing can naturally be performed in chronological order according to the order of description, but it is not necessary to perform them in chronological order. Some steps can be performed in parallel or independently of each other.

[0184] The above are preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for processing text data, characterized in that: include: Get the text data to be retrieved; Vectorizing the characters in the text data to be retrieved to obtain a retrieval text vector; Obtaining search word list data according to the similarity between the search text vector and multiple document vectors, wherein the multiple document vectors are obtained by vectorizing each character in the multiple document data; Mapping and converting the search word table data with the word table file to obtain target search data of the text data to be searched, wherein the word table file is obtained by traversing the characters in the plurality of document data; The traversal process of the characters in the plurality of document data includes: Obtain each character in the plurality of document data and construct a character set; Setting identification data for each character to construct an identification data set, wherein each character uniquely corresponds to one identification data; Traversing all character combinations in the plurality of document data to obtain frequency data of character occurrences; Combining the two characters with the largest frequency data to obtain a combined character, and appending the combined character to the character set; Setting combined character identification data for the combined character, and adding the combined character identification data to the identification data set; The character set is combined with the identification data set to obtain the vocabulary file, wherein each character in the character set corresponds to an identification data in the identification data set.

2. The text data processing method according to claim 1, characterized in that: Vectorizing the characters in the text data to be retrieved to obtain a retrieval text vector includes: Obtaining the position vector of each character in the text data to be retrieved and performing linearization processing to obtain an intermediate result; The intermediate result is input into the text conversion model for processing to obtain the search text vector.

3. The text data processing method according to claim 2, characterized in that: Obtaining the position vector of each character in the text data to be retrieved and performing linearization processing to obtain an intermediate result, including: according to , Get the position vector of each character, where PE pos is the position vector, p os is the absolute position of the character in the text, i is a positive integer, d model is the vector dimension; Perform linear weighting on the position vector of each character to obtain an intermediate result.

4. The text data processing method according to claim 3, characterized in that: The intermediate result is input into the text conversion model for processing to obtain the search text vector, including: The text conversion model uses the objective function: FFN ( x )=max(0, xW 1+ b 1) W 2+ b 2, Process the intermediate results to obtain the retrieved text vector, where: x is the intermediate result, W 1. W 2. b 1. b 2 is the preset parameters of the text conversion model.

5. The text data processing method according to claim 4, characterized in that: According to the similarity between the search text vector and multiple document vectors, the search word list data is obtained, including: according to , The similarity data between the retrieved text vector and multiple document vectors are obtained, wherein: a 1, a 2,…, a n is a vector a The elements in b 1, b 2,…, b n is a vector b The elements in || a || and || b || are vectors a and vector b The modulus of the vector a is the retrieved text vector, vector b are multiple document vectors; According to the similarity data, multiple similar documents are obtained; According to the identification data of the plurality of similar documents in the vocabulary file, retrieval vocabulary data is obtained.

6. The text data processing method according to claim 1, characterized in that: The retrieval word table data and the word table file are mapped and converted to obtain the target retrieval data of the text data to be retrieved, including: Comparing the search word table data with the identification data in the word table file in sequence, and when the search word table data and the identification data are equal, obtaining the characters in the word table file corresponding to the identification data; The acquired characters are outputted in sequence to obtain target search data of the text data to be searched.

7. A text data processing device, characterized in that: The processing device comprises: An acquisition module, used for acquiring text data to be retrieved; A processing module, used for vectorizing the characters in the text data to be retrieved to obtain a retrieval text vector; A generating module, used to obtain the search word list data according to the similarity between the search text vector and a plurality of document vectors, wherein the plurality of document vectors are obtained by vectorizing each character in the plurality of document data; A conversion module, used for mapping and converting the search word table data with a word table file to obtain target search data of the text data to be searched, wherein the word table file is obtained by traversing the characters in the plurality of document data; The traversal process of the characters in the plurality of document data includes: Obtain each character in the plurality of document data and construct a character set; Setting identification data for each character to construct an identification data set, wherein each character uniquely corresponds to one identification data; Traversing all character combinations in the plurality of document data to obtain frequency data of character occurrences; Combining the two characters with the largest frequency data to obtain a combined character, and appending the combined character to the character set; Setting combined character identification data for the combined character, and adding the combined character identification data to the identification data set; The character set is combined with the identification data set to obtain the vocabulary file, wherein each character in the character set corresponds to an identification data in the identification data set.

8. A computing device, characterized in that include: A processor and a memory storing a computer program, wherein when the computer program is executed by the processor, the method according to any one of claims 1 to 6 is executed.

9. A computer-readable storage medium, characterized in that: Instructions are stored, and when the instructions are executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Theme-based medical literature retrieval method and system, storage medium and terminal

    CN115658851A

  • Method and device for retrieving similar document

    JP2006146355A