Data processing method and device, equipment, medium and product

Abstract

The invention discloses a data processing method and device, equipment, a medium and a product, and relates to the technical field of artificial intelligence, and the method comprises the steps: responding to the received input content, carrying out the word segmentation of the input content, and obtaining each target lexical element; labeling each target lexical element based on a preset hash table to obtain a labeling result; the method comprises the following steps: obtaining a labeling result, screening each target lexical element according to the labeling result to obtain target content, labeling each target lexical element obtained by word segmentation through a preset hash table to determine valuable words in the input content, screening each target lexical element according to the labeling result to obtain the target content, and labeling each target lexical element obtained by word segmentation through the preset hash table to determine the valuable words in the input content. It can be ensured that the target content does not contain valuable words, the number of lexical elements in the target content is reduced, and then the calculation unit overhead for follow-up processing of the target content can be saved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a data processing method, device, equipment, medium and product. Background Art

[0002] Large Language Models (LLMs) are natural language processing models based on deep learning that have the ability to handle complex language tasks.

[0003] When reasoning with a large language model, the input sequence needs to be segmented to decompose the text into smaller units - tokens. Since some words in the input sequence (such as auxiliary words and conjunctions in Chinese) cannot provide valuable information, the tokens contain some worthless information after word segmentation, which leads to an increase in the number of tokens. More tokens mean a larger tensor dimension, which in turn leads to more computing unit overhead.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide a data processing method, device, equipment, medium and product, aiming to reduce the computing unit overhead in the model reasoning process.

[0006] To achieve the above objectives, the present application provides a data processing method, the method comprising:

[0007] In response to receiving input content, segmenting the input content to obtain target word units;

[0008] Marking each target word based on a preset hash table to obtain a marking result;

[0009] The target word-grams are screened according to the marking results to obtain target content.

[0010] In one embodiment, before the step of labeling each target word based on a preset hash table to obtain a labeling result, the step further includes:

[0011] Constructing a preset auxiliary word table, wherein the auxiliary word table includes each preset word and the part-of-speech configuration corresponding to each preset word;

[0012] An original hash table is created, and each preset word in the preset auxiliary word table is inserted into the original hash table according to the part-of-speech configuration corresponding to each preset word, so as to obtain the preset hash table.

[0013] In one embodiment, the step of labeling each target word based on a preset hash table to obtain a labeling result includes:

[0014] Performing hash processing on each target word to determine an index value corresponding to each target word;

[0015] Search the preset hash table according to the index value to determine the found preset word;

[0016] The target word-units are annotated according to the found preset words to obtain the annotated result.

[0017] In one embodiment, the step of searching the preset hash table according to the index value to determine the found preset word includes:

[0018] According to the index value, the preset words corresponding to the target word-units are searched in parallel in the preset hash table through multiple threads; or,

[0019] According to the segmentation order of each target word unit, the preset words corresponding to each target word unit are searched in sequence in the preset hash table according to the index value.

[0020] In one embodiment, the step of screening the target word-grams according to the annotation results to obtain the target content includes:

[0021] Obtaining the semantic understanding accuracy corresponding to the input content, and determining the part of speech to be filtered according to the semantic understanding accuracy;

[0022] The target word-grams that match the part of speech to be filtered are deleted from the target word-grams according to the marking results to obtain the target content.

[0023] In one embodiment, after the step of screening the target words according to the marking results to obtain the target content, the step further includes:

[0024] The target content is input into a large language model so that the large language model makes predictions based on the target content to obtain reply content corresponding to the target content.

[0025] In addition, to achieve the above-mentioned purpose, the present application also proposes a data processing device, the data processing device comprising:

[0026] A word segmentation module, configured to segment the input content in response to receiving the input content, to obtain target word units;

[0027] A labeling module, used to label each target word based on a preset hash table to obtain a labeling result;

[0028] The screening module is used to screen the target words according to the marking results to obtain the target content.

[0029] In addition, to achieve the above objectives, the present application also proposes a data processing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data processing method described above.

[0030] In addition, to achieve the above objectives, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data processing method described above are implemented.

[0031] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the data processing method described above are implemented.

[0032] One or more technical solutions proposed in this application have at least the following technical effects:

[0033] In response to receiving input content, the input content is segmented to obtain target word-grams; the target word-grams are labeled based on a preset hash table to obtain a labeling result; the target word-grams are screened according to the labeling result to obtain target content, and the target word-grams obtained by the segmentation are labeled using a preset hash table to determine worthless words in the input content; the target word-grams are screened according to the labeling result to obtain the target content, which can ensure that the target content does not contain worthless words, reduce the number of word-grams in the target content, and thus save the computing unit overhead for subsequent processing of the target content. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0035] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0036] Figure 1 A flowchart of the first embodiment of the data processing method of the present application is provided;

[0037] Figure 2A flowchart of the second embodiment of the data processing method of this application is provided;

[0038] Figure 3 A flowchart of the third embodiment of the data processing method of this application is provided;

[0039] Figure 4 This is a schematic diagram of the overall process in the embodiment of the present application;

[0040] Figure 5 This is a schematic diagram of the module structure of the data processing device according to an embodiment of the present application;

[0041] Figure 6 Schematic diagram of the device structure of the hardware operating environment involved in the data processing method in the embodiment of the present application.

[0042] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0043] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0044] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0045] The main solution of the embodiment of the present application is: in response to receiving input content, the input content is segmented to obtain target word elements; the target word elements are labeled based on a preset hash table to obtain a labeling result; the target word elements are screened according to the labeling result to obtain the target content, and the target word elements obtained by the segmentation are labeled by the preset hash table to determine the worthless words in the input content; the target word elements are screened according to the labeling result to obtain the target content, which can ensure that the target content does not contain worthless words, reduce the number of word elements in the target content, and thus save the computing unit overhead for subsequent processing of the target content.

[0046] In this embodiment, for the convenience of description, the following description is made by taking the identification data processing device as the execution subject.

[0047] Large Language Models (LLMs) are natural language processing models based on deep learning, which have the ability to handle complex language tasks. These models are usually composed of billions or even hundreds of billions of parameters, and can perform multiple tasks such as text generation, translation, question and answer. However, as the scale of the model increases, its reasoning performance faces many challenges, especially in real-time applications and resource-constrained environments. Large language model reasoning is usually divided into two main stages: the prefill stage and the decoder stage. In the prefill stage, the model generates context representations through training data, and in the decoder stage, the model gradually generates output text based on these representations. The main indicator of the prefill stage is the delay of the first token (word unit), and the main indicator of the decoder stage is what is commonly referred to as how many tokens are generated per second. The optimization technology of the prefill stage is mainly concerned in the embodiments of the present application.

[0048] At present, in the prior art, the pre-filling stage does not perform preprocessing after receiving the input sequence. First, tokenization is performed to decompose the text into smaller units (tokens). Then the index of the token in the vocabulary is found, and the corresponding vector representation is obtained by looking up the embedding matrix through the index. Next, position encoding is added to retain the position information of the token, and the matrix obtained by adding the position encoding is input to the large model. The input processed vector is sent to the self-attention layer. In this layer, the model calculates the query, key, and value vectors for each token. The model evaluates the relationship between each token in the input sequence. By calculating the similarity between the query and the key, the model generates attention weights and applies these weights to the value vector to aggregate relevant information from the input sequence. The aggregated information is further processed by a feedforward neural network and generates output. After processing through multiple layers, the model generates a probability distribution of the first token, usually obtained through a softmax layer. Then, the model selects the token with the highest probability as the output.

[0049] In the prior art, when a large language model is inferred, some words in the input sequence (hereinafter referred to as auxiliary words, such as auxiliary words and conjunctions in Chinese) cannot provide valuable information. If the input sequence is not preprocessed, it may contain some worthless words, which means that after word segmentation, some tokens containing worthless information will increase the number of tokens. More tokens mean a larger tensor dimension. Since the calculation process needs to save the intermediate results, it consumes more memory and bandwidth, and a larger tensor dimension requires more computing unit overhead.

[0050] This application provides a solution by introducing auxiliary word technology to filter out some words that have little effect on model reasoning, so as to achieve efficient reasoning while saving resource overhead such as reasoning memory, bandwidth memory computing unit, etc.

[0051] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a data processing device, etc. The following takes a data processing device as an example to illustrate this embodiment and the following embodiments.

[0052] Based on this, the present application embodiment provides a data processing method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the data processing method of the present application.

[0053] In this embodiment, the data processing method includes steps S10 to S30:

[0054] Step S10, in response to receiving the input content, segmenting the input content to obtain target word units;

[0055] Exemplarily, the input content of large language models (such as GPT series, BERT, etc.) usually refers to the text data provided to these models for processing. These large language models are designed to understand and generate natural language, so they can process various types of text input.

[0056] Exemplarily, the text type of the input content type may include sentences or phrases, such as "Today's weather is so nice"; it may also include paragraphs, such as "Beijing is the capital of China, with a long history and rich cultural heritage. It is not only a political center, but also a cultural and educational center."; it may also include text types such as articles or documents. In order to facilitate the explanation of the principles in the embodiments of the present application, sentences are used as an example for explanation.

[0057] Exemplarily, the format of the input content may include plain text, that is, text that is directly input without any special processing; it may also include structured data, such as JSON format, which contains multiple fields, each of which may correspond to different information, such as questions, contexts, etc.; it may also include special tags, and some models may require specific start or end tags, or separators for distinguishing different parts. For example, in some cases, there may be a tag such as ˋˋ to indicate the end of the text. The embodiments of this application are described by taking the plain text format as an example.

[0058] For example, tokenization of the input content of a large language model is a key step in Natural Language Processing (NLP). The tokenization process divides the text into smaller units, which are called "words" or "tokens". These word units can be words, subwords, characters, or other meaningful language units. Word segmentation is crucial for the model to understand and generate text because it determines how the model processes and learns text data. Word segmentation can ensure that all input text is processed in a unified manner. By using a limited number of word units to represent the text, the vocabulary of the model can be effectively managed and limited. Appropriate word segmentation can reduce the amount of data that the model needs to process, thereby increasing the speed of training and reasoning. A reasonable word segmentation strategy can help the model better capture the semantic information in the text.

[0059] Exemplarily, common word segmentation methods include rule-based word segmentation, dictionary matching, statistical word segmentation, and subword tokenization, among which rule-based word segmentation is suitable for languages ​​with obvious word boundaries such as English, for example, text is segmented according to spaces or punctuation marks; dictionary matching is to use a pre-defined dictionary to find and segment words in the text, which is suitable for languages ​​without obvious word boundaries such as Chinese; statistical word segmentation is to use statistical methods to determine the most likely word segmentation method, and contextual information can be combined to improve the accuracy of word segmentation; subword segmentation is to further decompose words into smaller subword units, and common technologies include Byte Pair Encoding (BPE), WordPiece and SentencePiece, etc. This method helps to deal with the problem of Out-of-Vocabulary (OOV) and can better handle morphologically rich languages.

[0060] Exemplarily, assume the input content is "I like to eat apples." Rule-based tokenization (such as for Chinese) may result in: ["I", "like", "to", "eat", "apples", "."], while subword tokenization (such as using BPE) may yield a more fine-grained result, for example: ["I", "li", "ke", "to", "ea", "t", "ap", "ples", "."]. During the pre-training phase of a large language model, the model learns how to predict the next token based on the tokenized data. During fine-tuning for a specific task, the same tokenizer is used to process the new input data to ensure consistency with the pre-training phase. During the inference phase, that is, when using the model to make predictions or generate text, the input text also needs to go through the same tokenization process. The Hugging Face Transformers library provides a variety of popular tokenization tools that support different models and tokenization strategies. Libraries such as spaCy and NLTK also provide powerful tokenization capabilities, suitable for different languages and application scenarios.

[0061] Exemplarily, in the embodiments of the present application, rule-based tokenization is taken as an example for illustration. For example, using the colloquial input sequence "Ha, the weather is really nice today!", the large language model tokenizes using the BPE technique, and the tokenization result is {"Ha", ",", "today", "weather", "really", "nice", "!", "!"}, and it can be calculated that there are a total of 8 tokens after tokenization.

[0062] Step S20, label each of the target tokens based on a preset hash table to obtain a labeling result;

[0063] Further, after receiving the input content and tokenizing the input content to obtain each target token, each of the target tokens can be labeled based on a preset hash table to obtain a labeling result.

[0064] Exemplarily, after tokenizing the input content to obtain each target token, the obtained target tokens may include tokens such as adverbs, conjunctions, and modal particles. For example, "Ha", "really", and "!" in the aforementioned example tokenization result. Such tokens are usually used to express mood, tense, or modify the sentence, and have little impact on the semantics that the input content needs to express, but usually consume more memory, bandwidth, and computing unit overhead. Therefore, in the embodiments of the present application, it is considered to label such tokens through the created hash table.

[0065] Exemplarily, as a data structure, a hash table can map a key to an index value in a fixed range through a hash function, thereby realizing fast insertion and search operations. The core idea of ​​a hash table is to use a hash function to reduce the search time. The hash function in the hash table is used to convert a key into an array index, and the array is used to store data items. The insertion method of a hash table includes calculating a hash value, checking conflicts, and inserting data, wherein calculating a hash value refers to using a hash function to convert a key into an index of an array, and then checking whether the index position is already occupied. If it is occupied, a conflict resolution method (such as a chain address method or an open address method) is required to find an available position. If there is no conflict, the key-value pair is directly inserted at the calculated index position. If there is a conflict, it is processed according to the selected conflict resolution method and the key-value pair is inserted. Similarly, the search method of a hash table includes calculating a hash value, checking a key-value pair, and a conflict resolution method, wherein calculating a hash value refers to using a hash function to convert a key into an index of an array; checking a key-value pair refers to searching for a key-value pair at the calculated index position. If a matching key-value pair is found, the corresponding value is returned. If not found, continue to search for other possible locations according to the selected conflict resolution method. In the conflict resolution method, the Separate Chaining method is to maintain a linked list or other data structure (such as a red-black tree) at each array index, and all key-value pairs hashed to the same index are stored in this data structure; the Open Addressing method is to find the next free position in the array when a conflict occurs. Common probing sequences include linear probing, quadratic probing, and double hashing, among which linear probing is to check the next position in sequence, quadratic probing is to check the next position according to a quadratic sequence, and double hashing is to use a second hash function to determine the probing step size. The specific conflict resolution method and probing method used in the embodiments of the present application are not specifically limited.

[0066] For example, before creating a hash table, you need to maintain a table of auxiliary particles. In this process, you need to define the scope, that is, to clarify what the auxiliary particle table needs to cover, such as whether it includes all possible variants and usages. You can collect information about auxiliary particles through literature, corpora, online resources, etc., organize the collected data into a format that is easy to understand and use, update the table regularly to add new auxiliary particles, and proofread to ensure that the information is accurate. You can also test the effectiveness of the auxiliary particle table in actual application scenarios and make adjustments based on feedback. In addition, since the meaning and usage of many auxiliary particles depend on the specific context, you need to pay attention to the context when using them. Different dialects or regions may have different usages of auxiliary particles, which also needs to be considered when maintaining the table. Some auxiliary particles may have specific cultural meanings or emotional colors, which are particularly evident in informal occasions or literary works.

[0067] For example, in the auxiliary word table, the parts of speech of the words (such as adverbs, conjunctions, and modal particles, etc.) can be marked. Through part-of-speech marking, different projects can be adapted. For example, some projects have high requirements for semantic understanding accuracy, so only modal particles can be filtered out. Some projects have low requirements for semantic understanding accuracy, so adverbs, conjunctions, and modal particles can all be filtered.

[0068] Exemplarily, after determining the secondary particle table, the secondary particle table can be burned into the specified storage directory of the target device. After the target device is turned on, the secondary particle table in the specified storage directory is read into the memory, a resident hash table is created, and an insertion method and a search method for the hash table are provided. The UTF-8 encoded words are hashed using the SHA-256 algorithm, and an index value is returned to determine the storage location of the word in the hash table. Determine whether to insert each secondary particle in the secondary particle table into the hash table through the insertion method according to the marked part-of-speech configuration item.

[0069] Exemplarily, each target word element obtained by word segmentation can be searched according to a preset hash table, and the word element found in the storage location can be marked to obtain a marking result.

[0070] Step S30, screening the target word-grams according to the marking results to obtain target content.

[0071] Exemplarily, each target word is labeled based on a preset hash table, and after obtaining the labeling result, each target word can be screened according to the labeling result to obtain the target content.

[0072] Exemplarily, the step of screening the target word-grams according to the annotation results to obtain the target content includes:

[0073] Obtaining the semantic understanding accuracy corresponding to the input content, and determining the part of speech to be filtered according to the semantic understanding accuracy;

[0074] The target word-grams that match the part of speech to be filtered are deleted from the target word-grams according to the marking results to obtain the target content.

[0075] Exemplarily, the annotation results include the part of speech corresponding to the auxiliary particles in each target word, and the relevant auxiliary particles can be filtered out according to the part of speech to obtain the filtered words as the target content. Different filtering rules can be set according to different accuracy requirements or annotations. For example, for the first level of accuracy requirements, adverbs, conjunctions and modal particles can be filtered; for the second level of accuracy requirements, conjunctions and modal particles can be filtered; for the third level of accuracy requirements, only modal particles can be filtered, so as to obtain the filtered target words, which can be used as the target content.

[0076] Exemplarily, each target word is screened according to the annotation results to obtain the target content, that is, the input content is simplified, and then the obtained target content is input into the large language model according to the original process to continue prediction, and the reply content corresponding to the target content is obtained, which can save the computing unit overhead in the reasoning process.

[0077] Exemplarily, through the data processing method in the embodiment of the present application, the example sentence "Ha, the weather is so nice today!" in the aforementioned embodiment can be simplified to "The weather is good today!" After word segmentation using Byte Pair Encoding (BPE) technology, {"today", "weather", "good", "!"} is obtained, a total of 4 tokens. The meanings expressed by the sentences before and after optimization are almost the same, but the number of tokens is reduced by half. From this, it can be calculated that when using key-value cache (KV cache), the memory and bandwidth are reduced by half, and the amount of self-attention calculation is reduced to 1 / 4 of the original, which can effectively reduce the first token delay problem.

[0078] This embodiment adopts the above scheme, specifically by responding to receiving input content, segmenting the input content to obtain target word-grams; marking the target word-grams based on a preset hash table to obtain a marking result; screening the target word-grams according to the marking result to obtain target content, marking the target word-grams obtained by segmentation through a preset hash table to determine worthless words in the input content, screening the target word-grams according to the marking result to obtain the target content, ensuring that the target content does not contain worthless words, reducing the number of word-grams in the target content, and thus saving the computing unit overhead for subsequent processing of the target content.

[0079] Based on the first embodiment of the present application, the second embodiment of the present application is proposed. In the second embodiment of the present application, the same or similar contents as those of the above-mentioned embodiment 1 can be referred to the above introduction and will not be repeated later. Figure 2, before step S20 or before step S10, the data processing method further includes steps S01-S02:

[0080] S01: constructing a preset auxiliary word table, wherein the auxiliary word table includes each preset word and the part-of-speech configuration corresponding to each preset word;

[0081] Exemplarily, the step of constructing a preset auxiliary word table further includes:

[0082] Burning the preset auxiliary particle table into a designated storage directory of a target device;

[0083] After the target device is turned on, the preset auxiliary word table is read from the designated storage directory into a memory, wherein the original hash table is created in the memory.

[0084] Exemplarily, in the embodiment of the present application, a preset hash table is obtained by maintaining a secondary particle table. In this process, it is necessary to define the scope, that is, to clarify what the secondary particle table needs to cover, such as whether it contains all possible variants and usages. Information on secondary particles can be collected through literature, corpora, online resources, etc., and the collected data can be organized into a format that is easy to understand and use. The table is updated regularly to add new secondary particles, and proofread to ensure that the information is accurate. The effectiveness of the secondary particle table can also be tested in actual application scenarios and adjusted based on feedback. In addition, since the meaning and usage of many secondary particles depend on the specific context, it is necessary to pay attention to the context when using them. Different dialects or regions may have different usages of secondary particles, which also needs to be considered when maintaining the table. Some secondary particles may have specific cultural meanings or emotional colors, which are particularly evident in informal occasions or literary works.

[0085] For example, in the auxiliary word table, the parts of speech of the words (such as adverbs, conjunctions, and modal particles, etc.) can be marked. Through part-of-speech marking, different projects can be adapted. For example, some projects have high requirements for semantic understanding accuracy, so only modal particles can be filtered out. Some projects have low requirements for semantic understanding accuracy, so adverbs, conjunctions, and modal particles can all be filtered.

[0086] Exemplarily, after the secondary particle table is determined, the secondary particle table can be burned into a designated storage directory of the target device. After the target device is powered on, the secondary particle table in the designated storage directory is read into the memory.

[0087] S02: creating an original hash table, and inserting each preset word into the original hash table according to the part-of-speech configuration corresponding to each preset word in the preset auxiliary word table, to obtain the preset hash table.

[0088] Exemplarily, create a resident hash table, and provide the hash table insertion method and search method. Convert each character in the auxiliary particle table into UTF-8 encoding format. UTF-8 is a variable-length character encoding method that can represent all characters in the Unicode standard. For example, suppose we want to encode the word "hello", the Unicode code point of "you" is U+4F60, and the Unicode code point of "good" is U+597D. According to the encoding rules of UTF-8, for three-byte characters (U+0800 to U+FFFF), the encoding method is as follows:

[0089] The first byte starts with 1110xxxx, and the next two bytes both start with 10xxxxxx.

[0090] Specifically, the encoding format of three-byte characters is 1110xxxx 10xxxxxx 10xxxxxx.

[0091] The encoding process includes:

[0092] "You" (U+4F60):

[0093] Unicode code point: 01001111 01100000

[0094] Convert to UTF-8:

[0095] First byte: 11100100 (E4)

[0096] Second byte: 10111111 (BF)

[0097] The third byte: 10100000 (80)

[0098] Therefore, the UTF-8 encoding of "you" is E4 BF 80.

[0099] "Good" (U+597D):

[0100] Unicode code point: 01011001 01111101

[0101] Convert to UTF-8:

[0102] First byte: 11100101 (E5)

[0103] Second byte: 10100111 (A5)

[0104] The third byte: 10111101 (BD)

[0105] Therefore, a "good" UTF-8 encoding is E5 A5 BD.

[0106] The UTF-8 encoding of the complete UTF-8 encoding "你好" is E4 BF 80E5 A5 BD.

[0107] Exemplarily, the SHA-256 algorithm is used to hash the UTF-8 encoded words, and an index value is returned to determine the storage location of the words in the hash table. For each auxiliary particle in the auxiliary particle table, it is determined whether to insert the auxiliary particle into the hash table by the insertion method according to the marked part-of-speech configuration item.

[0108] Exemplarily, SHA-256 (Secure Hash Algorithm 256-bit) is a cryptographic hash function. The SHA-256 algorithm is used to hash the UTF-8 encoded words, including creating a new SHA-256 hash object using hashlib.sha256(), adding the UTF-8 encoded byte sequence to the hash object using the update() method, and obtaining the hexadecimal representation of the hash value using the hexdigest() method, so as to obtain the final hash result and return the index value to determine the storage location of the words in the preset hash table.

[0109] In this embodiment, through the above solution, specifically by constructing a preset auxiliary particle table, where the auxiliary particle table includes each preset word and the corresponding part-of-speech configuration of each preset word; creating an original hash table, and inserting each preset word into the original hash table according to the corresponding part-of-speech configuration of each preset word in the preset auxiliary particle table to obtain the preset hash table. The insertion of each preset word in the hash table is realized through the constructed preset auxiliary particle table, and then it is used to label each target token after word segmentation, reducing the number of tokens of the final input to the large language model for prediction to obtain the reply content corresponding to the target content, and thus the bandwidth, memory and computing unit overhead can be saved.

[0110] Based on any of the foregoing embodiments of the present application, the third embodiment of the present application is proposed. In the third embodiment of the present application, the same or similar content as any of the above embodiments can be referred to the above introduction and will not be repeated hereinafter. On this basis, please refer to Figure 3 , step S20 further includes steps S201 to S202:

[0111] S201: Hash each of the target tokens to determine the index value corresponding to each of the target tokens;

[0112] S202: Search the preset hash table according to the index value to determine the preset word found;

[0113] S203: Label each of the target tokens according to the preset word found to obtain the labeling result.

[0114] Exemplarily, the step of searching the preset hash table according to the index value to determine the found preset word includes:

[0115] According to the index value, the preset words corresponding to the target word-units are searched in parallel in the preset hash table through multiple threads; or,

[0116] According to the segmentation order of each target word unit, the preset words corresponding to each target word unit are searched in sequence in the preset hash table according to the index value.

[0117] Reference Figure 4 , Figure 4 It is a schematic diagram of the overall process in the embodiment of the present application, such as Figure 4 As shown, after receiving the text input (i.e., input content), the target device performs text segmentation to obtain each target word, and then uses the SHA-256 algorithm to hash the UTF-8 encoded segmented words, and returns an index value for finding the storage location of the word in the hash table.

[0118] Exemplarily, each character in each target word is converted into the UTF-8 encoding format. UTF-8 is a variable-length character encoding method that can represent all characters in the Unicode standard. The encoding format for encoding each target word by UTF-8 can refer to the second embodiment above, which is not described in detail in this embodiment. Similarly, using the SHA-256 algorithm to hash the UTF-8 encoded word segmentation words includes using hashlib.sha256() to create a new SHA-256 hash object, using the update() method to add the UTF-8 encoded byte sequence to the hash object, and using the hexdigest() method to obtain the hexadecimal representation of the hash value, thereby obtaining the final hash result, and returning the index value to determine the storage location of the word in the preset hash table.

[0119] For example, in the embodiment of the present application, each target word can be searched in turn, and multi-threaded concurrent acceleration of word search can be used. If the corresponding word is found, it will be marked. If the corresponding word is not found, it will not be processed, and then the optimized and streamlined text (i.e., the target content) is output according to the marking result. The optimized text is input into the large language model according to the original process to continue prediction, and the reply content corresponding to the target content is obtained, which can save the computing unit overhead in the reasoning process.

[0120] This embodiment adopts the above scheme, specifically by performing hash processing on each target word to determine the index value corresponding to each target word; searching the preset hash table according to the index value to determine the preset word found; marking each target word according to the preset word found to obtain the marking result, marking each target word obtained by word segmentation through the preset hash table, it is possible to determine the worthless words in the input content, and screening each target word according to the marking result to obtain the target content, which can ensure that the target content does not contain worthless words, reduce the number of words in the target content, and thus save the computing unit overhead for subsequent processing of the target content.

[0121] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the data processing method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0122] This application also provides a data processing device, please refer to Figure 5 , the data processing device comprises:

[0123] A word segmentation module, configured to segment the input content in response to receiving the input content, to obtain target word units;

[0124] A labeling module, used to label each target word based on a preset hash table to obtain a labeling result;

[0125] The screening module is used to screen the target words according to the marking results to obtain the target content.

[0126] The data processing device provided by the present application adopts the data processing method in the above embodiment to solve the technical problem of data processing. Compared with the prior art, the beneficial effects of the data processing device provided by the present application are the same as the beneficial effects of the data processing method provided by the above embodiment, and other technical features in the data processing device are the same as the features disclosed in the above embodiment method, which will not be described in detail here.

[0127] The present application provides a data processing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data processing method in the above-mentioned embodiment 1.

[0128] Reference below Figure 6, which shows a schematic diagram of the structure of a data processing device suitable for implementing an embodiment of the present application. The data processing device in the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The data processing device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0129] like Figure 6 As shown, the data processing device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the data processing device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the data processing device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a data processing device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0130] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0131] The data processing device provided by the present application adopts the data processing method in the above embodiment to solve the technical problem of data processing. Compared with the prior art, the beneficial effects of the data processing device provided by the present application are the same as the beneficial effects of the data processing method provided by the above embodiment, and other technical features in the data processing device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0132] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0133] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0134] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the data processing method in the above-mentioned embodiment.

[0135] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0136] The computer-readable storage medium may be included in a data processing device, or may exist independently without being incorporated into a data processing device.

[0137] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by a data processing device, the data processing device: in response to receiving input content, segment the input content to obtain each target word element; mark the each target word element based on a preset hash table to obtain a marking result; filter the each target word element according to the marking result to obtain the target content, and mark the each target word element obtained by the segmentation through the preset hash table to determine the worthless words in the input content; filter the each target word element according to the marking result to obtain the target content, which can ensure that the target content does not contain worthless words, reduce the number of words in the target content, and thus save the computing unit overhead for subsequent processing of the target content.

[0138] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0139] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0140] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0141] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned data processing method, and can solve the technical problems of data processing. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the data processing method provided in the above-mentioned embodiment, and will not be repeated here.

[0142] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned data processing method when executed by a processor.

[0143] The computer program product provided by this application can solve the technical problem of data processing. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the data processing method provided by the above embodiment, which will not be repeated here.

[0144] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: In response to receiving input content, segmenting the input content to obtain target word units; Marking each target word based on a preset hash table to obtain a marking result; The target word-grams are screened according to the marking results to obtain target content.

2. The method according to claim 1, characterized in that Before the step of labeling the target words based on the preset hash table to obtain the labeling result, the step further includes: Constructing a preset auxiliary word table, wherein the auxiliary word table includes each preset word and the part-of-speech configuration corresponding to each preset word; An original hash table is created, and each preset word in the preset auxiliary word table is inserted into the original hash table according to the part-of-speech configuration corresponding to each preset word, so as to obtain the preset hash table.

3. The method according to claim 1, characterized in that The step of labeling each target word based on a preset hash table to obtain a labeling result comprises: Performing hash processing on each target word to determine an index value corresponding to each target word; Search the preset hash table according to the index value to determine the found preset word; The target word-units are annotated according to the found preset words to obtain the annotated result.

4. The method according to claim 3, characterized in that The step of searching the preset hash table according to the index value to determine the found preset word includes: According to the index value, the preset words corresponding to the target word-units are searched in parallel in the preset hash table through multiple threads; or, According to the segmentation order of each target word unit, the preset words corresponding to each target word unit are searched in sequence in the preset hash table according to the index value.

5. The method according to claim 1, characterized in that The step of screening the target words according to the marking results to obtain the target content includes: Obtaining the semantic understanding accuracy corresponding to the input content, and determining the part of speech to be filtered according to the semantic understanding accuracy; The target word-grams that match the part of speech to be filtered are deleted from the target word-grams according to the marking results to obtain the target content.

6. The method according to claim 1, characterized in that After the step of screening the target words according to the marking results to obtain the target content, the following step further includes: The target content is input into a large language model so that the large language model makes predictions based on the target content to obtain reply content corresponding to the target content.

7. A data processing device, characterized in that: The device comprises: A word segmentation module, configured to segment the input content in response to receiving the input content, to obtain target word units; A labeling module, used to label each target word based on a preset hash table to obtain a labeling result; The screening module is used to screen the target words according to the marking results to obtain the target content.

8. A data processing device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data processing method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 6 are implemented.

Images