Text processing system based on large model

Abstract

The invention discloses a text processing system based on a large model, and the method comprises the steps: determining target text processing result information corresponding to to-be-processed text information based on first text feature information;

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically to a text processing system based on a large model. Background Technology

[0002] With the continuous development of Large Language Models (LLMs), people can generate text content using LLMs. The text content generated by LLMs is almost comparable to human-generated text content, and the source of the text cannot be distinguished by human judgment. Therefore, it is necessary to identify whether the text content was generated by a LLM. However, existing methods for identifying text information are computationally expensive. Summary of the Invention

[0003] This application provides a text processing system based on a large model.

[0004] In a first aspect, this application provides a method comprising:

[0005] Obtain the text information to be processed;

[0006] Based on the text information to be processed, determine the first text feature information;

[0007] Based on the first text feature information, the target text processing result information corresponding to the text information to be processed is determined.

[0008] Secondly, this application provides a system comprising:

[0009] The information acquisition module is used to acquire text information to be processed;

[0010] The information determination module is used to determine the first text feature information based on the text information to be processed;

[0011] The text processing module is used to determine the target text processing result information corresponding to the text information to be processed based on the first text feature information.

[0012] Thirdly, this application also provides a computer device, which includes:

[0013] One or more processors;

[0014] Memory; and

[0015] One or more applications, wherein the applications are stored in memory and configured to be executed by a processor to implement the methods of any one of the first aspects.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, the computer program being loaded by a processor to perform the steps of the method in any of the first aspects. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of a scenario for the text processing system provided in an embodiment of the present invention;

[0019] Figure 2 This is a flowchart of an embodiment of the text processing method provided by the present invention;

[0020] Figure 3 This is a schematic diagram of a specific scenario of the text processing method provided in the embodiments of the present invention;

[0021] Figure 4 This is a flowchart illustrating a specific embodiment of determining first text feature information provided by this invention.

[0022] Figure 5 This is a schematic diagram of a specific embodiment of the present invention for determining the target text processing result information;

[0023] Figure 6 This is a flowchart of another embodiment of the text processing method provided in this invention;

[0024] Figure 7 This is a schematic diagram of the text processing system provided in the embodiments of the present invention;

[0025] Figure 8 This is a schematic diagram of an embodiment of the computer device provided in this invention. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] In the description of this application, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. Furthermore, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, features defined with "first," "second," "third," etc., may explicitly or implicitly include one or more of the stated features.

[0028] In this application, the term "exemplary" is used to mean "used as an example, illustration, or description." Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use this application. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that this application can be made without using these specific details. In other instances, well-known structures and processes are not described in detail to avoid obscuring the description of this application with unnecessary detail. Therefore, this application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0029] It should be noted that since the method in this application embodiment is executed in a computer device, the processing objects of each computer device exist in the form of data or information, such as time, which is essentially time information. It is understood that if size, quantity, position, etc. are mentioned in subsequent embodiments, they are all corresponding data that exist so that the computer device can process them. Specific details will not be elaborated here.

[0030] This application provides a text processing method, system, device, and storage medium based on a large model, which will be described in detail below.

[0031] Please see Figure 1 , Figure 1 This is a schematic diagram of a scenario for a text processing system provided in an embodiment of this application. The text processing system may include a computer device 100, which integrates a text processing system, such as... Figure 1 Computer equipment in the country.

[0032] In this embodiment, the computer device 100 is mainly used to acquire text information to be processed; determine first text feature information based on the text information to be processed; and determine the target text processing result information corresponding to the text information to be processed based on the first text feature information. The text processing process does not require access to the model, and the text processing computation cost is low and the text processing speed is fast.

[0033] In this embodiment, the computer device 100 can be a standalone server, a server network, or a server cluster. For example, the computer device 100 described in this embodiment includes, but is not limited to, a computer, a network host, a single network server, a set of multiple network servers, or a cloud server composed of multiple servers. The cloud server is composed of a large number of computers or network servers based on cloud computing.

[0034] It is understood that the computer device 100 used in the embodiments of this application can be a device that includes both receiving and transmitting hardware, that is, a device having receiving and transmitting hardware capable of performing bidirectional communication on a bidirectional communication link. Such a device may include: cellular or other communication devices having a single-line display, a multi-line display, or a cellular or other communication device without a multi-line display. Specifically, the computer device 100 may be a desktop terminal or a mobile terminal, and may also be one of a mobile phone, tablet computer, laptop computer, etc.

[0035] Those skilled in the art will understand that Figure 1 The application environment shown is merely one application scenario for the solution in this application and does not constitute a limitation on the application scenario of the solution in this application. Other application environments may include those that are more specific to this application. Figure 1 The number of computer devices shown is more or less, for example Figure 1 Only one computer device is shown in the document. It is understood that the text processing system may also include one or more other services, which are not limited here.

[0036] In addition, such as Figure 1 As shown, the text processing system may also include a memory 200 for storing data, such as vocabulary information, such as first target vocabulary information, second target vocabulary information, third target vocabulary information, etc., and feature information, such as first feature information, second feature information, third feature information, etc.

[0037] It should be noted that, Figure 1The schematic diagram of the text processing system shown is merely an example. The text processing system and scenario described in this application are intended to more clearly illustrate the technical solutions of this application and do not constitute a limitation on the technical solutions provided in this application. As those skilled in the art will know, with the evolution of text processing systems and the emergence of new business scenarios, the technical solutions provided in this application are also applicable to similar technical problems.

[0038] First, this application provides a text processing method, system, device, and storage medium based on a large model. The text processing method based on the large model is executed by a text processing system applied to a computer device. The text processing method includes: acquiring text information to be processed; determining first text feature information based on the text information to be processed; and determining target text processing result information corresponding to the text information to be processed based on the first text feature information.

[0039] like Figure 2 The diagram shown is a flowchart of an embodiment of the text processing method in this application. The text processing method may include the following steps S201 to S203, as detailed below:

[0040] S201. Obtain the text information to be processed.

[0041] The text information to be processed is the text information acquired by the computer device that needs to be determined whether it was generated by the target model. The text information to be processed can be a sentence, a paragraph, an entire article, a paper, etc., and this embodiment does not limit it. In addition, the computer device can receive the text information to be processed input by the user through input devices such as keyboard, mouse, and touch screen. The computer device can also acquire the text information to be processed from other devices through networks, Bluetooth, etc., and this embodiment does not limit it.

[0042] For example, refer to Figure 3 As shown, the text information to be processed is a sentence entered by the user through input devices such as keyboard, mouse, and touch screen: "My hometown is the most beautiful memory in my life, the home of my soul..." The computer device needs to identify whether the sentence was generated by the Large Language Model (LLM) on the left.

[0043] It's important to note that large language models refer to artificial neural network models with an extremely large number of parameters. In the field of artificial intelligence, large models typically refer to models with hundreds of millions to trillions of parameters. These models usually need to be trained on massive datasets and require significant computational resources for optimization and tuning. Large models are commonly used to solve complex tasks such as natural language processing, computer vision, and speech recognition.

[0044] In this embodiment of the application, the large language model can be a language model of the scale of ChatGPT, BERT, XLNet, Zhipu model, Claude, Moonshot AI model, ChatGLM model, Qianyitongwen model, MiniMax model, Xinghuo model, Llama model, 360GPT model, Qwen model, Baichuan model, Yunque model, vivoLM model, and Wenxin Yiyan, etc., and this embodiment of the application does not limit it.

[0045] S202. Based on the text information to be processed, determine the first text feature information.

[0046] In one specific embodiment, to detect whether the text information to be processed was generated by the target model, this embodiment divides a preset vocabulary into a first vocabulary, a second vocabulary, and a third vocabulary, and makes the target model tend to generate words from the first vocabulary when generating text content. Here, the vocabulary is the set of words used by the target model in the process of generating text content, containing all the words that the target model may encounter in the process of generating text content. For example, the first vocabulary, the second vocabulary, and the third vocabulary are respectively a green vocabulary, a white vocabulary, and a red vocabulary, making the target model tend to generate words from the green vocabulary when generating text content.

[0047] Furthermore, the first text feature information represents the number of words in the text to be processed that belong to the first vocabulary. Since the target model tends to generate words from the first vocabulary when generating text content, the number of words in the text to be processed that belong to the first vocabulary can determine whether the text to be processed was generated by the target model. For example, the first vocabulary, the second vocabulary, and the third vocabulary are the green vocabulary, the white vocabulary, and the red vocabulary, respectively, and the first text feature information is the number of words in the text to be processed that belong to the green vocabulary.

[0048] In a specific implementation method, refer to Figure 4 As shown, the step S202 above, which determines the first text feature information based on the text information to be processed, may include the following steps S301 to S302, as detailed below:

[0049] S301. Determine the first target vocabulary information corresponding to each word in the text information to be processed.

[0050] In one specific embodiment, the first target vocabulary information is the vocabulary information corresponding to words belonging to the first vocabulary in a preset vocabulary determined based on each word in the text information to be processed. For example, the preset vocabulary includes n words from a1 to an. If a1, a2, and ak are determined to be words belonging to the first vocabulary based on the i-th word in the text information to be processed, then the first target vocabulary information corresponding to the i-th word in the text information to be processed is {a1, a2, ak}.

[0051] In one specific embodiment, the step of determining the first target vocabulary information corresponding to each word in the text information to be processed specifically includes: determining the first feature information corresponding to each word in the text information to be processed; processing the first feature information to obtain the initial value information corresponding to each word; classifying the second target vocabulary information based on the initial value information and the first vocabulary feature information corresponding to the second target vocabulary information to obtain the first target vocabulary information corresponding to each word.

[0052] In this text, the first feature information represents the feature information of the preceding word corresponding to each word in the text to be processed. For example, if the text to be processed is "Hometown is the most beautiful memory in my life, the…", the first feature information corresponding to "hometown" is the word vector of its preceding word "home", and the first feature information corresponding to "life" is the word vector of its preceding word "life". The first feature information corresponding to each word in the text to be processed can be obtained by mapping the preceding word to the vector space through a word vector model. The word vector model can be built based on the word2vec model, the one-hot model, or the FastText model; this embodiment does not limit the specific model.

[0053] Furthermore, initial value information refers to the initial value used in the pseudo-random number generator to generate pseudo-random numbers. The steps of processing the first feature information to obtain the initial value information corresponding to each word specifically include: creating a hash table of a preset length; processing the first feature information based on the hash table and a preset key to obtain candidate initial value information corresponding to each word; and processing the candidate initial value information to obtain the initial value information corresponding to each word. For example, the process of creating a hash table of length 1000003 can be represented as: hashtable = torch.randperm(1000003), where the torch.randperm() function is used to generate a random integer permutation from 0 to n-1. The process of processing the first feature information based on the hash table and a preset key to obtain candidate initial value information corresponding to each word can be represented as: seed1 = hashtable[salt_key*input_ids%len(hashtable)], where seed1 represents the candidate initial value information, salt_key represents the key, input_ids represents the first feature information, and len(hashtable) represents the length of the hash table. The process of processing candidate initial value information to obtain the initial value information corresponding to each word can be represented as: seed = torch.min(seed).item(), where seed represents the seed (i.e., initial value information) of the pseudo-random generator, torch.min() function is used to calculate the minimum value of all elements in the tensor, and item() represents converting a Tensor variable into a Python scalar.

[0054] In one specific embodiment, the step of classifying the second target vocabulary information based on the first vocabulary feature information corresponding to the initial value information and the second target vocabulary information to obtain the first target vocabulary information corresponding to each word specifically includes: performing information generation processing on the first vocabulary feature information corresponding to the initial value information and the second target vocabulary information to obtain the index information corresponding to each word; multiplying the first vocabulary feature information and the first parameter to obtain the second vocabulary feature information corresponding to the first target vocabulary information; and filtering the second target vocabulary information based on the index information and the second vocabulary feature information to obtain the first target vocabulary information corresponding to each word.

[0055] The second target vocabulary information is the vocabulary information of the preset target model, and the first vocabulary feature information represents the size of the vocabulary of the second target vocabulary information. The process of generating information from the initial value information and the first vocabulary feature information corresponding to the second target vocabulary information to obtain the index information corresponding to each word can be represented as: vocab_permutation = torch.randperm(vocab_size, generator = seed), where vocab_permutation represents the index information, the torch.randperm() function is used to generate a random integer permutation from 0 to n-1, vocab_size represents the first vocabulary feature information, and seed represents the initial value information.

[0056] Furthermore, the second lexical feature information represents the lexical size corresponding to the first target lexical information. The process of determining the second lexical feature information can be expressed as: V1 = γ|V|, where V1 represents the second lexical feature information, |V| represents the first lexical feature information, and γ represents the first parameter, which can be set according to user needs.

[0057] In one specific embodiment, the process of filtering the second target vocabulary information based on index information and second vocabulary feature information to obtain the first target vocabulary information corresponding to each word can be represented as: greenlist_ids = vocab_permutation[:greenlist_size], where greenlist_ids represents the first target vocabulary information, vocab_permutation represents the index information, and greenlist_size represents the second vocabulary feature information.

[0058] S302. Based on each word in the text to be processed and the first target vocabulary information corresponding to each word, determine the first text feature information.

[0059] In one specific embodiment, the step of determining the first text feature information based on each word in the text information to be processed and the first target vocabulary information corresponding to each word specifically includes: performing matching processing on each word in the text information to be processed and the first target vocabulary information corresponding to each word to obtain a number of first feature words; and determining the number of the number of first feature words as the first text feature information.

[0060] Among them, several first feature words are words in the text information to be processed that match the first target vocabulary information. Words in the text information to be processed that match the first target vocabulary information refer to words in the text information to be processed that belong to their corresponding first target vocabulary information. For example, if the text information to be processed is "Hometown is the most beautiful memory in my life", and the first target vocabulary information is {home, hometown, is, I, among, most, beautiful, good…}, then the words "home", "hometown", "is", "I", "am", "most", "beautiful", and "good" in the text information to be processed match the first target vocabulary information. Several first feature words include {home, hometown, is, I, among, most, beautiful, good}, and the first text feature information is 8.

[0061] S203. Based on the first text feature information, determine the target text processing result information corresponding to the text information to be processed.

[0062] In one specific embodiment, the target text processing result information includes first processing result information and second processing result information. The first processing result information indicates that the text information to be processed was generated by the target model, and the second processing result information indicates that the text information to be processed was not generated by the target model. This embodiment determines the first text feature information based on the text information to be processed, and determines whether the text information to be processed was generated by the target model based on the first text feature information. The text processing process does not require access to the model, resulting in low computational cost and high speed. The target model can be a Large Language Model (LLM).

[0063] In a specific implementation method, refer to Figure 5 As shown, in step S203 above, determining the target text processing result information corresponding to the text information to be processed based on the first text feature information may include the following steps S401 to S402, as detailed below:

[0064] S401. Determine the first probability information based on the first text feature information and the second text feature information of the text to be processed.

[0065] In one specific embodiment, the second text feature information represents the text length of the text information to be processed. Based on hypothesis H0, it is determined whether the text information to be processed was generated by the target model. Hypothesis H0 is: the text was generated without knowledge of the watermarking rules. The process of determining the first probability information can be expressed as: p1 = p1 represents the first probability information, |s| G =n represents the first text feature information, T represents the second text feature information, I represents the incomplete beta function for regularization, y represents the first parameter, and γ can be set according to user needs. For example, if γ is 0.3, n = 51, T = 100, then p1 = I 0.3(51, 100-51-1)=9×10 -6 .

[0066] S402. The first probability information is identified and processed to obtain the target text processing result information corresponding to the text information to be processed.

[0067] In one specific embodiment, the target text processing result information includes first processing result information and second processing result information. The first processing result information indicates that the text information to be processed is generated by the target model, and the second processing result information indicates that the text information to be processed is not generated by the target model. The step of identifying and processing the first probability information to obtain the target text processing result information corresponding to the text information to be processed specifically includes: comparing the first probability information with a first probability threshold; if the probability value corresponding to the first probability information is less than the first probability threshold, determining the target text processing result information corresponding to the text information to be processed as the first processing result information; if the probability value corresponding to the first probability information is greater than or equal to the first probability threshold, determining the target text processing result information corresponding to the text information to be processed as the second processing result information.

[0068] The first probability threshold is a pre-set probability threshold used to measure whether the text information to be processed was generated by the target model. The first probability threshold can be set according to user needs; for example, the first probability threshold can be set to 10. -5 If the first probability information p1 = 9 × 10 -6 <10 -5 The target text processing result information corresponding to the text information to be processed is determined as the first processing result information, that is, the text information to be processed is generated by the target model.

[0069] In one specific embodiment, the target model generates the text information to be processed through the following steps: fusing the third target vocabulary information and the prompt information to obtain fused information; extracting features from the fused information to obtain second feature information; determining the probability distribution information based on the third feature information and the second feature information; and performing sampling processing based on the probability distribution information to obtain the text information to be processed.

[0070] It should be noted that when generating text information to be processed, the target model predicts the next word based on the prompt information and the words already generated. The third target vocabulary information is the vocabulary information corresponding to the word currently generated by the target model. For example, if the prompt information is "Write an article about my hometown", and the currently generated word is "My hometown is the most beautiful thing in my life", then concatenating the currently generated word "My hometown is the most beautiful thing in my life" with the prompt information "Write an article about my hometown" will yield the fused information.

[0071] In one specific embodiment, the third feature information represents the feature information of the word preceding the currently generated word in the target model. For example, if the currently generated word is "hometown is the most beautiful thing in my life", then the third feature information is the word vector corresponding to the word "good". The third feature information can be obtained by mapping the word preceding the currently generated word to the vector space through a word vector model. The word vector model can be built based on the word2vec model, the one-hot model, or the FastText model. This embodiment does not limit the specific model.

[0072] In one specific embodiment, the second feature information includes several feature information to be processed. The step of determining the probability distribution information based on the third feature information and the second feature information specifically includes: classifying the second target vocabulary information based on the third feature information to obtain several fourth target vocabulary information; matching the several fourth target vocabulary information and several feature information to be processed to obtain vocabulary association information; and calculating the several feature information to be processed based on the vocabulary association information to obtain probability distribution information.

[0073] In one specific embodiment, the plurality of fourth target vocabulary information are vocabulary information corresponding to words belonging to several word lists in the target vocabulary information obtained by classifying the second target vocabulary information based on the third feature information. For example, the second target vocabulary information is {home, hometown, is, I, life, fate, among, most, beautiful, good, of, memory}, {home, hometown, is, I, among, most, beautiful, good} belongs to the first word list, {life, fate} belongs to the second word list, and {of, memory} belongs to the third word list. Then the plurality of fourth target vocabulary information includes {home, hometown, is, I, among, most, beautiful, good}, {life, fate}, and {of, memory}. The process of classifying the second target vocabulary information based on the third feature information to obtain the plurality of fourth target vocabulary information is the same as the aforementioned process of classifying the second target vocabulary information based on the first feature information. Specifically, the steps of classifying the second target vocabulary information based on the first feature information can be referred to. To avoid repetition, this embodiment will not repeat them here.

[0074] In one specific embodiment, the lexical association information represents the correspondence between several feature information to be processed and several fourth target lexical information. For example, the several fourth target lexical information includes first lexical information, second lexical information and third lexical information. Feature information A corresponds to the first lexical information, feature information B corresponds to the second lexical information and feature information C corresponds to the third lexical information.

[0075] In one specific embodiment, the step of calculating and processing several unprocessed feature information based on lexical association information to obtain probability distribution information specifically includes: performing weighted summation processing on several unprocessed feature information based on lexical association information to obtain fourth feature information; for any unprocessed feature information among several unprocessed feature information, performing calculation and processing on the fourth feature information and the unprocessed feature information based on lexical association information to obtain transformed feature information corresponding to each feature information; and performing fusion processing on the transformed feature information corresponding to several unprocessed feature information to obtain probability distribution information.

[0076] For example, several fourth target vocabulary information items include first vocabulary information, second vocabulary information, and third vocabulary information. The process of determining the fourth feature information can be represented as follows: S represents the fourth feature information, G represents the first vocabulary information, W represents the second vocabulary information, and R represents the third vocabulary information. This represents the i-th feature information in the second feature information, and δ represents the bias information, which can be set according to user needs.

[0077] In one specific embodiment, the step of calculating and processing a fourth feature information and the feature information to be processed based on lexical association information to obtain the transformed feature information corresponding to each feature information specifically includes: for any feature information to be processed among the several feature information to be processed, if the fourth target lexical information corresponding to the feature information to be processed is a first lexical information, adding the feature information to be processed and the bias information to obtain a first preprocessed feature information, and calculating and processing the first preprocessed feature information and the fourth feature information to obtain the transformed feature information corresponding to the feature information to be processed; if the fourth target lexical information corresponding to the feature information to be processed is a second lexical information, exponentially operating the feature information to be processed to obtain a second preprocessed feature information, and dividing the second preprocessed feature information and the fourth feature information to obtain the transformed feature information corresponding to the feature information to be processed; if the fourth target lexical information corresponding to the feature information to be processed is a third lexical information, subtracting the feature information to be processed and the bias information to obtain a third preprocessed feature information, and calculating and processing the third preprocessed feature information and the fourth feature information to obtain the transformed feature information corresponding to the feature information to be processed.

[0078] For example, This represents the k-th feature information in the second feature information. If the corresponding fourth target vocabulary information is the first vocabulary information, then... The process of determining the corresponding transformation feature information can be represented as follows: like The corresponding fourth target vocabulary information is the second vocabulary information, then The process of determining the corresponding transformation feature information can be represented as follows: like The corresponding fourth target vocabulary information is the third vocabulary information, then The process of determining the corresponding transformation feature information can be represented as follows: express The corresponding transformation feature information, S represents the fourth feature information, δ represents the bias information, and δ can be set according to user needs.

[0079] In a specific implementation method, refer to Figure 6 As shown, after determining the target text processing result information corresponding to the text information to be processed based on the first text feature information in step S203 above, the following steps S501 to S503 may be included, as follows:

[0080] S501. Based on the second text feature information and the first text length information of the text information to be processed, determine several first candidate text information of the text information to be processed.

[0081] In one specific embodiment, the second text feature information represents the text length of the text information to be processed, the first text length information is the minimum number of words in a pre-set paragraph, and several first candidate text information can be represented as: [1, L min ],[1,L min +1],[1,L min +2],…,[1,T],[2,L min +1],[2,L min +2],…[2,T],…,[T-10,T],T represents the second text feature information,L min This represents the length information of the first text, where [1, L] min [] indicates the first to Lth words in the text information to be processed. min Text segments between words, [1, L] min +1] indicates the second to the Lth word in the text information to be processed. min +1 word intervals, and so on, [T-10, T] represents the text interval between the (T-10)th word and the Tth word in the text information to be processed. For example, L min If set to 10, then several first candidate text information can be represented as: [1, 10], [1, 11], [1, 12], ..., [1, T], [2, 11], [2, 12], ..., [2, T], ..., [T-10, T].

[0082] S502. Determine the second probability information corresponding to each first candidate text information.

[0083] In this embodiment, each first candidate text information includes first boundary information and second boundary information. The first boundary information is the start position information of the text segment corresponding to the first candidate text information in the text information to be processed, and the second boundary information is the end position information of the text segment corresponding to the first candidate text information in the text information to be processed. For example, the first candidate text information [1, L] min Includes first boundary information 1 and second boundary information L min The first boundary information 1 indicates that the text segment corresponding to the first candidate text information starts from the first word of the text information to be processed, and the second boundary information L min This represents the Lth position of the text segment corresponding to the first candidate text information. min The first candidate text segment is the text segment from the first word to the Lth word in the text to be processed. min For example, the first candidate text information [2, T] includes the first boundary information 2 and the second boundary information T. The first boundary information 2 indicates that the text segment corresponding to the first candidate text information starts from the second word of the text information to be processed, and the second boundary information T indicates that the text segment corresponding to the first candidate text information ends at the Tth word of the text information to be processed. That is, the text segment corresponding to the first candidate text information is the text segment between the second word and the Tth word in the text information to be processed.

[0084] In one specific embodiment, the step of determining the second probability information corresponding to each first candidate text information specifically includes: determining first quantity information based on first boundary information; determining second quantity information based on second boundary information; subtracting the first quantity information and the second quantity information to obtain third quantity information; subtracting the first boundary information and the second boundary information to obtain second text length information corresponding to each first candidate text information; and determining the second probability information corresponding to each first candidate text information based on the third quantity information and the second text length information.

[0085] In one specific embodiment, the process of determining the second probability information can be expressed as: p i，j =I γ (n i，j T i，j -n i，j +1), p i，j This represents the second probability information corresponding to the first candidate text information [i, j], where n is the number of candidates. i，j =n j -n i T i，j =ji,n i，j Represents the third quantity information, n iLet n represent the first quantity. j T represents the second quantity. i，j The second text length information is represented by i, the first boundary information is represented by j, the second boundary information is represented by I, the incomplete beta function of the regular expression is represented by γ, and the first parameter is represented by γ, which can be set according to user needs.

[0086] In one specific embodiment, the step of determining the first quantity information based on the first boundary information specifically includes: determining the first text fragment information based on the first boundary information; determining the fifth target vocabulary information corresponding to each word in the first text fragment information; performing matching processing on each word in the first text fragment information and the fifth target vocabulary information corresponding to each word to obtain a number of second feature words; and determining the number of the number of second feature words as the first quantity information.

[0087] The first text fragment information is the text fragment between the first word and the first boundary information in the text information to be processed. For example, if the first boundary information is 1, then the first text fragment information is the first word in the text information to be processed; if the first boundary information is 5, then the first text fragment information is the text fragment between the first word and the fifth word in the text information to be processed; if the first boundary information is T-10, then the first text fragment information is the text fragment between the first word and the T-10th word in the text information to be processed.

[0088] In one specific embodiment, the step of determining the fifth target vocabulary information corresponding to each word in the first text fragment information is the same as the step of determining the first target vocabulary information corresponding to each word in the text information to be processed in the aforementioned step S301. For details, please refer to the discussion of the aforementioned step S301. To avoid repetition, this embodiment will not repeat it here.

[0089] In one specific embodiment, several second feature words are words in the first text fragment information that match the fifth target vocabulary information. The words in the first text fragment information that match the fifth target vocabulary information refer to words in the first text fragment information that belong to their corresponding fifth target vocabulary information. For example, if the first text fragment information is "Hometown is the most beautiful memory in my life," and the fifth target vocabulary information is {home, hometown, is, I, among, most, beautiful, good…}, then the words "home," "hometown," "is," "I," "am," "most," "beautiful," and "good" in the first text fragment information match the fifth target vocabulary information. If several second feature words include {home, hometown, is, I, among, most, beautiful, good}, then the first quantity information is 8.

[0090] It should be noted that the process of determining the second quantity information based on the second boundary information is the same as the process of determining the first quantity information based on the first boundary information. For details, please refer to the specific steps of determining the first quantity information based on the first boundary information. To avoid repetition, this embodiment will not elaborate further.

[0091] S503. Based on the second probability information, several first candidate text information are fused to obtain the target text information corresponding to the text information to be processed.

[0092] In one specific embodiment, the target text information represents the text position of the text segment generated by the target model in the text information to be processed. The step of fusing several first candidate text information based on the second probability information to obtain the target text information corresponding to the text information to be processed specifically includes: determining several second candidate text information from several first candidate text information based on the second probability information; fusing several second candidate text information based on the first boundary information to obtain several third candidate text information; and fusing several third candidate text information to obtain the target text information corresponding to the text information to be processed.

[0093] Among these, several second candidate text pieces are text pieces with a second probability value less than a second probability threshold among several first candidate text pieces. The second probability threshold is a pre-set probability threshold used to measure whether the text fragment corresponding to the first candidate text piece was generated by the target model. The second probability threshold can be set according to actual needs; for example, the second probability threshold can be set to 10. -5 .

[0094] In one specific embodiment, fusing several second candidate text information based on the first boundary information refers to merging second candidate text information that have the same first boundary information. For example, the several second candidate text information include: For any k, Therefore It can be fused into And so on, It can be fused into

[0095] In one specific embodiment, the step of fusing several third candidate text information to obtain the target text information corresponding to the text information to be processed specifically includes: starting from the first third candidate text information among several third candidate text information, determining whether there is an intersection between two adjacent third candidate text information; if there is an intersection between two adjacent third candidate text information, then performing a union operation on the two adjacent third candidate text information until the last third candidate text information among several third candidate text information is traversed.

[0096] For example, several third candidate text information can be represented as: [i1, j1], [i2, j2], ..., [i K j K When fusing several third-party candidate text information, you can first set [p] l p r ] = [i1, j1], and determine [p l p r Does [p] and [i2, j2] have an intersection? If [p] l p r [i1, j2] and [i2, j2] have an intersection, set [p] l p r ] = [i1, j1] ∪ [i2, j2], and continue to execute the judgment [p l p r The step of checking if there is an intersection between [i] and [i3, j3] continues until [i] is reached. K j K ]; if [p l p r If [i1, j2] and [i2, j2] have no intersection, then set [p] to [i2, j2]. l p r ] = [i2, j2], and continue to execute the judgment [p l p r The step of checking if there is an intersection between [i] and [i3, j3] continues until [i] is reached. K j K ].

[0097] In summary, the text processing method provided in this implementation scheme obtains the text information to be processed, determines the first text feature information based on the text information, and determines the target text processing result information corresponding to the text information to be processed based on the first text feature information. In this scheme, the first text feature information is determined based on the text information to be processed, and the target text processing result information corresponding to the text information to be processed is determined based on the first text feature information. The text processing process does not require access to the model, resulting in low computational cost and fast processing speed. Furthermore, the target model transforms the second feature information based on the third feature information to obtain probability distribution information, and performs sampling processing based on the probability distribution information. Compared with existing text generation methods, this allows for less modification to the generated text distribution. Even further, based on the second text feature information and the first text length information of the text information to be processed, several first candidate text information are determined, and the second probability information corresponding to each first candidate text information is determined. Based on the second probability information, the several first candidate text information are fused to obtain the target text information corresponding to the text information to be processed. This allows for the identification of text fragments generated by the target model in the text information to be processed.

[0098] To better implement the text processing method in the embodiments of this application, a text processing system is also provided in the embodiments of this application, such as... Figure 7 As shown, the text processing system 600 includes:

[0099] Information acquisition module 610 is used to acquire text information to be processed;

[0100] The information determination module 620 is used to determine the first text feature information based on the text information to be processed;

[0101] The text processing module 630 is used to determine the target text processing result information corresponding to the text information to be processed based on the first text feature information.

[0102] In this embodiment, first text feature information is determined based on the text information to be processed, and target text processing result information corresponding to the text information to be processed is determined based on the first text feature information. The text processing process does not require access to the model, and the text processing computation cost is low and the text processing speed is fast.

[0103] In some embodiments of this application, the information determination module 620 determines first text feature information based on the text information to be processed, including:

[0104] Determine the first target vocabulary information corresponding to each word in the text information to be processed;

[0105] Based on each word in the text to be processed and the first target vocabulary information corresponding to each word, the first text feature information is determined.

[0106] In some embodiments of this application, the information determination module 620 determines the first target vocabulary information corresponding to each word in the text information to be processed, including:

[0107] Determine the first feature information corresponding to each word in the text information to be processed; the first feature information represents the feature information of the preceding word corresponding to each word in the text information to be processed;

[0108] The first feature information is processed to obtain the initial value information corresponding to each word;

[0109] Based on the initial value information and the first vocabulary feature information corresponding to the second target vocabulary information, the second target vocabulary information is classified to obtain the first target vocabulary information corresponding to each word.

[0110] In some embodiments of this application, the information determination module 620 classifies the second target vocabulary information based on the initial value information and the first vocabulary feature information corresponding to the second target vocabulary information, to obtain the first target vocabulary information corresponding to each word, including:

[0111] Information generation processing is performed on the first vocabulary feature information corresponding to the initial value information and the second target vocabulary information to obtain the index information corresponding to each word;

[0112] The first lexical feature information and the first parameter are multiplied to obtain the second lexical feature information corresponding to the first target lexical information;

[0113] The second target vocabulary information is filtered based on the index information and the second vocabulary feature information to obtain the first target vocabulary information corresponding to each word.

[0114] In some embodiments of this application, the information determination module 620 determines first text feature information based on each word in the text information to be processed and the first target vocabulary information corresponding to each word, including:

[0115] The process involves matching each word in the text to be processed with the corresponding first target vocabulary information to obtain several first feature words; these first feature words are words in the text to be processed that match the first target vocabulary information.

[0116] The number of several first feature words is determined as the first text feature information.

[0117] In some embodiments of this application, the text processing module 630 determines the target text processing result information corresponding to the text information to be processed based on the first text feature information, including:

[0118] Based on the first text feature information and the second text feature information of the text information to be processed, the first probability information is determined; the second text feature information represents the text length of the text information to be processed.

[0119] The first probability information is identified and processed to obtain the target text processing result information corresponding to the text information to be processed.

[0120] In some embodiments of this application, the target text processing result information includes first processing result information and second processing result information. The text processing module 630 performs recognition processing on the first probability information to obtain the target text processing result information corresponding to the text information to be processed, including:

[0121] Compare the first probability information with the first probability threshold;

[0122] If the probability value corresponding to the first probability information is less than the first probability threshold, the target text processing result information corresponding to the text information to be processed is determined to be the first processing result information; the first processing result information indicates that the text information to be processed is generated by the target model; and / or,

[0123] If the probability value corresponding to the first probability information is greater than or equal to the first probability threshold, the target text processing result information corresponding to the text information to be processed is determined as the second processing result information; the second processing result information indicates that the text information to be processed is not generated by the target model.

[0124] In some embodiments of this application, the target model generates the text information to be processed through the following steps:

[0125] The third-target vocabulary information and the prompt information are fused to obtain fused information; the third-target vocabulary information represents the vocabulary information corresponding to the word currently generated by the target model.

[0126] Feature extraction is performed on the fused information to obtain the second feature information;

[0127] Based on the third feature information and the second feature information, the probability distribution information is determined; the third feature information represents the feature information of the previous word corresponding to the currently generated word of the target model.

[0128] Sampling is performed based on probability distribution information to obtain the text information to be processed.

[0129] In some embodiments of this application, the second feature information includes several feature information to be processed. The target model determines probability distribution information based on the third feature information and the second feature information, including:

[0130] Based on the third feature information, the second target vocabulary information is classified and processed to obtain several fourth target vocabulary information;

[0131] Information matching is performed on several fourth target vocabulary pieces and several feature pieces to be processed to obtain vocabulary association information; the vocabulary association information represents the correspondence between several feature pieces to be processed and several fourth target vocabulary pieces.

[0132] Based on lexical association information, several feature information to be processed are calculated to obtain probability distribution information.

[0133] In some embodiments of this application, the target model calculates and processes several feature information to be processed based on lexical association information to obtain probability distribution information, including:

[0134] Based on lexical association information, a weighted summation of several feature information to be processed is performed to obtain the fourth feature information;

[0135] For any one of the several unprocessed feature information, the fourth feature information and the unprocessed feature information are calculated and processed based on the word association information to obtain the transformation feature information corresponding to each feature information;

[0136] The transformed feature information corresponding to several feature information to be processed is fused to obtain probability distribution information.

[0137] In some embodiments of this application, several fourth target vocabulary information includes first vocabulary information, and / or second vocabulary information, and / or third vocabulary information. The target model, for any one of the several feature information to be processed, calculates and processes the fourth feature information and the feature information to be processed based on vocabulary association information to obtain the transformed feature information corresponding to each feature information, including:

[0138] For any one of several unprocessed feature information, if the fourth target vocabulary information corresponding to the unprocessed feature information is the first vocabulary information, the unprocessed feature information and the bias information are added together to obtain the first preprocessed feature information. The first preprocessed feature information and the fourth feature information are then calculated to obtain the transformed feature information corresponding to the unprocessed feature information; and / or,

[0139] If the fourth target vocabulary information corresponding to the feature information to be processed is the second vocabulary information, perform an exponential operation on the feature information to be processed to obtain the second preprocessed feature information, and divide the second preprocessed feature information and the fourth feature information to obtain the transformed feature information corresponding to the feature information to be processed; and / or,

[0140] If the fourth target vocabulary information corresponding to the feature information to be processed is the third vocabulary information, the feature information to be processed and the bias information are subtracted to obtain the third preprocessed feature information. The third preprocessed feature information and the fourth feature information are calculated to obtain the transformation feature information corresponding to the feature information to be processed.

[0141] In some embodiments of this application, after the text processing module 630 determines the target text processing result information corresponding to the text information to be processed based on the first text feature information, the text processing module 630 is further configured to:

[0142] Based on the second text feature information and the first text length information of the text information to be processed, several first candidate text information of the text information to be processed are determined; the second text feature information represents the text length of the text information to be processed.

[0143] Determine the second probability information corresponding to each first candidate text information;

[0144] Based on the second probability information, several first candidate text information are fused to obtain the target text information corresponding to the text information to be processed; the target text information represents the text position of the text segment generated by the target model in the text information to be processed.

[0145] In some embodiments of this application, each first candidate text information includes first boundary information and second boundary information. The text processing module 630 determines the second probability information corresponding to each first candidate text information, including:

[0146] Based on the first boundary information, determine the first quantity information;

[0147] Based on the second boundary information, determine the second quantity information;

[0148] Subtract the first quantity information from the second quantity information to obtain the third quantity information;

[0149] Subtract the first boundary information from the second boundary information to obtain the second text length information corresponding to each first candidate text information;

[0150] Based on the third quantity information and the second text length information, the second probability information corresponding to each first candidate text information is determined.

[0151] In some embodiments of this application, the text processing module 630 determines first quantity information based on first boundary information, including:

[0152] Based on the first boundary information, determine the information of the first text fragment;

[0153] Determine the fifth target vocabulary information corresponding to each word in the first text segment information;

[0154] The first text fragment information is matched with the fifth target vocabulary information corresponding to each word to obtain several second feature words; the several second feature words are the words in the first text fragment information that match the fifth target vocabulary information;

[0155] The quantity of several second feature words is determined as the first quantity information.

[0156] In some embodiments of this application, the text processing module 630 performs fusion processing on several first candidate text information based on second probability information to obtain target text information corresponding to the text information to be processed, including:

[0157] Based on the second probability information, several second candidate text information are determined from several first candidate text information; the several second candidate text information are text information whose probability value corresponding to the second probability information is less than the second probability threshold among the several first candidate text information.

[0158] Based on the first boundary information, several second candidate text information are fused to obtain several third candidate text information;

[0159] Several third-party candidate text information are fused to obtain the target text information corresponding to the text information to be processed.

[0160] This application also provides a computer device that integrates any of the text processing systems provided in this application. The computer device includes:

[0161] One or more processors;

[0162] Memory; and

[0163] One or more applications, wherein the applications are stored in memory and configured to be executed by a processor from the steps of the text processing method in any of the embodiments described above.

[0164] This application also provides a computer device that integrates any of the text processing systems provided in this application. For example... Figure 8 As shown, it illustrates a structural schematic diagram of the computer device involved in the embodiments of this application, specifically:

[0165] The computer device may include components such as a processor 801 with one or more processing cores, a memory 802 with one or more computer-readable storage media, a power supply 803, and an input unit 804. Those skilled in the art will understand that... Figure 8The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0166] The processor 801 is the control center of the computer device. It connects various parts of the computer device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 802, and by calling data stored in the memory 802, it performs various functions of the computer device and processes data, thereby providing overall monitoring of the computer device. Optionally, the processor 801 may include one or more processing cores; preferably, the processor 801 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 801.

[0167] The memory 802 can be used to store software programs and modules. The processor 801 executes various functional applications and data processing by running the software programs and modules stored in the memory 802. The memory 802 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 802 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 802 may also include a memory controller to provide the processor 801 with access to the memory 802.

[0168] The computer device also includes a power supply 803 that supplies power to the various components. Preferably, the power supply 803 can be logically connected to the processor 801 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 803 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0169] The computer device may also include an input unit 804, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0170] Although not shown, the computer device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 801 in the computer device loads the executable files corresponding to the processes of one or more application programs into the memory 802 according to the following instructions, and the processor 801 runs the application programs stored in the memory 802 to realize various functions, as follows:

[0171] Obtain the text information to be processed;

[0172] Based on the text information to be processed, determine the first text feature information;

[0173] Based on the first text feature information, the target text processing result information corresponding to the text information to be processed is determined.

[0174] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0175] Therefore, embodiments of this application provide a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk, etc. A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in any of the text processing methods provided in embodiments of this application. For example, the computer program loaded by the processor can execute the following steps:

[0176] Obtain the text information to be processed;

[0177] Based on the text information to be processed, determine the first text feature information;

[0178] Based on the first text feature information, the target text processing result information corresponding to the text information to be processed is determined.

[0179] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the detailed descriptions of other embodiments above, which will not be repeated here.

[0180] In practice, each of the above units or structures can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units or structures, please refer to the previous method embodiments, which will not be repeated here.

[0181] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0182] The foregoing has provided a detailed description of a text processing method, system, device, and storage medium based on a large model provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method, characterized in that, include: Obtain the text information to be processed; Based on the text information to be processed, the first text feature information is determined; Based on the first text feature information, the target text processing result information corresponding to the text information to be processed is determined.

2. The method according to claim 1, characterized in that, The step of determining the first text feature information based on the text information to be processed includes: Determine the first target vocabulary information corresponding to each word in the text information to be processed; Based on each word in the text to be processed and the first target vocabulary information corresponding to each word, the first text feature information is determined.

3. The method according to claim 2, characterized in that, The step of determining the first target vocabulary information corresponding to each word in the text information to be processed includes: Determine the first feature information corresponding to each word in the text information to be processed; the first feature information represents the feature information of the preceding word corresponding to each word in the text information to be processed; The first feature information is processed to obtain the initial value information corresponding to each word; Based on the initial value information and the first vocabulary feature information corresponding to the second target vocabulary information, the second target vocabulary information is classified to obtain the first target vocabulary information corresponding to each word.

4. The method according to claim 3, characterized in that, The process of classifying the second target vocabulary information based on the initial value information and the first vocabulary feature information corresponding to the second target vocabulary information to obtain the first target vocabulary information corresponding to each word includes: Information generation processing is performed on the first vocabulary feature information corresponding to the initial value information and the second target vocabulary information to obtain the index information corresponding to each word; The first lexical feature information and the first parameter are multiplied together to obtain the second lexical feature information corresponding to the first target lexical information. Based on the index information and the second vocabulary feature information, the second target vocabulary information is filtered to obtain the first target vocabulary information corresponding to each word.

5. The method according to claim 2, characterized in that, The step of determining the first text feature information based on each word in the text to be processed and the first target vocabulary information corresponding to each word includes: The text information to be processed is matched with the first target vocabulary information corresponding to each word to obtain a number of first feature words; the number of first feature words are words in the text information to be processed that match the first target vocabulary information. The number of several of the first feature words is determined as the first text feature information.

6. The method according to claim 1, characterized in that, The step of determining the target text processing result information corresponding to the text information to be processed based on the first text feature information includes: Based on the first text feature information and the second text feature information of the text information to be processed, a first probability information is determined; the second text feature information represents the text length of the text information to be processed. The first probability information is processed to obtain the target text processing result information corresponding to the text information to be processed.

7. The method according to claim 6, characterized in that, The target text processing result information includes first processing result information and second processing result information. The step of identifying and processing the first probability information to obtain the target text processing result information corresponding to the text information to be processed includes: Compare the first probability information with the first probability threshold; If the probability value corresponding to the first probability information is less than the first probability threshold, the target text processing result information corresponding to the text information to be processed is determined to be the first processing result information; the first processing result information indicates that the text information to be processed is generated by the target model; and / or, If the probability value corresponding to the first probability information is greater than or equal to the first probability threshold, the target text processing result information corresponding to the text information to be processed is determined to be the second processing result information; the second processing result information indicates that the text information to be processed is not generated by the target model.

8. The method according to claim 7, characterized in that, The target model generates the text information to be processed through the following steps: The third target vocabulary information and the prompt information are fused to obtain fused information; the third target vocabulary information represents the vocabulary information corresponding to the word currently generated by the target model. Feature extraction is performed on the fused information to obtain second feature information; Based on the third feature information and the second feature information, probability distribution information is determined; the third feature information represents the feature information of the word preceding the word currently generated by the target model. Sampling processing is performed based on the probability distribution information to obtain the text information to be processed.

9. The method according to claim 8, characterized in that, The second feature information includes several feature information to be processed. The step of determining the probability distribution information based on the third feature information and the second feature information includes: Based on the third feature information, the second target vocabulary information is classified and processed to obtain several fourth target vocabulary information; Information matching is performed on several fourth target vocabulary pieces and several feature pieces to be processed to obtain vocabulary association information; the vocabulary association information represents the correspondence between several feature pieces to be processed and several fourth target vocabulary pieces. Based on the lexical association information, several of the features to be processed are calculated and processed to obtain probability distribution information.

10. The method according to claim 9, characterized in that, The step of calculating and processing several feature information to be processed based on the vocabulary association information to obtain probability distribution information includes: Based on the lexical association information, a weighted summation of several feature information to be processed is performed to obtain the fourth feature information; For any one of the several features to be processed, the fourth feature and the feature to be processed are calculated and processed based on the word association information to obtain the transformation feature information corresponding to each feature to be processed. The transformed feature information corresponding to several of the feature information to be processed is fused to obtain probability distribution information.

11. The method according to claim 10, characterized in that, The plurality of fourth target vocabulary information includes first vocabulary information, and / or second vocabulary information, and / or third vocabulary information. For any one of the plurality of target feature information, the fourth feature information and the target feature information are calculated and processed based on the vocabulary association information to obtain the transformation feature information corresponding to each target feature information, including: For any one of the several unprocessed feature information, if the fourth target vocabulary information corresponding to the unprocessed feature information is the first vocabulary information, the unprocessed feature information and the bias information are added together to obtain the first preprocessed feature information, and the first preprocessed feature information and the fourth feature information are calculated to obtain the transformation feature information corresponding to the unprocessed feature information; and / or, If the fourth target vocabulary information corresponding to the feature information to be processed is the second vocabulary information, perform an exponential operation on the feature information to be processed to obtain the second preprocessed feature information, and perform a division operation between the second preprocessed feature information and the fourth feature information to obtain the transformation feature information corresponding to the feature information to be processed; and / or, If the fourth target vocabulary information corresponding to the feature information to be processed is the third vocabulary information, the feature information to be processed and the bias information are subtracted to obtain the third preprocessed feature information, and the third preprocessed feature information and the fourth feature information are calculated to obtain the transformation feature information corresponding to the feature information to be processed.

12. The method according to any one of claims 1 to 11, characterized in that, After determining the target text processing result information corresponding to the text information to be processed based on the first text feature information, the process includes: Based on the second text feature information and the first text length information of the text information to be processed, several first candidate text information of the text information to be processed are determined; the second text feature information represents the text length of the text information to be processed. Determine the second probability information corresponding to each of the first candidate text information; Based on the second probability information, several first candidate text information are fused to obtain target text information corresponding to the text information to be processed; the target text information represents the text position of the text segment generated by the target model in the text information to be processed.

13. The method according to claim 12, characterized in that, Each of the first candidate text pieces includes first boundary information and second boundary information. Determining the second probability information corresponding to each of the first candidate text pieces includes: Based on the first boundary information, the first quantity information is determined; Based on the second boundary information, determine the second quantity information; Subtract the first quantity information from the second quantity information to obtain the third quantity information; Subtract the first boundary information from the second boundary information to obtain the second text length information corresponding to each first candidate text information; Based on the third quantity information and the second text length information, the second probability information corresponding to each of the first candidate text information is determined.

14. The method according to claim 13, characterized in that, The step of determining the first quantity information based on the first boundary information includes: Based on the first boundary information, the information of the first text segment is determined; Determine the fifth target vocabulary information corresponding to each word in the first text fragment information; The first text fragment information is matched with the fifth target vocabulary information corresponding to each word to obtain a number of second feature words; the number of second feature words are words in the first text fragment information that match the fifth target vocabulary information; The quantity of several second feature words is determined as the first quantity information.

15. The method according to claim 13, characterized in that, The step of fusing several first candidate text information based on the second probability information to obtain the target text information corresponding to the text information to be processed includes: Based on the second probability information, several second candidate text information are determined from several first candidate text information; the several second candidate text information are text information whose probability value corresponding to the second probability information is less than the second probability threshold among several first candidate text information. Based on the first boundary information, several second candidate text information are fused to obtain several third candidate text information; The target text information corresponding to the text information to be processed is obtained by fusing several of the third candidate text information.

16. A system, characterized in that, include: The information acquisition module is used to acquire text information to be processed; The information determination module is used to determine first text feature information based on the text information to be processed; The text processing module is used to determine the target text processing result information corresponding to the text information to be processed based on the first text feature information; Optionally, the information determination module determines first text feature information based on the text information to be processed, including: Determine the first target vocabulary information corresponding to each word in the text information to be processed; Based on each word in the text to be processed and the first target vocabulary information corresponding to each word, the first text feature information is determined; Optionally, the information determination module determines the first target vocabulary information corresponding to each word in the text information to be processed, including: Determine the first feature information corresponding to each word in the text information to be processed; the first feature information represents the feature information of the preceding word corresponding to each word in the text information to be processed; The first feature information is processed to obtain the initial value information corresponding to each word; Based on the initial value information and the first vocabulary feature information corresponding to the second target vocabulary information, the second target vocabulary information is classified to obtain the first target vocabulary information corresponding to each word; Optionally, the information determination module classifies the second target vocabulary information based on the initial value information and the first vocabulary feature information corresponding to the second target vocabulary information to obtain the first target vocabulary information corresponding to each word, including: Information generation processing is performed on the first vocabulary feature information corresponding to the initial value information and the second target vocabulary information to obtain the index information corresponding to each word; The first lexical feature information and the first parameter are multiplied together to obtain the second lexical feature information corresponding to the first target lexical information. Based on the index information and the second vocabulary feature information, the second target vocabulary information is filtered to obtain the first target vocabulary information corresponding to each word; Optionally, the information determination module determines first text feature information based on each word in the text information to be processed and the first target vocabulary information corresponding to each word, including: The text information to be processed is matched with the first target vocabulary information corresponding to each word to obtain a number of first feature words; the number of first feature words are words in the text information to be processed that match the first target vocabulary information. The number of several first feature words is determined as the first text feature information; Optionally, the text processing module determines the target text processing result information corresponding to the text information to be processed based on the first text feature information, including: Based on the first text feature information and the second text feature information of the text information to be processed, a first probability information is determined; the second text feature information represents the text length of the text information to be processed. The first probability information is identified and processed to obtain the target text processing result information corresponding to the text information to be processed. Optionally, the target text processing result information includes first processing result information and second processing result information. The text processing module performs recognition processing on the first probability information to obtain the target text processing result information corresponding to the text information to be processed, including: Compare the first probability information with the first probability threshold; If the probability value corresponding to the first probability information is less than the first probability threshold, the target text processing result information corresponding to the text information to be processed is determined to be the first processing result information; the first processing result information indicates that the text information to be processed is generated by the target model; and / or, If the probability value corresponding to the first probability information is greater than or equal to the first probability threshold, the target text processing result information corresponding to the text information to be processed is determined to be the second processing result information; the second processing result information indicates that the text information to be processed is not generated by the target model. Optionally, the target model generates the text information to be processed through the following steps: The third target vocabulary information and the prompt information are fused to obtain fused information; the third target vocabulary information represents the vocabulary information corresponding to the word currently generated by the target model. Feature extraction is performed on the fused information to obtain second feature information; Based on the third feature information and the second feature information, probability distribution information is determined; the third feature information represents the feature information of the word preceding the word currently generated by the target model. Sampling processing is performed based on the probability distribution information to obtain the text information to be processed; Optionally, the second feature information includes several feature information to be processed, and the target model determines probability distribution information based on the third feature information and the second feature information, including: Based on the third feature information, the second target vocabulary information is classified and processed to obtain several fourth target vocabulary information; Information matching is performed on several fourth target vocabulary pieces and several feature pieces to be processed to obtain vocabulary association information; the vocabulary association information represents the correspondence between several feature pieces to be processed and several fourth target vocabulary pieces. Based on the vocabulary association information, several feature information to be processed are calculated and processed to obtain probability distribution information; Optionally, the target model calculates and processes several of the features to be processed based on the lexical association information to obtain probability distribution information, including: Based on the lexical association information, a weighted summation of several feature information to be processed is performed to obtain the fourth feature information; For any one of the several features to be processed, the fourth feature and the feature to be processed are calculated and processed based on the word association information to obtain the transformation feature information corresponding to each feature to be processed. The transformed feature information corresponding to several of the feature information to be processed is fused to obtain probability distribution information; Optionally, the plurality of fourth target vocabulary information includes first vocabulary information, and / or second vocabulary information, and / or third vocabulary information. The target model, for any one of the plurality of feature information to be processed, calculates and processes the fourth feature information and the feature information to be processed based on the vocabulary association information to obtain the transformation feature information corresponding to each feature information to be processed, including: For any one of the several unprocessed feature information, if the fourth target vocabulary information corresponding to the unprocessed feature information is the first vocabulary information, the unprocessed feature information and the bias information are added together to obtain the first preprocessed feature information, and the first preprocessed feature information and the fourth feature information are calculated to obtain the transformation feature information corresponding to the unprocessed feature information; and / or, If the fourth target vocabulary information corresponding to the feature information to be processed is the second vocabulary information, perform an exponential operation on the feature information to be processed to obtain the second preprocessed feature information, and perform a division operation between the second preprocessed feature information and the fourth feature information to obtain the transformation feature information corresponding to the feature information to be processed; and / or, If the fourth target vocabulary information corresponding to the feature information to be processed is the third vocabulary information, the feature information to be processed and the bias information are subtracted to obtain the third preprocessed feature information, and the third preprocessed feature information and the fourth feature information are calculated to obtain the transformation feature information corresponding to the feature information to be processed. Optionally, after the text processing module determines the target text processing result information corresponding to the text information to be processed based on the first text feature information, the text processing module is further configured to: Based on the second text feature information and the first text length information of the text information to be processed, several first candidate text information of the text information to be processed are determined; the second text feature information represents the text length of the text information to be processed. Determine the second probability information corresponding to each of the first candidate text information; Based on the second probability information, several first candidate text information are fused to obtain target text information corresponding to the text information to be processed; the target text information represents the text position of the text segment generated by the target model in the text information to be processed. Optionally, each of the first candidate text information includes first boundary information and second boundary information, and the text processing module determines the second probability information corresponding to each of the first candidate text information, including: Based on the first boundary information, the first quantity information is determined; Based on the second boundary information, determine the second quantity information; Subtract the first quantity information from the second quantity information to obtain the third quantity information; Subtract the first boundary information from the second boundary information to obtain the second text length information corresponding to each first candidate text information; Based on the third quantity information and the second text length information, determine the second probability information corresponding to each of the first candidate text information; Optionally, the text processing module determines the first quantity information based on the first boundary information, including: Based on the first boundary information, the information of the first text segment is determined; Determine the fifth target vocabulary information corresponding to each word in the first text fragment information; The first text fragment information is matched with the fifth target vocabulary information corresponding to each word to obtain a number of second feature words; the number of second feature words are words in the first text fragment information that match the fifth target vocabulary information; The quantity of several second feature words is determined as the first quantity information; Optionally, the text processing module performs fusion processing on several first candidate text information based on the second probability information to obtain target text information corresponding to the text information to be processed, including: Based on the second probability information, several second candidate text information are determined from several first candidate text information; the several second candidate text information are text information whose probability value corresponding to the second probability information is less than the second probability threshold among several first candidate text information. Based on the first boundary information, several second candidate text information are fused to obtain several third candidate text information; The target text information corresponding to the text information to be processed is obtained by fusing several of the third candidate text information.

17. A computer device, characterized in that, The computer device includes: One or more processors; Memory; and One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the method of any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that, It contains a computer program that is loaded by a processor to perform the steps of the method according to any one of claims 1 to 15.

Images