Text analysis method, device and equipment, medium and vehicle
By analyzing and sorting the pending text, filtering the appropriate target text, and inputting it with the question statement into the language processing model, the problem of reducing analysis efficiency caused by text length differences is solved and the work efficiency of the language processing model is improved.
Patent Information
- Application Number
- CN202311008653.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-10
- Publication Date
- 2025-02-25
AI Technical Summary
When the language processing model processes text in actual applications, the text length is far from the text length during training, resulting in reduced analysis efficiency and reduced work efficiency.
By receiving pending text and questioning statements, the semantic correlation between the text and the questioning statements is analyzed, the pending text is sorted, the target text that meets the preset number of texts, and the target text is input to the questioning statement for analysis.
The number of text that the language processing model needs to analyze is reduced, and the efficiency and work efficiency of text analysis are improved.
Smart Images
Figure CN119513236A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of natural language processing, and in particular to a text analysis method, device, equipment, medium and vehicle. Background Art
[0002] Currently, language processing models can handle a variety of natural language tasks, such as text classification, question and answer, dialogue, etc., and are an important path to artificial intelligence.
[0003] When using a language processing model to process natural language, due to the need to record contextual history information of unknown length, it is inevitable to input text with a very large number of characters (very long text length), that is, text with a very long text length, into the language processing model. However, language processing models are usually trained using text with a small number of characters (smaller text length), which makes the text length of the language processing model in actual application significantly different from the text length during training, resulting in reduced analysis efficiency of the language processing model for the text, and reducing the working efficiency of the language processing model. Summary of the invention
[0004] In order to solve the above technical problems, the present disclosure provides a text analysis method, device, equipment, medium and vehicle.
[0005] A first aspect of the present disclosure provides a text analysis method, comprising:
[0006] Receiving multiple texts to be processed and question statements corresponding to the texts to be processed;
[0007] Analyze the semantic relevance between the text to be processed and the question sentence to obtain the semantic relevance between each text to be processed and the question sentence;
[0008] Sort the texts to be processed in descending order of semantic relevance to obtain a text sequence;
[0009] Starting from the starting text of the text sequence, the text to be processed that is less than or equal to the preset number of characters is selected as the target text, and the preset number of characters is less than or equal to the maximum number of characters that can be processed by the language processing model;
[0010] The target text and question sentence are input into the language processing model, language analysis is performed based on the language processing model, and the answer sentence corresponding to the question sentence is output.
[0011] A second aspect of the present disclosure provides a text analysis device, comprising:
[0012] A receiving module, used for receiving a plurality of texts to be processed and question statements corresponding to the texts to be processed;
[0013] An analysis module is used to analyze the semantic relevance between the text to be processed and the question sentence, and obtain the semantic relevance between each text to be processed and the question sentence;
[0014] A sorting module is used to sort the texts to be processed in descending order of semantic relevance to obtain a text sequence;
[0015] A screening module is used to start from the starting text of the text sequence and screen the to-be-processed text corresponding to less than or equal to a preset number of characters as the target text, where the preset number of characters is less than or equal to the maximum number of characters that can be processed by the language processing model;
[0016] The output module is used to input the target text and the question sentence into the language processing model, perform language analysis based on the language processing model, and output the answer sentence corresponding to the question sentence.
[0017] A third aspect of the present disclosure provides a computer device, comprising:
[0018] A memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the text analysis method of the first aspect mentioned above can be implemented.
[0019] A fourth aspect of the present disclosure provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the text analysis method of the first aspect can be implemented.
[0020] A fifth aspect of the present disclosure provides a vehicle, comprising the text analysis device of the second aspect or the computer device of the third aspect or the computer-readable storage medium of the fourth aspect, which can implement the text analysis method of the first aspect.
[0021] Compared with the prior art, the technical solution provided by the present invention has the following advantages:
[0022] The present invention receives a plurality of to-be-processed texts and question sentences corresponding to the to-be-processed texts; analyzes the semantic relevance between the to-be-processed texts and the question sentences to obtain the semantic relevance between each to-be-processed text and the question sentence; sorts the to-be-processed texts in descending order of the semantic relevance to obtain a text sequence; starts from the starting text of the text sequence, filters out to-be-processed texts corresponding to a number less than or equal to a preset number of characters as target texts, wherein the preset number of characters is less than or equal to the maximum number of characters that can be processed by a language processing model; inputs the target texts and the question sentences into a language processing model, performs language analysis based on the language processing model, and outputs an answer sentence corresponding to the question sentence. According to the semantic relevance between the to-be-processed texts and the question sentences and the number of characters in the to-be-processed texts, the to-be-processed texts having a greater semantic relevance to the question sentences and a number of characters that does not exceed the processing capacity of the language processing model are filtered out, and the filtered to-be-processed texts and question sentences are input into the language processing model for analysis, thereby reducing the number of characters that the language processing model needs to analyze, and improving the text analysis efficiency and work efficiency of the language processing model. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0025] Figure 1 is a flowchart of a text analysis method provided by an embodiment of the present disclosure;
[0026] Figure 2 is a flowchart of another text analysis method provided by an embodiment of the present disclosure;
[0027] Figure 3 is a flow chart of a target coding model training method provided by an embodiment of the present disclosure;
[0028] Figure 4 is a flowchart of another target coding model training method provided by an embodiment of the present disclosure;
[0029] Figure 5 is a structural schematic diagram of a text analysis device provided by an embodiment of the present disclosure;
[0030] Figure 6 It is a structural diagram of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0031] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.
[0032] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.
[0033] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0034] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0035] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0036] Related technologies: Currently, language processing models can handle a variety of natural language tasks, such as text classification, question and answer, dialogue, etc., and are an important path to artificial intelligence.
[0037] The embodiments of the present disclosure provide a text analysis method, apparatus, device, medium and vehicle, which can screen out texts to be processed that have a greater semantic relevance to a question statement and a number of characters that does not exceed the processing capacity of a language processing model according to the semantic relevance between the texts to be processed and the question statement and the number of characters in the texts to be processed, and input the screened texts to be processed and the question statement into the language processing model for analysis, thereby reducing the number of characters that need to be analyzed by the language processing model and improving the text analysis efficiency and work efficiency of the language processing model.
[0038] The text analysis method provided by the embodiments of the present disclosure may be executed by a computer device, which may be understood as any device with processing and computing capabilities, and may include but is not limited to mobile terminals such as smart phones, laptops, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), vehicle-mounted terminals, wearable devices, and fixed electronic devices such as digital TVs, desktop computers, smart home devices, and the like.
[0039] In order to better understand the inventive concept of the embodiments of the present disclosure, the technical solutions of the embodiments of the present disclosure are described below in conjunction with exemplary embodiments.
[0040] Figure 1 is a flowchart of a text analysis method provided by an embodiment of the present disclosure, such as Figure 1 As shown, the text analysis method provided in this embodiment includes the following steps:
[0041] Step 110: Receive a plurality of texts to be processed and question statements corresponding to the texts to be processed.
[0042] In the disclosed embodiment, the text to be processed can be understood as a text composed of multiple context sentences and / or historical sentences. The text to be processed can be a text composed of at least one text sentence, or a text composed of at least one text chunk, each of which contains at least one text sentence.
[0043] The question sentence corresponding to the to-be-processed text can be understood as a sentence with a question or problem meaning related to the to-be-processed text. For example, the question sentence can include a question sentence in scenarios such as text classification, question-answering, and dialogue.
[0044] In the disclosed embodiment, the computer device may receive a plurality of texts to be processed and question statements corresponding to the texts to be processed.
[0045] Step 120: Analyze the semantic relevance between the text to be processed and the question sentence to obtain the semantic association degree between each text to be processed and the question sentence.
[0046] In the embodiment of the present disclosure, the degree of semantic association can be understood as the degree of correlation between the semantics expressed by the text to be processed and the semantics expressed by the question sentence.
[0047] In the disclosed embodiment, the computer device may analyze the semantic relevance between the text to be processed and the question sentence to obtain the degree of semantic association between each text to be processed and the question sentence.
[0048] Step 130 , sort the texts to be processed in descending order of semantic relevance to obtain a text sequence.
[0049] In the embodiment of the present disclosure, after obtaining the semantic association degree between each to-be-processed text and the question sentence, the computer device can sort the to-be-processed texts in descending order of the semantic association degree to obtain a text sequence. That is, the semantic association degree corresponding to the to-be-processed texts in the sentence sequence decreases from the front to the back.
[0050] Step 140 , starting from the starting text of the text sequence, filter out the to-be-processed text corresponding to a preset number of characters as the target text, wherein the preset number of characters is less than or equal to the maximum number of characters that can be processed by the language processing model.
[0051] In the disclosed embodiment, the computer device may start from the starting text of the text sequence and filter out the to-be-processed text corresponding to a number less than or equal to a preset number of characters as the target text, that is, the number of characters contained in the target text is less than or equal to the preset number of characters.
[0052] The preset number of characters is less than or equal to the maximum number of characters that can be processed by the language processing model. The language processing model can be understood as a language processing model, for example, a large language model (LLM). The maximum number of characters that can be processed by the language processing model is the maximum number of characters that the language processing model can process. The maximum number of characters that can be processed by the language processing model is determined by the number of characters in the text when training the language processing model and / or the computing power of the computer.
[0053] Step 150: Input the target text and the question sentence into a language processing model, perform language analysis based on the language processing model, and output an answer sentence corresponding to the question sentence.
[0054] In the disclosed embodiment, the computer device can input the target text and the question statement into the language processing model, perform language analysis based on the language processing model, and output the answer statement corresponding to the question statement, that is, the language processing model can generate the answer statement corresponding to the question statement based on the target text.
[0055] The disclosed embodiment receives a plurality of to-be-processed texts and question sentences corresponding to the to-be-processed texts; analyzes the semantic relevance between the to-be-processed texts and the question sentences to obtain the semantic relevance between each to-be-processed text and the question sentence; sorts the to-be-processed texts in descending order of the semantic relevance to obtain a text sequence; starts from the starting text of the text sequence, filters out to-be-processed texts corresponding to a number of characters less than or equal to a preset number as target texts, wherein the number of characters is less than or equal to the maximum number of characters that can be processed by a language processing model; inputs the target texts and the question sentence into a language processing model, performs language analysis based on the language processing model, and outputs an answer sentence corresponding to the question sentence. According to the semantic relevance between the to-be-processed texts and the question sentence and the number of characters in the to-be-processed texts, the to-be-processed texts having a greater semantic relevance to the question sentence and a number of characters that does not exceed the processing capacity of the language processing model are filtered out, and the filtered to-be-processed texts and the question sentences are input into the language processing model for analysis, thereby reducing the number of characters that the language processing model needs to analyze and improving the text analysis efficiency and work efficiency of the language processing model.
[0056] Figure 2 is a flowchart of a text analysis method provided by an embodiment of the present disclosure, such as Figure 2 As shown, the text analysis method provided in this embodiment includes the following steps:
[0057] Step 210: Receive a plurality of texts to be processed and question statements corresponding to the texts to be processed.
[0058] Step 220: Encode the texts to be processed and the question sentences to obtain first sentence vectors corresponding to the texts to be processed and second sentence vectors corresponding to the question sentences.
[0059] In the embodiment of the present disclosure, after receiving the text to be processed and the question sentence corresponding to the text to be processed, the computer device can encode the text to be processed to obtain the first sentence vector corresponding to each text to be processed, and encode the question sentence to obtain the second sentence vector corresponding to the question sentence.
[0060] In some embodiments, encoding the text to be processed and the question sentence to obtain the first sentence vector corresponding to each text to be processed and the second sentence vector corresponding to the question sentence may include steps 2201-2204:
[0061] Step 2201: for each text to be processed, perform word segmentation encoding processing on the text sentences in the text to be processed to obtain at least one first word vector corresponding to each text to be processed, and perform word segmentation encoding processing on the question sentence to obtain at least one second word vector corresponding to the question sentence.
[0062] In the disclosed embodiment, word segmentation is to decompose a character, word or phrase into multiple sub-characters, sub-words or phrases to better understand and analyze the text.
[0063] For each text to be processed, the computer device may perform word segmentation processing on a text sentence in the text to be processed to obtain at least one word segmentation contained in each text to be processed, and then encode each word segmentation to obtain a first word vector corresponding to each word segmentation, thereby obtaining at least one first word vector corresponding to each text to be processed;
[0064] The computer device can also perform word segmentation processing on the question sentence to obtain at least one word segmentation contained in the question sentence, and then encode each word segmentation to obtain a second word vector corresponding to each word segmentation, thereby obtaining at least one second word vector corresponding to the question sentence.
[0065] For example, word segmentation processing can be performed using word segmentation algorithms such as Jieba and BPE. For details, please refer to related technologies, which will not be described here.
[0066] Step 2202: For each text to be processed, combine the first word vector corresponding to the text to be processed with the second word vector corresponding to the question sentence to obtain a first word vector sequence corresponding to the text to be processed; combine the second word vectors corresponding to the question sentence with each other to obtain a second word vector sequence corresponding to the question sentence.
[0067] In the disclosed embodiment, for each text to be processed, the computer device can combine the first word vector corresponding to the text to be processed with the second word vector corresponding to the question statement to obtain a first word vector sequence corresponding to the text to be processed, and combine the second word vectors corresponding to the question statement with each other to obtain a second word vector sequence corresponding to the question statement.
[0068] For example, the text a to be processed contains text sentences S1 and S2, and the first word vector corresponding to text sentence S1 is e token1 and e token2 , the first word vector corresponding to the text sentence S2 is e token3 , then the first word vector corresponding to the text a to be processed is e token1 、e token2 and e token3 3 in total;
[0069] The second word vector corresponding to question sentence b is e token4 and e token5 2 in total;
[0070] The first word vector sequence corresponding to the text a to be processed can be e token1 , e token2 , e token3 , e token4 , etoken5 ;
[0071] The second word vector sequence corresponding to question sentence b can be e token4 , e token5 .
[0072] Step 2203: Based on the preset target encoding model, the first word vector sequence corresponding to each text to be processed and the second word vector sequence corresponding to the question sentence are encoded to obtain the first sentence vector corresponding to each first word vector sequence and the second sentence vector corresponding to the second word vector sequence.
[0073] In the disclosed embodiment, the target encoding model can be understood as a model that can encode each word vector sequence into a sentence vector.
[0074] The computer device can encode the first word vector sequence corresponding to each text to be processed and the second word vector sequence corresponding to the question sentence based on a preset target encoding model to obtain the first sentence vector corresponding to each first word vector sequence and the second sentence vector corresponding to the second word vector sequence.
[0075] For example, the first word vector sequence corresponding to the text a to be processed is e token1 , e token2 , e token3 , e token4 , e token5 , then the first sentence vector corresponding to the first word vector sequence can be f1(e token1 , e token2 , e token3 , e token4 , e token5 ), the second word vector sequence corresponding to question sentence b is e token4 , e token5 , then the second sentence vector corresponding to the second word vector sequence can be f2(e token4 , e token5 ).
[0076] Step 230: For each first sentence vector, calculate the similarity between the first sentence vector and the second sentence vector.
[0077] In the disclosed embodiment, the computer device may calculate the similarity between the first sentence vector corresponding to each to-be-processed text and the second sentence vector corresponding to the question sentence, for example, the similarity may be cosine similarity, and the specific calculation process may refer to the relevant technology, which will not be described in detail here.
[0078] Step 240: Determine the similarity between the first sentence vector corresponding to the text to be processed and the second sentence vector corresponding to the question sentence as the degree of semantic association between the text sentence and the question sentence.
[0079] In the embodiment of the present disclosure, the similarity between the first sentence vector and the second sentence vector can represent the distance between the semantics expressed by the first sentence vector and the semantics expressed by the second sentence vector. Therefore, the similarity between the first sentence vector corresponding to the text to be processed and the second sentence vector corresponding to the question statement can be determined as the degree of semantic association between the text to be processed and the question statement.
[0080] Step 250: Sort the texts to be processed in descending order of semantic relevance to obtain a text sequence.
[0081] Step 260 , starting from the starting text of the text sequence, filter out the to-be-processed text corresponding to a preset number of characters as the target text, where the preset number of characters is less than or equal to the maximum number of characters that can be processed by the language processing model.
[0082] Step 270: Input the target text and the question sentence into the language processing model, perform language analysis based on the language processing model, and output an answer sentence corresponding to the question sentence.
[0083] The contents of steps 250-270 of the embodiment of the present disclosure may refer to the contents of steps 130-150 described above, and will not be repeated here.
[0084] Therefore, according to the semantic relevance between the text to be processed and the question statement and the number of characters in the text to be processed, the text to be processed that has a large semantic relevance to the question statement and the number of characters that does not exceed the processing capacity of the language processing model can be screened, and the screened text to be processed and the question statement can be input into the language processing model for analysis, which can reduce the number of characters that the language processing model needs to analyze and improve the text analysis efficiency and work efficiency of the language processing model.
[0085] In some embodiments of the present disclosure, before encoding the first word vector sequence corresponding to each to-be-processed text and the second word vector sequence corresponding to the question sentence based on the preset target encoding model, the computer device may execute Figure 3 A flowchart of a target coding model training method is provided, and the target coding model is obtained by training, such as Figure 3 As shown, the target coding model training method provided in this embodiment includes the following steps:
[0086] Step 310: Obtain training text, where the training text includes at least one target training text, a training question sentence, and a training answer sentence.
[0087] In the embodiment of the present disclosure, the computer device can obtain training text for training the model to be trained.
[0088] The training text may include at least one target training text, a training question sentence and a training answer sentence.
[0089] The target training text can be understood as a training text composed of multiple context sentences and / or historical sentences. The target training text can be a text composed of at least one text sentence, or a text composed of at least one text chunk, each of which contains at least one text sentence.
[0090] Step 320: Train the model to be trained based on the target training text, the training question sentence and the training answer sentence to obtain a target encoding model.
[0091] In the disclosed embodiment, the computer device may train the model to be trained based on the target training text, the training question sentences and the training answer sentences to obtain the target coding model.
[0092] In some embodiments, training the training model based on the target training text, the training question sentence and the training answer sentence to obtain the target encoding model may include steps 3301-3308:
[0093] Step 3301: For each target training text block, perform word segmentation encoding processing on the training text sentences in the target training text block to obtain at least one first training word vector corresponding to each target training text, perform word segmentation encoding processing on the training question sentences to obtain at least one second training word vector corresponding to the training question sentences, and perform word segmentation encoding processing on the training answer sentences to obtain at least one third training word vector corresponding to the training answer sentences.
[0094] In the embodiment of the present disclosure, the computer device may perform word segmentation encoding processing on the training text sentences in the target training text for each target training text, and obtain at least one first training word vector corresponding to each target training text;
[0095] Performing word segmentation encoding processing on the training question sentence to obtain at least one second training word vector corresponding to the training question sentence;
[0096] Perform word segmentation encoding processing on the training answer sentence to obtain at least one third training word vector corresponding to the training answer sentence.
[0097] Step 3302: For each target training text, combine the first training word vector corresponding to the target training text with the second training word vector corresponding to the training question sentence to obtain a first training word vector sequence corresponding to the target training text; combine the second training word vectors corresponding to the training question sentence to obtain a second training word vector sequence corresponding to the training question sentence; and combine the third training word vectors corresponding to the training answer sentence to obtain a third training word vector sequence corresponding to the training answer sentence.
[0098] In the embodiment of the present disclosure, the computer device may combine, for each target training text, a first training word vector corresponding to the target training text and a second training word vector corresponding to the training question sentence to obtain a first training word vector sequence corresponding to the target training text;
[0099] The second training word vectors corresponding to the training question sentences are combined with each other to obtain a second training word vector sequence corresponding to the training question sentences;
[0100] The third training word vectors corresponding to the training answer sentences are combined with each other to obtain a third training word vector sequence corresponding to the training answer sentences.
[0101] For example, Figure 4 is a flowchart of a target encoding model training method, such as Figure 4 As shown, if the target training text is a text block, including text Chunk1, text Chunk2, text Chunk3 and text Chunk4, the training question sentence is Q, and the training answer sentence is A;
[0102] The first training word vector corresponding to the text Chunk1 is e t1 、e t2 and e t3 There are 3 in total. The first training word vector corresponding to text Chunk2 is e t4 and e t5 There are 2 in total. The first training word vector corresponding to text Chunk3 is e t6 and e t7 There are 2 in total. The first training word vector corresponding to text Chunk4 is e t8 and e t9 2 in total;
[0103] The second training word vector corresponding to the training question sentence Q is e t10 and e t11 2 in total;
[0104] The third training word vector corresponding to the training answer sentence A is e t12 and e t13 2 in total;
[0105] Then the first training word vector sequence corresponding to the text Chunk1 can be e t1 , e t2 , e t3 ;
[0106] The first training word vector sequence corresponding to the text Chunk2 can be e t4 , e t5 ;
[0107] The first training word vector sequence corresponding to text Chunk3 can be e t6 , e t7 ;
[0108] The first training word vector sequence corresponding to text Chunk4 can be e t8 , e t9 ;
[0109] The second training word vector sequence corresponding to the training question sentence Q can be e t10 , e t11 ;
[0110] The third training word vector sequence corresponding to the training answer sentence A can be e t12 , e t13 .
[0111] Step 3303: Based on the model to be trained, encode the first training word vector sequence, the second training word vector sequence and the third training word vector sequence to obtain a first training sentence vector corresponding to the first training word vector sequence, a second training sentence vector corresponding to the second training word vector sequence, and a third training sentence vector corresponding to the third training word vector sequence.
[0112] In an embodiment of the present disclosure, a computer device can encode a first training word vector sequence, a second training word vector sequence, and a third training word vector sequence based on a model to be trained to obtain a first training sentence vector corresponding to the first training word vector sequence, a second training sentence vector corresponding to the second training word vector sequence, and a third training sentence vector corresponding to the third training word vector sequence.
[0113] Among them, the encoding method of the first training word vector sequence, the second training word vector sequence and the third training word vector sequence can refer to relevant encoding technologies. For example, a bidirectional encoding method similar to Bert pre-training can be used for encoding processing, which will not be repeated here.
[0114] For example, the first training sentence vector corresponding to the text Chunk1 can be f T1 (e t1 , e t2 , e t3 );
[0115] The first training sentence vector corresponding to text Chunk2 can be f T2 (e t4 , e t5 );
[0116] The first training sentence vector corresponding to text Chunk3 can be f T3 (e t6 , e t7 );
[0117] The first training sentence vector corresponding to text Chunk4 can be f T4 (e t8 , e t9 );
[0118] The second training sentence vector corresponding to the training question sentence Q can be f Q (e t10 , e t11 );
[0119] The third training sentence vector corresponding to the training answer sentence A can be f A (e t12 , e t13 ).
[0120] Step 3304: Based on a preset combination method, all first training sentence vectors are combined with second training sentence vectors to obtain a first text vector, and each word vector in the third training sentence vector is combined to obtain a second text vector.
[0121] In the disclosed embodiment, the computer device may combine all first training sentence vectors with second training sentence vectors based on a preset combination method to obtain a first text vector, and combine each word vector in the third training sentence vector to obtain a second text vector.
[0122] The preset combination method can be used to determine the weight of each first training sentence vector, and the weight of the first training sentence vector is a value that measures the correlation between the first training sentence vector and the second training sentence vector.
[0123] Specifically, based on a preset combination method, all first training sentence vectors are combined with second training sentence vectors to obtain a first text vector, which may include S11-S13:
[0124] S11. Determine the semantic relevance between each first training sentence vector and the second training sentence vector based on a preset combination method.
[0125] S12. For each first training sentence vector, multiply the semantic relevance corresponding to the first training sentence vector by the first training sentence vector to obtain a weight vector of the first training sentence vector.
[0126] S13: Combine the weight vectors of the first training sentence vectors with the second training sentence vectors to obtain a first text vector.
[0127] The preset combination method may include at least one of a fully connected network and an attention mechanism. Among them, the fully connected network is also called a multilayer perceptron (MLP), and the layers of the multilayer perceptron are fully connected. For details, please refer to the relevant technology, which will not be repeated here.
[0128] For example, suppose the first training sentence vector f corresponding to the text Chunk1 T1 , the first training sentence vector f corresponding to text Chunk2 T2 , the first training sentence vector f corresponding to text Chunk3 T3 , the first training sentence vector f corresponding to text Chunk4 T4 The second training sentence vector f corresponding to the training question sentence Q Q If the semantic relevance between them is 1, then the first training sentence vector f T1 、f T2 、f T3 and f T4 With the second training sentence vector f Q Combine them to get the first text vector, that is, text vector, denoted as D vec , D vec = fun1(f T1 ,f T2 ,f T3 ,f T4 ,f Q ), D vec Denoted as U.
[0129] Step 3305: Combine the word vectors in the third training sentence vector to obtain a second text vector.
[0130] In the disclosed embodiment, the computer device may combine the word vectors in the third training sentence vector to obtain a second text vector.
[0131] For example, the word vectors in the third training sentence vector corresponding to the training answer sentence A can be combined to obtain the second text vector, namely Avector, denoted as A vec ,A vec = fun2(f A ), A vecDenoted as V.
[0132] Step 3306: Combine each first text vector and the second text vector to obtain a text vector group.
[0133] In the embodiment of the present disclosure, the computer device may combine each first text vector and the second text vector to obtain a text vector group.
[0134] Step 3307: input each text vector group into a preset loss function, evaluate the semantic relevance between the first text vector and the second text vector in the text vector group, and obtain an evaluation result.
[0135] In the disclosed embodiment, the computer device may input each text vector group into a preset loss function, evaluate the semantic relevance between the first text vector and the second text vector in the text vector group, and obtain an evaluation result.
[0136] For example, the preset loss function can be a normalized exponential function (softmax):
[0137] Y = softmax(W(U,V,|UV|));
[0138] Among them, U represents the first text vector, V represents the second text vector, W represents the model parameter, and Y represents the function value, that is, the evaluation result. The larger the Y is, the greater the semantic correlation between the first text vector and the second text vector is.
[0139] Step 3308: Based on the evaluation results, update the model parameters of the model to be trained, and iteratively train the model to be trained until the preset loss function converges to obtain the target coding model.
[0140] In the disclosed embodiment, the computer device can update the model parameters of the model to be trained based on the evaluation results, iteratively train the model to be trained until the preset loss function converges, determine the model parameters of the model to be trained, and obtain the target coding model.
[0141] In this way, a target encoding model can be trained to encode the word vector sequence corresponding to each sentence into a sentence vector.
[0142] Figure 5 is a schematic diagram of the structure of a text analysis device provided by an embodiment of the present disclosure, and the device can be understood as the above-mentioned computer device or some functional modules in the above-mentioned computer device. Figure 5 As shown, the text analysis device 500 includes:
[0143] A receiving module 510 is used to receive a plurality of texts to be processed and question sentences corresponding to the texts to be processed;
[0144] An analysis module 520 is used to analyze the semantic relevance between the text to be processed and the question sentence, and obtain the semantic association degree between each text to be processed and the question sentence;
[0145] A sorting module 530 is used to sort the texts to be processed in descending order of semantic relevance to obtain a text sequence;
[0146] A screening module 540 is used to screen, starting from the starting text of the text sequence, texts to be processed that are less than or equal to a preset number of characters as target texts, where the preset number of characters is less than or equal to a maximum number of characters that can be processed by the language processing model;
[0147] The output module 550 is used to input the target text and the question sentence into the language processing model, perform language analysis based on the language processing model, and output the answer sentence corresponding to the question sentence.
[0148] Optionally, the analysis module 520 includes:
[0149] The encoding submodule is used to encode the text to be processed and the question sentence to obtain a first sentence vector corresponding to each text to be processed and a second sentence vector corresponding to the question sentence;
[0150] A calculation submodule, used for calculating the similarity between the first sentence vector and the second sentence vector for each first sentence vector;
[0151] The determination submodule is used to determine the similarity between the first sentence vector corresponding to the text to be processed and the second sentence vector corresponding to the question sentence as the semantic association degree between the text to be processed and the question sentence.
[0152] Optionally, the encoding submodule includes:
[0153] A first word segmentation encoding unit is used to perform word segmentation encoding processing on a text sentence in each to-be-processed text to obtain at least one first word vector corresponding to each to-be-processed text, and to perform word segmentation encoding processing on a question sentence to obtain at least one second word vector corresponding to the question sentence;
[0154] A combining unit is used to combine, for each text to be processed, the first word vector corresponding to the text to be processed and the second word vector corresponding to the question sentence to obtain a first word vector sequence corresponding to the text to be processed; and to combine the second word vectors corresponding to the question sentence with each other to obtain a second word vector sequence corresponding to the question sentence;
[0155] The encoding unit is used to encode the first word vector sequence corresponding to each text to be processed and the second word vector sequence corresponding to the question sentence based on a preset target encoding model to obtain the first sentence vector corresponding to each first word vector sequence and the second sentence vector corresponding to the second word vector sequence.
[0156] Optionally, the encoding submodule includes:
[0157] An acquisition unit, used to acquire a training text, wherein the training text includes at least one target training text, a training question sentence and a training answer sentence;
[0158] The training unit is used to train the training model based on the target training text, the training question sentence and the training answer sentence to obtain the target encoding model.
[0159] Optionally, the training unit includes:
[0160] A word segmentation encoding subunit is used to perform word segmentation encoding processing on the training text sentences in the target training text for each target training text to obtain a first training word vector corresponding to each target training text, and to perform word segmentation encoding processing on the training question sentence to obtain a second training word vector corresponding to the training question sentence, and to perform word segmentation encoding processing on the training answer sentence to obtain a third training word vector corresponding to the training answer sentence;
[0161] The first combining subunit is used to combine, for each target training text, the first training word vector corresponding to the target training text with the second training word vector corresponding to the training question sentence to obtain a first training word vector sequence corresponding to the target training text; combine the second training word vectors corresponding to the training question sentence with each other to obtain a second training word vector sequence corresponding to the training question sentence; and combine the third training word vectors corresponding to the training answer sentence with each other to obtain a third training word vector sequence corresponding to the training answer sentence;
[0162] The encoding subunit is used to encode the first training word vector sequence, the second training word vector sequence and the third training word vector sequence based on the model to be trained, so as to obtain a first training sentence vector corresponding to the first training word vector sequence, a second training sentence vector corresponding to the second training word vector sequence and a third training sentence vector corresponding to the third training word vector sequence;
[0163] A second combining subunit is used to combine all the first training sentence vectors with the second training sentence vectors based on a preset combining method to obtain a first text vector;
[0164] A third combining subunit, used for combining each word vector in the third training sentence vector to obtain a second text vector;
[0165] a fourth combining subunit, configured to combine each first text vector with the second text vector to obtain a text vector group;
[0166] An evaluation subunit, used for inputting each text vector group into a preset loss function, and evaluating the semantic relevance between the first text vector and the second text vector in the text vector group based on the preset loss function to obtain an evaluation result;
[0167] The training subunit is used to update the model parameters of the model to be trained based on the evaluation results, and iteratively train the model to be trained until the preset loss function converges to obtain the target encoding model.
[0168] Optionally, the second combination subunit includes:
[0169] A determination component, used to determine the semantic relevance between each first training sentence vector and the second training sentence vector based on a preset combination method;
[0170] A multiplication component, used for multiplying the semantic relevance corresponding to the first training sentence vector by the first training sentence vector for each first training sentence vector, to obtain a weight vector of the first training sentence vector;
[0171] The combining component is used to combine the weight vectors of each first training sentence vector with the second training sentence vector to obtain a first text vector.
[0172] The text analysis device provided in the embodiments of the present disclosure can implement the method of any of the above embodiments, and its execution method and beneficial effects are similar, which will not be repeated here.
[0173] The embodiments of the present disclosure also provide a computer device, which includes a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the method of any of the above embodiments can be implemented, and its execution method and beneficial effects are similar and will not be repeated here.
[0174] Figure 6 is a schematic diagram of the structure of a computer device provided by an embodiment of the present disclosure, such as Figure 6 As shown, the computer device 600 may include a processor 610 and a memory 620, wherein the memory 620 stores a computer program 621, and when the computer program 621 is executed by the processor 610, the method provided by any of the above embodiments can be implemented, and its execution method and beneficial effects are similar and will not be repeated here.
[0175] Of course, to simplify, Figure 6Only some of the components related to the present invention in the computer device 600 are shown, and components such as a bus, an input / output interface, an input device, and an output device are omitted. In addition, according to specific application conditions, the computer device 600 may also include any other appropriate components.
[0176] An embodiment of the present disclosure provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the method of any of the above embodiments can be implemented. The execution method and beneficial effects are similar and will not be repeated here.
[0177] The above-mentioned computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0178] The computer program may be written in any combination of one or more programming languages to write program codes for performing the operations of the disclosed embodiments, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer device, partially on the user's device, as a separate software package, partially on the user's computer device and partially on a remote computer device, or entirely on a remote computer device or server.
[0179] An embodiment of the present disclosure provides a vehicle, which includes the above-mentioned text analysis device or the above-mentioned computer device or the above-mentioned computer-readable storage medium, and can implement the method of any of the above-mentioned embodiments. The execution method and beneficial effects are similar and will not be repeated here.
[0180] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.
[0181] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.
[0182] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A text analysis method, characterized in that: include: Receiving a plurality of texts to be processed and question statements corresponding to the texts to be processed; Analyzing the semantic relevance between the text to be processed and the question sentence to obtain the semantic association degree between each text to be processed and the question sentence; Sorting the texts to be processed in descending order of the semantic association degree to obtain a text sequence; Starting from the starting text of the text sequence, selecting the to-be-processed text corresponding to a number less than or equal to a preset number of characters as the target text, wherein the preset number of characters is less than or equal to the maximum number of characters that can be processed by the language processing model; The target text and the question sentence are input into the language processing model, language analysis is performed based on the language processing model, and an answer sentence corresponding to the question sentence is output.
2. The method according to claim 1, characterized in that The step of analyzing the semantic relevance between the text to be processed and the question sentence to obtain the semantic association degree between each text to be processed and the question sentence includes: Encoding the text to be processed and the question sentence to obtain a first sentence vector corresponding to each of the text to be processed and a second sentence vector corresponding to the question sentence; For each of the first sentence vectors, calculating the similarity between the first sentence vector and the second sentence vector; The similarity between the first sentence vector corresponding to the text to be processed and the second sentence vector corresponding to the question sentence is determined as the degree of semantic association between the text to be processed and the question sentence.
3. The method according to claim 2, characterized in that The encoding process of the to-be-processed text and the question sentence to obtain a first sentence vector corresponding to each of the to-be-processed text and a second sentence vector corresponding to the question sentence includes: For each text to be processed, performing word segmentation encoding processing on the text sentences in the text to be processed to obtain at least one first word vector corresponding to each of the text to be processed, and performing word segmentation encoding processing on the question sentence to obtain at least one second word vector corresponding to the question sentence; For each of the to-be-processed texts, the first word vector corresponding to the to-be-processed text is combined with the second word vector corresponding to the question sentence to obtain a first word vector sequence corresponding to the to-be-processed text; the second word vectors corresponding to the question sentence are combined with each other to obtain a second word vector sequence corresponding to the question sentence; Based on a preset target encoding model, the first word vector sequence corresponding to each of the texts to be processed and the second word vector sequence corresponding to the question sentence are encoded to obtain the first sentence vector corresponding to each of the first word vector sequences and the second sentence vector corresponding to the second word vector sequence.
4. The method according to claim 3, characterized in that Before encoding the first word vector sequence corresponding to each of the to-be-processed texts and the second word vector sequence corresponding to the question sentence based on the preset target encoding model, the method further includes: Acquire training text, wherein the training text includes at least one target training text, a training question sentence, and a training answer sentence; The model to be trained is trained based on the target training text, the training question sentences and the training answer sentences to obtain a target encoding model.
5. The method according to claim 4, characterized in that The training model is trained based on the target training text, the training question sentence and the training answer sentence to obtain the target encoding model, including: For each of the target training texts, performing word segmentation encoding processing on the training text sentences in the target training text to obtain a first training word vector corresponding to each of the target training texts, performing word segmentation encoding processing on the training question sentences to obtain a second training word vector corresponding to the training question sentences, and performing word segmentation encoding processing on the training answer sentences to obtain a third training word vector corresponding to the training answer sentences; For each of the target training texts, the first training word vector corresponding to the target training text is combined with the second training word vector corresponding to the training question sentence to obtain a first training word vector sequence corresponding to the target training text; the second training word vectors corresponding to the training question sentence are combined with each other to obtain a second training word vector sequence corresponding to the training question sentence; the third training word vectors corresponding to the training answer sentence are combined with each other to obtain a third training word vector sequence corresponding to the training answer sentence; Based on the model to be trained, encoding the first training word vector sequence, the second training word vector sequence, and the third training word vector sequence is performed to obtain a first training sentence vector corresponding to the first training word vector sequence, a second training sentence vector corresponding to the second training word vector sequence, and a third training sentence vector corresponding to the third training word vector sequence; Based on a preset combination method, all the first training sentence vectors are combined with the second training sentence vectors to obtain a first text vector; Combining each word vector in the third training sentence vector to obtain a second text vector; Combining each of the first text vectors with the second text vector to obtain a text vector group; Inputting each of the text vector groups into a preset loss function, and evaluating the semantic relevance between the first text vector and the second text vector in the text vector group based on the preset loss function to obtain an evaluation result; Based on the evaluation result, the model parameters of the model to be trained are updated, and the model to be trained is iteratively trained until the preset loss function converges to obtain the target coding model.
6. The method according to claim 5, characterized in that The combining of all the first training sentence vectors and the second training sentence vectors based on a preset combination method to obtain a first text vector includes: Determining the semantic relevance between each of the first training sentence vectors and the second training sentence vectors based on a preset combination method; For each of the first training sentence vectors, multiplying the semantic relevance corresponding to the first training sentence vector by the first training sentence vector to obtain a weight vector of the first training sentence vector; The weight vectors of the first training sentence vectors are combined with the second training sentence vectors to obtain the first text vector.
7. A text analysis device, characterized in that: include: A receiving module, used for receiving a plurality of texts to be processed and question statements corresponding to the texts to be processed; An analysis module, used to analyze the semantic relevance between the text to be processed and the question sentence, and obtain the semantic association degree between each text to be processed and the question sentence; A sorting module, used to sort the texts to be processed in descending order of the semantic association degree to obtain a text sequence; A screening module, for screening, starting from the starting text of the text sequence, text to be processed corresponding to a number less than or equal to a preset number of characters as target text, wherein the preset number of characters is less than or equal to a maximum number of characters that can be processed by the language processing model; The output module is used to input the target text and the question sentence into the language processing model, perform language analysis based on the language processing model, and output an answer sentence corresponding to the question sentence.
8. A computer device, characterized in that: include: A memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the text analysis method according to any one of claims 1 to 6 is implemented.
9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the text analysis method according to any one of claims 1 to 6 is implemented.
10. A vehicle, characterized in that: The method comprises the text analysis apparatus as claimed in claim 7 or the computer device as claimed in claim 8 or the computer-readable storage medium as claimed in claim 9.