Text processing method, device, electronic device and readable storage medium

Abstract

The present application provides a text processing method, device, electronic device and readable storage medium. The method includes: processing the text to be processed to obtain multiple sentence vectors; classifying each sentence vector according to the title vector of the text to be processed to obtain a sentence set corresponding to each category, and the sentence set includes at least one sentence vector; for each sentence set, according to each sentence vector, the sentences corresponding to the sentence vector representing the same topic are merged to obtain at least one text block. The present application classifies the sentence vectors based on the title vector representing the large topic, and merges the sentences describing the same topic in combination with the small topic represented by the sentence vector itself, so that the non-standardized text and the standardized text can be accurately understood, so that the text can be accurately segmented and the accurate text segmentation result can be obtained.

Description

Technical Field

[0001] The present application belongs to the field of computer technology, and in particular, relates to a text processing method, device, electronic device and readable storage medium. Background Art

[0002] At present, text segmentation is achieved through fixed rules or simple statistical methods. This method can handle standard texts well, but it often has poor processing effects when facing network texts, colloquial texts or multilingual mixed texts. Summary of the invention

[0003] The embodiments of the present application provide a text processing method, device, electronic device, readable storage medium and computer program product, which can solve the problem that the existing text segmentation method has poor processing effect.

[0004] In a first aspect, an embodiment of the present application provides a text processing method, comprising:

[0005] Process the text to be processed to obtain multiple sentence vectors;

[0006] Classifying each of the sentence vectors according to the title vector of the text to be processed to obtain a sentence set corresponding to each category, wherein the sentence set includes at least one sentence vector;

[0007] For each of the sentence sets, based on each of the sentence vectors, sentences corresponding to the sentence vectors representing the same topic are merged to obtain at least one text block.

[0008] In one embodiment, the sentence vectors are classified according to the title vector of the text to be processed to obtain a sentence set corresponding to each category, including:

[0009] Determining a first correlation between the title vector and each of the sentence vectors;

[0010] Sorting the sentence vectors according to the first relevances to obtain a sorted vector set;

[0011] Obtaining sentence vectors corresponding to each of the categories from the sorted vector set to obtain at least two cluster center vectors;

[0012] Based on each of the cluster center vectors, the sentence vectors are classified to obtain a sentence set of each category.

[0013] In one embodiment, acquiring the sentence vectors corresponding to each of the categories from the sorted vector set to obtain at least two cluster center vectors includes:

[0014] Obtaining coordinate information of each category;

[0015] According to the coordinate information of each of the categories, a corresponding sentence vector is obtained from the sorted vector set to obtain at least two cluster center vectors;

[0016] The coordinate information is determined according to the number of categories and the number of sentence vectors.

[0017] In one embodiment, for each of the sentence sets, according to each of the sentence vectors, sentences corresponding to the sentence vectors representing the same topic are merged to obtain at least one text block, including:

[0018] For each of the sentence sets, the second correlation between two adjacent sentence vectors is determined according to the text order, and if the second correlation is greater than a preset threshold, the sentences corresponding to the two adjacent sentence vectors are merged until the sentence vectors of the sentence set are traversed to obtain the at least one text block.

[0019] In one embodiment, determining the second correlation between two adjacent sentence vectors according to the text order, and merging the two adjacent target text segments if the second correlation is greater than a preset threshold, until the sentence vectors of the sentence set are traversed, includes:

[0020] Sorting the sentence vectors in the sentence set according to the text order to obtain a sorted sentence set;

[0021] Obtain the next sentence vector adjacent to the current sentence vector according to the order of the sorted sentence set;

[0022] Determine the second correlation between the current sentence vector and the next sentence vector, and if the second correlation is greater than a preset threshold, merge the sentence corresponding to the current sentence vector with the sentence corresponding to the next sentence vector;

[0023] The next sentence vector is used as the current sentence vector, and the execution step is returned: according to the order of the sorted sentence set, the next sentence vector adjacent to the current sentence vector is obtained until the sentence vectors of the sorted sentence set are traversed.

[0024] In one embodiment, the processing of the text to be processed to obtain multiple sentence vectors includes:

[0025] Segment the text to be processed according to the preset delimiter priority to obtain multiple sentences;

[0026] Each of the sentences is vectorized to obtain a plurality of sentence vectors.

[0027] In one embodiment, before classifying each of the sentence vectors according to the title vector of the text to be processed, the method further includes:

[0028] If the text to be processed does not have title information, inputting the text to be processed and the prompt word into a trained large language model to obtain the title vector output by the trained large language model;

[0029] If the text to be processed has the title information, the title information is processed to obtain the title vector.

[0030] In a second aspect, an embodiment of the present application provides a text processing device, including:

[0031] A processing module is used to process the text to be processed and obtain multiple sentence vectors;

[0032] A classification module, used to classify each of the sentence vectors according to the title vector of the text to be processed, and obtain a sentence set corresponding to each category, wherein the sentence set includes at least one sentence vector;

[0033] The merging module is used to merge the sentences corresponding to the sentence vectors representing the same topic for each of the sentence sets according to each of the sentence vectors to obtain at least one text block.

[0034] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a method as described in any one of the above-mentioned first aspects when executing the computer program.

[0035] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method as described in any one of the above-mentioned first aspects is implemented.

[0036] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes any one of the methods described in the first aspect.

[0037] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0038] The embodiment of the present application obtains multiple sentence vectors by processing the text to be processed; classifies each sentence vector according to the title vector of the text to be processed to obtain a sentence set corresponding to each category, and the sentence set includes at least one sentence vector; for each sentence set, according to each sentence vector, merges the sentences corresponding to the sentence vector representing the same topic to obtain at least one text block, classifies the sentence vectors based on the title vector representing the large topic, and merges the sentences describing the same topic in combination with the small topics represented by the sentence vector itself, so that non-normalized text and normalized text can be accurately understood, so that the text can be accurately segmented to obtain accurate text segmentation results.

[0039] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 This is a first flow chart of a text processing method provided by an embodiment of the present application;

[0042] Figure 2 This is a second flow chart of the text processing method provided by an embodiment of the present application;

[0043] Figure 3 This is a third flow chart of a text processing method provided in an embodiment of the present application;

[0044] Figure 4 This is a fourth flow chart of a text processing method provided in an embodiment of the present application;

[0045] Figure 5 is a structural schematic diagram of a text processing device provided by an embodiment of the present application;

[0046] Figure 6 It is a schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0047] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0048] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0049] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0050] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0051] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0052] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0053] Please refer to Figure 1 , Figure 1 This is a first flow chart of a text processing method provided by an embodiment of the present application. Figure 1 As shown, the method comprises:

[0054] S11: Process the text to be processed to obtain multiple sentence vectors.

[0055] In the application, the text to be processed is divided into high-granularity parts to obtain multiple sentences expressing a single content. Multiple sentences are vectorized to obtain sentence vectors.

[0056] S12: Classify each sentence vector according to the title vector of the text to be processed to obtain a sentence set corresponding to each category.

[0057] The sentence set includes at least one sentence vector.

[0058] In the application, the title of the text to be processed is used to represent the theme and central content of the text to be processed. Generally, the title is the clue throughout the text. Based on the content expressed by the title of the text to be processed and the content expressed by the sentences, the sentence vectors are classified to obtain multiple sentence sets. The sentences corresponding to each sentence vector in the sentence set express similar content.

[0059] S13: For each sentence set, based on each sentence vector, sentences corresponding to the sentence vectors representing the same topic are merged to obtain at least one text block.

[0060] In the application, the sentence set also includes sentence vectors representing the same topic, and the relationship between the sentence vectors is determined to merge the sentences expressing the same topic. After merging, the text to be processed is divided into at least one text block, each text block expresses the same topic, and a text segmentation result is obtained.

[0061] It is understandable that in addition to segmenting the text according to fixed rules or simple statistical methods, the text will also be segmented by a neural network to achieve text segmentation. However, the neural network segmentation method requires training the neural network before it can be used, and a large amount of labeled data is required during training, and the generalization ability of the model and the ability to process long texts need to be improved. And some neural networks need to understand the semantics of the text before they can be segmented, which will consume a lot of computing resources and is not conducive to deployment in resource-constrained environments. Some neural networks regenerate the original text to achieve direct segmentation of the text, but the long text has many words, resulting in a large number of generated words, a long time and a large amount of calculation. Some neural networks summarize the key points of the text and then divide the text to reduce the amount of text generated, which may lose detailed information. This embodiment does not involve the use of a neural network, does not require training of the neural network, and does not require the regeneration of the original text. Compared with the neural network method, it can shorten the application time of the text segmentation method, and it is short in time and small in calculation. This embodiment divides the text according to the title and text sentence itself of the text, and does not involve the use of a neural network, so that the text can be segmented quickly and can be deployed in an environment with few resources. This embodiment directly processes the entire text to be processed, and the article content will not be lost during the segmentation process.

[0062] In addition, this embodiment divides the text to be processed and merges the sentences corresponding to the sentence vectors representing the same topic, thereby saving the vector storage space after the text is vectorized and improving the recall rate and precision rate of subsequent vector retrieval.

[0063] This embodiment obtains multiple sentence vectors by processing the text to be processed; classifies each sentence vector according to the title vector of the text to be processed to obtain a sentence set corresponding to each category, and the sentence set includes at least one sentence vector; for each sentence set, according to each sentence vector, merges the sentences corresponding to the sentence vector representing the same topic to obtain at least one text block, classifies the sentence vectors based on the title vector representing the large topic, and merges the sentences describing the same topic in combination with the small topics represented by the sentence vector itself, so that non-normalized text and normalized text can be accurately understood, so that the text can be accurately segmented to obtain accurate text segmentation results.

[0064] Please refer to Figure 2 , Figure 2 This is a second flow chart of the text processing method provided by an embodiment of the present application. Figure 2 As shown, step S11 includes:

[0065] S111: Segment the text to be processed according to the preset delimiter priority to obtain multiple sentences.

[0066] In the application, the usage of punctuation marks is analyzed, the priority of each punctuation mark is determined, and the preset delimiter priority is obtained. According to the preset delimiter priority, the text to be processed is segmented to obtain multiple minimum units with complete semantic information, namely sentences. The specific expression is: T = {t i |i∈[1,m],i∈Z}, where t i is the i-th sentence, m is the number of sentences, and Z represents a set of integers.

[0067] For example, the delimiter list is L = [(back quotation mark"), (period.), (exclamation mark!), (question mark?)]. The default delimiter priority is to decrease from the front to the back, and the back quotation mark is used as a delimiter only when it is after three other punctuation marks.

[0068] Why does "Timeout when waiting for searchstring200OK" appear when starting or upgrading a cluster? The " and " in the string are not used together. The " is used as a separator when splitting.

[0069] Scientists named the cloned sheep "Dolly" after a country singer. " before ", " as a separator when dividing.

[0070] Xu Beihong gave her a sentence: "No venture, no gain." " appears after " and is used as a separator when segmenting.

[0071] S112: Process each sentence to obtain multiple sentence vectors.

[0072] In the application, each sentence can be input into the trained Embedding model to obtain the vectorized sentence output by the trained Embedding model.

[0073] Among them, the vectorized vector is a high-dimensional vector. The specific expression is: V = {v i |v i =E(t i ),t i ∈T,i∈[1,m],i∈Z}, where v i is the i-th n-dimensional sentence vector, and E is the Embedding model.

[0074] Then, the vectorized sentences are normalized to obtain the sentence vector. The specific expression is Among them, n is the dimension of the sentence vector, x j is the j-th dimension data in the sentence vector. At this time, V={v i `|i∈[1,m],i∈Z}.

[0075] This embodiment segments the text to be processed according to the preset delimiter priority to accurately divide the text to be processed, obtains multiple sentences, vectorizes each sentence, and obtains multiple sentence vectors to better process the sentences later.

[0076] In one embodiment, sentence vector classification is achieved by determining the relevance between the title vector and the sentence vector, and classifying the sentence vector according to the relevance. Figure 3 , Figure 3 FIG. 1 is a third flow chart of a text processing method provided in an embodiment of the present application. Figure 3 As shown, step S12 includes:

[0077] S121: Determine a first correlation between the title vector and each sentence vector.

[0078] In the application, the cosine similarity between the title vector and each sentence vector is calculated, and the cosine similarity represents the correlation, and the first correlation is obtained.

[0079] The specific expression is: v theme is the title vector, and v` is the sentence vector.

[0080] In a possible implementation, before step S11, the method further includes:

[0081] S21: If the text to be processed does not have title information, the text to be processed and the prompt word are input into the trained large language model to obtain the title vector output by the trained large language model.

[0082] In the application, when the text to be processed does not have title information, the title information of the text to be processed must be obtained first. The title of the text to be processed can be extracted through the trained large language model (LLM). To ensure an accurate title, the text to be processed and the prompt word are input into the trained large language model. The trained large language model analyzes the text to be processed based on the prompt word, determines and outputs the topic vector.

[0083] In order to reduce the long prediction time of the trained large language model, the length of the output topic vector is limited.

[0084] S22: If the text to be processed has title information, the title information is processed to obtain a title vector.

[0085] In the application, when there is title information in the text to be processed, the title information can be directly vectorized and normalized to obtain the title vector.

[0086] S122: Sort the sentence vectors according to the first relevances to obtain a sorted vector set.

[0087] In the application, the sentence vectors are sorted in descending order according to the first relevances to obtain a sorted vector set. The sorted vector set includes m sentence vectors.

[0088] S123: Obtain sentence vectors corresponding to each category from the sorted vector set to obtain at least two cluster center vectors.

[0089] In the application, the first correlation between the sentence vector and the title vector not only reflects the correlation between the content of the sentence vector and the title vector, but also reflects the correlation between the contents of each sentence vector. The cluster center vector that can represent the category characteristics is obtained from the sorted vector set, providing a basis for classifying each sentence vector.

[0090] In a possible implementation, step S123 includes:

[0091] S31: Obtain coordinate information of each category.

[0092] Among them, the coordinate information is determined according to the number of categories and the number of sentence vectors.

[0093] In the application, the number of categories is first determined according to the type of text to be processed and the number of sentence vectors. Then the coordinates of the categories are determined according to the number of categories and the number of sentence vectors.

[0094] For example, the formula for determining the number of categories is: Where p and q are positive integers and are determined according to the type of text to be processed and the number of sentence vectors. k∈[1,m]. The formula for determining the coordinate information is:

[0095] S32: According to the coordinate information of each category, obtain the corresponding sentence vector from the sorted vector set to obtain at least two cluster center vectors.

[0096] In the application, according to the coordinate information of the category, the corresponding sequence number is determined in the sorted vector set, and then the corresponding sentence vector is obtained to obtain the cluster center vector.

[0097] S124: Classify the sentence vectors based on the cluster center vectors to obtain sentence sets of each category.

[0098] In a possible implementation, based on each cluster center vector, a clustering algorithm is used to classify the sentence vectors to obtain a set of sentences in each category. For example, the clustering algorithm is a Kmeans clustering algorithm (k-means clustering algorithm, an iterative clustering analysis algorithm).

[0099] This embodiment determines the first correlation between the title vector and each sentence vector, sorts the sentence vectors according to each first correlation, obtains a sorted vector set, obtains the sentence vectors corresponding to each category from the sorted vector set, obtains at least two cluster center vectors, and classifies the sentence vectors based on each cluster center vector to obtain sentence sets of each category. This embodiment can accurately classify the sentence vectors and obtain accurate sentence sets of each category.

[0100] Please refer to Figure 4 , Figure 4 This is a fourth flow chart of a text processing method provided by an embodiment of the present application. Figure 4 As shown, step S13 includes:

[0101] S131: For each sentence set, determine the second correlation between two adjacent sentence vectors according to the text order, and if the second correlation is greater than a preset threshold, merge the sentences corresponding to the two adjacent sentence vectors until all sentence vectors of the sentence set are traversed to obtain at least one text block.

[0102] In applications, generally, the greater the correlation between two adjacent sentences, the greater the possibility that they describe the same topic. According to the second correlation between the two adjacent sentence vectors, it is determined whether the two sentences are to be merged.

[0103] In a possible implementation, step S131 includes:

[0104] S41: Sort the sentence vectors in the sentence set according to the text order to obtain a sorted sentence set.

[0105] For example, the text to be processed includes a, b, c, d, e, f, and g in text order. A certain sentence set includes c, e, and a. The sentence set is sorted in text order, and the sorted sentence set includes a, c, and e.

[0106] S42: Obtain the next sentence vector adjacent to the current sentence vector according to the order of the sorted sentence set.

[0107] S43: Determine a second correlation between the current sentence vector and the next sentence vector, and if the second correlation is greater than a preset threshold, merge the sentence corresponding to the current sentence vector with the sentence corresponding to the next sentence vector.

[0108] In the application, the cosine similarity between the current sentence vector and the next sentence vector is calculated. The cosine similarity represents the correlation, and the second correlation is obtained. The specific expression is: v` c is the current sentence vector, v` c+1 is the next sentence vector.

[0109] Among them, the preset threshold can be determined according to the working effect of the Embedding model of multiple texts and the corresponding text segmentation results. The preset threshold is greater than 0 and less than 1.

[0110] When the second correlation is greater than the preset threshold, it means that the current sentence vector is similar to the next sentence vector, and the sentence corresponding to the current sentence vector and the sentence corresponding to the next sentence vector describe the same topic, and the sentence corresponding to the current sentence vector and the sentence corresponding to the next sentence vector are merged. When the second correlation is less than or equal to the preset threshold, it means that the current sentence vector and the next sentence vector are not similar, and the sentence corresponding to the current sentence vector and the sentence corresponding to the next sentence vector do not describe the same topic, and the sentence corresponding to the current sentence vector and the sentence corresponding to the next sentence vector are not merged and remain independent.

[0111] S44: Take the next sentence vector as the current sentence vector, and return to the execution step: obtain the next sentence vector adjacent to the current sentence vector in the order of the sorted sentence set, until the sentence vectors of the sorted sentence set are traversed.

[0112] In the application, if the sorted sentence set includes h sentence vectors, the similarity calculation is performed h-1 times.

[0113] This embodiment determines the second correlation between two adjacent sentence vectors for each sentence set according to the text order, and if the second correlation is greater than a preset threshold, merges the sentences corresponding to the two adjacent sentence vectors until all sentence vectors of the sentence set are traversed to obtain at least one text block. It is possible to merge sentences that describe the same topic in the text to be processed, and obtain an accurate text segmentation result of the text to be processed.

[0114] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0115] Corresponding to the method described in the above embodiment, for the convenience of explanation, only the part related to the embodiment of the present application is shown.

[0116] Please refer to Figure 5 , Figure 5 It is a structural diagram of a text processing device provided in one embodiment of the present application.

[0117] like Figure 5 As shown, the device comprises:

[0118] A processing module 10 is used to process the text to be processed and obtain multiple sentence vectors;

[0119] A classification module 11, used to classify each sentence vector according to the title vector of the text to be processed, and obtain a sentence set corresponding to each category, wherein the sentence set includes at least one sentence vector;

[0120] The merging module 12 is used to merge the sentences corresponding to the sentence vectors representing the same topic for each sentence set according to each sentence vector, so as to obtain at least one text block.

[0121] In one embodiment, the classification module is specifically used to determine the first correlation between the title vector and each sentence vector; sort the sentence vectors according to each first correlation to obtain a sorted vector set; obtain the sentence vector corresponding to each category from the sorted vector set to obtain at least two cluster center vectors; based on each cluster center vector, classify the sentence vectors to obtain a sentence set of each category.

[0122] In one embodiment, the classification module is specifically used to determine the coordinate information of each category; according to the coordinate information of each category, the corresponding sentence vector is obtained from the sorted vector set to obtain at least two cluster center vectors; wherein the coordinate information is determined according to the number of categories and the number of sentence vectors.

[0123] In one embodiment, the merging module is specifically used to determine, for each sentence set, the second correlation between two adjacent sentence vectors according to the text order, and if the second correlation is greater than a preset threshold, merge the sentences corresponding to the two adjacent sentence vectors until the sentence vectors of the sentence set are traversed to obtain at least one text block.

[0124] In one embodiment, the merging module is specifically used to sort the sentence vectors in the sentence set according to the text order to obtain the sorted sentence set; obtain the next sentence vector adjacent to the current sentence vector according to the order of the sorted sentence set; determine the second correlation between the current sentence vector and the next sentence vector, and if the second correlation is greater than a preset threshold, merge the sentence corresponding to the current sentence vector with the sentence corresponding to the next sentence vector; use the next sentence vector as the current sentence vector, and return to the execution step: obtain the next sentence vector adjacent to the current sentence vector according to the order of the sorted sentence set, until the sentence vectors of the sorted sentence set are traversed.

[0125] In one embodiment, the processing module is specifically used to segment the text to be processed according to the preset delimiter priority to obtain multiple sentences; and process each sentence to obtain multiple sentence vectors.

[0126] In one embodiment, the processing module is further used to input the text to be processed and the prompt word into a trained large language model to obtain a title vector output by the trained large language model if there is no title information in the text to be processed; if there is title information in the text to be processed, process the title information to obtain a title vector.

[0127] Figure 6 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 6 As shown, the electronic device 2 of this embodiment includes: at least one processor 20 ( Figure 6 Only one is shown in the figure), a memory 21, and a computer program 22 stored in the memory 21 and executable on the at least one processor 20, wherein the processor 20 implements the steps of any of the above-mentioned method embodiments when executing the computer program 22.

[0128] The electronic device 2 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The electronic device 2 may include, but is not limited to, a processor 20 and a memory 21. Those skilled in the art will appreciate that Figure 6 It is only an example of the electronic device 2 and does not constitute a limitation on the electronic device 2. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0129] The processor 20 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0130] In some embodiments, the memory 21 may be an internal storage unit of the electronic device 2, such as a hard disk or memory of the electronic device 2. In other embodiments, the memory 21 may also be an external storage device of the electronic device 2, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 2. Further, the memory 21 may also include both an internal storage unit of the electronic device 2 and an external storage device. The memory 21 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory 21 may also be used to temporarily store data that has been output or is to be output.

[0131] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0132] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0133] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0134] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0135] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some cases, the computer-readable medium cannot be an electric carrier signal and a telecommunication signal.

[0136] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0137] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0138] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0139] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0140] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A text processing method, characterized in that: include: Process the text to be processed to obtain multiple sentence vectors; Classifying each of the sentence vectors according to the title vector of the text to be processed to obtain a sentence set corresponding to each category, wherein the sentence set includes at least one sentence vector; For each of the sentence sets, based on each of the sentence vectors, sentences corresponding to the sentence vectors representing the same topic are merged to obtain at least one text block.

2. The method according to claim 1, characterized in that The step of classifying the sentence vectors according to the title vector of the text to be processed to obtain a sentence set corresponding to each category includes: Determining a first correlation between the title vector and each of the sentence vectors; Sorting the sentence vectors according to the first relevances to obtain a sorted vector set; Obtaining sentence vectors corresponding to each of the categories from the sorted vector set to obtain at least two cluster center vectors; Based on each of the cluster center vectors, the sentence vectors are classified to obtain a sentence set of each category.

3. The method according to claim 2, characterized in that The acquiring the sentence vectors corresponding to each of the categories from the sorted vector set to obtain at least two cluster center vectors includes: Obtaining coordinate information of each category; According to the coordinate information of each of the categories, a corresponding sentence vector is obtained from the sorted vector set to obtain at least two cluster center vectors; The coordinate information is determined according to the number of categories and the number of sentence vectors.

4. The method according to any one of claims 1 to 3, characterized in that: The step of merging sentences corresponding to the sentence vectors representing the same topic for each of the sentence sets according to each of the sentence vectors to obtain at least one text block includes: For each of the sentence sets, the second correlation between two adjacent sentence vectors is determined according to the text order, and if the second correlation is greater than a preset threshold, the sentences corresponding to the two adjacent sentence vectors are merged until the sentence vectors of the sentence set are traversed to obtain the at least one text block.

5. The method according to claim 4, characterized in that The step of determining a second correlation between two adjacent sentence vectors according to the text order, and merging sentences corresponding to the two adjacent sentence vectors if the second correlation is greater than a preset threshold, until all the sentence vectors of the sentence set are traversed, includes: Sorting the sentence vectors in the sentence set according to the text order to obtain a sorted sentence set; Obtain the next sentence vector adjacent to the current sentence vector according to the order of the sorted sentence set; Determine the second correlation between the current sentence vector and the next sentence vector, and if the second correlation is greater than a preset threshold, merge the sentence corresponding to the current sentence vector with the sentence corresponding to the next sentence vector; The next sentence vector is used as the current sentence vector, and the execution step is returned: according to the order of the sorted sentence set, the next sentence vector adjacent to the current sentence vector is obtained until the sentence vectors of the sorted sentence set are traversed.

6. The method according to claim 5, characterized in that The text to be processed is processed to obtain multiple sentence vectors, including: Segment the text to be processed according to the preset delimiter priority to obtain multiple sentences; Each of the sentences is processed to obtain a plurality of sentence vectors.

7. The method according to claim 6, characterized in that Before classifying each of the sentence vectors according to the title vector of the text to be processed, the method further includes: If the text to be processed does not have title information, inputting the text to be processed and the prompt word into a trained large language model to obtain the title vector output by the trained large language model; If the text to be processed has the title information, the title information is processed to obtain the title vector.

8. A text processing device, characterized in that: include: A processing module is used to process the text to be processed and obtain multiple sentence vectors; A classification module, used to classify each of the sentence vectors according to the title vector of the text to be processed, and obtain a sentence set corresponding to each category, wherein the sentence set includes at least one sentence vector; The merging module is used to merge the sentences corresponding to the sentence vectors representing the same topic for each of the sentence sets according to each of the sentence vectors to obtain at least one text block.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images