A text segmentation method and device

Abstract

The present application provides a text segmentation method and device, wherein the text segmentation method comprises: receiving a segmentation instruction for a target text; performing sentence processing on the target text in response to the segmentation instruction to generate a sentence sequence; inputting each sentence in the sentence sequence into a sentence feature extraction model in turn to obtain a sentence feature vector corresponding to each sentence output by the sentence feature extraction model, wherein each sentence feature vector represents feature information of the corresponding sentence; inputting the sentence feature vector corresponding to each sentence into a feature classification model to obtain a sentence category corresponding to each sentence output by the feature classification model; performing segmentation processing on the target text according to the sentence category corresponding to each sentence to obtain a paragraph set. Through this method, there is no need to truncate or compress the text, and the contextual semantic information of the text can be effectively referenced to achieve a better text segmentation effect and enhance the user experience.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence in the field of computer technology, and in particular to a text segmentation method and apparatus, a computing device, and a computer-readable storage medium. Background Art

[0002] Artificial intelligence (AI) refers to the ability of engineered (i.e. designed and manufactured) systems to perceive the environment and to acquire, process, apply and represent knowledge. The development status of key technologies in the field of artificial intelligence, including machine learning, knowledge graphs, natural language processing, computer vision, human-computer interaction, biometrics, virtual reality / augmented reality and other key technologies. In recent years, the most impressive progress in the field of machine learning has appeared in the subfield of deep learning (also known as deep neural network learning). Deep learning refers to the method of creating rich hierarchical representations by training neural networks with many hidden layers. Deep learning uses simple computing units (i.e. "neurons") to form a multi-layer network. Each unit combines a set of input values ​​to produce an output value and passes the value to other neurons downstream. The neural networks in deep learning are composed of several hidden layers. Natural language processing is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between humans and computers in natural language, and segmentation of text, especially long text is an important branch of natural language processing. Deep learning technology can be used to solve various technical problems in the field of natural language processing.

[0003] With the development of computer technology, neural network models have become more and more mature in processing texts. However, neural network models usually have character restrictions when processing texts. For long texts with more words, they usually cannot be input into the neural network model for processing at one time. Therefore, the text needs to be segmented. Text segmentation refers to dividing a piece of text into smaller paragraphs according to different semantics or structures. At present, the text is usually truncated by the processing character limit of the neural network model, or the text is divided into multiple sentences. Useful sentences are screened by pre-trained neural network models to achieve the purpose of shortening the number of words in the long text. However, for a text, there is usually a relationship between the previous and next sentences. Using the above method will cause the text to lose a lot of semantic information, resulting in poor segmentation effect and failure to meet user requirements. Summary of the invention

[0004] In view of this, embodiments of the present application provide a text segmentation method and apparatus, a computing device, and a computer-readable storage medium to address the technical deficiencies in the prior art.

[0005] According to a first aspect of an embodiment of the present application, a text segmentation method is provided, comprising:

[0006] receiving a segmentation instruction for a target text;

[0007] In response to the segmentation instruction, the target text is segmented to generate a sentence sequence;

[0008] Inputting each sentence in the sentence sequence into a sentence feature extraction model in sequence, obtaining a sentence feature vector corresponding to each sentence output by the sentence feature extraction model, wherein each sentence feature vector represents feature information of the corresponding sentence;

[0009] Inputting the sentence feature vector corresponding to each sentence into the feature classification model to obtain the sentence category corresponding to each sentence output by the feature classification model;

[0010] The target text is segmented according to the sentence category corresponding to each sentence to obtain a paragraph set.

[0011] According to a second aspect of an embodiment of the present application, a text segmentation device is provided, comprising:

[0012] A receiving module is configured to receive a segmentation instruction for a target text;

[0013] A sentence segmentation module is configured to perform sentence segmentation processing on the target text in response to the segmentation instruction to generate a sentence sequence;

[0014] A feature extraction module is configured to input each sentence in the sentence sequence into a sentence feature extraction model in sequence, and obtain a sentence feature vector corresponding to each sentence output by the sentence feature extraction model, wherein each sentence feature vector represents feature information of the corresponding sentence;

[0015] A classification module is configured to input a sentence feature vector corresponding to each sentence into a feature classification model to obtain a sentence category corresponding to each sentence output by the feature classification model;

[0016] The segmentation module is configured to segment the target text according to the sentence category corresponding to each sentence to obtain a paragraph set.

[0017] According to a third aspect of an embodiment of the present application, a computing device is provided, comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor implements the steps of the text segmentation method when executing the computer instructions.

[0018] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores computer instructions, and when the computer instructions are executed by a processor, the steps of the text segmentation method are implemented.

[0019] According to a fifth aspect of an embodiment of the present application, a chip is provided, which stores computer instructions, and the computer instructions implement the steps of the text segmentation method when executed by the chip.

[0020] The text segmentation method provided in the embodiment of the present application includes receiving a segmentation instruction for a target text; performing sentence processing on the target text in response to the segmentation instruction to generate a sentence sequence; inputting each sentence in the sentence sequence into a sentence feature extraction model in turn to obtain a sentence feature vector corresponding to each sentence output by the sentence feature extraction model, wherein each sentence feature vector represents feature information of the corresponding sentence; inputting the sentence feature vector corresponding to each sentence into a feature classification model to obtain a sentence category corresponding to each sentence output by the feature classification model; performing segmentation processing on the target text according to the sentence category corresponding to each sentence to obtain a paragraph set. Through this method, there is no need to truncate or compress the text, and the contextual semantic information of the text can be effectively referenced to achieve a better text segmentation effect and enhance the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a structural block diagram of a computing device provided in an embodiment of the present application;

[0022] Figure 2 is a flowchart of a text segmentation method provided by an embodiment of the present application;

[0023] Figure 3 It is a schematic diagram of the structure of the text segmentation method provided in the embodiment of the present application;

[0024] Figure 4 is a processing flow chart of a text segmentation method for segmenting target long text data provided by an embodiment of the present application;

[0025] Figure 5 is a structural schematic diagram of a text segmentation device provided in an embodiment of the present application;

[0026] Figure 6 It is an architecture diagram of a resume processing system provided in an embodiment of the present application;

[0027] Figure 7 It is a processing flow chart of a text segmentation method applied to resume text provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0029] The terms used in one or more embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present application. The singular forms of "a", "said" and "the" used in one or more embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of the present application refers to and includes any or all possible combinations of one or more associated listed items.

[0030] It should be understood that, although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present application, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "in response to determination".

[0031] First, the terms involved in one or more embodiments of the present invention are explained.

[0032] Natural Language Processing is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between humans and computers using natural language. It refers to the use of computers to process the form, sound, meaning and other information of natural language, that is, the operation and processing of input, output, recognition, analysis, understanding and generation of characters, words, sentences and passages.

[0033] A neural network (neural network / neural net / artificial neural network) refers to a network in which each primitive processing element is interconnected by weighted links with adjustable weights. Each element generates a value by applying a nonlinear function to the input value, and then transmits this value to other elements or represents it as an output value.

[0034] A neural-network model refers to an abstract model of a neural network, which can be simulated by software or implemented as a neural computer.

[0035] Bidirectional Encoder Representation from Transformers (BERT) is a deep bidirectional, unsupervised language representation model that is pre-trained using only a plain text corpus. It is used as a pre-training technology for natural language processing (NLP).

[0036] BiLSTM: Bidirectional Long Short-Term Memory artificial neural network model.

[0037] End-to-end framework: refers to the user directly inputting raw materials and getting usable results directly without having to worry about the intermediate products.

[0038] Fully connected layer: Each node in the fully connected layer is connected to all the nodes in the previous layer to integrate the features extracted previously.

[0039] Text segmentation specifically refers to dividing a long text into smaller paragraphs according to different semantics or structures. There are several main methods for text segmentation: (1) Truncation method, which can be divided into head truncation, tail truncation, and head-tail truncation. The head truncation method refers to the number of words from the beginning of the text to the BERT limit, the tail truncation method refers to the number of words from the end of the text to the BERT limit, and the head-tail truncation method refers to retaining a part of the head and tail until the BERT limit. (2) Pooling method: truncate a text into multiple sentences, and then connect a fully connected layer to extract features. (3) Compression method: divide the text into multiple sentences, select meaningful sentences through rules or pre-trained small models, and discard meaningless sentences to shorten the text. However, in actual applications, due to the large number of words in long texts, the longest sequence length supported by BERT is 512, and the number of valid characters is 510, which cannot be input into the BERT model at one time to extract text features. At the same time, there is a relationship between the previous and next sentences in the text. Direct truncation or compression will lose a lot of semantic information of the previous and next contexts, resulting in inaccurate segmentation.

[0040] Based on this, in the present application, a text segmentation method and apparatus, a computing device and a computer-readable storage medium are provided, which are described in detail one by one in the following embodiments.

[0041] Figure 1 The structure block diagram of a computing device 100 according to an embodiment of the present application is shown. The components of the computing device 100 include but are not limited to a memory 110 and a processor 120. The processor 120 is connected to the memory 110 via a bus 130, and the database 150 is used to store data.

[0042] The computing device 100 also includes an access device 140 that enables the computing device 100 to communicate via one or more networks 160. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 140 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0043] In one embodiment of the present application, the above components of the computing device 100 and Figure 1 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 1 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of the present application. Those skilled in the art may add or replace other components as needed.

[0044] The computing device 100 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 100 may also be a mobile or stationary server.

[0045] The processor 120 may execute Figure 2 Steps in the text segmentation method shown. Figure 2 A flowchart of a text segmentation method according to an embodiment of the present application is shown, including steps 202 to 210.

[0046] Step 202: Receive segmentation instructions for the target text.

[0047] The target text is the text that needs to be segmented in this application. Usually, the target text is a long text, such as an article, a news article, a resume document, etc., and the target text has a large number of words.

[0048] The segmentation instruction is an instruction for segmenting the target text. The operation of segmenting the target text is performed according to the segmentation instruction.

[0049] In a specific implementation provided by the present application, taking the target text as a resume document as an example, the resume document includes basic information, skill information, education experience, work experience, etc., and receiving a segmentation instruction for the resume document. It should be noted that in actual application, if the resume document is in a table format, it is also necessary to perform OCR recognition and information extraction on the resume document in advance, and extract the information in the resume to generate resume text information.

[0050] Step 204: Sentence processing is performed on the target text in response to the segmentation instruction to generate a sentence sequence.

[0051] The target text is usually a text including many sentences. In the present application, the target text is processed into sentences in response to the segmentation instruction to obtain a sentence sequence consisting of multiple sentences. Specifically, the target text is processed into sentences in response to the segmentation instruction to generate a sentence sequence, including S2042-S2046:

[0052] S2042. Acquire a preset sentence identifier in response to the segmentation instruction.

[0053] The preset sentence identifier is an identifier used to indicate that a sentence has been fully expressed. It can include semicolons, periods, exclamation marks, line breaks ( / n), question marks, and the full-width and half-width formats of the above symbols. It does not include commas, semicolons, colons, brackets, ellipsis, book title marks, spaces, etc.

[0054] As a specific embodiment, the preset sentence identifiers include semicolons, periods, exclamation marks, line breaks, and the like. Other special characters include mathematical symbols, units, geometric figures, formulas, serial numbers, Latin letters, pinyin, radicals, and the like. Regarding the preset sentence identifiers of this application, we believe that Japanese, Korean, and Russian are not identifiers, but text. Roman numerals and Chinese numerals are both text, and are not identifiers. For some combinations of English with numbers and symbols, such as MBA III, PETS4, ￥35K, etc., they are also regarded as text in this application, rather than identifiers; for cases where there are text or symbols before and after the numbers, such as "in 1998.", "January 2010-May 2020", etc., they are also regarded as text in this application, rather than identifiers.

[0055] The preset sentence identifier can be directly stored in the relevant code for sentence segmentation in the text segmentation system; it can also be placed in the sentence identifier database of the text segmentation system. When the text sentence needs to be processed, the preset sentence identifier is read from the sentence identifier database.

[0056] S2044: Sentence the target text according to the sentence identifier to obtain multiple sentences, and record the position identifier of each sentence.

[0057] After obtaining the preset sentence identifier, the target text is compared with the preset sentence identifier one by one. As long as the preset sentence identifier is encountered, the target text is broken thereby to obtain a sentence. After the entire target text is traversed, multiple sentences separated by sentence identifiers are obtained.

[0058] The resume text information is identified in sequence, and after the sentence identifier is identified, it is divided into a sentence. After all the resume text information is identified, multiple sentences are obtained. For example, taking the period and line break as an example, the target text information is sentence-segmented to obtain multiple sentences corresponding to the target text.

[0059] For example, the target text is: "Zhang San / nWork experience / n From January 2010 to May 2020, he worked as a patent engineer in Kingsoft Software.". According to the preset sentence identifiers " / n", ".", the target text can be divided into three sentences, namely "Zhang San", "Work experience", and "From January 2010 to May 2020, he worked as a patent engineer in Kingsoft Software.".

[0060] While the target text is being processed into sentences according to the sentence identifier, the position identifier of each sentence in the target text is also recorded. The position identifier specifically refers to the position information used to mark each sentence in the target text. For example, the first sentence is recorded as 01, the second sentence is recorded as 02, and so on.

[0061] As a specific embodiment, the position identifier refers to the position coordinates of the text where the first character of the sentence is located, which can be a vector identifier or an array. The position identifier and the sentence are stored in the database of pending sentences corresponding to each other, so as to facilitate obtaining the pending sentence from the database in the subsequent processing process.

[0062] Specifically, the sentence processing of the target text according to the sentence identifier to obtain multiple sentences includes:

[0063] Get the preset statement threshold;

[0064] The target text is sentence-processed according to the sentence identifier and the sentence threshold to obtain a plurality of sentences, wherein the number of sentences obtained by the sentence processing is less than or equal to the sentence threshold.

[0065] The preset sentence threshold is the upper limit of the number of sentences obtained after the target text is processed by sentence segmentation. In practical applications, the sentence threshold can be determined according to the processing capacity of the subsequent feature classification model. For example, if the subsequent feature classification model can process 64 data at a time, the target document can be divided into 64 sentences at most. After obtaining the sentence threshold, the target text is sentence segmented so that the number of sentences in the sentence sequence is less than the sentence threshold.

[0066] It should be noted that in actual applications, the number of sentences may be greater than the sentence threshold. In this case, the sentences are divided according to the sentence threshold to obtain at least two sentence sets. For example, after a target text is processed by sentence segmentation, a total of 80 sentences are obtained. The preset sentence threshold is 64, and two sentence sets are generated, where the first sentence set includes sentences 1-64 and the second sentence set includes sentences 65-80. The sentence threshold is determined based on the processing power of the subsequent feature classification model. When input into the feature classification model, it is also input according to the settings of the feature classification model. For example, the second sentence set actually has only 16 sentences, and it is also necessary to add 0 to the second sentence set to add 64 sentences to the second sentence set.

[0067] S2046. Obtain a statement sequence corresponding to the multiple statements according to the position identifier of each statement.

[0068] After the sentence processing is completed, multiple statements are obtained, and then the multiple statements are combined into a statement sequence according to the position identifier of each statement.

[0069] In a specific implementation provided in the present application, following the above example, after sentence segmentation of the resume document, 50 sentences are obtained, and the position identifiers of the sentences are respectively determined as S01, S02...S50, and these position identifiers are used to form a sentence sequence (S01, S02...S50), wherein 01 in S01 represents that this sentence is the first sentence in the resume document, 02 in S02 represents that this sentence is the second sentence in the resume document, and so on.

[0070] Step 206: Input each sentence in the sentence sequence into the sentence feature extraction model in sequence, and obtain a sentence feature vector corresponding to each sentence output by the sentence feature extraction model, wherein each sentence feature vector represents feature information of the corresponding sentence.

[0071] The feature extraction model is a neural network model used to extract feature information of each sentence. The feature extraction model can be a BERT model, an Ernie model, a Robert model, etc. Preferably, the feature extraction model is a BERT model (Bidirectional Encoder Representation from Transformers). In an embodiment of the present application, a BERT model including sequentially connected embedding layers and an encoder, wherein the encoder includes 12 sequentially connected encoding layers, is taken as an example for explanation.

[0072] Before a sentence is input into the BERT model, a [CLS] mark is usually added to the front of the sentence. The [CLS] mark is a symbol for the beginning of the sentence. After the sentence with the [CLS] mark is input into the BERT model, it is processed by the embedding layer to obtain the matrix to be encoded corresponding to the sentence. The matrix to be encoded is then input into the encoder for encoding processing to obtain the encoding matrix of the sentence. The vector corresponding to the CLS mark in the encoding matrix is ​​the CLS vector. After each sentence in the sentence sequence is processed by the feature extraction model, the corresponding sentence feature vector of each sentence is obtained, that is, the CLS vector corresponding to each sentence is obtained.

[0073] Specifically, the step of sequentially inputting each sentence in the sentence sequence into a sentence feature extraction model to obtain a sentence feature vector corresponding to each sentence output by the sentence feature extraction model includes:

[0074] S2062: Determine a target sentence in the sentence sequence.

[0075] The target sentence is the sentence that enters the feature extraction model for feature extraction. Each sentence can be used as a target sentence when performing feature extraction.

[0076] For example, if the sentence that needs to extract features from the sentence sequence is "Zhang San, male, age 24.", then this sentence is the target sentence.

[0077] S2064: Perform word segmentation processing on the target sentence to obtain a word unit set corresponding to the target sentence.

[0078] Among them, the target words are segmented in order to convert each word into a corresponding word vector according to the embedding dictionary in the subsequent embedding process. Based on this, after determining the target sentence, it is necessary to segment the target sentence to obtain the word unit set corresponding to the target sentence. It should be noted that at this time, the CLS identifier needs to be added specifically for each target, and the word unit set obtained is a word unit set including the CLS identifier.

[0079] For example, if the target sentence is "Zhang San, gender, male, age, 24.", after word segmentation, the word unit set obtained is [CLS, Zhang San, gender, male, age, 24].

[0080] S2066. Input the word unit set into the embedding layer for embedding processing to obtain a word unit vector set corresponding to the word unit set.

[0081] The purpose of inputting the word unit set into the embedding layer for embedding processing is to convert the natural language into a vector that can be recognized by the computer, so that the computer can subsequently extract feature information from the word unit vector.

[0082] For example, using the above example, the word unit set is [CLS, Zhang, San, Sex, Gender, Male, Year, Age, 2, 4], and after embedding processing by the embedding layer of the feature extraction model, the word unit vector set (A CLS , A 张 , A 三 , A 性 , A 别 , A 男 , A 年 , A 龄 , A2, A4).

[0083] S2068. Input the word unit vector set into the encoder for encoding processing to obtain a sentence feature vector corresponding to the target sentence.

[0084] The word unit vector set is input into the encoder, and after the encoding process of 12 sequentially connected encoding layers in the encoder, the position features, keyword features, semantic features, and inter-sentence association features between the word unit vectors in the word unit vector set are extracted to obtain the encoding vector corresponding to the target sentence. The CLS vector in the target sentence is the sentence feature vector of the target sentence. After obtaining the CLS vector, it is convenient to calculate the probability of the target sentence as the starting sentence based on the CLS vector.

[0085] For example, the word unit vector set (A CLS , A 张 , A 三 , A 性 , A 别 , A 男 , A 年 , A 龄 , A2, A4) are input into the embedding layer of the sentence feature extraction model for encoding processing to obtain the sentence feature vector (B CLS , B 张 , B 三 , B 性 , B 别 , B 男 , B 年 , B 龄 , B2, B4), where B CLS Vector is the sentence feature vector corresponding to the target sentence. CLS The vector is the CLS vector, which is used to identify the characteristics of the sentence and serves as the semantic representation of the sentence.

[0086] In a specific embodiment provided in the present application, following the above example, after the sentence sequence (S01, S02...S50) is subjected to the feature extraction model, the CLS vector corresponding to each sentence is obtained to form a vector sequence (B CLS-01 , B CLS-02 ……B CLS-50), where B CLS-01 Represents the CLS vector of the 01th sentence.

[0087] Step 208: Input the sentence feature vector corresponding to each sentence into the feature classification model to obtain the sentence category corresponding to each sentence output by the feature classification model.

[0088] After obtaining the CLS vector corresponding to each sentence, each CLS vector can be input into the feature classification model together. The feature classification model is used to determine the category of each sentence according to the feature vector of each sentence.

[0089] In a specific embodiment provided in the present application, the feature classification model includes a feature classification model using a recurrent neural network;

[0090] The step of inputting the sentence feature vector corresponding to each sentence into the feature classification model to obtain the sentence category corresponding to each sentence output by the feature classification model includes:

[0091] The sentence feature vector corresponding to each sentence is input into a feature classification model using a recurrent neural network, and a sentence category corresponding to each sentence output by the feature classification model using a recurrent neural network is obtained.

[0092] In another specific implementation provided in the present application, the feature classification model includes a binary classification model.

[0093] The step of inputting the sentence feature vector corresponding to each sentence into the feature classification model using the recurrent neural network to obtain the sentence category corresponding to each sentence output by the feature classification model using the recurrent neural network includes:

[0094] The sentence feature vector corresponding to each sentence is input into a binary classification model using a recurrent neural network, and the first category or the second category corresponding to each sentence output by the binary classification model using the recurrent neural network is obtained.

[0095] In practical applications, the feature classification model needs to use a feature classification model that applies a recurrent neural network, such as an LSTM network, a BiLSTM network, an RNN network, etc. The feature classification model that applies a recurrent neural network can refer to the previous and next relationships between each vector. Each sentence feature vector (CLS vector) is used to represent the semantic features of each sentence. The type of sentence corresponding to each CSL vector is calculated by the binary classification function of the feature classification model at the output layer. Specifically, the sentence type is divided into the first category or the second category, where the first category is the starting sentence, which can be represented by 1; the second category is the middle sentence, which can be represented by 0. After determining the starting sentence and the middle sentence, the target text can be segmented according to the starting sentence and the middle sentence, and each segment includes a starting sentence and the middle sentence following the starting sentence.

[0096] In a specific embodiment provided in the present application, the vector sequence (B CLS-01 , B CLS-02 ……B CLS-50 ) is input into the feature classification model built with BiLSTM network. The features of each CLS vector are extracted and classified by BiLSTM. The classification of each CLS vector is judged and the sentence category corresponding to each CLS vector in the vector sequence is determined as (1 001 , 0 002 , …, 1 007 , …, 0 050 ). Among them, 1 001 Indicates that sentence 001 is the starting sentence, 0 002 Indicates that sentence 002 is the middle sentence..., 1 007 Indicates that sentence 001 is the starting sentence, ... 0 050 Indicates that sentence 050 is the middle sentence.

[0097] Step 210: Segment the target text according to the sentence category corresponding to each sentence to obtain a paragraph set.

[0098] After determining the sentence category corresponding to each sentence, the target text may be segmented according to the sentence category of each sentence. Specifically, the target text is segmented according to the sentence category corresponding to each sentence to obtain a paragraph set, including:

[0099] The target text is segmented according to the first category or the second category corresponding to each sentence to obtain a paragraph set.

[0100] In practical applications, the target text can be segmented by the sentence category corresponding to each sentence. Since the sentence categories are divided into the first category and the second category, where the first category is the starting sentence and the second category is the middle sentence, the target text can be segmented according to the sentence category and sentence sequence corresponding to each sentence to obtain multiple paragraph sets. The specific segmentation rule is: the first sentence of each paragraph is the first category, and the other sentences are the second category. That is, each sentence of the first category is taken as the first sentence of a paragraph, and the middle sentence between two sentences of the first category is taken as the sentence of the paragraph where the previous sentence of the first category is located.

[0101] The target text is segmented according to the first category or the second category corresponding to each sentence to obtain a paragraph set, specifically including:

[0102] Sort each statement according to its corresponding position identifier;

[0103] The target text is segmented according to the sentences corresponding to the first category to obtain a plurality of paragraphs, wherein the first sentence of each paragraph is of the first category, and the other sentences except the first sentence are of the second category.

[0104] Each statement to be processed is obtained from the aforementioned statement database, and each statement to be processed is segmented according to the category corresponding to each statement to be processed. Specifically, each statement in the first category is used as the first sentence of each paragraph, and the statements in the second category of the two first categories are used as the statements of the previous paragraph. For example, there are 10 statements in total, and the vector sequence is (1 001 , 0 002 , 0 003 , 1 004 , 0 005 , 1 006 , 0 007 , 0 008 , 1 009 , 0 010 ), where the categories corresponding to the 1st, 4th, 6th, and 9th sentences are the first category, and the target text is divided into 4 paragraphs, where the 1st paragraph includes the 1st to 3rd sentences, the 2nd paragraph includes the 4th to 5th sentences, the 3rd paragraph includes the 6th to 8th sentences, and the 4th paragraph includes the 9th to 10th sentences.

[0105] In a specific embodiment provided in the present application, the vector sequence (B CLS-01 , B CLS-02 ……B CLS-50 ) is input into the feature classification model built with BiLSTM network. The features of each CLS vector are extracted and classified by BiLSTM. The classification of each CLS vector is judged and the sentence category corresponding to each CLS vector in the vector sequence is determined as (1 001 , 0 002 , …, 1 007 , …, 0 050). Among them, the 1st, 7th, 18th and 23rd sentences are of the first category (i.e., the starting sentences), and the other sentences are of the second category (i.e., the middle sentences). Therefore, it can be determined that the 1st to 6th sentences are the first paragraph, the 7th to 17th sentences are the second paragraph, the 18th to 22nd sentences are the third paragraph, and the 23rd to 50th sentences are the fourth paragraph. At the same time, the semantics corresponding to the CLS vector corresponding to the 1st sentence is represented as "basic information", the semantics corresponding to the CLS vector corresponding to the 7th sentence is represented as "skill information", the semantics corresponding to the CLS vector corresponding to the 18th sentence is represented as "education experience", and the semantics corresponding to the CLS vector corresponding to the 23rd sentence is represented as "work experience". Therefore, it can be determined that the first paragraph is the basic information in the target resume, the second paragraph is the skill information in the target resume, the third paragraph is the education experience in the target resume, and the fourth paragraph is the work experience in the target resume.

[0106] The text segmentation method provided by the embodiment of the present application includes receiving a segmentation instruction for a target text; performing sentence processing on the target text in response to the segmentation instruction to generate a sentence sequence; inputting each sentence in the sentence sequence into a sentence feature extraction model in turn to obtain a sentence feature vector corresponding to each sentence output by the sentence feature extraction model, wherein each sentence feature vector represents the feature information of the corresponding sentence; inputting the sentence feature vector corresponding to each sentence into a feature classification model to obtain a sentence category corresponding to each sentence output by the feature classification model; performing segmentation processing on the target text according to the sentence category corresponding to each sentence to obtain a paragraph set. Through the text segmentation method provided by the present application, the sentence feature extraction model can obtain the feature information of each sentence, and then classify the feature information of each sentence according to the feature classification model, and complete the text segmentation task without intercepting or compressing the text, and can effectively refer to the contextual semantic information of the text to achieve a better text segmentation effect and enhance the user experience.

[0107] Combine the following Figure 3 and Figure 4 Taking the text segmentation method provided in the present application for segmenting target long text data as an example, the text segmentation method is further explained. Figure 3 A schematic diagram of the architecture of the text segmentation method provided in an embodiment of the present application is shown. In the embodiment of the present application, a model architecture combining BERT+BiLSTM is adopted, wherein the sentence feature extraction model adopts the BERT model and the feature classification model adopts the BiLSTM model.

[0108] Figure 4 A processing flow chart of a text segmentation method for segmenting target long text data provided in an embodiment of the present application is shown, including steps 402 to 410.

[0109] Step 402: Receive a segmentation instruction for target long text data.

[0110] Step 404: Sentence processing is performed on the target long text data according to preset sentence identifiers to generate a sentence sequence consisting of multiple sentences.

[0111] Step 406: Input each sentence in the sentence sequence into the BERT model in sequence to obtain the CLS vector corresponding to each sentence output by the BERT model.

[0112] Step 408: Input the CLS vector corresponding to each sentence into the BiLSTM model for binary classification processing to obtain the sentence category corresponding to each sentence output by the BiLSTM model.

[0113] Step 410: Segment the target long text data according to the sentence category corresponding to each clause to obtain a paragraph set.

[0114] The text segmentation method provided by the embodiment of the present application includes receiving a segmentation instruction for a target text; performing sentence processing on the target text in response to the segmentation instruction to generate a sentence sequence; inputting each sentence in the sentence sequence into a sentence feature extraction model in turn to obtain a sentence feature vector corresponding to each sentence output by the sentence feature extraction model, wherein each sentence feature vector represents the feature information of the corresponding sentence; inputting the sentence feature vector corresponding to each sentence into a feature classification model to obtain a sentence category corresponding to each sentence output by the feature classification model; performing segmentation processing on the target text according to the sentence category corresponding to each sentence to obtain a paragraph set. Through the text segmentation method provided by the present application, the sentence feature extraction model can obtain the feature information of each sentence, and then classify the feature information of each sentence according to the feature classification model, and complete the text segmentation task without intercepting or compressing the text, and can effectively refer to the contextual semantic information of the text to achieve a better text segmentation effect and enhance the user experience.

[0115] Corresponding to the above text segmentation method embodiment, the present application also provides a text segmentation device embodiment, Figure 5 FIG. 2 shows a schematic diagram of the structure of a text segmentation device according to an embodiment of the present application. Figure 5 As shown, the device comprises:

[0116] A receiving module 502 is configured to receive a segmentation instruction for a target text;

[0117] The sentence segmentation module 504 is configured to perform sentence segmentation processing on the target text in response to the segmentation instruction to generate a sentence sequence;

[0118] The feature extraction module 506 is configured to input each sentence in the sentence sequence into the sentence feature extraction model in sequence, and obtain a sentence feature vector corresponding to each sentence output by the sentence feature extraction model, wherein each sentence feature vector represents feature information of the corresponding sentence;

[0119] The classification module 508 is configured to input the sentence feature vector corresponding to each sentence into the feature classification model to obtain the sentence category corresponding to each sentence output by the feature classification model;

[0120] The segmentation module 510 is configured to segment the target text according to the sentence category corresponding to each sentence to obtain a paragraph set.

[0121] Optionally, the sentence segmentation module 504 is further configured to:

[0122] Responding to the segmentation instruction, obtaining a preset sentence identifier;

[0123] Sentence processing is performed on the target text according to the sentence identifier to obtain multiple sentences, and a position identifier of each sentence is recorded;

[0124] A statement sequence corresponding to the multiple statements is obtained according to the position identifier of each statement.

[0125] Optionally, the sentence segmentation module 504 is further configured to:

[0126] Get the preset statement threshold;

[0127] The target text is sentence-processed according to the sentence identifier and the sentence threshold to obtain a plurality of sentences, wherein the number of sentences obtained by the sentence processing is less than or equal to the sentence threshold.

[0128] Optionally, the sentence feature extraction model includes an embedding layer and an encoder;

[0129] The feature extraction module 506 is further configured to:

[0130] determining a target sentence in the sentence sequence;

[0131] Performing word segmentation processing on the target sentence to obtain a word unit set corresponding to the target sentence;

[0132] Inputting the word unit set into the embedding layer for embedding processing to obtain a word unit vector set corresponding to the word unit set;

[0133] The word unit vector set is input into the encoder for encoding processing to obtain a sentence feature vector corresponding to the target sentence.

[0134] Optionally, the feature classification model includes a feature classification model applying a recurrent neural network;

[0135] The classification module 508 is further configured to:

[0136] The sentence feature vector corresponding to each sentence is input into a feature classification model using a recurrent neural network, and a sentence category corresponding to each sentence output by the feature classification model using a recurrent neural network is obtained.

[0137] Optionally, the feature classification model includes a binary classification model;

[0138] The classification module 508 is further configured to:

[0139] The sentence feature vector corresponding to each sentence is input into a binary classification model using a recurrent neural network, and the first category or the second category corresponding to each sentence output by the binary classification model using the recurrent neural network is obtained.

[0140] Optionally, the segmentation module 510 is further configured to:

[0141] The target text is segmented according to the first category or the second category corresponding to each sentence to obtain a paragraph set.

[0142] Optionally, each statement corresponds to a position marker;

[0143] The segmentation module 510 is further configured to:

[0144] Sort each statement according to its corresponding position identifier;

[0145] The target text is segmented according to the sentences corresponding to the first category to obtain a plurality of paragraphs, wherein the first sentence of each paragraph is of the first category, and the other sentences except the first sentence are of the second category.

[0146] The text segmentation device provided by the embodiment of the present application includes receiving a segmentation instruction for a target text; performing sentence processing on the target text in response to the segmentation instruction to generate a sentence sequence; inputting each sentence in the sentence sequence into a sentence feature extraction model in turn to obtain a sentence feature vector corresponding to each sentence output by the sentence feature extraction model, wherein each sentence feature vector represents feature information of the corresponding sentence; inputting the sentence feature vector corresponding to each sentence into a feature classification model to obtain a sentence category corresponding to each sentence output by the feature classification model; performing segmentation processing on the target text according to the sentence category corresponding to each sentence to obtain a paragraph set. Through the text segmentation device provided by the present application, the sentence feature extraction model can obtain the feature information of each sentence, and then classify the feature information of each sentence according to the feature classification model, completing the text segmentation task without truncation or compression of the text, and can effectively refer to the contextual semantic information of the text to achieve a better text segmentation effect and enhance the user experience.

[0147] The above is a schematic scheme of a text segmentation device of this embodiment. It should be noted that the technical scheme of the text segmentation device and the technical scheme of the above text segmentation method belong to the same concept, and the details of the technical scheme of the text segmentation device that are not described in detail can be found in the description of the technical scheme of the above text segmentation method.

[0148] It should be noted that the components in the device claim should be understood as the functional modules that must be established to implement the steps of the program flow or the steps of the method, and the functional modules are not actually divided or separated. The device claim defined by such a group of functional modules should be understood as the functional module architecture of the solution mainly implemented by the computer program recorded in the specification, and should not be understood as the physical device that mainly implements the solution by hardware.

[0149] See also Figure 6 and Figure 7 Taking the application of the text segmentation method for resume documents as an example, the text segmentation method of this application is further explained, see Figure 6 , Figure 6 The architecture diagram of the resume processing system provided in the embodiment of the present application is shown as follows: Figure 6As shown, the resume processing system 600 includes a resume document database 602, a text extraction system 604, a text segmentation system 606, and an information extraction system 608. Among them, the resume document database 602 stores multiple resume documents, and the resume documents are divided into Chinese and foreign resumes, and have tables, images, headers and footers and other contents. The text extraction system 604 is responsible for extracting text information from the resume document and converting the resume document into corresponding resume text. The text segmentation system 606 is responsible for segmenting each resume text, obtaining multiple paragraph sets, and then transmitting each paragraph set to the information extraction system. The information extraction system 608 extracts information from the paragraph set.

[0150] See also Figure 7 , Figure 7 A processing flow chart of a text segmentation method for segmenting a resume document provided in an embodiment of the present application is shown, including steps 702 to 710.

[0151] Step 702: Receive a resume document and extract the resume text in the resume document.

[0152] In the embodiment provided in this application, the resume document is divided into several parts, such as basic information, education experience, work experience, skill description, personal summary, etc. The text content in the resume document can be extracted by text extraction and / or OCR recognition, and the extracted text content is sequentially connected and saved.

[0153] Step 704: Sentence processing is performed on the resume text according to preset sentence identifiers to generate a sentence sequence consisting of multiple sentences.

[0154] The preset sentence identifier is an identifier used to indicate that a sentence has been fully expressed. It can include semicolons, periods, exclamation marks, line breaks ( / n), question marks, and the full-width and half-width formats of the above symbols. It does not include commas, commas, colons, brackets, ellipsis, book title marks, spaces, etc. The preset sentence identifier can be directly saved in the relevant code for sentence segmentation in the text segmentation system; it can also be placed in the sentence identifier database of the text segmentation system. When the text sentence needs to be processed, the preset sentence identifier is read from the sentence identifier database.

[0155] In the embodiment provided by the present application, the resume text is matched one by one with the preset sentence identifiers, and according to the preset semicolon, period, exclamation mark, line break, question mark and other sentence identifiers, the resume text is disconnected as long as the sentence identifier is encountered. The number of bytes of the sentence after disconnection may be greater than 510 characters, may be less than 510 characters, or even very small. Since 510 characters is the maximum processable sequence length of the BERT model, 510 characters are used as the maximum length of characters retained for a sentence. When the sentence length is less than 510 characters, the sentence is stored in the database of pending sentences for subsequent processing; when the sentence length is greater than 510 characters, only the first 510 characters of the sentence are retained, the characters after exceeding 510 characters are deleted, and the sentence is stored in the database of pending sentences. Since the method provided by the present application is to identify the sentence type of each sentence, the sentence type of the sentence can be identified by the first 510 characters, so the part exceeding 510 characters can be discarded. Therefore, a resume text is divided into n sentences to be processed, and n sentences to be processed constitute a sentence sequence. The length of each sentence to be processed is less than 510 characters, among which 510 characters is the maximum sequence length that the BERT model can process.

[0156] In actual applications, the number of sentences is related to the sentence threshold set by the subsequent feature classification model. The sentence threshold is saved in the relevant code of the text segmentation system. The number of sentences n is less than or equal to the sentence threshold set by the feature classification model. When the number of sentences is determined, the number of sentences is adjusted according to the sentence threshold. For example, if the sentence threshold is 64 and the number of sentences is 90, sentences 1-64 are taken as the first batch and sentences 65-90 are taken as the second batch. In the second batch, if there are less than 64 sentences, it is necessary to add 0 to supplement the sentences of the second batch to 64.

[0157] Step 706: Input each sentence in the sentence sequence into the sentence feature extraction model in turn, and obtain the CLS vector corresponding to each sentence output by the sentence feature extraction model.

[0158] In the embodiment provided in the present application, n sentences in a sentence sequence are sequentially input into a sentence feature extraction model based on a BERT model for processing. The sentence feature extraction model adds a "CLS" identifier before each sentence and converts the sentence into a sentence vector.

[0159] Each sentence vector is input into the sentence feature extraction model. The sentence feature extraction model extracts the position feature, keyword feature, semantic feature, sentence association feature, etc. of each sentence vector. Each sentence in the sentence sequence shares the model parameters of the sentence feature extraction model. The CSL vector "CLS1, CLS2, ...CLSn ”.

[0160] Step 708: Input the CLS vector corresponding to each sentence into the feature classification model for binary classification processing to obtain the sentence category corresponding to each sentence output by the feature classification model.

[0161] In the embodiment provided in the present application, the feature classification model adopts a BiLSTM model, and the full-text features of the resume text are extracted by the BiLSTM model, referring to the contextual information of the resume text. Each resume document is a Batch (processing batch), and the BiLSTM model performs feature extraction and feature classification on each Batch. Batch_Size represents the number of batches processed (i.e., the sentence threshold). Batch_Size is a hyperparameter of the model. When the size of Batch_Size is set to 64, after the resume document is processed by sentence segmentation, n of the obtained n sentences is less than or equal to 64. Taking the dimension of the CSL vector as 768 bits and n=64 as an example, the size of the feature vector of the resume document is 64*768.

[0162] The feature classification model performs feature classification based on the CLS vector of each sentence and outputs the sentence category corresponding to each sentence. The classification task of the feature classification model is binary classification, which determines whether the sentence is a starting sentence. If the sentence is a starting sentence, its corresponding sentence category is 1, and if the sentence is not a starting sentence, its corresponding sentence category is 0. The feature classification model is trained to identify whether a sentence is a starting sentence, and is used to determine whether the sentence is the first sentence of a paragraph, and then the text can be segmented by the starting sentence. Taking the resume text as an example, the name, work experience, education experience, professional skills, qualification certificates, etc. can all be used as starting sentences to express that the information after the sentence is related to the sentence.

[0163] Step 710: Segment the resume text according to the sentence category corresponding to each sentence to obtain a paragraph set.

[0164] For example, the resume text is "

[0165] Resume

[0166] Name: Zhang San Male 22 years old (April 1999) 1 year of work experience Bachelor degree

[0167] Current residence: Beijing | Hukou: Beijing | Operation and maintenance engineer

[0168] Mobile phone: 185********

[0169] E-mail: *******@qq.com

[0170] Job intention

[0171] Expected working area: Beijing

[0172] Expected monthly salary: 10,000-20,000 yuan / month

[0173] Desired job type: Full-time

[0174] Desired occupation: Operation and maintenance engineer

[0175] Expected industry: Internet

[0176] Self-evaluation

[0177] I graduated from the School of Computer Science at Tsinghua University with excellent academic performance and a responsible attitude.

[0178] Work Experience

[0179] 2021.08-PresentXXX Technology Co., Ltd.

[0180] Operation and maintenance engineer 10001-15000 yuan / month

[0181] Job Description: 1. Mainly responsible for operation and maintenance related work

[0182] 2. Be fully responsible for operation and maintenance related work

[0183] Project Experience

[0184] 2021.10-Present Search Algorithm

[0185] Project Description: Search Algorithms.

[0186] Responsible for handling search algorithms in the project.

[0187] 2021.08-2021.10 Intelligent Dialogue

[0188] Project Description: Intelligent Dialogue.

[0189] Responsible for natural language processing in the project.

[0190] Education

[0191] 2016.09-2020.07 Tsinghua University, Bachelor of Computer Science

[0192] Certificate

[0193] 2018.08Computer Level 2

[0194] 2019.12 College English

[0195] Professional skills

[0196] Proficient in using relevant computer skills. "After the resume text is processed by sentence segmentation, multiple sentences are obtained, see Table 1 below:

[0197]

[0198]

[0199] The above-mentioned sentences are input into the sentence feature extraction model for processing to obtain the CLS vector corresponding to each sentence, namely “CLS1, CLS2, …CLS 32 Each CLS vector is input into the feature classification model for binary classification. The binary classification function in the feature classification model calculates the probability of the CLS vector being the starting sentence based on each CLS vector. When the probability is greater than a preset threshold, the CLS vector is the starting sentence (1), otherwise the CLS vector is the middle sentence (0). The binary classification function can be a sigmoid function, a softmax function, etc. Taking the sigmoid function as an example, see the following formula 1:

[0200]

[0201] Where x is the CLS vector and e is the Euler number.

[0202] Based on this, the statement category corresponding to each CLS vector is shown in the following Table 2:

[0203] <![CDATA[CLS1]]> 1 <![CDATA[CLS2]]> 0 <![CDATA[CLS3]]> 0 <![CDATA[CLS4]]> 0 <![CDATA[CLS5]]> 0 <![CDATA[CLS6]]> 1 <![CDATA[CLS7]]> 0 <![CDATA[CLS8]]> 0 <![CDATA[CLS9]]> 0 <![CDATA[CLS 10 ]]> 0 <![CDATA[CLS 11 ]]> 0 <![CDATA[CLS 12 ]]> 1 <![CDATA[CLS 13 ]]> 0 <![CDATA[CLS 14 ]]> 1 <![CDATA[CLS 15 ]]> 0 <![CDATA[CLS 16 ]]> 0 <![CDATA[CLS 17 ]]> 0 <![CDATA[CLS 18 ]]> 0 <![CDATA[CLS 19 ]]> 1 <![CDATA[CLS 20 ]]> 0 <![CDATA[CLS 21 ]]> 0 <![CDATA[CLS 22 ]]> 0 <![CDATA[CLS 23 ]]> 0 <![CDATA[CLS 24 ]]> 0 <![CDATA[CLS 25 ]]> 0 <![CDATA[CLS 26 ]]> 1 <![CDATA[CLS 27 ]]> 0 <![CDATA[CLS 28 ]]> 1 <![CDATA[CLS 29 ]]> 0 <![CDATA[CLS 30 ]]> 0 <![CDATA[CLS 31 ]]> 1 <![CDATA[CLS 32 ]]> 0

[0204] 1 is the first sentence of a paragraph, 0 is the middle sentence of a paragraph, then 1-5 is a paragraph, 6-11 is a paragraph, 12-13 is a paragraph, 14-18 is a paragraph, 19-25 is a paragraph, 26-27 is a paragraph, 28-30 is a paragraph, 31-32 is a paragraph. There are 8 paragraphs in total. That is:

[0205] 1. Personal resume

[0206] Name: Zhang San Male 22 years old (April 1999) 1 year of work experience Bachelor degree

[0207] Current residence: Beijing | Hukou: Beijing | Operation and maintenance engineer

[0208] Mobile phone: 18573123456

[0209] E-mail: 458888888@qq.com”

[0210] 2. "Job seeking intention"

[0211] Expected working area: Beijing

[0212] Expected monthly salary: 10,000-20,000 yuan / month

[0213] Desired job type: Full-time

[0214] Desired occupation: Operation and maintenance engineer

[0215] Expected industry: Internet

[0216] 3. “Self-evaluation

[0217] I graduated from the School of Computer Science at Tsinghua University with excellent academic performance and a responsible attitude. "

[0218] 4. Work experience

[0219] 2021.08-PresentXXX Technology Co., Ltd.

[0220] Operation and maintenance engineer 10001-15000 yuan / month

[0221] Job Description: 1. Mainly responsible for operation and maintenance related work

[0222] 2. Be fully responsible for operation and maintenance related work"

[0223] 5. Project Experience

[0224] 2021.10-Present Search Algorithm

[0225] Project Description: Search Algorithms.

[0226] Responsible for handling search algorithms in the project.

[0227] 2021.08-2021.10 Intelligent Dialogue

[0228] Project Description: Intelligent Dialogue.

[0229] Responsible for natural language processing in the project. ”

[0230] 6. Educational experience

[0231] 2016.09-2020.07 Tsinghua University, Bachelor of Computer Science

[0232] 7. Certificate

[0233] 2018.08Computer Level 2

[0234] 2019.12 College English"

[0235] 8. “Professional skills

[0236] Proficiency in the use of relevant computer skills.”

[0237] The divided paragraph set is saved in the to-be-processed paragraph database for subsequent resume information processing, and can be input into the information extraction module of the resume processing system to extract the information in each paragraph.

[0238] The text segmentation method provided by this application is based on sentence segmentation. The features of each sentence are extracted through the BERT model, and the features of the sentence are extracted and classified through the BiLSTM model. The text segmentation task is completed according to the classification results. There is no need to screen the sentences of the text, nor is there any need to use the BERT model to extract features and then use the fully connected layer for classification. This is an end-to-end text segmentation method, which achieves better text segmentation effect and improves the user experience.

[0239] In one embodiment of the present application, a computing device is also provided, including a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor implements the steps of the text segmentation method when executing the computer instructions.

[0240] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned text segmentation method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above-mentioned text segmentation method.

[0241] An embodiment of the present application further provides a computer-readable storage medium storing computer instructions, which implement the steps of the text segmentation method as described above when executed by a processor.

[0242] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned text segmentation method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned text segmentation method.

[0243] An embodiment of the present application discloses a chip storing computer instructions, which implement the steps of the text segmentation method as described above when executed by a processor.

[0244] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0245] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0246] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0247] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0248] The preferred embodiments of the present application disclosed above are only used to help explain the present application. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the present application. The present application selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can understand and use the present application well. The present application is only limited by the claims and their full scope and equivalents.

Claims

1. A text segmentation method, characterized in that: include: receiving a segmentation instruction for a target text; In response to the segmentation instruction, the target text is segmented to generate a sentence sequence; Inputting each sentence in the sentence sequence into a sentence feature extraction model in sequence, obtaining a sentence feature vector corresponding to each sentence output by the sentence feature extraction model, wherein each sentence feature vector represents feature information of the corresponding sentence; Inputting a sentence feature vector corresponding to each sentence into a feature classification model to obtain a sentence category corresponding to each sentence output by the feature classification model, wherein the feature classification model includes a feature classification model using a recurrent neural network, and the sentence category includes an initial sentence or an intermediate sentence; The target text is segmented according to the sentence category corresponding to each sentence to obtain a paragraph set.

2. The text segmentation method according to claim 1, characterized in that: In response to the segmentation instruction, the target text is segmented to generate a sentence sequence, including: Responding to the segmentation instruction, obtaining a preset sentence identifier; Sentence processing is performed on the target text according to the sentence identifier to obtain multiple sentences, and a position identifier of each sentence is recorded; A statement sequence corresponding to the multiple statements is obtained according to the position identifier of each statement.

3. The text segmentation method according to claim 2, characterized in that: The sentence processing of the target text according to the sentence identifier to obtain a plurality of sentences includes: Get the preset statement threshold; The target text is sentence-processed according to the sentence identifier and the sentence threshold to obtain a plurality of sentences, wherein the number of sentences obtained by the sentence processing is less than or equal to the sentence threshold.

4. The text segmentation method according to claim 1, characterized in that: The sentence feature extraction model includes an embedding layer and an encoder; The step of sequentially inputting each sentence in the sentence sequence into a sentence feature extraction model to obtain a sentence feature vector corresponding to each sentence output by the sentence feature extraction model comprises: determining a target sentence in the sentence sequence; Performing word segmentation processing on the target sentence to obtain a word unit set corresponding to the target sentence; Inputting the word unit set into the embedding layer for embedding processing to obtain a word unit vector set corresponding to the word unit set; The word unit vector set is input into the encoder for encoding processing to obtain a sentence feature vector corresponding to the target sentence.

5. The text segmentation method according to claim 1, characterized in that: The step of inputting the sentence feature vector corresponding to each sentence into the feature classification model to obtain the sentence category corresponding to each sentence output by the feature classification model includes: The sentence feature vector corresponding to each sentence is input into a feature classification model using a recurrent neural network to obtain a sentence category corresponding to each sentence output by the feature classification model using the recurrent neural network.

6. The text segmentation method according to claim 5, characterized in that: The feature classification model includes a binary classification model; The step of inputting the sentence feature vector corresponding to each sentence into the feature classification model using the recurrent neural network to obtain the sentence category corresponding to each sentence output by the feature classification model using the recurrent neural network includes: The sentence feature vector corresponding to each sentence is input into a binary classification model using a recurrent neural network, and the first category or the second category corresponding to each sentence output by the binary classification model using the recurrent neural network is obtained.

7. The text segmentation method according to claim 6, characterized in that: The target text is segmented according to the sentence category corresponding to each sentence to obtain a paragraph set, including: The target text is segmented according to the first category or the second category corresponding to each sentence to obtain a paragraph set.

8. The text segmentation method according to claim 7, characterized in that: Each statement corresponds to a position marker; The target text is segmented according to the first category or the second category corresponding to each sentence to obtain a paragraph set, including: Sort each statement according to its corresponding position identifier; The target text is segmented according to the sentences corresponding to the first category to obtain a plurality of paragraphs, wherein the first sentence of each paragraph is of the first category, and the other sentences except the first sentence are of the second category.

9. A text segmentation device, characterized in that: include: A receiving module is configured to receive a segmentation instruction for a target text; A sentence segmentation module is configured to perform sentence segmentation processing on the target text in response to the segmentation instruction to generate a sentence sequence; A feature extraction module is configured to input each sentence in the sentence sequence into a sentence feature extraction model in sequence, and obtain a sentence feature vector corresponding to each sentence output by the sentence feature extraction model, wherein each sentence feature vector represents feature information of the corresponding sentence; A classification module is configured to input a sentence feature vector corresponding to each sentence into a feature classification model to obtain a sentence category corresponding to each sentence output by the feature classification model, wherein the feature classification model includes a feature classification model using a recurrent neural network, and the sentence category includes an initial sentence or an intermediate sentence; The segmentation module is configured to segment the target text according to the sentence category corresponding to each sentence to obtain a paragraph set.

10. A computing device comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, characterized in that: When the processor executes the computer instructions, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium storing computer instructions, characterized in that: When the computer instruction is executed by a processor, the steps of the method described in any one of claims 1 to 8 are implemented.

Images