Method, device, equipment and storage medium for extracting target text fragments

Abstract

The present application discloses a method, device, equipment and storage medium for extracting target text fragments, the method comprising: segmenting the target text to obtain a word sequence vector; obtaining a word weight vector based on the word sequence vector and the text category vector obtained through training; obtaining N target words with weights higher than a threshold value in each text category according to the weight of each word in each text category; obtaining target text fragments belonging to each text category based on the N target words. It can be seen that the method utilizes the text category vector obtained through model training and the word sequence vector obtained through segmentation of the target text to obtain the weight of each word in each text category. Since the weight can characterize the importance of the word in each text category, the target text is categorized to extract the target text fragments, thereby improving the accuracy of the target text fragment extraction.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and storage medium for extracting a target text segment. Background Art

[0002] With the development of Internet technology, online public opinions (e.g., advertisements) are everywhere. However, since it is difficult to fully cover the Internet supervision, there are a lot of false online public opinions that do not conform to the actual situation on the Internet, which will mislead the public. Since online public opinions are generally long, in order to efficiently identify them, it is necessary to identify the target text fragments containing important information.

[0003] The existing unsupervised solutions mainly use TextRank, TF-IDF and other unsupervised solutions, which can mine text fragments but cannot extract them by category.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of the present application is to provide a method, device, equipment and storage medium for extracting target text segments, aiming to solve the technical problem that the existing target text segments cannot be extracted by classification.

[0006] To achieve the above object, the present application provides a method for extracting a target text segment, the method comprising:

[0007] Segmenting the target text to obtain a word sequence vector; wherein the word sequence vector includes a plurality of words obtained by segmenting the target text;

[0008] Based on the word sequence vector and the text category vector obtained through training, a word weight vector is obtained; wherein the word weight vector includes the weight of each word in each text category, and the text category vector includes multiple text categories;

[0009] According to the weight of each word in each text category, obtain N target words whose weight in each text category is higher than the threshold, where N is a positive integer;

[0010] Based on the N target words, target text segments belonging to each text category are obtained.

[0011] Optionally, before the step of segmenting the target text to obtain a word sequence vector, the method further includes:

[0012] Acquire a sample text set including a plurality of sample texts and a text category set of the sample texts; wherein the text category set includes a plurality of the text categories;

[0013] Based on the sample text set and the text category set, obtaining a text weight vector representing the importance of each sample text in each text category;

[0014] Based on the text weight vector, obtaining a category vector of each of the text categories on each of the sample texts;

[0015] Based on the text weight vector and the category vector, the text category vector is obtained through model training.

[0016] Optionally, the step of obtaining a text weight vector characterizing the importance of each sample text in each text category based on the sample text set and the text category set specifically includes:

[0017] Based on the text category set, obtaining an initial text category vector;

[0018] Perform segmentation processing on each of the sample texts to obtain a sample word sequence vector of each of the sample texts;

[0019] The text weight vector of each of the sample texts is obtained based on the initial text category vector and the sample word sequence vector.

[0020] Optionally, the sample word sequence vector includes a plurality of sample words obtained by segmenting the target text; and the step of obtaining the text weight vector of each sample text based on the initial text category vector and the sample word sequence vector specifically includes:

[0021] Based on the initial text category vector and the sample word sequence vector, obtaining a sample word weight vector for each sample word in each text category;

[0022] Normalizing the sample word weight vector of each sample word in each text category to obtain a processed sample word weight vector;

[0023] Multiplying the sample word weight vector by the processed sample word weight vector element by element to obtain a plurality of element-by-element product results;

[0024] A sum operation is performed on multiple element product results to obtain the text weight vector of each sample text.

[0025] Optionally, the step of obtaining the text category vector through model training based on the text weight vector and the category vector specifically includes:

[0026] Based on the text weight vector and the category vector, obtaining a sample text vector;

[0027] The sample text vector is input into the model to be trained to obtain the text category vector.

[0028] Optionally, the step of segmenting the target text to obtain a word sequence vector specifically includes:

[0029] The target text is input into the trained language representation model to obtain the word sequence vector.

[0030] Optionally, the step of obtaining a word weight vector based on the word sequence vector and the text category vector obtained through training specifically includes:

[0031] The word sequence vector is multiplied by the text category vector obtained through training to obtain the word weight vector.

[0032] Optionally, the step of obtaining target text segments belonging to each text category based on the N target words specifically includes:

[0033] Determine whether there are M target words with continuous semantics among the N target words in the same category, where M is a positive integer and M is less than or equal to N;

[0034] If so, the M semantically continuous target words are concatenated to obtain the target text segment.

[0035] Optionally, the target text is a long text including news public opinion, and the target text segment includes fraud risk public opinion;

[0036] The step of segmenting the target text to obtain a word sequence vector specifically includes:

[0037] Segment the long text including news and public opinion to obtain word sequence vectors;

[0038] The step of obtaining target text segments belonging to each text category based on the N target words specifically includes:

[0039] Based on the N target words, target text segments including fraud risk public opinion belonging to each text category are obtained.

[0040] In addition, to achieve the above purpose, the present application also proposes a device for extracting a target text segment, the device comprising:

[0041] A text segmentation module, used for segmenting the target text to obtain a word sequence vector; wherein the word sequence vector includes a plurality of words obtained by segmenting the target text;

[0042] A weight obtaining module, used to obtain a word weight vector based on the word sequence vector and the text category vector obtained through training; wherein the word weight vector includes the weight of each word in each text category, and the text category vector includes multiple text categories;

[0043] A word acquisition module is used to acquire N target words whose weights are higher than a threshold in each text category according to the weights of each word in each text category, where N is a positive integer;

[0044] The segment acquisition module is used to obtain target text segments belonging to each text category based on the N target words.

[0045] In addition, to achieve the above objectives, the present application also provides an electronic device, which includes a processor, a memory, and a computer program stored in the memory, and the computer program implements the steps of the above method when executed by the processor.

[0046] In addition, to achieve the above objectives, the present application also provides a computer storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0047] In addition, to achieve the above objectives, the present application also provides a computer program product, including a computer program, which implements the steps of the above method when executed by a processor.

[0048] The beneficial effects that can be achieved by this application.

[0049] The present application proposes a method for extracting target text fragments, the method comprising: segmenting the target text to obtain a word sequence vector; wherein the word sequence vector includes a plurality of words obtained by segmenting the target text; based on the word sequence vector and the text category vector obtained by training, a word weight vector is obtained; wherein the word weight vector includes the weight of each word in each text category, and the text category vector includes a plurality of text categories; according to the weight of each word in each text category, N target words with a weight higher than a threshold in each text category are obtained, where N is a positive integer; based on the N target words, target text fragments belonging to each text category are obtained. It can be seen that the method uses the text category vector obtained by model training and the word sequence vector obtained by segmenting the target text to obtain the weight of each word in each text category. Since the weight can represent the importance of the word in each text category, the target text fragment belonging to each text category can be obtained according to the weight of each word in each text category, so that the target text is categorized and the target text fragment is extracted, and the accuracy of the target text fragment extraction is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A schematic diagram of the structure of an electronic device of a hardware operating environment involved in an embodiment of the present application;

[0051] Figure 2 A scene graph for implementing the method for extracting a target text segment according to an embodiment of the present application;

[0052] Figure 3 This is a flow chart of a method for extracting target text segments in an embodiment of the present application;

[0053] Figure 4 This is a flow chart of a method for obtaining a text category vector in an embodiment of the present application;

[0054] Figure 5 Schematic diagram of a device for extracting target text segments in an embodiment of the present application.

[0055] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0056] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of this application.

[0058] It should be noted that the descriptions of "first", "second", etc. in this application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in this field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0059] The main solution of the embodiment of the present application is: a method for extracting target text fragments, the method comprising: segmenting the target text to obtain a word sequence vector; wherein the word sequence vector includes multiple words obtained by segmenting the target text; based on the word sequence vector and the text category vector obtained by training, a word weight vector is obtained; wherein the word weight vector includes the weight of each word in each text category, and the text category vector includes multiple text categories; according to the weight of each word in each text category, N target words with weights higher than a threshold in each text category are obtained, where N is a positive integer; based on the N target words, target text fragments belonging to each text category are obtained.

[0060] In an embodiment of the present application, the target text may refer to a text that includes public opinion information that is not in line with the facts, such as false advertisements, deceptive advertisements, and speeches that are not in line with the facts. Generally speaking, there will be a large amount of text content on a web page, which may include some target public opinions. For example, in the field of automobile sales, there are many false advertisements. Since public opinion web page texts are generally long texts, and long texts have a large amount of information and many information dimensions, in order to obtain key information, it is necessary to accurately extract the target text fragments from the template text; for example, risk text fragments of automobile fraud. In addition, since fraudulent text fragments are often updated faster and use newer words, it is necessary to update the fraudulent text corpus more frequently. At this time, a method for automatically mining fraudulent text fragments is needed.

[0061] The existing methods for identifying target text fragments mainly use unsupervised schemes such as TextRank (text ranking) and TF-IDF (TermFrequency / Inverse Document Frequency). Since unsupervised learning performs clustering without labeled training samples, that is, modeling without constraints, although some text fragments can be mined, the target text fragments cannot be classified and extracted according to established categories.

[0062] Compared with the existing unsupervised solutions, the present application provides a solution that uses the text category vector obtained by model training and the word sequence vector obtained by segmenting the target text to obtain the weight of each word in each text category. Since the weight can characterize the importance of the word in each text category, the target text fragment belonging to each text category can be obtained according to the weight of each word in each text category. Therefore, the solution of the present application only uses the parameters of model training to obtain the target text fragment through vector calculation, and uses the text category vector including the category in the parameters to realize classified text extraction, rather than directly adopting the unsupervised solution, so as to realize the extraction of target text fragments by categorizing the target text, thereby improving the accuracy of the extraction of target text fragments.

[0063] Reference Figure 1 , Figure 1 A schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiment of the present application.

[0064] like Figure 1 As shown, the electronic device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (WIreless-FIdelity, WI-FI) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM) memory, or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0065] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0066] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a data storage module, a network communication module, a user interface module and an electronic program.

[0067] exist Figure 1In the electronic device shown, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the electronic device of the present invention can be set in the electronic device, and the electronic device calls the method program for extracting the target text fragment stored in the memory 1005 through the processor 1001, and executes the method for extracting the target text fragment provided in the embodiment of the present application.

[0068] Reference Figure 2 , is an implementation scenario diagram of the solution of this embodiment, which includes a terminal device 201, which is connected to a server 202, and the terminal device 201 can obtain the target text through the server 202.

[0069] Embodiment 1

[0070] Reference Figure 3 , an embodiment of the present application provides a method for extracting a target text segment, the method comprising:

[0071] S20, segmenting the target text to obtain a word sequence vector; wherein the word sequence vector includes a plurality of words obtained by segmenting the target text;

[0072] S40, obtaining a word weight vector based on the word sequence vector and the text category vector obtained through training; wherein the word weight vector includes the weight of each word in each text category, and the text category vector includes multiple text categories;

[0073] S60, according to the weight of each word in each text category, obtaining N target words whose weight in each text category is higher than a threshold, where N is a positive integer;

[0074] S80: Based on the N target words, obtain target text segments belonging to each text category.

[0075] It should be noted that in this embodiment, the target text is generally a long text, or a short text. The long text refers to a text with multiple segmented words (i.e., words with actual meanings). For a long text, there may be multiple semantics, and each or two semantics are represented by a short text or text fragment.

[0076] Correspondingly, the target text segment is the text containing key information in the target text, which may be composed of key words or may be composed of only key words.

[0077] Combine the following Figure 3 , the specific execution process of the method in this embodiment is described in detail.

[0078] First, execute S20 to segment the target text to obtain a word sequence vector.

[0079] In the specific implementation process, the word sequence vector includes multiple words (tokens) obtained by segmenting the target text. The word sequence vector is a vector sequence including multiple words. The word sequence vector can be regarded as a vector matrix. The format of the matrix is ​​expressed as Time Steps*emb_size, where emb_size represents the vector length of each token, and Time Steps is the time step, which represents the time step occupied by the token in this embodiment. Among them, time steps*emb size, referred to as TE, represents a word sequence vector of a target text in this embodiment.

[0080] As an optional implementation, segmenting the target text to obtain a word sequence vector may specifically include:

[0081] The target text is input into the trained language representation model to obtain the word sequence vector.

[0082] In the specific implementation process, the language representation model can be a BERT series model, and specifically, models such as Transformer and Google Bert can be used.

[0083] Among them, transformer is a language model, which abandons the traditional CNN and RNN in its structure, and the entire network structure is completely composed of the Attention mechanism. Due to the use of the Attention mechanism, the distance between any two positions in the word sequence is reduced to a constant. Secondly, since it is not a sequential structure similar to RNN, it has better parallelism and conforms to the existing GPU framework. Therefore, it can be used to segment text into words, and in this embodiment, the segmentation of the target text is achieved.

[0084] BERT, the full name of which is Bidirectional Encoder Representation from Transformers, is also a language model that trains pre-trained deep bidirectional representations by jointly adjusting the bidirectional Transformers in all layers. The input of this model can be a single text or a pair of texts (for example, [question, answer]). Therefore, using this model, taking the target text as input, a token sequence can be obtained.

[0085] Therefore, using these language representation models, the template text can be segmented into token sequences, and then the vector representation of each token of the text can be output.

[0086] Text segmentation can be achieved manually, that is, division based on the language knowledge learned by people, but its efficiency is not high. Since the language representation model is a mature model obtained through training, using the language representation model to achieve text segmentation is more efficient and has a higher segmentation accuracy.

[0087] Next, S40 is executed to obtain a word weight vector based on the word sequence vector and the text category vector obtained through training.

[0088] In a specific implementation process, the word weight vector includes the weight of each word in each text category, and the text category vector includes multiple text categories.

[0089] A text category vector is a vector that includes multiple text categories, such as {high risk, medium risk, low risk, no risk}. The text category vector is a model parameter obtained by training a multi-label classification model using samples. The text category vector can also be regarded as a vector matrix. Assuming there are N categories, and the length of the vector of each category is emb_size, a vector representation matrix of N*emb_size is formed. The values ​​in this matrix are all parameters obtained by model training. The format of this matrix is ​​(class size*embedding size, referred to as CE).

[0090] As an optional implementation, in order to obtain a text category vector, before the step of segmenting the target text to obtain a word sequence vector, the method of this embodiment further includes:

[0091] S101, obtaining a sample text set including a plurality of sample texts and a text category set of the sample texts; wherein the text category set includes a plurality of the text categories;

[0092] S102, based on the sample text set and the text category set, obtaining a text weight vector representing the importance of each sample text in each text category;

[0093] S103, based on the text weight vector, obtaining a category vector of each of the text categories on each of the sample texts;

[0094] S104. Obtain the text category vector through model training based on the text weight vector and the category vector.

[0095] In the specific implementation process, since it is based on sample text and the text type as a label, this embodiment obtains the text category vector through supervised model training as the basic data for target text extraction, thereby obtaining the target text through vector calculation. Among them, since the text category vector includes the category of the text, the classified text extraction can be finally realized through vector calculation, without the need to directly identify and obtain the unclassified target text through an unsupervised model.

[0096] See also Figure 4 In this embodiment, the method steps for obtaining the text category vector are as follows:

[0097] First, S101 is executed to obtain a sample text set including a plurality of sample texts and a text category set of the sample texts.

[0098] In the specific implementation process, the sample text set can be collected from a web page, generally including multiple long texts (i.e., sample texts). Specifically, the text category set LongClasses can be obtained by reading and analyzing each sample text in the sample text set and manually summarizing it, and the text category set contains multiple text categories. For example, the text category set LongClasses = {high risk, medium risk, low risk, no risk}. In this embodiment, the elements in the text category set can be continuously added by manual annotation and regular matching.

[0099] Next, S102 is performed to obtain a text weight vector representing the importance of each sample text in each text category based on the sample text set and the text category set.

[0100] In the specific implementation process, the text weight vector is used to characterize the importance of each sample text in each text category. The importance can be understood as the degree of matching between the sample text and a certain text category based on the semantics of the sample text. In other words, the higher the importance of a sample text in a certain text category (the larger the element value corresponding to the sample text in the text weight vector), the higher the matching degree between the sample text and the text category.

[0101] As an optional embodiment, the step of obtaining a text weight vector characterizing the importance of each sample text in each text category based on the sample text set and the text category set specifically includes:

[0102] Based on the text category set, obtaining an initial text category vector;

[0103] Perform segmentation processing on each of the sample texts to obtain a sample word sequence vector of each of the sample texts;

[0104] The text weight vector of each of the sample texts is obtained based on the initial text category vector and the sample word sequence vector.

[0105] In a specific implementation process, the initial text category vector is a text category vector initialized before using sample text to train a multi-label classification model.

[0106] Referring to the method of S20, each of the sample texts can be segmented to obtain a sample word sequence vector of each sample text. Since the model training is based on batch training, the size of the sample word sequence vectors of multiple sample texts is (batch size*time steps*embedding size, referred to as BTE), where batch size represents the number of samples selected for one training.

[0107] After obtaining the initial text category vector and the sample word sequence vector, the text weight vector of each sample text can be obtained based on the category attention mechanism.

[0108] As an optional embodiment, the sample word sequence vector includes a plurality of sample words obtained by segmenting the target text;

[0109] The step of obtaining the text weight vector of each of the sample texts based on the initial text category vector and the sample word sequence vector specifically includes:

[0110] Based on the initial text category vector and the sample word sequence vector, obtaining a sample word weight vector for each sample word in each text category;

[0111] Normalizing the sample word weight vector of each sample word in each text category to obtain a processed sample word weight vector;

[0112] Multiplying the sample word weight vector by the processed sample word weight vector element by element to obtain a plurality of element-by-element product results;

[0113] A sum operation is performed on multiple element product results to obtain the text weight vector of each sample text.

[0114] In a specific implementation process, the sample category weight vector on each time step can be first obtained by multiplying the initial text category vector and the sample word sequence vector, that is, the sample word weight vector of each sample word in each text category.

[0115] That is, BTC=matmul(BTE*CE0), which can be specifically expressed as BTE*CE0=BTC, where BTC represents the sample word sequence vector and CE0 represents the initial text category vector.

[0116] Then, the sample word weight vector of each sample word in each text category is normalized to obtain the processed sample word weight vector, that is, on the BTC multi-dimensional vector representation, a softmax operation is performed on the Tth to obtain the processed sample word weight vector BTC_softmax, that is, BTC_softmax=softmax(BTC*dim=1).

[0117] Then, the product of BTC_softmax and BTC is calculated by element-by-element multiplication in the vector, and then the product is summed up in the dim=1 dimension to obtain the BTC vector representation, that is, the text weight vector of each sample text. It represents the importance of each input text in each category, that is, BC=sum(BTC_softmax*BTC, dim=1)

[0118] Next, S103 is executed to obtain a category vector of each of the text categories on each of the sample texts based on the text weight vector.

[0119] In a specific implementation process, the category vector of each text category on each sample text may be obtained by multiplying the text weight vector with the sample word sequence vector.

[0120] That is, BCE=matmul(BTC_softmax.permute(0,2,1),BTE).

[0121] Next, S104 is executed to obtain the text category vector through model training based on the text weight vector and the category vector.

[0122] In the specific implementation process, the text category vector can be obtained by training a multi-label classification model.

[0123] As an optional embodiment, the step of obtaining the text category vector through model training based on the text weight vector and the category vector specifically includes:

[0124] Based on the text weight vector and the category vector, obtaining a sample text vector;

[0125] The sample text vector is input into the model to be trained to obtain the text category vector.

[0126] Specifically, the text weight vector and the category vector can be spliced ​​to obtain a sample text vector. Specifically, vectors with the same format can be spliced ​​into one vector, that is, the text weight vector and the category vector are connected end to end to form a sample text vector. Then, the sample text vector is input into the model to be trained to obtain the text category vector.

[0127] It should be noted that when the sample text vector is input into the model to be trained, when the training end condition is reached, a multi-label classification model can be trained and the model parameters - parameters of the text category vector can be obtained at the same time, thereby obtaining the text category vector.

[0128] At this point, S40 is completed.

[0129] After S40 is executed, S60 is executed next to obtain N target words with weights higher than a threshold in each text category according to the weight of each word in each text category, where N is a positive integer.

[0130] In the specific implementation process, each column can be sorted in reverse order according to the weight of each word in each text category, so as to obtain a token list in descending order of weight in each category. Then, the first N target words above the threshold are taken.

[0131] Specifically, after sorting in reverse order, the words that are ranked at the front have a greater weight and a higher degree of conformity with the category. Therefore, the first N target words that are above the threshold can be used as the basis of the target segment.

[0132] It should be noted that the threshold can be set according to industry experience or experiments. For example, if the threshold is 70%, it means that words with a weight greater than 70% are identified as target words. Since the threshold can be set and adjusted as needed, and the weight of the target words obtained according to the threshold judgment meets certain requirements, the accuracy of the corresponding extracted target text fragments for their respective categories can be guaranteed.

[0133] Next, S80 is executed to obtain target text segments belonging to each text category based on the N target words.

[0134] In a specific implementation process, the target text segment may be a single word or a segment consisting of multiple words.

[0135] In an optional embodiment, when the target text segment is a segment consisting of multiple words, the step of obtaining the target text segment belonging to each text category based on the N target words specifically includes:

[0136] Determine whether there are M target words with continuous semantics among the N target words in the same category, where M is a positive integer and M is less than or equal to N;

[0137] If so, the M semantically continuous target words are concatenated to obtain the target text segment.

[0138] In the specific implementation process, semantically continuous words can form a target text segment that can express a meaning. Since the weight of each word constituting the target text segment is greater than the threshold, the target text segment is also of high importance in the text type to which it belongs.

[0139] In one implementation scenario, the target text of this embodiment is a long text including news public opinion, and the target text segment includes fraud risk public opinion.

[0140] Specifically, when executing the method of this embodiment, it is necessary to implement:

[0141] Segmenting the long text including news public opinion to obtain a word sequence vector; wherein the word sequence vector includes a plurality of words obtained by segmenting the target text including fraud risk public opinion;

[0142] Based on the word sequence vector and the text category vector obtained through training, a word weight vector is obtained; wherein the word weight vector includes the weight of each word in each text category, and the text category vector includes multiple text categories;

[0143] According to the weight of each word in each text category, obtain N target words whose weight in each text category is higher than the threshold, where N is a positive integer;

[0144] Based on the N target words, target text segments including fraud risk public opinion belonging to each text category are obtained.

[0145] For example, in the "medium risk" category of the text category set LongClasses = {high risk, medium risk, low risk, no risk}, the target words extracted from the long text including automobile sales public opinion include: no limit, household registration, no limit, car model, no check, credit report. After combination, the target text fragments obtained may include: no limit on household registration, no limit on car model, no check on credit report.

[0146] It can be seen from this that for the scenario of extracting fraud risk public opinion, target text fragments including fraud risk public opinion can be extracted according to multiple categories, which is convenient for users to classify text fragments to different degrees, thereby accurately identifying fraud public opinion with different risk levels.

[0147] It should be understood that the above is only an example and does not constitute any limitation to the technical solution of the present application. Those skilled in the art can make settings based on needs in actual applications and no limitation is made here.

[0148] From the above description, it is not difficult to find that in the existing technology, since fraudulent text fragments are often updated quickly and use newer words, the use of unsupervised models for identification will lead to inaccurate extraction of target text fragments due to untimely model updates, and the existing unsupervised solutions cannot achieve classification extraction.

[0149] The method for extracting target text fragments provided in this embodiment uses the text category vector obtained by model training and the word sequence vector obtained by segmenting the target text to obtain the weight of each word in each text category. Since the weight can represent the importance of the word in each text category, the target text fragments belonging to each text category can be obtained according to the weight of each word in each text category. Compared with the prior art, the extraction result is not directly obtained based on the unsupervised model, but the parameters of the supervised model are adopted to realize the extraction of target text fragments by categorizing the target text, thereby improving the accuracy of the extraction of the target text fragments.

[0150] Embodiment 2

[0151] See also Figure 5 Based on the same inventive idea as that of the first embodiment, the embodiment of the present application further proposes a device for extracting a target text segment, the device comprising:

[0152] A text segmentation module, used for segmenting the target text to obtain a word sequence vector; wherein the word sequence vector includes a plurality of words obtained by segmenting the target text;

[0153] A weight obtaining module, used to obtain a word weight vector based on the word sequence vector and the text category vector obtained through training; wherein the word weight vector includes the weight of each word in each text category, and the text category vector includes multiple text categories;

[0154] A word acquisition module is used to acquire N target words whose weights are higher than a threshold in each text category according to the weights of each word in each text category, where N is a positive integer;

[0155] The segment acquisition module is used to obtain target text segments belonging to each text category based on the N target words.

[0156] As an optional embodiment, the device further includes:

[0157] A sample acquisition module, used for acquiring a sample text set including a plurality of sample texts and a text category set of the sample texts before the step of segmenting the target text to obtain a word sequence vector; wherein the text category set includes a plurality of the text categories;

[0158] A first obtaining module, configured to obtain a text weight vector representing the importance of each sample text in each text category based on the sample text set and the text category set;

[0159] A second obtaining module is used to obtain a category vector of each of the text categories on each of the sample texts based on the text weight vector;

[0160] The third obtaining module is used to obtain the text category vector through model training based on the text weight vector and the category vector.

[0161] As an optional embodiment, the first acquisition module is also used to obtain an initial text category vector based on the text category set; segment each of the sample texts to obtain a sample word sequence vector of each of the sample texts; and obtain the text weight vector of each of the sample texts based on the initial text category vector and the sample word sequence vector.

[0162] As an optional embodiment, the sample word sequence vector includes a plurality of sample words obtained by segmenting the target text;

[0163] The first acquisition module is also used to obtain a sample word weight vector for each sample word in each text category based on the initial text category vector and the sample word sequence vector; normalize the sample word weight vector for each sample word in each text category to obtain a processed sample word weight vector; multiply the sample word weight vector by the processed sample word weight vector element by element to obtain a plurality of element multiplication results; and perform a sum operation on the plurality of element multiplication results to obtain the text weight vector of each of the sample texts.

[0164] As an optional embodiment, the third obtaining module is further used to obtain a sample text vector based on the text weight vector and the category vector; and input the sample text vector into the model to be trained to obtain the text category vector.

[0165] As an optional embodiment, the text segmentation module is further used to input the target text into a trained language representation model to obtain the word sequence vector.

[0166] As an optional embodiment, the weight obtaining module is further used to multiply the word sequence vector with the text category vector obtained through training to obtain the word weight vector.

[0167] As an optional embodiment, the fragment acquisition module is also used to determine whether there are M semantically continuous target words among the N target words in the same category, where M is a positive integer and M is less than or equal to N; if so, the M semantically continuous target words are spliced ​​to obtain the target text fragment.

[0168] It should be noted that the device for extracting target text segments in this embodiment corresponds one-to-one to the method for extracting target text segments in the aforementioned embodiment 1. Therefore, its various implementation methods and the technical effects achieved can refer to the aforementioned embodiment 1 and will not be repeated here.

[0169] From the above description, it is not difficult to find that in the existing technology, since fraudulent text fragments are often updated quickly and use newer words, the use of unsupervised models for identification will lead to inaccurate extraction of target text fragments due to untimely model updates, and the existing unsupervised solutions cannot achieve classification extraction.

[0170] The device for extracting target text fragments provided in this embodiment uses the text category vector obtained by model training and the word sequence vector obtained by segmenting the target text to obtain the weight of each word in each text category. Since the weight can represent the importance of the word in each text category, the target text fragments belonging to each text category can be obtained according to the weight of each word in each text category. Compared with the prior art, the extraction result is not directly obtained based on the unsupervised model, but the parameters of the supervised model are adopted to realize the extraction of target text fragments by categorizing the target text, thereby improving the accuracy of the extraction of the target text fragments.

[0171] In addition, in one embodiment, an electronic device is provided, the device comprising a processor, a memory, and a computer program stored in the memory, wherein the computer program implements the steps of the method in the aforementioned embodiment 1 when executed by the processor.

[0172] In addition, in one embodiment, the present application further provides a computer storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method in the aforementioned embodiment 1 are implemented.

[0173] In some embodiments, the computer readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or various devices including one or any combination of the above memories. The computer may be various computing devices including intelligent terminals and servers.

[0174] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0175] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0176] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0177] In addition, in one embodiment, the present application also provides a computer program product, including a computer program, which implements the steps of the method in the aforementioned embodiment 1 when executed by a processor.

[0178] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0179] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0180] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned implementation methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a disk, or an optical disk), and includes a number of instructions for enabling a multimedia terminal device (which can be a mobile phone, a computer, a television receiver, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0181] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for extracting a target text segment, characterized in that: The method comprises: Segmenting the target text to obtain a word sequence vector; wherein the word sequence vector includes a plurality of words obtained by segmenting the target text; Based on the word sequence vector and the text category vector obtained through training, a word weight vector is obtained; wherein the word weight vector includes the weight of each word in each text category, and the text category vector includes multiple text categories; According to the weight of each word in each text category, obtain N target words whose weight in each text category is higher than the threshold, where N is a positive integer; Based on the N target words, obtaining target text segments belonging to each text category; The step of obtaining target text segments belonging to each text category based on the N target words specifically includes: Determine whether there are M target words with continuous semantics among the N target words in the same category, where M is a positive integer and M is less than or equal to N; If so, the M semantically continuous target words are concatenated to obtain the target text segment.

2. The method according to claim 1, characterized in that Before the step of segmenting the target text to obtain a word sequence vector, the method further includes: Acquire a sample text set including a plurality of sample texts and a text category set of the sample texts; wherein the text category set includes a plurality of the text categories; Based on the sample text set and the text category set, obtaining a text weight vector representing the importance of each sample text in each text category; Based on the text weight vector, obtaining a category vector of each of the text categories on each of the sample texts; Based on the text weight vector and the category vector, the text category vector is obtained through model training.

3. The method according to claim 2, characterized in that The step of obtaining a text weight vector characterizing the importance of each sample text in each text category based on the sample text set and the text category set specifically includes: Based on the text category set, obtaining an initial text category vector; Perform segmentation processing on each of the sample texts to obtain a sample word sequence vector of each of the sample texts; The text weight vector of each of the sample texts is obtained based on the initial text category vector and the sample word sequence vector.

4. The method according to claim 3, characterized in that The sample word sequence vector includes a plurality of sample words obtained by segmenting the target text; the step of obtaining the text weight vector of each sample text based on the initial text category vector and the sample word sequence vector specifically includes: Based on the initial text category vector and the sample word sequence vector, obtaining a sample word weight vector for each sample word in each text category; Normalizing the sample word weight vector of each sample word in each text category to obtain a processed sample word weight vector; Multiplying the sample word weight vector by the processed sample word weight vector element by element to obtain a plurality of element-by-element product results; A sum operation is performed on multiple element product results to obtain the text weight vector of each sample text.

5. The method according to claim 2, characterized in that The step of obtaining the text category vector through model training based on the text weight vector and the category vector specifically includes: Based on the text weight vector and the category vector, obtaining a sample text vector; The sample text vector is input into the model to be trained to obtain the text category vector.

6. The method according to claim 1, characterized in that The step of obtaining a word weight vector based on the word sequence vector and the text category vector obtained through training specifically includes: The word sequence vector is multiplied by the text category vector obtained through training to obtain the word weight vector.

7. The method according to any one of claims 1 to 6, characterized in that The target text is a long text including news public opinion, and the target text segment includes fraud risk public opinion; The step of segmenting the target text to obtain a word sequence vector specifically includes: Segment the long text including news and public opinion to obtain word sequence vectors; The step of obtaining target text segments belonging to each text category based on the N target words specifically includes: Based on the N target words, target text segments including fraud risk public opinion belonging to each text category are obtained.

8. A device for extracting a target text segment, characterized in that: The device comprises: A text segmentation module, used for segmenting the target text to obtain a word sequence vector; wherein the word sequence vector includes a plurality of words obtained by segmenting the target text; A weight obtaining module, used to obtain a word weight vector based on the word sequence vector and the text category vector obtained through training; wherein the word weight vector includes the weight of each word in each text category, and the text category vector includes multiple text categories; A word acquisition module is used to acquire N target words whose weights are higher than a threshold in each text category according to the weights of each word in each text category, where N is a positive integer; A segment acquisition module, used for acquiring target text segments belonging to each text category based on the N target words; The fragment acquisition module is also used to determine whether there are M semantically continuous target words among the N target words in the same category, where M is a positive integer and M is less than or equal to N; if so, the M semantically continuous target words are spliced ​​to obtain the target text fragment.

9. An electronic device, characterized in that: The device comprises a processor, a memory and a computer program stored in the memory, and the computer program implements the steps of the method according to any one of claims 1 to 7 when executed by the processor.

10. A computer storage medium, characterized in that: The computer storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are performed.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images