Text processing method and device, electronic equipment and storage medium

Abstract

Embodiments of the present disclosure provide a text processing method and device, electronic equipment and storage medium. The method comprises: obtaining a to-be-processed text, and performing feature extraction on the to-be-processed text to obtain a semantic feature space, wherein the semantic feature space comprises semantic feature vectors of at least two text segments constituting the to-be-processed text; obtaining a scene feature space corresponding to at least two text processing scenes, wherein the scene feature space comprises semantic feature vectors of processing content keywords corresponding to the at least two text processing scenes; and determining a target text processing scene according to space features of the semantic feature space and space features of the at least two scene feature spaces, wherein the space features represent distribution features of the semantic feature vectors in the feature space. The semantic features corresponding to the to-be-processed text are converted into the semantic feature space, the semantic features corresponding to the text processing scenes are converted into the scene feature space, and comparison is performed at the feature space level, thereby achieving accurate detection of the text processing scene.

Description

Technical Field

[0001] This disclosure relates to the field of Internet technology, and in particular to a text processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Currently, with the popularization of the Internet and the rapid development of digital technology, a large amount of text data has been created. By processing and utilizing text data, various functions such as risk detection and precise information push can be realized.

[0003] In existing technologies, text data processing is typically based on pre-trained text processing models. However, when the text data to be processed needs to be processed using a corresponding text processing model based on its specific text processing scenario, human experience is still required to read and understand the entire text data before determining the text processing scenario and then identifying the appropriate text processing model and solution for processing the text.

[0004] Therefore, existing solutions suffer from low efficiency in text data processing and poor text quality because they cannot recognize text processing scenarios. Summary of the Invention

[0005] This disclosure provides a text processing method, apparatus, electronic device, and storage medium to overcome the problem that existing technologies cannot efficiently and accurately identify text processing scenarios.

[0006] In a first aspect, embodiments of this disclosure provide a text processing method, including:

[0007] The text to be processed is acquired, and features are extracted from the text to obtain a semantic feature space, which includes semantic feature vectors of at least two text segments constituting the text to be processed; scene feature spaces corresponding to at least two text processing scenarios are acquired, which include semantic feature vectors of processing content keywords corresponding to at least two text processing scenarios; a target text processing scenario is determined based on the spatial features of the semantic feature space and the spatial features of the at least two scene feature spaces, wherein the spatial features characterize the distribution features of the semantic feature vectors in the feature space.

[0008] Secondly, embodiments of this disclosure provide a text processing apparatus, including:

[0009] The first extraction module is used to acquire the text to be processed and to extract features from the text to be processed to obtain a semantic feature space, wherein the semantic feature space includes semantic feature vectors of at least two text segments constituting the text to be processed.

[0010] The second extraction module is used to obtain scene feature spaces corresponding to at least two text processing scenarios, wherein the scene feature spaces include semantic feature vectors of processing content keywords corresponding to at least two text processing scenarios;

[0011] The processing module is used to determine the target text processing scenario based on the spatial features of the semantic feature space and the spatial features of at least two of the scene feature spaces, wherein the spatial features characterize the distribution features of the semantic feature vectors in the feature space.

[0012] Thirdly, embodiments of this disclosure provide an electronic device, including: a processor and a memory;

[0013] The memory stores computer-executed instructions;

[0014] The processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the text processing method as described in the first aspect and various possible designs of the first aspect.

[0015] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the text processing method described in the first aspect and various possible designs of the first aspect.

[0016] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the text processing method described in the first aspect and various possible designs of the first aspect.

[0017] The text processing method, apparatus, electronic device, and storage medium provided in this embodiment acquire the text to be processed and extract features from the text to obtain a semantic feature space, which includes semantic feature vectors of at least two text segments constituting the text to be processed; acquire scene feature spaces corresponding to at least two text processing scenarios, which include semantic feature vectors of processing content keywords corresponding to at least two text processing scenarios; and determine a target text processing scenario based on the spatial features of the semantic feature space and the spatial features of the at least two scene feature spaces, wherein the spatial features characterize the distribution features of the semantic feature vectors in the feature space. By converting the semantic features corresponding to the text to be processed into a semantic feature space, converting the semantic features corresponding to the text processing scenarios into scene feature spaces, and comparing them at the feature space level, a target text processing scenario matching the text to be processed is determined, thereby improving text processing efficiency and text processing effect. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 An application scenario diagram of the text processing method provided in this disclosure embodiment;

[0020] Figure 2 Flowchart of the text processing method provided in the embodiments of this disclosure Figure 1 ;

[0021] Figure 3 for Figure 2 A flowchart illustrating the specific implementation process of step S101 in the illustrated embodiment;

[0022] Figure 4 A schematic diagram illustrating the process of generating a semantic feature space provided in this embodiment of the disclosure;

[0023] Figure 5 for Figure 3 A flowchart illustrating the specific implementation process of step S1011 in the illustrated embodiment;

[0024] Figure 6 A schematic diagram of a scene feature space provided in an embodiment of this disclosure;

[0025] Figure 7 A flowchart illustrating the text processing method provided in this disclosure embodiment. Figure 2 ;

[0026] Figure 8 for Figure 7 A flowchart illustrating the specific implementation process of step S207 in the illustrated embodiment;

[0027] Figure 9 for Figure 7 A flowchart illustrating the specific implementation process of step S211 in the illustrated embodiment;

[0028] Figure 10 A schematic diagram illustrating the generation process of a first spatial feature provided in an embodiment of this disclosure;

[0029] Figure 11 A schematic diagram illustrating the generation process of a second spatial feature provided in an embodiment of this disclosure;

[0030] Figure 12 This is a schematic diagram illustrating a process for performing a hypothesis test on the population mean, provided by an embodiment of this disclosure.

[0031] Figure 13 A structural block diagram of a text processing apparatus provided in an embodiment of this disclosure;

[0032] Figure 14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure;

[0033] Figure 15 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0035] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0036] The application scenarios of the embodiments of this disclosure are explained below:

[0037] Figure 1 This diagram illustrates an application scenario of the text processing method provided in this disclosure. The text processing method can be applied to applications and services with long text processing capabilities. More specifically, it can be applied to various application scenarios such as text search for long texts, document topic recognition, and security detection. The executing entity in this embodiment can be a terminal device running the aforementioned application with text processing capabilities, such as a personal computer or smartphone. It can also be a server running the server corresponding to the aforementioned application, or other electronic devices performing similar functions. (See reference...) Figure 1As shown in the diagram, taking the server as an example, after receiving a long text (Text_1) to be processed from the terminal device, the server uses the text processing method provided in this embodiment to process the long text and identify the corresponding text processing scenario. These scenarios include, for example, detecting content that infringes on user privacy, detecting descriptions of product requirements, and extracting the main idea of ​​the text. Then, based on the specific text processing scenario, a corresponding strategy is used to process the text and obtain the processing result. For example, processing the long text based on the scenario yields a short text (Text_2) representing the main idea of ​​the long text. This short text is then returned to the terminal device, thus completing the implementation of the text processing method. Further processing and transmission of the short text can then be performed as needed, which will not be elaborated here.

[0038] In existing technologies, long texts with complex content, such as software development requirements documents, are characterized by rigorous logic and abundant key points. However, the contextual dependencies in such long texts are not very strong. Therefore, globally encoding long texts can easily lead to significant errors in contextual dependency information. Furthermore, the objects of interest and focus that need to be processed may only constitute a small portion of the entire document's content. For example, they might be found in background information, objective descriptions, or benefit descriptions. Other content, such as feature introductions and activity descriptions, may occupy a much larger portion. Directly processing such long texts fails to extract the content that users truly care about and are interested in, resulting in text processing results that are difficult to achieve the desired effect. Other implementation solutions, which rely on human experience to understand and analyze long texts, determine the matching text processing scenario, and then apply corresponding strategies, often suffer from low processing efficiency.

[0039] This disclosure provides a text processing method to solve the above-mentioned problems.

[0040] refer to Figure 2 , Figure 2 Flowchart of the text processing method provided in the embodiments of this disclosure Figure 1 The method of this embodiment can be applied to terminal devices and servers. This text processing method includes:

[0041] Step S101: Obtain the text to be processed and extract features from the text to obtain a semantic feature space, which includes the semantic feature vectors of at least two text segments that constitute the text to be processed.

[0042] For example, refer to Figure 1The illustrated application scenario, using a terminal device as the execution entity, demonstrates that after receiving the text to be processed, the terminal device first divides the long text into at least two shorter texts. There are several ways to divide the text. For example, it can divide the text with a fixed step size to obtain a corresponding number of short texts, i.e., text segments. Another example is dividing the text based on segmentation markers to obtain several short sentences, i.e., text segments; or, based on the segmentation based on segmentation markers, further dividing the short sentences using the semantics of the obtained short sentences to obtain short texts at the character or word level as text segments. The specific implementation method can be set as needed. Segmentation markers include, for example, punctuation marks, line breaks, and page breaks in the text, used to segment the text.

[0043] Furthermore, after dividing the text to be processed into at least two text segments, features are extracted from each text segment to obtain the semantic features corresponding to that segment. These semantic features are expressed as semantic feature vectors. That is, semantic feature vectors for the two text segments are obtained. These semantic feature vectors can be achieved by embedding the text segments; the specific implementation method is existing technology and will not be elaborated further. Then, the semantic feature vectors corresponding to each text segment are merged to obtain a set of multi-dimensional features, i.e., a semantic feature vector. This semantic feature vector represents the features of the text to be processed in multiple content dimensions.

[0044] In one possible implementation, such as Figure 3 As shown, the specific implementation of step S101 includes:

[0045] Step S1011: Based on the target window length, slide the text to be processed to obtain at least two text segments.

[0046] Step S1012: Extract features from at least two text segments to obtain the corresponding first semantic feature vector.

[0047] Step S1013: Obtain the semantic feature space based on at least two first semantic feature vectors.

[0048] For example, firstly, information representing the length of the text segment is obtained, namely the target window length, more specifically, the target window length is, for example, the number of characters. Then, starting from the beginning of the text to be processed, the text of the target window length is sequentially segmented as text segments, and after each segment, the text is slid backward a certain distance until the end of the text to be processed is reached. Then, for example, each text segment is feature-encoded to obtain the first semantic information corresponding to each text segment. The semantic information corresponding to each text segment is then merged to obtain a semantic feature space, which can be identified based on a feature matrix, where each row (or each column) of the feature matrix represents the first semantic feature vector corresponding to a text segment.

[0049] Figure 4 This is a schematic diagram illustrating a process for generating a semantic feature space, as provided in an embodiment of the present disclosure. Figure 4 As shown, for the text to be processed, firstly, punctuation marks and line breaks (i.e., segmentation markers) are removed to obtain the corresponding plain text. Then, starting from the beginning of the plain text, a segment is extracted with a target window length N (i.e., the length of N characters, shown as N in the figure), resulting in text segment P_1. After shifting the window by M characters, another segment is extracted with the target window length N, resulting in text segment P_2, and so on, until the end of the plain text is reached, at which point the text is extracted again, resulting in text segment P_n. Next, feature encoding is performed on text segments P_1 to P_n, resulting in corresponding feature vectors F_1 to F_n. Finally, the semantic feature space is obtained based on the set of feature vectors F_1 to F_n. It should be noted that, depending on the relationship between the target window length, the shift step size, and the plain text, the length of the last segment obtained from the plain text extraction may be shorter than other segments. This can be addressed by padding with zeros or by leaving it unprocessed, having minimal impact on the final processing result, and can be set as needed.

[0050] Furthermore, in the process of sliding and dividing the text to be processed, the larger the translation step size after each segmentation, the smaller the overlap rate between the two text segments; conversely, the smaller the translation step size, the higher the overlap rate between the two text segments. A higher overlap rate allows for better extraction of contextual information between text segments, thereby improving the accuracy of semantic features. Therefore, in this embodiment, the overlap rate of two text segments during the translation process is dynamically set according to the content of the text to be processed, thereby improving the rationality of the set overlap rate (i.e., translation step size) and thus improving the accuracy of the extracted semantic features.

[0051] Specifically, in one possible implementation, such as Figure 5 As shown, the specific implementation steps of step S1011 include:

[0052] Step S1011A: Obtain the target overlap rate based on the average distance of the sentence break markers in the text to be processed.

[0053] Step S1011B: Based on the target overlap rate and target window length, slide the plain text corresponding to the text to be processed to obtain at least two text segments, where the plain text is the text to be processed after removing the sentence break markers.

[0054] For example, based on the previous introduction to overlap rates, after obtaining the text to be processed, the positions of sentence segmentation markers (such as punctuation marks, paragraph marks, etc.) in the text are parsed, and the intervals between sentence segmentation markers are calculated to obtain the average distance between them. Since the average distance between sentence segmentation markers can reflect sentence length and paragraph length, and in long texts, sentence segmentation is usually based on expressing a complete semantic meaning, the average distance between sentence segmentation markers reflects the information density in long texts to some extent. That is, the shorter the average distance between sentence segmentation markers, the more concise the sentence, meaning the more information represented by each character and the higher the information density; conversely, the longer the average distance between sentence segmentation markers, the richer the sentence content, meaning the less information represented by each character and the lower the information density. Therefore, based on the average distance, a matching overlap rate, i.e., the target overlap rate, is mapped using the above relationship. Among them, the shorter the average distance of the sentence segmentation markers and the higher the information density of the sentences, the lower the corresponding target overlap rate, thus avoiding excessive contextual information from affecting the recognition of the main semantics in the extracted text segments; while the longer the average distance and the lower the information density of the sentences, the higher the corresponding target overlap rate, thus improving the information utilization rate in the text, repeatedly using contextual information to determine the main semantics in the extracted text segments, thereby improving the rationality of semantic text segment division and the accuracy of the semantic feature space of the text segments to be processed.

[0055] Step S102: Obtain the scene feature space corresponding to at least two text processing scenarios, wherein the scene feature space includes the semantic feature vectors of the processing content keywords corresponding to at least two text processing scenarios.

[0056] For example, on the other hand, a scenario feature space corresponding to at least two preset text processing scenarios is obtained. Similar to the semantic feature space, the scenario feature space is also a collection of semantic feature vectors, but the semantic feature vectors in the scenario feature space represent the semantics of the scenario keywords corresponding to the text processing scenario. Specifically, for example, when the text to be processed is a software development requirements document, one text processing scenario is "summarizing the business requirements in the requirements document," and another document text processing scenario is "detecting non-compliant content in the requirements document." Correspondingly, for the above two text processing scenarios, each text processing scenario corresponds to a scenario feature space, which contains semantic feature vectors representing the keywords of the processed content. Figure 6 This is a schematic diagram of a scene feature space provided in an embodiment of the present disclosure, such as... Figure 6 As shown, the pre-generated scene feature spaces M1, M2, and M3 correspond to text processing scenarios A, B, and C, respectively. The specific implementations of each text processing scenario are as illustrated in the previous examples and will not be repeated here. Correspondingly, each scene feature space contains multiple semantic feature vectors representing keywords of the processed content. For example, in scene feature space M1 corresponding to text processing scenario A ("business requirements in the requirements document"), there are N semantic feature vectors (featureA_N) representing the keyword "registration volume" and featureA_2 representing the keyword "user activity". Similarly, in scene feature space M2 corresponding to text processing scenario B ("non-compliant content in the requirements document"), there are M semantic feature vectors (featureB_M) representing the keyword "address" and featureB_2 representing the keyword "phone number". The specific implementation of scene feature space M3 is similar to that of scene feature space M1 and scene feature space M2 mentioned above, and will not be shown again.

[0057] The aforementioned scene feature space can be pre-generated and stored in the terminal device or server as a file. Upon triggering the corresponding instruction, the file is read and the scene feature space is loaded. The keywords for the processed content in the scene feature space can be data obtained based on human expert experience and / or labeled samples, thus enabling accurate matching of different text processing scenarios. Furthermore, the above steps can be triggered before feature extraction of the text to be processed in step S101, or after feature extraction of the text to be processed in step S101, or simultaneously. The specific implementation can be set as needed and is not specifically limited here.

[0058] Step S103: Determine the target text processing scenario based on the spatial features of the semantic feature space and the spatial features of at least two scene feature spaces, wherein the spatial features represent the distribution characteristics of the semantic feature vectors in the feature space.

[0059] For example, after obtaining the semantic feature space and at least two scene feature spaces, the spatial features of the semantic feature space are compared with the spatial features of each scene feature space to determine a scene feature space that matches the semantic feature space, i.e., the target scene feature space. The text processing scene corresponding to the target scene feature space is then identified as the target text processing scene. Here, spatial features represent the distribution characteristics of semantic feature vectors in the feature space. Specifically, a spatial-level comparison is made between the multi-dimensional vector space formed by the semantic feature vectors of each text segment in the text to be processed and the multi-dimensional vector space formed by the semantic feature vectors of the keywords corresponding to the processing content in different text processing scenes. This is equivalent to simulating human understanding of the full text content of the text to be processed and determining the text processing scene from the perspective of the full text content. Since the scene feature space is a multi-dimensional set of feature vectors generated based on human expert experience and / or manually annotated samples, it can accurately describe different text processing scenes. Therefore, by comparing it with the semantic feature space, accurate identification of the text processing scene of the text to be processed can be achieved.

[0060] Furthermore, after obtaining the target text processing scenario, the text to be processed can be further processed based on the knowledge base data and text processing model corresponding to the target text processing scenario to achieve corresponding processing results such as text detection and summarization. Specific settings can be configured as needed, and will not be elaborated here. If there is a scenario knowledge base corresponding to the target text processing scenario, the risk intensity, historical security events, and risk solutions corresponding to the scenario can be directly associated. If there is no corresponding scenario knowledge base, the knowledge base can be expanded based on the recognition results to achieve a knowledge loop.

[0061] In this embodiment, by acquiring the text to be processed and extracting its features, a semantic feature space is obtained, which includes semantic feature vectors of at least two text segments constituting the text to be processed. At least two scene feature spaces corresponding to text processing scenarios are also acquired, each including semantic feature vectors of the processing content keywords corresponding to the at least two text processing scenarios. Based on the spatial features of the semantic feature space and the spatial features of the at least two scene feature spaces, a target text processing scenario is determined, where spatial features characterize the distribution of semantic feature vectors in the feature space. By transforming the semantic features corresponding to the text to be processed into a semantic feature space, and transforming the semantic features corresponding to the text processing scenarios into scene feature spaces, and comparing them at the feature space level, a target text processing scenario matching the text to be processed is determined, and the text to be processed is then processed. This achieves the goal of automated text processing based on scene pairs, improving text processing efficiency and effectiveness.

[0062] refer to Figure 7 , Figure 7 A flowchart illustrating the text processing method provided in this disclosure embodiment. Figure 2 This embodiment is in Figure 2 Based on the illustrated embodiments, the steps of each embodiment are further refined, and this text processing method includes:

[0063] Step S201: Obtain the text to be processed, which includes the main text and the title text. The title text is used to represent the main content of the main text.

[0064] Step S202: Based on the target window length, slide the main text to obtain at least two text segments.

[0065] For example, in this embodiment, the text to be processed is divided into body text and title text. The body text is the main part of the text to be processed, such as text comprising multiple paragraphs, while the title text is a description of the main content of each paragraph, such as the title of each paragraph. Constructing a complete piece of text data using title text and body text is a common text structure, which will not be elaborated upon here. Further, refer to... Figure 2 The illustrated embodiment describes the process of segmenting the text to be processed. In this embodiment, based on the target window length, the main text is segmented to obtain at least two text segments corresponding to the main text. The specific process is similar to the process of segmenting the text to be processed in previous embodiments, and will not be repeated here. As for the title text (short text), it can be directly encoded, which is equivalent to it being a text segment itself, and subsequent steps will be performed.

[0066] Step S203: Use the pre-trained generative model to generate summary text corresponding to the text segment.

[0067] Step S204: Obtain the first semantic feature vector corresponding to the text segment based on the summary text.

[0068] Step S205: Obtain the second semantic feature vector corresponding to the title text, and obtain the semantic feature space based on the first semantic feature vectors corresponding to at least two text segments and the second semantic feature vector corresponding to the title text.

[0069] For example, after obtaining the text segments corresponding to the main text, a pre-trained generative model is used to generate summary texts corresponding to the text segments, i.e., summary texts representing the main content of each text segment. Specifically, for short texts such as document titles and descriptions in the text to be processed (title text), no additional operations are required; text information embedding encoding is performed directly, word vectors are constructed using word2vec, and then converted into sentence vectors through pooling operations, thereby obtaining the second semantic feature vector corresponding to the title text. For long texts containing multiple noises (main text), the text is first divided into text segments, and then key information is extracted from each segment. Here, the key information is the summary text of the text segment. This can be achieved using a pre-trained Large Language Model (LLM), and the operation is similar to that for document titles, descriptions, and other text segments, to obtain the first semantic feature vector. Finally, based on the first semantic feature vectors corresponding to at least two text segments and the second semantic feature vector corresponding to the title text, a multi-dimensional vector feature space containing the content information of the entire text to be processed is obtained, i.e., the semantic feature space.

[0070] Step S206: Obtain expert experience data corresponding to at least two text processing scenarios. The expert experience data represents the text content distribution rules for the text processing scenarios.

[0071] Step S207: Based on expert experience data, generate a third semantic feature vector representing the keywords of the processed content.

[0072] For example, on the other hand, a third semantic feature vector representing the keywords of the processed content is generated by acquiring expert experience data corresponding to at least two text processing scenarios. The expert experience data represents the text content distribution rules for the specific text processing scenario. For instance, a specific set of expert experience data describes the following text content distribution rules: For text processing scenario A, keywords key_word_1, key_word_2, and key_word_3 appear, with key_word_1 appearing first, followed by key_word_2 and key_word_3. That is, when the above keywords and distribution rules appear in the text, the corresponding text processing scenario is A.

[0073] For example, based on the aforementioned expert experience data, the keywords representing the processed content and their corresponding distribution rules are encoded to generate multiple corresponding third semantic feature vectors. These third semantic feature vectors have a specific order (corresponding to the distribution rules), and a corresponding scene feature space is subsequently generated based on this order.

[0074] For example, such as Figure 8 As shown, the specific implementation steps of step S207 include:

[0075] Step S2071: Perform data augmentation on the expert experience data to obtain augmented experience data.

[0076] Step S2072: Based on the enhanced empirical data, generate a third semantic feature vector representing the keywords of the processed content.

[0077] For example, after obtaining expert experience data, data augmentation is performed on the expert experience data to further improve data quality. For example, data augmentation includes at least one of the following: similar word replacement, synonym conversion, and syntax tree operations. This expands the original expert experience data, resulting in expert data with a wider coverage, thereby improving the recognition success rate and accuracy in text processing scenarios. The specific implementation methods of data augmentation such as similar word replacement, synonym conversion, and syntax tree operations are existing technologies and will not be elaborated here.

[0078] Optionally, the method in this embodiment further includes:

[0079] Step S208: Obtain labeled sample data corresponding to the text processing scenario. The labeled sample data includes sample text and the labeling information corresponding to the sample text. The labeling information is used to label the processing content keywords in the sample text.

[0080] Step S209: Generate the fourth semantic feature vector based on the labeled sample data.

[0081] Step S210: Based on the third semantic feature vector and the fourth semantic feature vector, obtain the scene feature space.

[0082] For example, while obtaining expert experience data, corresponding small-sample labeled sample data can also be acquired. This labeled sample data can be text labeled with annotation information based on human experience. The annotation information in this labeled sample data is used to indicate the keywords in the sample text that can represent its text processing scenario, i.e., processing content keywords. Specifically, a set of labeled sample data may, for example, label keywords such as "registration volume" and "activity level" in a piece of text as processing content keywords to represent the text processing scenario of "business requirements in the requirements document". The role of the labeled sample data is similar to that of expert experience data, used to describe the specific text processing scenario and generate corresponding semantic feature information representing the processing content keywords, i.e., the fourth semantic feature vector.

[0083] For example, the third semantic feature vector and the fourth semantic feature vector are then merged to obtain the scene feature space. For specific implementation details, please refer to [reference needed]. Figure 2 The steps for obtaining the scene feature space based on the semantic feature vectors corresponding to multiple text segments in the illustrated embodiment will not be repeated here.

[0084] Step S211: Obtain the spatial features of the semantic feature space.

[0085] Step S212: Obtain spatial features from at least two scene feature spaces.

[0086] For example, the spatial features of the semantic feature space are designated as the first spatial features, which characterize the distribution of similarity between each semantic feature vector in the semantic feature space and semantic feature vectors in the scene feature space. The spatial features of the scene feature space are designated as the second spatial features, which characterize the distribution of similarity between each current semantic feature vector and other semantic feature vectors in the scene feature space. Here, the current semantic feature vector is a semantic feature vector sequentially obtained from the scene feature space, and the other semantic feature vectors are semantic feature vectors in the scene feature space excluding the current semantic feature vector. The distribution features are statistical results of the distribution of similarity corresponding to the current semantic feature vector. In one possible implementation, the first spatial features and the second spatial features are used to calculate the cosine similarity between the semantic feature vectors to obtain the similarity of each semantic feature vector.

[0087] One possible implementation is, such as Figure 9 As shown, the specific implementation of step S211 includes:

[0088] Step S2111: Sequentially obtain the cosine similarity between each semantic feature vector in the semantic feature space and the semantic feature vectors in each scene feature space.

[0089] Step S2112: Based on the cosine similarity between each semantic feature vector in the semantic feature space and the semantic feature vectors in each scene feature space, obtain at least two similarity sequences.

[0090] Step S2113: Obtain the first spatial feature based on at least two similarity sequences.

[0091] Figure 10 This is a schematic diagram illustrating the generation process of a first spatial feature according to an embodiment of the present disclosure, with reference to... Figure 10 As shown, for example, the semantic feature space includes semantic feature vectors F_1 to F_N (shown as F_1 to F_N in the figure), where N is an integer greater than 1. The scene feature space includes semantic feature vectors C_1 to C_M (shown as C_1 to C_N in the figure), where M is an integer greater than 1. Then, the cosine similarity (similarity calculation) between semantic feature vectors F_1 to F_N and semantic feature vectors C_1 to C_M is calculated sequentially to obtain the similarity sequence corresponding to each semantic feature vector in the semantic feature space, i.e., similarity sequence List_1 to similarity sequence List_N (shown as List_1 to List_N in the figure). Each similarity sequence contains M sequence values, i.e., the cosine similarity value between semantic feature vector F_n and semantic feature vector C_m, where 1≤n≤N, 1≤m≤M. Then, by performing distribution statistics along the similarity sequence List_1 to the similarity sequence List_N, we can obtain the distribution characteristics of the similarity between each semantic feature vector in the semantic feature space and the semantic feature vector in the scene feature space, which is the first spatial feature. Here, the similarity can be cosine similarity.

[0092] In one possible implementation, step S212 is specifically implemented as follows:

[0093] Step S2121: Sequentially obtain the cosine similarity between each semantic feature vector in the scene feature space and other semantic feature vectors.

[0094] Step S2122: Obtain the similarity metric matrix based on the cosine similarity between each semantic feature vector and other semantic feature vectors.

[0095] Step S2123: Obtain the second spatial features based on the similarity metric matrix.

[0096] For example, the second spatial features of the scene feature space can be obtained by calculating the cosine similarity between every two semantic feature vectors in the scene feature space. Figure 11 This is a schematic diagram illustrating the generation process of a second spatial feature according to an embodiment of the present disclosure, with reference to... Figure 11As shown, exemplarily, the scene feature space includes semantic feature vectors C_1 to C_M (shown as C_1 to C_M in the figure), where M is an integer greater than 1. Then, the cosine similarity between semantic feature vectors C_1 to C_M and themselves is calculated sequentially to obtain a similarity metric matrix. Each matrix point in the similarity matrix represents the cosine similarity between the semantic feature vector C_n in the corresponding row and the semantic feature vector C_m in the corresponding column. Here, 1 ≤ n ≤ M, 1 ≤ m ≤ M. Subsequently, the similarity metric matrix is ​​statistically analyzed to obtain the distribution characteristics of the similarity between each current semantic feature vector and other semantic feature vectors in the scene feature space. Here, the current semantic feature vector is the semantic feature vector sequentially obtained in the scene feature space, and the other semantic feature vectors are the semantic feature vectors in the scene feature space excluding the current semantic feature vector. The distribution characteristics are the statistical results of the distribution of similarity corresponding to the current semantic feature vector, i.e., the second spatial features.

[0097] Furthermore, in one possible implementation, step S2123 is specifically implemented as follows:

[0098] Step S2123A: Based on the diagonal of the similarity metric matrix, divide the similarity metric matrix into two feature regions.

[0099] Step S2123B: For any feature region, perform statistical analysis on all cosine similarities along the rows or columns to obtain the distribution characteristics of the similarity between different cosine similarities corresponding to the similarity metric matrix.

[0100] For example, refer to Figure 11 The similarity metric matrix shown is symmetrical along the diagonal. Therefore, the similarity metric matrix is ​​divided into two symmetrical feature regions along the diagonal. Then, for any feature region, the cosine similarity of all rows or columns is statistically analyzed to obtain the distribution feature of the similarity between each current semantic feature vector and other semantic feature vectors in the scene feature space. Here, the current semantic feature vector is the semantic feature vector obtained sequentially in the scene feature space, and the other semantic feature vectors are the semantic feature vectors in the scene feature space other than the current semantic feature vector. The distribution feature is the statistical result of the distribution of similarity corresponding to the current semantic feature vector, that is, the second space feature.

[0101] Step S213: Determine the target text processing scenario based on the spatial features of the semantic feature space and the spatial features of at least two scene feature spaces.

[0102] For example, after obtaining the spatial features of the semantic feature space and the spatial features of at least two scene feature spaces, i.e., after obtaining the first spatial feature and the second spatial feature obtained in the above steps, a multi-dimensional comparison is performed based on the similarity distribution of multiple feature dimensions corresponding to the first spatial feature and the second spatial feature. Specifically, an overall mean hypothesis test is performed on the first spatial feature and the second spatial feature to obtain the target text processing scene to which the first spatial feature belongs under a preset confidence level.

[0103] Figure 12 This is a schematic diagram illustrating a process for performing a hypothesis test on the population mean, provided in an embodiment of this disclosure. (Refer to...) Figure 12 As shown in the figure, based on the distribution curve corresponding to the semantic feature space (i.e., the first spatial feature) and the distribution curve corresponding to multiple scene feature spaces (i.e., the second spatial feature), the overall mean hypothesis test is performed on the first spatial feature and the second spatial feature. Under the condition of meeting the preset confidence level, the target second spatial feature that matches the first spatial feature can be obtained. For example, as shown in the figure, the second spatial feature M2 in the second spatial feature M1, second spatial feature M2 and second spatial feature M3 is determined as the target second spatial feature, and then the text processing scene corresponding to the target second spatial feature is determined as the target text processing scene.

[0104] In this embodiment, the implementation of step S201 is the same as that in this disclosure. Figure 2 The implementation of step S101 in the illustrated embodiment is the same, and will not be described in detail here.

[0105] Corresponding to the text processing method in the above embodiments, Figure 13 This is a structural block diagram of a text processing apparatus provided in an embodiment of the present disclosure. For ease of explanation, only the parts relevant to the embodiments of the present disclosure are shown.

[0106] Reference Figure 13 The text processing device 3 includes:

[0107] The first extraction module 31 is used to acquire the text to be processed and extract features from the text to be processed to obtain a semantic feature space, which includes semantic feature vectors of at least two text segments constituting the text to be processed.

[0108] The second extraction module 32 is used to obtain the scene feature space corresponding to at least two text processing scenarios, wherein the scene feature space includes the semantic feature vector of the processing content keywords corresponding to at least two text processing scenarios.

[0109] Processing module 33 is used to determine the target text processing scenario based on the spatial features of the semantic feature space and the spatial features of at least two scene feature spaces, wherein the spatial features represent the distribution characteristics of the semantic feature vectors in the feature space.

[0110] In one embodiment of this disclosure, the spatial features of the semantic feature space are first spatial features, which characterize the distribution of similarity between each semantic feature vector in the semantic feature space and the semantic feature vector in the scene feature space; the spatial features of the scene feature space are second spatial features, which characterize the distribution of similarity between each current semantic feature vector and other semantic feature vectors in the scene feature space, wherein the current semantic feature vector is a semantic feature vector sequentially obtained in the scene feature space, and the other semantic feature vectors are semantic feature vectors in the scene feature space other than the current semantic feature vector, and the distribution features are the statistical results of the distribution of similarity corresponding to the current semantic feature vector; the processing module 33 is specifically used to: perform an overall mean hypothesis test on the first spatial features and the second spatial features to obtain the target text processing scene to which the first spatial feature belongs under a preset confidence level.

[0111] In one embodiment of this disclosure, the processing module 33 is further configured to: sequentially obtain the cosine similarity between each semantic feature vector in the semantic feature space and the semantic feature vector in each scene feature space; obtain at least two similarity sequences based on the cosine similarity between each semantic feature vector in the semantic feature space and the semantic feature vector in each scene feature space; and obtain a first spatial feature based on the at least two similarity sequences.

[0112] In one embodiment of this disclosure, the processing module 33 is further configured to: sequentially obtain the cosine similarity between each semantic feature vector and other semantic feature vectors in the scene feature space; obtain a similarity measurement matrix based on the cosine similarity between each semantic feature vector and other semantic feature vectors; and obtain a second spatial feature based on the similarity measurement matrix.

[0113] In one embodiment of this disclosure, when the processing module 33 obtains the second spatial features based on the similarity measurement matrix, it is specifically used to: divide the similarity measurement matrix into two feature regions based on the diagonal of the similarity measurement matrix; and for any feature region, perform statistics on all cosine similarities along the rows or columns to obtain the distribution features of the similarity between different cosine similarities corresponding to the similarity measurement matrix.

[0114] In one embodiment of this disclosure, the first extraction module 31 is specifically used for: sliding division of the text to be processed based on the target window length to obtain at least two text segments; extracting features from the at least two text segments to obtain corresponding first semantic feature vectors; and obtaining a semantic feature space based on the at least two first semantic feature vectors.

[0115] In one embodiment of this disclosure, when the first extraction module 31 performs sliding division of the text to be processed based on the target window length to obtain at least two text segments, it is specifically used to: obtain the target overlap rate based on the average distance of the sentence break markers in the text to be processed; and perform sliding division of the plain text corresponding to the text to be processed based on the target overlap rate and the target window length to obtain at least two text segments, wherein the plain text is the text to be processed after removing the sentence break markers.

[0116] In one embodiment of this disclosure, the text to be processed includes main text and title text, with the title text used to characterize the main content of the main text. When the first extraction module 31 slides and divides the text to be processed based on the target window length to obtain at least two text segments, it is specifically used to: slide and divide the main text based on the target window length to obtain at least two text segments. When the first extraction module 31 obtains a semantic feature space based on at least two first semantic feature vectors, it is specifically used to: obtain the second semantic feature vector corresponding to the title text; and obtain the semantic feature space based on the at least two first semantic feature vectors corresponding to the main text and the second semantic feature vector corresponding to the title text.

[0117] In one embodiment of this disclosure, when the first extraction module 31 extracts features from at least two text segments to obtain the corresponding first semantic feature vector, it is specifically used to: generate a summary text corresponding to the text segment using a pre-trained generative model; and obtain the first semantic feature vector corresponding to the text segment based on the summary text.

[0118] In one embodiment of this disclosure, the second extraction module 32 is specifically used to: acquire expert experience data corresponding to at least two text processing scenarios, wherein the expert experience data characterizes the text content distribution rules for the text processing scenarios; generate a third semantic feature vector characterizing the keywords of the processed content based on the expert experience data; and obtain the scenario feature space based on the third semantic feature vector.

[0119] In one embodiment of this disclosure, the second extraction module 32 is specifically used to: perform data augmentation on expert experience data to obtain augmented experience data; when the second extraction module 32 generates a third semantic feature vector representing the keywords of the processed content based on the expert experience data, it is specifically used to: generate a third semantic feature vector representing the keywords of the processed content based on the augmented experience data.

[0120] In one embodiment of this disclosure, data augmentation includes at least one of the following: similar word replacement, synonym conversion, and syntax tree operations.

[0121] In one embodiment of this disclosure, the second extraction module 32 is further configured to: obtain labeled sample data corresponding to the text processing scenario, wherein the labeled sample data includes sample text and labeled information corresponding to the sample text, and the labeled information is used to label the processing content keywords in the sample text; generate a fourth semantic feature vector based on the labeled sample data; and when the second extraction module 32 obtains the scenario feature space based on the third semantic feature vector, it is specifically configured to: obtain the scenario feature space based on the third semantic feature vector and the fourth semantic feature vector.

[0122] The first extraction module 31, the second extraction module 32, and the processing module 33 are connected sequentially. The text processing device 3 provided in this embodiment can execute the technical solution of the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0123] Figure 14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure, such as... Figure 14 As shown, the electronic device 4 includes:

[0124] Processor 41, and memory 42 communicatively connected to processor 41;

[0125] Memory 42 stores instructions executed by the computer;

[0126] The processor 41 executes computer execution instructions stored in the memory 42 to achieve, for example, Figures 2-12 The text processing method in the illustrated embodiment.

[0127] Optionally, the processor 41 and the memory 42 are connected via a bus 43.

[0128] For relevant instructions, please refer to the corresponding text. Figures 2-12 The relevant descriptions and effects of the steps in the corresponding embodiments are understood, and will not be elaborated on here.

[0129] This disclosure provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement this disclosure. Figures 2-12 The text processing method provided in any of the corresponding embodiments.

[0130] To implement the above embodiments, this disclosure also provides an electronic device.

[0131] refer to Figure 15The diagram illustrates a structural schematic of an electronic device 900 suitable for implementing embodiments of the present disclosure. The electronic device 900 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, personal digital assistants (PDAs), portable Android devices (PADs), portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 15 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0132] like Figure 15 As shown, the electronic device 900 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 into a random access memory (RAM) 903. The RAM 903 also stores various programs and data required for the operation of the electronic device 900. The processing unit 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0133] Typically, the following devices can be connected to I / O interface 905: input devices 906 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 907 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 908 including, for example, magnetic tapes, hard disks, etc.; and communication devices 909. Communication device 909 allows electronic device 900 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 15 An electronic device 900 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0134] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by a processing device 901, it performs the functions defined in the methods of embodiments of this disclosure.

[0135] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0136] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0137] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0138] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0139] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0140] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0141] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0142] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0143] In a first aspect, according to one or more embodiments of this disclosure, a text processing method is provided, comprising:

[0144] The text to be processed is acquired, and features are extracted from the text to obtain a semantic feature space, which includes semantic feature vectors of at least two text segments constituting the text to be processed; scene feature spaces corresponding to at least two text processing scenarios are acquired, which include semantic feature vectors of processing content keywords corresponding to at least two text processing scenarios; a target text processing scenario is determined based on the spatial features of the semantic feature space and the spatial features of the at least two scene feature spaces, wherein the spatial features characterize the distribution features of the semantic feature vectors in the feature space.

[0145] According to one or more embodiments of this disclosure, the spatial features of the semantic feature space are first spatial features, which characterize the distribution characteristics of the similarity between each semantic feature vector in the semantic feature space and the semantic feature vector in the scene feature space; the spatial features of the scene feature space are second spatial features, which characterize the distribution characteristics of the similarity between each current semantic feature vector and other semantic feature vectors in the scene feature space, wherein the current semantic feature vector is a semantic feature vector sequentially obtained in the scene feature space, and the other semantic feature vectors are semantic feature vectors in the scene feature space other than the current semantic feature vector, and the distribution characteristics are the statistical results of the distribution of similarity corresponding to the current semantic feature vector; the step of determining the target text processing scene based on the spatial features of the semantic feature space and the spatial features of at least two scene feature spaces includes: performing an overall mean hypothesis test on the first spatial feature and the second spatial feature to obtain the target text processing scene to which the first spatial feature belongs under a preset confidence level.

[0146] According to one or more embodiments of this disclosure, the method further includes: sequentially obtaining the cosine similarity between each semantic feature vector in the semantic feature space and the semantic feature vectors in each of the scene feature spaces; obtaining at least two similarity sequences based on the cosine similarity between each semantic feature vector in the semantic feature space and the semantic feature vectors in each of the scene feature spaces; and obtaining the first spatial feature based on the at least two similarity sequences.

[0147] According to one or more embodiments of this disclosure, the method further includes: sequentially obtaining the cosine similarity between each semantic feature vector and other semantic feature vectors in the scene feature space; obtaining a similarity measurement matrix based on the cosine similarity between each semantic feature vector and other semantic feature vectors; and obtaining the second spatial feature based on the similarity measurement matrix.

[0148] According to one or more embodiments of this disclosure, obtaining the second spatial feature based on the similarity measurement matrix includes: dividing the similarity measurement matrix into two feature regions based on the diagonal of the similarity measurement matrix; and for any feature region, statistically analyzing all cosine similarities along rows or columns to obtain the distribution characteristics of similarity between different cosine similarities corresponding to the similarity measurement matrix.

[0149] According to one or more embodiments of this disclosure, the step of extracting features from the text to be processed to obtain a semantic feature space includes: dividing the text to be processed into at least two text segments based on a target window length; extracting features from the at least two text segments to obtain corresponding first semantic feature vectors; and obtaining the semantic feature space based on the at least two first semantic feature vectors.

[0150] According to one or more embodiments of this disclosure, the step of slidingly dividing the text to be processed based on a target window length to obtain at least two text segments includes: obtaining a target overlap rate based on the average distance of the sentence break markers in the text to be processed; and slidingly dividing the plain text corresponding to the text to be processed based on the target overlap rate and the target window length to obtain at least two text segments, wherein the plain text is the text to be processed after removing the sentence break markers.

[0151] According to one or more embodiments of this disclosure, the text to be processed includes main text and title text, wherein the title text is used to characterize the main content of the main text; the step of sliding the text to be processed based on the target window length to obtain at least two text segments includes: sliding the main text based on the target window length to obtain at least two text segments; the step of obtaining the semantic feature space based on at least two first semantic feature vectors includes: obtaining the second semantic feature vector corresponding to the title text; and obtaining the semantic feature space based on the at least two first semantic feature vectors corresponding to the main text and the second semantic feature vector corresponding to the title text.

[0152] According to one or more embodiments of this disclosure, feature extraction is performed on the at least two text segments to obtain corresponding first semantic feature vectors, including: generating a summary text corresponding to the text to be processed using a pre-trained generative model; and obtaining the semantic feature vector of the text segment corresponding to the text to be processed based on the summary text.

[0153] According to one or more embodiments of this disclosure, obtaining the scene feature space corresponding to at least two text processing scenarios includes: obtaining expert experience data corresponding to at least two text processing scenarios, wherein the expert experience data represents the text content distribution rules for the text processing scenarios; generating a third semantic feature vector representing keywords of the processed content based on the expert experience data; and obtaining the scene feature space based on the third semantic feature vector.

[0154] According to one or more embodiments of this disclosure, the method further includes: performing data augmentation on the expert experience data to obtain augmented experience data; the step of generating a third semantic feature vector representing the keywords of the processed content based on the expert experience data includes: generating a third semantic feature vector representing the keywords of the processed content based on the augmented experience data.

[0155] According to one or more embodiments of this disclosure, the data enhancement includes at least one of the following: similar word replacement, synonym conversion, and syntax tree operation.

[0156] According to one or more embodiments of this disclosure, the method further includes: obtaining labeled sample data corresponding to the text processing scenario, wherein the labeled sample data includes sample text and labeled information corresponding to the sample text, and the labeled information is used to label the processing content keywords in the sample text; generating a fourth semantic feature vector based on the labeled sample data; and obtaining the scenario feature space based on the third semantic feature vector, which includes: obtaining the scenario feature space based on the third semantic feature vector and the fourth semantic feature vector.

[0157] Secondly, according to one or more embodiments of the present disclosure, a text processing apparatus is provided, comprising:

[0158] The first extraction module is used to acquire the text to be processed and to extract features from the text to be processed to obtain a semantic feature space, wherein the semantic feature space includes semantic feature vectors of at least two text segments constituting the text to be processed.

[0159] The second extraction module is used to obtain scene feature spaces corresponding to at least two text processing scenarios, wherein the scene feature spaces include semantic feature vectors of processing content keywords corresponding to at least two text processing scenarios;

[0160] The processing module is used to determine the target text processing scenario based on the spatial features of the semantic feature space and the spatial features of at least two of the scene feature spaces, wherein the spatial features characterize the distribution features of the semantic feature vectors in the feature space.

[0161] According to one or more embodiments of this disclosure, the spatial features of the semantic feature space are first spatial features, which characterize the distribution characteristics of the similarity between each semantic feature vector in the semantic feature space and the semantic feature vector in the scene feature space; the spatial features of the scene feature space are second spatial features, which characterize the distribution characteristics of the similarity between each current semantic feature vector and other semantic feature vectors in the scene feature space, wherein the current semantic feature vector is a semantic feature vector sequentially obtained in the scene feature space, and the other semantic feature vectors are semantic feature vectors in the scene feature space other than the current semantic feature vector, and the distribution characteristics are the statistical results of the distribution of similarity corresponding to the current semantic feature vector; the processing module is specifically used to: perform an overall mean hypothesis test on the first spatial feature and the second spatial feature to obtain the target text processing scene to which the first spatial feature belongs under a preset confidence level.

[0162] According to one or more embodiments of this disclosure, the processing module is further configured to: sequentially obtain the cosine similarity between each semantic feature vector in the semantic feature space and the semantic feature vectors in each of the scene feature spaces; obtain at least two similarity sequences based on the cosine similarity between each semantic feature vector in the semantic feature space and the semantic feature vectors in each of the scene feature spaces; and obtain the first spatial feature based on the at least two similarity sequences.

[0163] According to one or more embodiments of this disclosure, the processing module is further configured to: sequentially obtain the cosine similarity between each semantic feature vector and other semantic feature vectors in the scene feature space; obtain a similarity measurement matrix based on the cosine similarity between each semantic feature vector and other semantic feature vectors; and obtain the second spatial feature based on the similarity measurement matrix.

[0164] According to one or more embodiments of this disclosure, when the processing module obtains the second spatial feature based on the similarity measurement matrix, it is specifically used to: divide the similarity measurement matrix into two feature regions based on the diagonal of the similarity measurement matrix; and for any feature region, perform statistics on all cosine similarities along the rows or columns to obtain the distribution characteristics of the similarity between different cosine similarities corresponding to the similarity measurement matrix.

[0165] According to one or more embodiments of this disclosure, the first extraction module is specifically used for: sliding division of the text to be processed based on the target window length to obtain at least two text segments; extracting features from the at least two text segments to obtain corresponding first semantic feature vectors; and obtaining the semantic feature space based on the at least two first semantic feature vectors.

[0166] According to one or more embodiments of this disclosure, when the first extraction module performs sliding division of the text to be processed based on the target window length to obtain at least two text segments, it is specifically used to: obtain a target overlap rate based on the average distance of the sentence break markers in the text to be processed; and perform sliding division of the plain text corresponding to the text to be processed based on the target overlap rate and the target window length to obtain at least two text segments, wherein the plain text is the text of the text to be processed after removing the sentence break markers.

[0167] According to one or more embodiments of this disclosure, the text to be processed includes main text and title text, wherein the title text is used to characterize the main content of the main text; when the first extraction module performs sliding division of the text to be processed based on the target window length to obtain at least two text segments, it is specifically used to: perform sliding division of the main text based on the target window length to obtain at least two text segments; when the first extraction module obtains the semantic feature space based on at least two first semantic feature vectors, it is specifically used to: obtain the second semantic feature vector corresponding to the title text; and obtain the semantic feature space based on the at least two first semantic feature vectors corresponding to the main text and the second semantic feature vector corresponding to the title text.

[0168] According to one or more embodiments of this disclosure, when the first extraction module extracts features from the at least two text segments to obtain the corresponding first semantic feature vector, it is specifically used to: generate a summary text corresponding to the text to be processed using a pre-trained generative model; and obtain the semantic feature vector of the text segment corresponding to the text to be processed based on the summary text.

[0169] According to one or more embodiments of this disclosure, the second extraction module is specifically configured to: acquire expert experience data corresponding to at least two text processing scenarios, wherein the expert experience data characterizes the text content distribution rules for the text processing scenarios; generate a third semantic feature vector characterizing keywords of the processed content based on the expert experience data; and obtain the scenario feature space based on the third semantic feature vector.

[0170] According to one or more embodiments of this disclosure, the second extraction module is specifically used to: perform data augmentation on the expert experience data to obtain augmented experience data; when the second extraction module generates a third semantic feature vector representing the keywords of the processed content based on the expert experience data, it is specifically used to: generate a third semantic feature vector representing the keywords of the processed content based on the augmented experience data.

[0171] According to one or more embodiments of this disclosure, the data enhancement includes at least one of the following: similar word replacement, synonym conversion, and syntax tree operation.

[0172] According to one or more embodiments of this disclosure, the second extraction module is further configured to: obtain labeled sample data corresponding to the text processing scenario, the labeled sample data including sample text and labeled information corresponding to the sample text, the labeled information being used to label processing content keywords in the sample text; generate a fourth semantic feature vector based on the labeled sample data; when the second extraction module 32 obtains the scenario feature space based on the third semantic feature vector, it is specifically configured to: obtain the scenario feature space based on the third semantic feature vector and the fourth semantic feature vector.

[0173] Thirdly, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor and a memory;

[0174] The memory stores computer-executed instructions;

[0175] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the text processing method as described in the first aspect and various possible designs of the first aspect.

[0176] Fourthly, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, and when a processor executes the computer-executable instructions, the text processing method described in the first aspect and various possible designs of the first aspect is implemented.

[0177] Fifthly, according to one or more embodiments of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the text processing method as described in the first aspect and various possible designs of the first aspect.

[0178] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0179] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0180] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A text processing method, characterized in that, include: The text to be processed is obtained, and features are extracted from the text to be processed to obtain a semantic feature space, which includes semantic feature vectors of at least two text segments constituting the text to be processed. Obtain scene feature spaces corresponding to at least two text processing scenarios, wherein the scene feature spaces include semantic feature vectors of the processing content keywords corresponding to at least two text processing scenarios; Obtain spatial features of the semantic feature space and spatial features of the scene feature space; the spatial features of the semantic feature space characterize the distribution characteristics of the similarity between each semantic feature vector in the semantic feature space and the semantic feature vector in the scene feature space, and the spatial features of the scene feature space characterize the distribution characteristics of the similarity between each current semantic feature vector and other semantic feature vectors in the scene feature space. For the spatial features of the semantic feature space and the spatial features of the scene feature space, an overall mean hypothesis test is performed to obtain the target text processing scene to which the spatial features of the semantic feature space belong under a preset confidence level.

2. The method according to claim 1, characterized in that, The current semantic feature vector is a semantic feature vector obtained sequentially in the scene feature space. The other semantic feature vectors are semantic feature vectors in the scene feature space other than the current semantic feature vector. The distribution feature is the statistical result of the distribution of similarity corresponding to the current semantic feature vector.

3. The method according to claim 2, characterized in that, The method further includes: The cosine similarity between each semantic feature vector in the scene feature space and other semantic feature vectors is obtained sequentially. The similarity metric matrix is ​​obtained based on the cosine similarity between each semantic feature vector and other semantic feature vectors. Based on the diagonal of the similarity measurement matrix, the similarity measurement matrix is ​​divided into two feature regions; For any of the aforementioned feature regions, all cosine similarities are statistically analyzed along rows or columns to obtain the distribution characteristics of the similarity between different cosine similarities corresponding to the similarity metric matrix; Based on the similarity metric matrix, the spatial features of the scene feature space are obtained.

4. The method according to claim 1, characterized in that, The step of extracting features from the text to be processed to obtain a semantic feature space includes: Based on the target window length, the text to be processed is divided into at least two text segments by sliding. Feature extraction is performed on the at least two text segments to obtain the corresponding first semantic feature vector; The semantic feature space is obtained based on at least two of the first semantic feature vectors.

5. The method according to claim 4, characterized in that, The text to be processed is divided into at least two text segments based on the target window length, including: The target overlap rate is obtained based on the average distance of the sentence break markers in the text to be processed; Based on the target overlap rate and the target window length, the plain text corresponding to the text to be processed is divided by sliding to obtain at least two text segments, wherein the plain text is the text of the text to be processed after removing the sentence segmentation marker.

6. The method according to claim 4, characterized in that, The text to be processed includes main text and title text, and the title text is used to characterize the main content of the main text. The text to be processed is divided into at least two text segments based on the target window length, including: Based on the target window length, the main text is divided into at least two text segments by sliding. The step of obtaining the semantic feature space based on at least two of the first semantic feature vectors includes: Obtain the second semantic feature vector corresponding to the title text; The semantic feature space is obtained based on at least two first semantic feature vectors corresponding to the main text and a second semantic feature vector corresponding to the title text.

7. The method according to claim 1, characterized in that, Obtain the text to be processed, and extract features from the text to obtain a semantic feature space, including: Using a pre-trained generative model, a summary text corresponding to the text to be processed is generated; Based on the abstract text, the semantic feature vector of the text segment corresponding to the text to be processed is obtained.

8. The method according to claim 1, characterized in that, The acquisition of scene feature spaces corresponding to at least two text processing scenarios includes: Obtain expert experience data corresponding to at least two text processing scenarios, wherein the expert experience data represents the text content distribution rules for the text processing scenarios; Based on the expert experience data, a third semantic feature vector representing the keywords of the processed content is generated; The scene feature space is obtained based on the third semantic feature vector.

9. The method according to claim 8, characterized in that, Also includes: Obtain labeled sample data corresponding to the text processing scenario. The labeled sample data includes sample text and labeling information corresponding to the sample text. The labeling information is used to label the processing content keywords in the sample text. Based on the labeled sample data, a fourth semantic feature vector is generated; The step of obtaining the scene feature space based on the third semantic feature vector includes: The scene feature space is obtained based on the third semantic feature vector and the fourth semantic feature vector.

10. A text processing device, characterized in that, include: The first extraction module is used to acquire the text to be processed and to extract features from the text to be processed to obtain a semantic feature space, wherein the semantic feature space includes semantic feature vectors of at least two text segments constituting the text to be processed. The second extraction module is used to obtain scene feature spaces corresponding to at least two text processing scenarios, wherein the scene feature spaces include semantic feature vectors of processing content keywords corresponding to at least two text processing scenarios; The processing module is used to obtain the spatial features of the semantic feature space and the spatial features of the scene feature space. The spatial features of the semantic feature space represent the distribution characteristics of the similarity between each semantic feature vector in the semantic feature space and the semantic feature vector in the scene feature space. The spatial features of the scene feature space represent the distribution characteristics of the similarity between each current semantic feature vector in the scene feature space and other semantic feature vectors. For the spatial features of the semantic feature space and the spatial features of the scene feature space, a general mean hypothesis test is performed to obtain the target text processing scene to which the spatial features of the semantic feature space belong under a preset confidence level.

11. An electronic device, characterized in that, include: Processor and memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the text processing method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the text processing method as described in any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the text processing method as described in any one of claims 1 to 9.

Images