Text splitting mark generation method and device, medium and electronic equipment

Abstract

The invention relates to the field of text processing, in particular to a text splitting mark generation method and device, a medium and electronic equipment. Comprising the following steps: obtaining summary information of a sub-text corresponding to each sliding window in a to-be-processed text; generating a semantic change amplitude between any two adjacent summary information; and when any semantic change amplitude is greater than a change threshold, generating a splitting mark at the tail position of the sliding window corresponding to the front summary information in two adjacent summary information. According to the method, a complete text is split into a plurality of sub-texts, then the change size between scenes described by two adjacent sub-texts is reflected through the cosine distance between semantic vectors corresponding to the two adjacent sub-texts, and the change size is used as the basis of text splitting, so that the whole to-be-processed text is reasonably split and marked. Therefore, clearer plot arrangement structures of works can be provided for creators, follow-up modification is facilitated, and meanwhile creation efficiency and quality are improved.

Description

Background Art

[0002] In recent years, literary works in electronic narrative form, such as literary texts, have gradually been loved and appreciated by a wider range of people. As described in the narrative framework of J. Hillis Miller, a good narrative text usually has clear character traits and a series of events that change over time. Among them, a more appropriate series of events that change over time, that is, scene changes, is the key to whether a literary work can provide sufficient appeal. In the prior art, the evaluation of whether the scene change settings in a completed literary text are reasonable is usually conducted subjectively in a human way, which will not be able to form a more objective and accurate evaluation opinion, and will not be able to assist the creator to more clearly understand the plot structure of the work, which is not conducive to subsequent modifications, and the evaluation efficiency is also relatively low. Summary of the invention

[0003] In view of the above technical problems, the technical solution adopted by the present invention is:

[0004] According to one aspect of the present invention, a method for generating text splitting tags is provided, the method comprising the following steps:

[0005] Obtain summary information of a subtext corresponding to each sliding window in the text to be processed; each subtext includes at least one complete sentence;

[0006] Generate the semantic change amplitude between any two adjacent summary information according to the cosine distance between the semantic vectors corresponding to any two adjacent summary information;

[0007] When the amplitude of any semantic change is greater than the change threshold, a split mark is generated at the tail position of the sliding window corresponding to the earlier summary information in two adjacent summary information;

[0008] The change threshold is obtained according to the following steps:

[0009] Performing clustering processing on the semantic vectors corresponding to all summary information to generate at least one cluster;

[0010] According to the cluster group to which each semantic vector belongs, a cluster label of the subtext corresponding to each semantic vector is generated;

[0011] According to the cluster proportions of the paragraphs to which the two sub-texts belong corresponding to each semantic change amplitude, a change threshold corresponding to each semantic change amplitude is generated; the change threshold is negatively correlated with the cluster proportion, and the cluster proportion is the ratio of the total number of cluster marker types possessed by the paragraphs to which the two sub-texts belong to and the total number of cluster marker types possessed by the processed text.

[0012] Furthermore, the summary information of the subtext corresponding to each sliding window in the text to be processed is obtained, including:

[0013] Call the summary extraction API in Cloud Natural Language to process each subtext obtained to generate corresponding summary information.

[0014] Furthermore, before obtaining the subtext corresponding to each sliding window in the text to be processed, the method further includes:

[0015] The sliding window slides to the initial window position according to the preset sliding step size;

[0016] Get the first character distance and the second character distance corresponding to the inner and outer sides of the tail of the initial window position respectively; the first character distance is the character distance between the sentence end identifier closest to the inner side of the tail of the initial window position and the tail of the initial window position; the second character distance is the character distance between the sentence end identifier closest to the outer side of the tail of the initial window position and the tail of the initial window position;

[0017] If the first character distance is less than or equal to the second character distance, the tail of the initial window position is moved to the position corresponding to the nearest end-of-sentence identifier inside to generate a target window position.

[0018] Furthermore, after obtaining the first character distance and the second character distance corresponding to the inner and outer sides of the tail of the initial window position, the method further includes:

[0019] If the second character distance is smaller than the first character distance, the tail of the initial window position is moved to the position corresponding to the nearest end-of-sentence identifier on the outside to generate the target window position.

[0020] Furthermore, sentence end identifiers include a period, a question mark, an exclamation mark, and an ellipsis.

[0021] Furthermore, according to the cosine distance between the semantic vectors corresponding to any two adjacent summary information, the semantic change amplitude between any two adjacent summary information is generated, including:

[0022] Through the BERT model, obtain the global semantic feature vector corresponding to the text to be processed;

[0023] Through the BERT model, the local semantic feature vectors corresponding to two adjacent summary information are obtained respectively;

[0024] The local semantic feature vector corresponding to each summary information is fused with the global semantic feature vector to generate a semantic vector corresponding to each summary information;

[0025] The cosine distance between the semantic vectors corresponding to two adjacent summary information is taken as the semantic change amplitude between the two adjacent summary information.

[0026] Furthermore, according to the cluster proportions of the paragraphs to which the two sub-texts corresponding to each semantic change amplitude belong, a change threshold corresponding to each semantic change amplitude is generated, including:

[0027] Get the mean μ corresponding to all semantic change amplitudes c And the distribution concentration σ c ;

[0028] According to the cluster proportions of the paragraphs to which the two sub-texts belong corresponding to each semantic change amplitude, an adjustment weight corresponding to each semantic change amplitude is generated; the adjustment weight is inversely proportional to the cluster proportion;

[0029] According to μ c , σ c and the adjustment weight corresponding to each semantic change amplitude, generating a change threshold corresponding to each semantic change amplitude;

[0030] Among them, the change threshold Yi corresponding to the i-th semantic change amplitude satisfies the following conditions:

[0031] Y=μ c +Ki×σ c ;

[0032] Ki is the adjustment weight corresponding to the i-th semantic change amplitude.

[0033] Furthermore, Ki satisfies the following conditions:

[0034]

[0035] Among them, G i is the total number of cluster marker types in the paragraphs of the two sub-texts corresponding to the i-th semantic change amplitude, G sum To handle the total number of cluster marker types that the text has, round() is a rounding function.

[0036] According to a second aspect of the present invention, there is provided a device for generating text splitting marks, the device comprising:

[0037] A summary acquisition module is used to acquire summary information of a subtext corresponding to each sliding window in the text to be processed; each subtext includes at least one complete sentence;

[0038] A semantic change amplitude generating module, used for generating a semantic change amplitude between any two adjacent summary information according to a cosine distance between semantic vectors corresponding to any two adjacent summary information;

[0039] A split mark generation module is used to generate a split mark at the tail position of the sliding window corresponding to the earlier summary information in two adjacent summary information when any semantic change amplitude is greater than the change threshold;

[0040] The split marker generation module is also used to obtain the change threshold according to the following steps:

[0041] Performing clustering processing on the semantic vectors corresponding to all summary information to generate at least one cluster;

[0042] According to the cluster group to which each semantic vector belongs, a cluster label of the subtext corresponding to each semantic vector is generated;

[0043] According to the cluster proportions of the paragraphs to which the two sub-texts belong corresponding to each semantic change amplitude, a change threshold corresponding to each semantic change amplitude is generated; the change threshold is negatively correlated with the cluster proportion, and the cluster proportion is the ratio of the total number of cluster marker types possessed by the paragraphs to which the two sub-texts belong to and the total number of cluster marker types possessed by the processed text.

[0044] According to a third aspect of the present invention, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned method for generating splitting tags for a text is implemented.

[0045] According to a fourth aspect of the present invention, there is provided an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method for generating splitting tags for a text when executing the computer program.

[0046] The present invention has at least the following beneficial effects:

[0047] In the present invention, a complete text is split into multiple sub-texts by sliding a window, and then the cosine distance (i.e., the amplitude of semantic change) between the semantic vectors corresponding to two adjacent sub-texts is calculated to evaluate the semantic change between the two sub-texts, thereby reflecting the change between the scenes described by the two adjacent sub-texts, and using this as the basis for text splitting to reasonably split and mark the entire text to be processed. The split text will be marked at each position where there is an obvious plot change, thereby providing the creator with a clearer plot layout structure of the work, which is conducive to subsequent modifications and improves the creation efficiency and quality.

[0048] At the same time, the present invention further highlights and clarifies the content to be expressed by each sub-text by extracting the sub-text summary. In order to obtain a more accurate semantic vector through the summary information, the accuracy of the subsequent clustering process can be improved. And by extracting the summary, the amount of data to be processed for the subsequent text vector encoding can be further reduced to improve the processing efficiency.

[0049] In addition, the change threshold used to determine the magnitude of the semantic change amplitude in this embodiment will also be dynamically reversed according to the cluster proportion of the paragraph to which the two sub-texts currently belong. Since the cluster proportion in this embodiment can reflect the richness of the change in the description content in the paragraph to which the two sub-texts currently belong (that is, the richness of the semantic change), when the cluster proportion is low, it means that the semantic change of the description text currently involved is small, and the change threshold needs to be increased accordingly to avoid excessive splitting of content with similar semantics. When the cluster proportion is high, it means that the semantic change of the description text currently involved is large, and the change threshold needs to be lowered accordingly to split the description content more carefully to avoid missing splitting. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0051] Figure 1 A flowchart of a method for generating text splitting tags provided by an embodiment of the present invention.

[0052] Figure 2 A structural block diagram of a device for generating text splitting marks provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0054] As a possible embodiment of the present invention, Figure 1 As shown, a method for generating text splitting tags is provided, and the method comprises the following steps:

[0055] S100: Obtain summary information of a subtext corresponding to each sliding window in the text to be processed. Each subtext includes at least one complete sentence.

[0056] Specifically, S100 includes:

[0057] S101: calling the summary extraction API in Cloud Natural Language to process each acquired subtext to generate corresponding summary information.

[0058] The main purpose of this step is to extract the summary information of each sub-text to prepare for the subsequent generation of the semantic vector corresponding to each sub-text. Extracting the summary information of the text is a common application scenario in natural language processing (NLP). Many existing APIs can achieve the above functions, such as the Google Cloud Natural Language API. In addition, in this embodiment, in order to ensure the integrity of the meaning expressed by each sub-text, it is necessary to limit each sub-text to consist of complete sentences.

[0059] Before S100, the method further includes:

[0060] S110: The sliding window slides to an initial window position according to a preset sliding step length.

[0061] In this embodiment, the size of the sliding window and the size of the sliding step can be determined according to the writing habits of the text in the actual application scenario. For example, for some texts with more intense conflicts, the size of the sliding window and the sliding step can be smaller values. For example, for suspense, reasoning, science fiction and war texts, the size of the sliding window can be 100 characters, and the sliding step can be 20 characters. For some texts with more gentle conflicts, the size of the sliding window and the sliding step can be larger values. For example, for family-like, biography, and natural description texts, the size of the sliding window can be 300 characters, and the sliding step can be 50 characters.

[0062] S120: Obtain the first character distance and the second character distance corresponding to the inner and outer sides of the tail of the initial window position. The first character distance is the character distance between the sentence end identifier closest to the inner side of the tail of the initial window position and the tail of the initial window position. The second character distance is the character distance between the sentence end identifier closest to the outer side of the tail of the initial window position and the tail of the initial window position. Specifically, the sentence end identifier includes a period, a question mark, an exclamation mark, and an ellipsis.

[0063] S130: If the first character distance is less than or equal to the second character distance, the tail of the initial window position is moved to a position corresponding to the nearest end-of-sentence identifier inside to generate a target window position.

[0064] S140: If the second character distance is smaller than the first character distance, the tail of the initial window position is moved to a position corresponding to the nearest end-of-sentence identifier on the outside to generate a target window position.

[0065] In order to ensure that each subtext selected by the sliding window is composed of sentences expressing complete meanings, the sliding window needs to be expanded or reduced according to the position of the sliding window tail during the movement of the sliding window. In this embodiment, by comparing the number of characters between the tail of the initial window position and the nearest sentence end identifiers on both sides thereof, it is determined which side of the sentence the tail of the initial window position is closer to, and then the tail of the initial window position is moved to the nearest sentence end in the bureau.

[0066] Since, in the process of writing a text, different plot change degrees (i.e., semantic change degrees) are set in different parts according to actual needs, in order to more accurately and quickly split different parts of the text into sub-texts of more appropriate lengths, the size of the sliding window and / or the size of the sliding step need to be adaptively adjusted.

[0067] In a possible embodiment of the present invention, the size of the sliding window and the size of the sliding step may be determined according to the following steps:

[0068] S150: Generate a next initial adjacent window corresponding to the current sliding window according to the initial sliding step length.

[0069] S160: Obtaining semantic change level identifiers corresponding to the text paragraphs to which the two sub-texts selected by the current sliding window and the next initial adjacent window belong respectively.

[0070] In this embodiment, the user needs to mark the semantic change speed of each text paragraph in the processed text in advance, that is, mark the semantic change level mark for each text paragraph. Specifically, the judgment standard of the semantic change level mark can be manually set according to the actual text usage scenario.

[0071] S170: If the semantic change level identifiers corresponding to the text paragraphs to which the two sub-texts belong are the same, then the sliding step length and / or the size of the sliding window corresponding to the current sliding window is obtained according to the semantic change level identifier.

[0072] In this embodiment, if the semantic change represented by the semantic change level identifier is faster, the sliding step size and / or sliding window size corresponding to the semantic change level identifier is larger. Conversely, if the semantic change represented by the semantic change level identifier is slower, the sliding step size and / or sliding window size corresponding to the semantic change level identifier is smaller. Specifically, the corresponding relationship can be determined by setting a mapping table. In this embodiment, the sliding step size and / or sliding window size are dynamically adjusted according to the speed of the semantic change, so that sub-texts can be extracted in a more sparse manner in texts with slower semantic changes, and sub-texts can be extracted in a more dense manner in texts with faster semantic changes, so as to facilitate subsequent text splitting.

[0073] S200: Generate a semantic change amplitude between any two adjacent summary information according to the cosine distance between the semantic vectors corresponding to any two adjacent summary information.

[0074] Specifically, S200 includes:

[0075] S201: Obtain a global semantic feature vector corresponding to the text to be processed through the BERT model.

[0076] If the length of the text to be processed exceeds the maximum input length, the text to be processed needs to be segmented and concatenated. The corresponding global semantic feature vector can be the vector corresponding to the [CLS] tag in the BERT model output, or the average vector of the entire article.

[0077] S202: Obtain local semantic feature vectors corresponding to two adjacent summary information respectively through the BERT model.

[0078] S203: Fusing the local semantic feature vector corresponding to each summary information with the global semantic feature vector to generate a semantic vector corresponding to each summary information.

[0079] The fusion processing in this step can be vector concatenation, weighted summation or attention mechanism.

[0080] In this embodiment, in order to ensure that each sub-text not only contains local information but also reflects the influence of the global context after vector encoding, the local semantic feature vector corresponding to a single summary information is fused with the global semantic feature vector corresponding to the text to be processed to generate a semantic vector corresponding to the summary information.

[0081] S204: taking the cosine distance between the semantic vectors corresponding to two adjacent summary information as the semantic change amplitude between the two adjacent summary information.

[0082] S300: When any semantic change amplitude is greater than a change threshold, a split mark is generated at the tail position of the sliding window corresponding to the earlier summary information in two adjacent summary information.

[0083] The change threshold is obtained according to the following steps:

[0084] S301: performing clustering processing on the semantic vectors corresponding to all summary information to generate at least one cluster.

[0085] The clustering method used in this embodiment can be a clustering method that requires the number of clusters to be specified in advance, such as the K-Means clustering method. The clustering in this embodiment is actually a classification based on the actual description content represented by each summary information, so the creator of the text can usually know how many types of content description representations there are in the entire text to be processed, so the number of clusters K can be specified by the creator of the text. In addition, the number of clusters K can also be determined based on information such as the writing outline of the text.

[0086] You can also use clustering methods that do not require the number of clusters to be specified in advance, such as DBSCAN (Density-Based Spatial Clustering of Applications with Noise), which is a density-based clustering algorithm that can find clusters of any shape and identify noise points. Or Affinity Propagation, which is a message-passing-based clustering algorithm that determines cluster centers by exchanging similarity information between data points.

[0087] S302: Generate a cluster label of the subtext corresponding to each semantic vector according to the cluster group to which each semantic vector belongs.

[0088] S303: Generate a change threshold corresponding to each semantic change amplitude according to the cluster proportion of the paragraphs to which the two sub-texts belong corresponding to each semantic change amplitude. The change threshold is negatively correlated with the cluster proportion, which is the ratio of the total number of cluster marker types in the paragraphs to which the two sub-texts belong to and the total number of cluster marker types in the processed text.

[0089] In this embodiment, the change threshold used to determine the magnitude of the semantic change amplitude will also be dynamically reversed according to the cluster proportion of the paragraph to which the two sub-texts currently belong. Since the cluster proportion in this embodiment can reflect the richness of the change in the description content in the paragraph to which the two sub-texts currently belong (that is, the richness of the semantic change), when the cluster proportion is low, it means that the semantic change of the description text currently involved is small, and the change threshold needs to be increased accordingly to avoid excessive splitting of content with similar semantics. When the cluster proportion is high, it means that the semantic change of the description text currently involved is large, and the change threshold needs to be lowered accordingly to split the description content more carefully to avoid missing splitting.

[0090] The text splitting mark generation method in this embodiment can be applied to the field of related intelligent services for vehicles. In the audio playback application of the vehicle, there will be a huge amount of audio and video files corresponding to the electronic text. The text corresponding to these audio and video files can be split and marked using the splitting mark method in this method, and the plot setting characteristics of the corresponding audio and video content can be judged according to the density of these splitting marks, and then corresponding recommendations can be made according to the preferences of the car owner.

[0091] In addition, it can also be used in the voice recognition service of the vehicle cabin system. Specifically, the voice recognition module in the vehicle can convert the long voice (such as chatting) uttered by the people in the car into corresponding text, and then segment the corresponding text through the splitting and marking method of this method. The emotional tendency of the current occupants or the atmosphere in the car can be judged according to the density of these splitting marks, and then relevant services can be recommended accordingly.

[0092] The text splitting mark generation method in this embodiment can also be applied to the field of text creation. The text creator can use the completed draft as the text to be processed in this embodiment, and split and mark it according to the method in this embodiment to help the creator adjust and modify the text more objectively and quickly.

[0093] Of course, the text splitting mark generation method in this embodiment can be applied to the fields of education and training or news media text review.

[0094] Its application in education and training is mainly reflected in the ability to help teachers efficiently evaluate the scenario setting structure of students' composition writing, thereby saving time and improving the objectivity of the evaluation, and providing corresponding targeted guidance to students' writing.

[0095] Its application in the field of journalism is mainly reflected in the ability to quickly analyze the scenario setting structure of large amounts of news texts, helping news editors or media organizations or related readers to understand and classify news information more efficiently.

[0096] As another possible embodiment of the present invention, specifically, S303 includes:

[0097] S313: Obtain the mean μ corresponding to all semantic change amplitudes c And the distribution concentration σ c .

[0098] After S200 processing, all the semantic change amplitudes included in the text to be processed can be obtained, based on which μ can be calculated. c and σ c In this embodiment, σ c It can be the standard deviation of all semantic change amplitudes.

[0099] S323: Generate an adjustment weight corresponding to each semantic change amplitude according to the cluster proportion of the paragraphs to which the two subtexts belong corresponding to each semantic change amplitude. The adjustment weight is inversely proportional to the cluster proportion.

[0100] S333: According to μ c , σ c and the adjustment weight corresponding to each semantic change amplitude, and generate the change threshold corresponding to each semantic change amplitude.

[0101] Among them, the change threshold Yi corresponding to the i-th semantic change amplitude satisfies the following conditions:

[0102] Yi=μ c +Ki×σ c .

[0103] Ki is the adjustment weight corresponding to the i-th semantic change amplitude.

[0104] Furthermore, Ki satisfies the following conditions:

[0105]

[0106] Among them, G i is the total number of cluster marker types in the paragraphs of the two sub-texts corresponding to the i-th semantic change amplitude, G sum To handle the total number of cluster marker types that the text has, round() is a rounding function.

[0107] In this embodiment, since the value range of Ki is -1 to 1, Yi can be adjusted in a relatively large range. i As increases, Ki decreases gradually, which makes Yi also decrease, in order to meet the above-mentioned characteristics of negative correlation between the change threshold and the cluster proportion.

[0108] At the same time, Ki is not a linearly changing state. Specifically, when G i ≤G sum When Ki increases with G i The increase of G i >G sum When G i More than G sum When the semantic change of the paragraphs to which the two sub-texts belong exceeds a certain threshold, Ki is always the smallest. That is, when the semantic change of the paragraphs to which the two sub-texts belong exceeds a certain threshold, it indicates that the content of the part has a relatively fast semantic change, so Ki should be the smallest. Therefore, the change threshold Yi can also be adjusted more accurately.

[0109] As another possible embodiment of the present invention, Figure 2As shown, a device for generating text splitting marks is also provided, the device comprising:

[0110] A summary acquisition module is used to acquire summary information of a subtext corresponding to each sliding window in the text to be processed; each subtext includes at least one complete sentence;

[0111] A semantic change amplitude generating module, used for generating a semantic change amplitude between any two adjacent summary information according to a cosine distance between semantic vectors corresponding to any two adjacent summary information;

[0112] A split mark generation module is used to generate a split mark at the tail position of the sliding window corresponding to the earlier summary information in two adjacent summary information when any semantic change amplitude is greater than the change threshold;

[0113] The split marker generation module is also used to obtain the change threshold according to the following steps:

[0114] Performing clustering processing on the semantic vectors corresponding to all summary information to generate at least one cluster;

[0115] According to the cluster group to which each semantic vector belongs, a cluster label of the subtext corresponding to each semantic vector is generated;

[0116] According to the cluster proportions of the paragraphs to which the two sub-texts belong corresponding to each semantic change amplitude, a change threshold corresponding to each semantic change amplitude is generated; the change threshold is negatively correlated with the cluster proportion, and the cluster proportion is the ratio of the total number of cluster marker types possessed by the paragraphs to which the two sub-texts belong to and the total number of cluster marker types possessed by the processed text.

[0117] Furthermore, the summary acquisition module is used to perform the following steps:

[0118] Call the summary extraction API in Cloud Natural Language to process each subtext obtained to generate corresponding summary information.

[0119] Furthermore, the device further includes a sliding window determination module, which is used to perform the following steps:

[0120] The sliding window slides to the initial window position according to the preset sliding step size;

[0121] Get the first character distance and the second character distance corresponding to the inner and outer sides of the tail of the initial window position respectively; the first character distance is the character distance between the sentence end identifier closest to the inner side of the tail of the initial window position and the tail of the initial window position; the second character distance is the character distance between the sentence end identifier closest to the outer side of the tail of the initial window position and the tail of the initial window position;

[0122] If the first character distance is less than or equal to the second character distance, the tail of the initial window position is moved to the position corresponding to the nearest end-of-sentence identifier inside to generate a target window position.

[0123] Furthermore, after obtaining the first character distance and the second character distance corresponding to the inner and outer sides of the tail of the initial window position, the sliding window determination module is further used to perform the following steps:

[0124] If the second character distance is smaller than the first character distance, the tail of the initial window position is moved to the position corresponding to the nearest end-of-sentence identifier on the outside to generate the target window position.

[0125] Furthermore, sentence end identifiers include a period, a question mark, an exclamation mark, and an ellipsis.

[0126] Furthermore, the semantic change amplitude generation module is used to perform the following steps:

[0127] Through the BERT model, obtain the global semantic feature vector corresponding to the text to be processed;

[0128] Through the BERT model, the local semantic feature vectors corresponding to two adjacent summary information are obtained respectively;

[0129] The local semantic feature vector corresponding to each summary information is fused with the global semantic feature vector to generate a semantic vector corresponding to each summary information;

[0130] The cosine distance between the semantic vectors corresponding to two adjacent summary information is taken as the semantic change amplitude between the two adjacent summary information.

[0131] Furthermore, the semantic change amplitude generation module is also used to perform the following steps:

[0132] Get the mean μ corresponding to all semantic change amplitudes c And the distribution concentration σ c ;

[0133] According to the cluster proportions of the paragraphs to which the two sub-texts belong corresponding to each semantic change amplitude, an adjustment weight corresponding to each semantic change amplitude is generated; the adjustment weight is inversely proportional to the cluster proportion;

[0134] According to μ c , σ c and the adjustment weight corresponding to each semantic change amplitude, generating a change threshold corresponding to each semantic change amplitude;

[0135] Among them, the change threshold Yi corresponding to the i-th semantic change amplitude satisfies the following conditions:

[0136] Y=μ c +Ki×σc ;

[0137] Ki is the adjustment weight corresponding to the i-th semantic change amplitude.

[0138] Furthermore, Ki satisfies the following conditions:

[0139]

[0140] Among them, G i is the total number of cluster marker types in the paragraphs of the two sub-texts corresponding to the i-th semantic change amplitude, G sum To handle the total number of cluster marker types that the text has, round() is a rounding function.

[0141] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0142] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0143] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0144] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0145] The electronic device according to this embodiment of the present invention is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0146] The electronic device is presented in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: the at least one processor mentioned above, the at least one storage device mentioned above, and a bus connecting different system components (including storage devices and processors).

[0147] The storage stores program codes, which can be executed by the processor, so that the processor executes the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification.

[0148] The memory may include readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory, and may further include read only memory (ROM).

[0149] The storage may also include a program / utility having a set (at least one) of program modules, such program modules including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0150] The bus may represent one or more of several types of bus structures including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures.

[0151] The electronic device may also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may communicate with one or more devices that enable a user to interact with the electronic device, and / or may communicate with any device (e.g., routers, modems, etc.) that enables the electronic device to communicate with one or more other computing devices. Such communication may be performed through an input / output (I / O) interface. Furthermore, the electronic device may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter. The network adapter communicates with other modules of the electronic device through a bus. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0152] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0153] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present invention may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0154] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0155] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0156] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0157] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0158] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0159] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0160] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for generating text splitting tags, characterized in that: The method comprises the following steps: Obtain summary information of a subtext corresponding to each sliding window in the text to be processed; each subtext includes at least one complete sentence; Generate a semantic change amplitude between any two adjacent summary information according to the cosine distance between the semantic vectors corresponding to any two adjacent summary information; When any semantic change amplitude is greater than the change threshold, a split mark is generated at the tail position of the sliding window corresponding to the earlier summary information in two adjacent summary information; The change threshold is obtained according to the following steps: Performing clustering processing on the semantic vectors corresponding to all summary information to generate at least one cluster; According to the cluster group to which each semantic vector belongs, a cluster label of the subtext corresponding to each semantic vector is generated; According to the cluster proportion of the paragraphs to which the two sub-texts belong corresponding to each semantic change amplitude, a change threshold corresponding to each semantic change amplitude is generated; the change threshold is negatively correlated with the cluster proportion, and the cluster proportion is the ratio of the total number of cluster mark types possessed by the paragraphs to which the two sub-texts belong to and the total number of cluster mark types possessed by the processed text.

2. The method according to claim 1, characterized in that Before obtaining the subtext corresponding to each sliding window in the text to be processed, the method further includes: The sliding window slides to the initial window position according to the preset sliding step size; Obtain the first character distance and the second character distance corresponding to the inner and outer sides of the tail of the initial window position, respectively; the first character distance is the character distance between the closest end-of-sentence identifier on the inner side of the tail of the initial window position and the tail of the initial window position; the second character distance is the character distance between the closest end-of-sentence identifier on the outer side of the tail of the initial window position and the tail of the initial window position; If the first character distance is less than or equal to the second character distance, the tail of the initial window position is moved to the position corresponding to the nearest inner end-of-sentence identifier to generate a target window position.

3. The method according to claim 2, characterized in that After obtaining the first character distance and the second character distance corresponding to the inner and outer sides of the tail of the initial window position, the method further includes: If the second character distance is smaller than the first character distance, the tail of the initial window position is moved to the position corresponding to the nearest end-of-sentence identifier on the outside to generate a target window position.

4. The method according to claim 3, characterized in that Sentence-ending markers include period, question mark, exclamation mark, and ellipsis.

5. The method according to claim 1, characterized in that Generating a semantic change amplitude between any two adjacent summary information according to a cosine distance between semantic vectors corresponding to any two adjacent summary information includes: Obtaining a global semantic feature vector corresponding to the text to be processed through the BERT model; Through the BERT model, the local semantic feature vectors corresponding to two adjacent summary information are obtained respectively; The local semantic feature vector corresponding to each summary information is fused with the global semantic feature vector to generate a semantic vector corresponding to each summary information; The cosine distance between the semantic vectors corresponding to two adjacent summary information is used as the semantic change amplitude between the two adjacent summary information.

6. The method according to claim 1, characterized in that According to the cluster proportions of the paragraphs to which the two sub-texts belong corresponding to each semantic change amplitude, a change threshold corresponding to each semantic change amplitude is generated, including: Get the mean μ corresponding to all semantic change amplitudes c And the distribution concentration σ c ; According to the cluster proportions of the paragraphs to which the two subtexts belong corresponding to each semantic change amplitude, an adjustment weight corresponding to each semantic change amplitude is generated; the adjustment weight is inversely proportional to the cluster proportion; According to μ c , σ c and the adjustment weight corresponding to each semantic change amplitude, generating a change threshold corresponding to each semantic change amplitude; Among them, the change threshold Yi corresponding to the i-th semantic change amplitude satisfies the following conditions: Y=μ c +Ki×σ c ; Ki is the adjustment weight corresponding to the i-th semantic change amplitude.

7. The method according to claim 6, characterized in that Ki satisfies the following conditions: Among them, G i is the total number of cluster marker types in the paragraphs of the two sub-texts corresponding to the i-th semantic change amplitude, G sum To handle the total number of cluster marker types that the text has, round() is a rounding function.

8. A text splitting mark generation device, characterized in that: The device comprises: A summary acquisition module is used to acquire summary information of a subtext corresponding to each sliding window in the text to be processed; each subtext includes at least one complete sentence; A semantic change amplitude generating module, used for generating a semantic change amplitude between any two adjacent summary information according to a cosine distance between semantic vectors corresponding to any two adjacent summary information; A split mark generating module, configured to generate a split mark at the tail position of the sliding window corresponding to the earlier summary information in two adjacent summary information when any semantic change amplitude is greater than a change threshold; The split mark generation module is further used to obtain the change threshold according to the following steps: Performing clustering processing on the semantic vectors corresponding to all summary information to generate at least one cluster; According to the cluster group to which each semantic vector belongs, a cluster label of the subtext corresponding to each semantic vector is generated; According to the cluster proportion of the paragraphs to which the two sub-texts belong corresponding to each semantic change amplitude, a change threshold corresponding to each semantic change amplitude is generated; the change threshold is negatively correlated with the cluster proportion, and the cluster proportion is the ratio of the total number of cluster mark types possessed by the paragraphs to which the two sub-texts belong to and the total number of cluster mark types possessed by the processed text.

9. A non-transitory computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for generating text splitting tags according to any one of claims 1 to 7 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for generating text splitting tags according to any one of claims 1 to 7 is implemented.

Images