Vector length variable-based text content intelligent fragmentation method, medium and equipment
Through the intelligent sharding method of text content based on variable vector length, the problem of incomplete sentences in the prior art is solved, and more accurate and operable text slices are achieved, ensuring the integrity and semantic retention of sentences.
Patent Information
- Application Number
- CN202411281334.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-13
- Publication Date
- 2024-12-13
AI Technical Summary
The existing text slicing method can easily lead to incomplete sentences and lack of semantics, increasing the difficulty of subsequent operations and reducing the accuracy of the results.
Using the intelligent sharding method of text content based on variable vector length, we determine the initial vector length by obtaining the total length and slicing accuracy of the text content, determine whether the character at the end of the slicing is a rest, and slice it with the current text according to the statement after the segment. The correlation between them is segmented and adjusted to ensure sentence integrity.
The problem of sentence truncation is solved, the accuracy of the results and the operability of sharding are improved, and the integrity and semantic retention of the sentence are ensured.
Smart Images
Figure CN119129594A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a method, medium and device for intelligently slicing text content based on variable vector length. Background Art
[0002] Currently, long text matching based on word meaning involves segmenting the long text into several relatively short texts. The traditional segmentation method is to simply split the long text according to a fixed length. This segmentation method will truncate the complete sentence, resulting in incomplete sentences and loss of semantics, increasing the difficulty of subsequent operations and reducing the accuracy of the results. Summary of the invention
[0003] In view of the above problems, the present application provides a technical solution for intelligent segmentation of text content with variable vector length to solve the problem that the existing text slicing method easily leads to incomplete sentences and missing content.
[0004] To achieve the above objectives, in a first aspect, the present application provides a text content intelligent segmentation method based on variable vector length, the method comprising:
[0005] Get the text content and determine the initial vector length based on the total length of the text content and the segmentation accuracy;
[0006] The text content is segmented according to the length of the initial vector, and it is determined whether the character at the end of the segmentation is a rest. If so, a text slice is obtained. If not, the text slice is obtained by extending backward from the current segmentation end to the first rest.
[0007] Furthermore, after each segmentation to obtain a text slice, the following steps are performed for the text slice:
[0008] Acquire a plurality of post-paragraph statements after the current text slice T, and analyze the relevance between the plurality of post-paragraph statements and the current text slice T and the relevance between the plurality of post-paragraph statements;
[0009] Determine a segmentation adjustment strategy according to the strength of the correlation between the plurality of post-paragraph sentences and the current text slice T and the strength of the correlation between the plurality of post-paragraph sentences;
[0010] The current text slice T is segmented and adjusted according to the segmentation adjustment strategy, and the segmentation pointer position is updated to segment the next text slice starting from the updated segmentation pointer position.
[0011] Further, the plurality of post-paragraph statements include a first post-paragraph statement H1 and a second post-paragraph statement H2, and the correlation between T and H1 is R1, the correlation between H1 and H2 is R2, and the correlation between T and H2 is R3;
[0012] Determining a segmentation adjustment strategy according to the strength of the correlation between the post-segment statements and the current text slice T and the strength of the correlation between the post-segment statements, performing segmentation adjustment on the current text slice T according to the segmentation adjustment strategy, and updating the segmentation pointer position includes:
[0013] If it is determined that R1 is not a strong association relationship, the current text slice T is kept unchanged, and the segmentation pointer position is updated to the end position of the current text slice T;
[0014] If it is determined that R1 is a strong association relationship but R2 is a non-strong association relationship, H1 is included in the current text slice T, and the segmentation pointer position is updated to the end position of H1;
[0015] If it is determined that R1 and R2 are both strongly associated but R3 is not strongly associated, then both H1 and H2 are included in the current text slice T, and the segmentation pointer position is updated to the end position of H1;
[0016] If it is determined that R1, R2, and R3 are all strongly associated, H1 and H2 are both included in the current text slice T, and the segmentation pointer position is updated to the end position of H2.
[0017] Furthermore, before including H1 and H2 in the current text slice T, the following steps are also included:
[0018] Determine whether the sum of the lengths of H1, H2, and text slice T exceeds the preset length limit. If so, merge H1 and H2 and slice them independently.
[0019] Furthermore, the method further comprises:
[0020] After the next text slice is generated, determine the correlation between the merged segment of H1 and H2 and the next text slice. If the correlation between the two is not strong, keep H1 and H2 as an independent slice after merging. If the overall correlation is strong, determine whether the sum of the lengths of H1, H2 and the next text slice exceeds the preset length limit. If not, merge H1 and H2 into the next text slice. If so, keep H1 and H2 as an independent slice after merging.
[0021] Furthermore, if it is determined that R1, R2, and R3 are all strongly associated, before H1 and H2 are included in the current text slice T, the following steps are also included:
[0022] Obtain several end-of-paragraph sentences located at the end of the current text slice T, analyze the correlation between the several end-of-paragraph sentences and H1 and the correlation between the several end-of-paragraph sentences respectively, and determine the segmentation adjustment strategy according to the correlation between the several end-of-paragraph sentences and H1 and the correlation between the several end-of-paragraph sentences, and decide whether to include H1 and H2 in the current text slice T according to the segmentation adjustment strategy.
[0023] Furthermore, the plurality of end-of-segment statements include the end-of-first-segment statement Q1 and the end-of-second-segment statement Q2, and the correlation between Q1 and H1 is R1', the correlation between Q1 and Q2 is R5', and the correlation between Q2 and H1 is R6';
[0024] Determining a segmentation adjustment strategy according to the correlation between the end-of-segment sentences and H1 and the correlation between the end-of-segment sentences, and determining whether to include H1 and H2 in the current text slice T according to the segmentation adjustment strategy includes:
[0025] If it is determined that R1' is not a strong association, the current text slice T is kept unchanged, H1 and H2 are merged and sliced independently, and the segmentation pointer position is updated to the end position of the current text slice T;
[0026] If it is determined that R1' is a strong correlation but R2' is a non-strong correlation, Q1 is removed from the current text slice T, Q1, H1 and H2 are merged and sliced independently, and the segmentation pointer position is updated to the end position of H2;
[0027] If it is determined that R1' and R2' are both strongly associated but R3' is not strongly associated, the current text slice T is kept unchanged, H1, H2, Q1 and Q2 are merged and sliced independently, and the segmentation pointer position is updated to the end position of H2, or Q1 and Q2 are removed from the current text slice T, Q1, Q2, H1 and H2 are merged and sliced independently, and the segmentation pointer position is updated to the end position of H2;
[0028] If it is determined that R1', R2', and R3' are all strongly associated, H1 and H2 are both included in the current text slice T, and the segmentation pointer position is updated to the end position of H2.
[0029] Furthermore, the method comprises:
[0030] Input the text slices into the trained neural network model in sequence, and output the labels corresponding to the text slices;
[0031] The labels are mapped to the respective text slices and stored.
[0032] In a second aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the text content intelligent segmentation method based on variable vector length as described in the first aspect of the present application.
[0033] In a third aspect, the present application provides an electronic device having a computer program stored thereon, including a processor and a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by the processor, the text content intelligent segmentation method based on variable vector length as described in the first aspect of the present application is implemented.
[0034] Different from the prior art, the above technical solution involves a text content intelligent segmentation method, medium and device based on variable vector length, the method comprising: obtaining text content, determining the initial vector length according to the total length of the text content and the segmentation accuracy; segmenting the text content according to the initial vector length, judging whether the character at the end of the segmentation is a rest, if so, obtaining a text slice, otherwise extending backward from the current segmentation end to the first rest to obtain a text slice. The above segmentation method can solve the problem of sentence truncation caused by the prior art, facilitate the use of subsequent segmentation, and improve the accuracy of the results and the operability of the segmentation.
[0035] The above-mentioned records related to the invention content are only an overview of the technical solution of the present application. In order to enable ordinary technicians in the field to more clearly understand the technical solution of the present application, and then implement it according to the text of the specification and the contents recorded in the drawings, and to make the above-mentioned purpose and other purposes, features and advantages of the present application easier to understand, the following is an explanation in combination with the specific implementation mode and drawings of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] The drawings are only used to illustrate the principles, implementation methods, applications, characteristics and effects of the specific embodiments of the present invention and other related contents, and shall not be considered as limiting the present application.
[0037] In the drawings of the specification:
[0038] Figure 1 This is a flowchart of the text content intelligent segmentation method based on variable vector length described in the first exemplary embodiment of the present application;
[0039] Figure 2 This is a flow chart of the text content intelligent segmentation method based on variable vector length described in the second exemplary embodiment of the present application;
[0040] Figure 3 This is a flowchart of the text content intelligent segmentation method based on variable vector length described in the third exemplary embodiment of the present application;
[0041] Figure 4 A schematic diagram of text slicing involved in the first exemplary embodiment of the present application;
[0042] Figure 5 A schematic diagram of an electronic device according to an exemplary embodiment of the present application;
[0043] The reference numerals in the above drawings are described as follows:
[0044] 10. Electronic equipment;
[0045] 101. Processor;
[0046] 102. Storage medium. DETAILED DESCRIPTION
[0047] In order to explain in detail the possible application scenarios, technical principles, specific schemes that can be implemented, and the purposes and effects that can be achieved, the following is a detailed description of the specific embodiments listed in conjunction with the accompanying drawings. The embodiments described herein are only used to more clearly illustrate the technical solutions of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.
[0048] Reference to "embodiment" herein means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The term "embodiment" appearing in various places in the specification does not necessarily refer to the same embodiment, nor does it particularly limit its independence or association with other embodiments. In principle, in the present application, as long as there is no technical contradiction or conflict, the various technical features mentioned in the embodiments can be combined in any way to form a corresponding implementable technical solution.
[0049] Unless otherwise defined, the technical terms used in this document have the same meanings as those generally understood by those skilled in the art to which this application belongs; the use of relevant terms in this document is only for describing specific embodiments and is not intended to limit this application.
[0050] In the description of this application, the term "and / or" is an expression used to describe the logical relationship between objects, indicating that three relationships may exist, for example, A and / or B, which means: A exists, B exists, and A and B exist at the same time. In addition, the character " / " in this article generally indicates that the objects before and after are in an "or" logical relationship.
[0051] In the present application, terms such as “first” and “second” are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship of quantity, priority or sequence between these entities or operations.
[0052] Without further limitations, in this application, the words "include", "comprises", "has" or other similar open-ended expressions used in sentences are intended to cover non-exclusive inclusion. These expressions do not exclude the presence of additional elements in the process, method or product including the elements, so that the process, method or product including a series of elements may include not only those limited elements, but also other elements not explicitly listed, or also include elements inherent to such process, method or product.
[0053] Similar to the understanding in the Examination Guidelines, in this application, expressions such as "greater than", "less than", "exceed" and the like are understood to exclude the number itself; expressions such as "above", "below", "within" and the like are understood to include the number itself. In addition, in the description of the embodiments of this application, "multiple" means more than two (including two), and similar expressions related to "multiple" are also understood in this way, such as "multiple groups", "multiple times", etc., unless otherwise clearly and specifically limited.
[0054] In the description of the embodiments of the present application, space-related expressions used, such as "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "vertical", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc., indicate the orientation or position relationship based on the orientation or position relationship shown in the specific embodiments or drawings, and are only for the convenience of describing the specific embodiments of the present application or facilitating the reader's understanding, and do not indicate or imply that the referred device or component must have a specific position, a specific orientation, or be constructed or operated in a specific orientation, and therefore cannot be understood as a limitation on the embodiments of the present application.
[0055] Unless otherwise expressly specified or limited, in the description of the embodiments of the present application, the terms such as "install", "connect", "connect", "fix", "set", etc. used should be understood in a broad sense. For example, the "connection" can be a fixed connection, a detachable connection, or an integrated setting; it can be a mechanical connection, an electrical connection, or a communication connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. For technicians in the technical field to which the present application belongs, the specific meanings of the above terms in the embodiments of the present application can be understood according to the specific circumstances.
[0056] like Figure 1 As shown, in a first aspect, the present application provides a text content intelligent segmentation method based on variable vector length, the method comprising:
[0057] First, go to step S101 to obtain text content, and determine the initial vector length according to the total length of the text content and the segmentation accuracy;
[0058] Then, the process proceeds to step S102 to segment the text content according to the length of the initial vector, and to determine whether the character at the end of the segmentation is a rest character;
[0059] If the judgment result of step S102 is yes, then proceed to step S103 to obtain text slices;
[0060] If the judgment result of step S102 is no, then the process proceeds to step S104 to extend backward from the end of the current segmentation to the first rest to obtain a text segment.
[0061] In step S101, the text content is a preset text article, for example, a policy document, a reading article, etc. When obtaining the text content, it is necessary to slice the text content to facilitate further analysis and interpretation.
[0062] The segmentation precision can include two precision parameters, namely the coherence parameter and the search matching precision parameter. The value range of the coherence parameter is 0-1, 1 represents the most coherent, 0 represents the least coherent, and the initial value can be set to 0.5 by default; the value range of the search matching precision parameter is 0-1, 1 represents the most accurate match, 0 represents the most fuzzy match, and the initial value can be set to 0.5 by default. The value range of the initial vector length can be 100-300, the minimum segment length is 100, and the maximum segment length is 300.
[0063] In step S102, the rest is a symbol indicating the end of a complete sentence, such as a period, an exclamation mark, a question mark, etc.
[0064] In the above manner, when a sentence in the text content is detected to be truncated with the initial vector as a unit, it will automatically extend to the next rest position for slicing, thus ensuring the integrity of the sentence. This facilitates the use of subsequent slicing and improves the accuracy of the analysis results and the operability of slicing.
[0065] like Figure 2 As shown, in some embodiments, after each segmentation to obtain a text slice, the following steps are performed for the text slice:
[0066] First, enter step S201 to obtain a plurality of post-paragraph sentences located after the current text slice T, and analyze the correlation between the plurality of post-paragraph sentences and the current text slice T and the correlation between the plurality of post-paragraph sentences;
[0067] Then, step S202 is entered to determine the segmentation adjustment strategy according to the correlation strength between the plurality of post-segment sentences and the current text slice T and the correlation strength between the plurality of post-segment sentences;
[0068] Then, the process proceeds to step S203 to adjust the segmentation of the current text slice T according to the segmentation adjustment strategy, and to update the segmentation pointer position so as to start segmenting the next text slice from the updated segmentation pointer position.
[0069] In this embodiment, the strength of the correlation between the post-paragraph sentences and the current text slice T, and the strength of the correlation between the post-paragraph sentences can be calculated by a semantic similarity algorithm based on text vectorization.
[0070] In this embodiment, after obtaining the text slice T, the correlation between several post-segment sentences after the current text slice and the current text slice T is further analyzed, and then the segmentation adjustment strategy is determined according to the correlation, and the current text slice is re-segmented and adjusted according to the segmentation adjustment strategy, so that the classification of the post-segment sentences is more accurate, and the content correlation in each adjusted text slice T is stronger, which is convenient for improving the accuracy of subsequent analysis.
[0071] In some embodiments, the plurality of post-paragraph statements include a first post-paragraph statement H1 and a second post-paragraph statement H2, and the correlation between T and H1 is R1, the correlation between H1 and H2 is R2, and the correlation between T and H2 is R3;
[0072] Determining a segmentation adjustment strategy according to the strength of the correlation between the post-segment statements and the current text slice T and the strength of the correlation between the post-segment statements, performing segmentation adjustment on the current text slice T according to the segmentation adjustment strategy, and updating the segmentation pointer position includes:
[0073] S1: If it is determined that R1 is not a strong association relationship, the current text slice T is kept unchanged, and the segmentation pointer position is updated to the end position of the current text slice T;
[0074] S2: If it is determined that R1 is a strong association relationship but R2 is a non-strong association relationship, H1 is included in the current text slice T, and the segmentation pointer position is updated to the end position of H1;
[0075] S3: If it is determined that R1 and R2 are both strongly associated but R3 is not strongly associated, then both H1 and H2 are included in the current text slice T, and the segmentation pointer position is updated to the end position of H1;
[0076] S4: If it is determined that R1, R2, and R3 are all strongly associated, H1 and H2 are both included in the current text slice T, and the segmentation pointer position is updated to the end position of H2.
[0077] In short, the situation of step S1 shows that the sentence after the current text slice T (i.e., H1) is not closely related to the current text slice T, so the current text slice T can be kept unchanged and the next slice starts from text slice T. The situation of step S2 shows that the sentence after the current text slice T (i.e., H1) is strongly related to the current text slice T, but the next sentence H2 is not closely related to H1. In order to make the semantic relevance in the text slice stronger, H1 needs to be included in the text slice T, and the next slice starts from the end of H1. The situation of step S3 shows that there is a strong correlation between the current text slice T and the next sentence H1, H1 and H2 (the sentence after H1), but H2 is weakly related to the text slice T. At this time, H1 and H2 will be included in the text slice T, but the next segmentation pointer starts from the end of H1. At this time, H2 is equivalent to a transition sentence between two adjacent slices, and exists in both the current text slice and the next text slice. The situation in step S4 shows that there is a strong correlation between the current text slice T and H1 and H2. At this time, the text slice T and H1 and H2 can be regarded as a whole. H1 and H2 are both included in the current text slice T, and the segmentation pointer position is updated to the end position of H2, that is, the next text slice is sliced starting from the end position of H2.
[0078] In some embodiments, before including both H1 and H2 in the current text slice T, it also includes: determining whether the sum of the lengths of H1, H2 and text slice T exceeds the preset length upper limit, and if so, merging H1 and H2 and slicing them independently. For example, if the length range of text slice T is 100-300, the preset length upper limit is 300, the length of text slice T is 250, and the lengths of H1 and H2 are both 50, then H1 and H2 will be merged and sliced independently of text slice T. For another example, if the length of text slice T is 150, and the lengths of H1 and H2 are both 50, then H1 and H2 will be included in the current text slice T, and an updated text slice T with a length of 250 will be obtained. In the above manner, the length of each text slice can be adaptively adjusted, so that the length of each text slice can be more balanced, which is convenient for subsequent analysis.
[0079] In some embodiments, the method further includes: after the next text slice is generated, determining the correlation between the merged segment of H1 and H2 and the next text slice; if the correlation between the two is not strongly correlated, keeping H1 and H2 as an independent slice after merging; if the overall correlation is strongly correlated, determining whether the sum of the lengths of H1, H2 and the next text slice exceeds a preset length upper limit; if not, merging H1 and H2 into the next text slice; if exceeded, keeping H1 and H2 as an independent slice after merging.
[0080] In short, if the length of the current text slice T and H1, H2 exceeds the preset length limit, H1, H2 can be merged and cached first, and after waiting for the next text slice to be generated, the correlation between the independent slice after the merger of H1, H2 and the next text slice as a whole can be judged again. If the correlation is strongly correlated and the sum of the lengths of H1, H2 and the next text slice does not exceed the preset length limit, the independent slice after the merger of H1, H2 can be incorporated into the next text slice, so that the length of the text slices is more balanced, thereby improving the accuracy of subsequent analysis.
[0081] In some embodiments, if it is determined that R1, R2, and R3 are all strongly associated, before including H1 and H2 in the current text slice T, the following is further included:
[0082] Obtain several end-of-paragraph sentences located at the end of the current text slice T, analyze the correlation between the several end-of-paragraph sentences and H1 and the correlation between the several end-of-paragraph sentences respectively, and determine the segmentation adjustment strategy according to the correlation between the several end-of-paragraph sentences and H1 and the correlation between the several end-of-paragraph sentences, and decide whether to include H1 and H2 in the current text slice T according to the segmentation adjustment strategy.
[0083] Furthermore, the plurality of end-of-segment statements include the end-of-first-segment statement Q1 and the end-of-second-segment statement Q2, and the correlation between Q1 and H1 is R1', the correlation between Q1 and Q2 is R5', and the correlation between Q2 and H1 is R6';
[0084] Determining a segmentation adjustment strategy according to the correlation between the end-of-segment sentences and H1 and the correlation between the end-of-segment sentences, and determining whether to include H1 and H2 in the current text slice T according to the segmentation adjustment strategy includes:
[0085] S5: If it is determined that R1' is not a strong association relationship, the current text slice T is kept unchanged, H1 and H2 are merged and sliced independently, and the segmentation pointer position is updated to the end position of the current text slice T;
[0086] S6: If it is determined that R1' is a strong correlation but R2' is a non-strong correlation, Q1 is removed from the current text slice T, Q1, H1 and H2 are merged and sliced independently, and the segmentation pointer position is updated to the end position of H2;
[0087] S7: If it is determined that R1' and R2' are both strongly associated but R3' is not strongly associated, keep the current text slice T unchanged, merge H1, H2, Q1 and Q2 into an independent slice, and update the segmentation pointer position to the end position of H2, or remove Q1 and Q2 from the current text slice T, merge Q1, Q2, H1 and H2 into an independent slice, and update the segmentation pointer position to the end position of H2;
[0088] S8: If it is determined that R1', R2', and R3' are all strongly associated, then H1 and H2 are both included in the current text slice T, and the segmentation pointer position is updated to the end position of H2.
[0089] In short, if Figure 4 As shown, the situation in step S5 indicates that the last sentence of the current text slice T is not closely related to the sentence after the text slice T (i.e., H1), so the current text slice T can be kept unchanged, and then according to the length of H1, H2 and the text slice T, it is decided to merge H1 and H2 into independent slices, or merge H1 and H2 into the text slice T.
[0090] The situation in step S6 shows that the last sentence of the current text slice T is strongly associated with the sentence after the text slice T (i.e., H1), but the last sentence of the text slice T is not closely associated with the penultimate sentence. At this time, Q1 can be removed from the current text slice T, and Q1, H1, and H2 can be merged and sliced independently.
[0091] Of course, after Q1, H1 and H2 are merged into independent slices, they can be cached first. After waiting for the generation of the next text slice, the overall correlation between the independent slices after the merger of Q1, H1, H2 and the next text slice can be judged again. If the correlation is strong and the sum of the lengths of Q1, H1, H2 and the next text slice does not exceed the preset length limit, the independent slices after the merger of Q1, H1, H2 can be incorporated into the next text slice, so that the length of the text slices is more balanced, thereby improving the accuracy of subsequent analysis.
[0092] The situation in step S7 shows that the last sentence of the current text slice T is strongly associated with the sentence after the text slice T (i.e., H1), and the last sentence of the text slice T is also strongly associated with the penultimate sentence, but the penultimate sentence of the text slice T is not highly associated with H1. At this time, Q1 can be removed from the current text slice T. At this time, H1, H2, Q1, and Q2 can be merged and sliced independently, or Q1 and Q2 can be removed from the current text slice T, and Q1, Q2, H1, and H2 can be merged and sliced independently, and the segmentation pointer position can be updated to the end position of H2.
[0093] The situation in step S8 shows that there is a strong correlation between Q1, Q2, H1 and H2. Based on the lengths of H1, H2 and the text slice T, it is decided whether to merge H1 and H2 and slice them independently, or to merge H1 and H2 and incorporate them into the text slice T.
[0094] like Figure 3 As shown, in some embodiments, the method includes:
[0095] First, enter step S301 to input the text slices into the trained neural network model in sequence, and output the labels corresponding to the text slices;
[0096] Then, the process proceeds to step S302 to map and store the label with each text slice.
[0097] In step S301, the neural network model may be a convolutional neural network (CNN) or a recurrent neural network (RNN) and its variants (such as LSTM, GRU).
[0098] By mapping and storing labels with each text slice, the full text interpretation of the text content can be completed, facilitating subsequent retrieval and search.
[0099] In a second aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the text content intelligent segmentation method based on variable vector length as described in the first aspect of the present invention.
[0100] The computer-readable storage medium may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories.
[0101] The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a ferromagnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD ROM); the magnetic surface memory can be a disk memory or a tape memory.
[0102] The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), direct memory bus random access memory (DRRAM). The computer-readable storage medium described in the embodiments of the present invention is intended to include these and any other suitable types of memory.
[0103] like Figure 5 As shown, in a third aspect, the present invention provides an electronic device 10, comprising a processor 101 and a storage medium 102, wherein a computer program is stored on the storage medium, and when the computer program is executed by the processor, the text content intelligent segmentation method based on variable vector length as described in the first aspect of the present invention is implemented.
[0104] In some embodiments, the processor can be implemented by software, hardware, firmware or a combination thereof, and can use a circuit, a single or multiple application-specific integrated circuits (Application Specific Integrated Circuit, ASIC), a digital signal processor (Digital Signal Processor, DSP), a digital signal processing device (Digital Signal Processing Device, DSPD), a programmable logic device (Programmable Logic Device, PLD), a field programmable gate array (Field Programmable Gate Array, FPGA), a central processing unit (Central Processing Unit, CPU), a controller, a microcontroller, at least one of a microprocessor, so that the processor can execute part of the steps or all of the steps or any combination of the steps in the method for intelligent segmentation of text content based on variable vector length described in the various embodiments of the present application.
[0105] Finally, it should be noted that although the above embodiments have been described in the specification and drawings of this application, this does not limit the scope of patent protection of this application. All technical solutions generated by replacing or modifying equivalent structures or equivalent processes based on the essential concept of this application using the contents recorded in the specification and drawings of this application, as well as directly or indirectly implementing the technical solutions of the above embodiments in other related technical fields, are included in the scope of patent protection of this application.
Claims
1. A text content intelligent segmentation method based on variable vector length, characterized in that: The method comprises: Get the text content and determine the initial vector length based on the total length of the text content and the segmentation accuracy; The text content is segmented according to the length of the initial vector, and it is determined whether the character at the end of the segmentation is a rest. If so, a text slice is obtained. If not, the text slice is obtained by extending backward from the current segmentation end to the first rest.
2. The text content intelligent segmentation method based on variable vector length as claimed in claim 1, characterized in that: After each segmentation to obtain a text slice, the following steps are performed for the text slice: Acquire a plurality of post-paragraph statements after the current text slice T, and analyze the relevance between the plurality of post-paragraph statements and the current text slice T and the relevance between the plurality of post-paragraph statements; Determine a segmentation adjustment strategy according to the strength of the correlation between the plurality of post-paragraph sentences and the current text slice T and the strength of the correlation between the plurality of post-paragraph sentences; The current text slice T is segmented and adjusted according to the segmentation adjustment strategy, and the segmentation pointer position is updated to segment the next text slice starting from the updated segmentation pointer position.
3. The text content intelligent segmentation method based on variable vector length as claimed in claim 2, characterized in that: The plurality of post-paragraph statements include a first post-paragraph statement H1 and a second post-paragraph statement H2, and the correlation between T and H1 is R1, the correlation between H1 and H2 is R2, and the correlation between T and H2 is R3; Determining a segmentation adjustment strategy according to the strength of the correlation between the post-segment statements and the current text slice T and the strength of the correlation between the post-segment statements, performing segmentation adjustment on the current text slice T according to the segmentation adjustment strategy, and updating the segmentation pointer position includes: If it is determined that R1 is not a strong association relationship, the current text slice T is kept unchanged, and the segmentation pointer position is updated to the end position of the current text slice T; If it is determined that R1 is a strong association relationship but R2 is a non-strong association relationship, H1 is included in the current text slice T, and the segmentation pointer position is updated to the end position of H1; If it is determined that R1 and R2 are both strongly associated but R3 is not strongly associated, then both H1 and H2 are included in the current text slice T, and the segmentation pointer position is updated to the end position of H1; If it is determined that R1, R2, and R3 are all strongly associated, H1 and H2 are both included in the current text slice T, and the segmentation pointer position is updated to the end position of H2.
4. The text content intelligent segmentation method based on vector length variation as claimed in claim 3, characterized in that: Before both H1 and H2 are included in the current text slice T, it also includes: Determine whether the sum of the lengths of H1, H2, and text slice T exceeds the preset length limit. If so, merge H1 and H2 and slice them independently.
5. The text content intelligent segmentation method based on variable vector length as claimed in claim 4, characterized in that: The method further comprises: After the next text slice is generated, determine the correlation between the merged segment of H1 and H2 and the next text slice. If the correlation between the two is not strong, keep H1 and H2 as an independent slice after merging. If the overall correlation is strong, determine whether the sum of the lengths of H1, H2 and the next text slice exceeds the preset length limit. If not, merge H1 and H2 into the next text slice. If so, keep H1 and H2 as an independent slice after merging.
6. The text content intelligent segmentation method based on variable vector length as claimed in claim 3, characterized in that: If it is determined that R1, R2, and R3 are all strongly associated, before H1 and H2 are included in the current text slice T, the following steps are also included: Obtain several end-of-paragraph sentences located at the end of the current text slice T, analyze the correlation between the several end-of-paragraph sentences and H1 and the correlation between the several end-of-paragraph sentences respectively, and determine the segmentation adjustment strategy according to the correlation between the several end-of-paragraph sentences and H1 and the correlation between the several end-of-paragraph sentences, and decide whether to include H1 and H2 in the current text slice T according to the segmentation adjustment strategy.
7. The text content intelligent segmentation method based on variable vector length as claimed in claim 3, characterized in that: The plurality of end-of-segment statements include the end-of-first-segment statement Q1 and the end-of-second-segment statement Q2. The correlation between Q1 and H1 is R1', the correlation between Q1 and Q2 is R5', and the correlation between Q2 and H1 is R6'; Determining a segmentation adjustment strategy according to the correlation between the end-of-segment sentences and H1 and the correlation between the end-of-segment sentences, and determining whether to include H1 and H2 in the current text slice T according to the segmentation adjustment strategy includes: If it is determined that R1' is not a strong association, the current text slice T is kept unchanged, H1 and H2 are merged and sliced independently, and the segmentation pointer position is updated to the end position of the current text slice T; If it is determined that R1' is a strong correlation but R2' is a non-strong correlation, Q1 is removed from the current text slice T, Q1, H1 and H2 are merged and sliced independently, and the segmentation pointer position is updated to the end position of H2; If it is determined that R1' and R2' are both strongly associated but R3' is not strongly associated, the current text slice T is kept unchanged, H1, H2, Q1 and Q2 are merged and sliced independently, and the segmentation pointer position is updated to the end position of H2, or Q1 and Q2 are removed from the current text slice T, Q1, Q2, H1 and H2 are merged and sliced independently, and the segmentation pointer position is updated to the end position of H2; If it is determined that R1', R2', and R3' are all strongly associated, H1 and H2 are both included in the current text slice T, and the segmentation pointer position is updated to the end position of H2.
8. The text content intelligent segmentation method based on variable vector length as claimed in claim 1, characterized in that: The method comprises: Input the text slices into the trained neural network model in sequence, and output the labels corresponding to the text slices; The labels are mapped to the respective text slices and stored.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for intelligent segmentation of text content based on variable vector length as described in any one of claims 1 to 8 is implemented.
10. An electronic device having a computer program stored thereon, characterized in that: It comprises a processor and a storage medium, wherein a computer program is stored on the storage medium, and when the computer program is executed by the processor, the text content intelligent segmentation method based on variable vector length as described in any one of claims 1 to 8 is implemented.