Data storage method and apparatus applied to long documents

Abstract

Embodiments of the present application provide a data storage method and device applied to long documents, the method comprising: obtaining a token sequence corresponding to a long document to be processed; dividing the token sequence into a plurality of token segments according to a preset division rule; generating a to-be-stored information unit of each token segment under different storage modalities; and storing the to-be-stored information unit of each token segment under different storage modalities in a preset database corresponding to the different storage modalities, so as to store the long document to be processed. The technical solution of the embodiments of the present application can avoid the problem of token truncation caused by division boundaries, and ensure that a subsequent language processing model can correctly analyze the document content.

Description

Technical Field

[0001] This application relates to the field of data storage technology, and more specifically, to a data storage method and apparatus for long documents. Background Technology

[0002] With the continuous development of science and technology, the application scenarios of language processing models are becoming increasingly diverse, requiring them to deal with a large number of long documents.

[0003] In related technologies, in order to facilitate language processing models to retrieve and call long documents, long documents are usually segmented according to a preset number of characters or words, and then the segmented text fragments are stored in a database for language processing models to retrieve and call.

[0004] However, language processing models use tokens as the basic processing unit, and there is no fixed mapping relationship between characters or word counts and tokens. Therefore, when a language processing model retrieves a long document, if the segmentation boundaries of the text fragments stored in the long document are inconsistent with the token boundaries, it will result in truncated tokens in the text fragments, causing the language processing model to fail to parse the long document correctly. Summary of the Invention

[0005] To address the aforementioned technical problems, embodiments of this application provide a data storage method and apparatus for long documents.

[0006] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0007] According to one aspect of the embodiments of this application, a data storage method for long documents is provided, comprising: obtaining a token sequence corresponding to the long document to be processed; dividing the token sequence into multiple token segments according to a preset segmentation rule; generating information units to be stored for each token segment under different storage modes; and storing the information units to be stored for each token segment under different storage modes in preset databases corresponding to different storage modes, so as to store the long document to be processed.

[0008] Through the above embodiments, in the process of segmenting long documents, by obtaining the token sequence corresponding to the long document to be processed, and then segmenting the token sequence into multiple token fragments according to preset segmentation rules, the problem of token truncation due to segmentation boundaries is avoided, thereby ensuring that the subsequent language processing model can correctly parse the document content. Simultaneously, by generating information units to be stored for each token fragment in different storage modes and storing them in the preset database of the corresponding storage mode, heterogeneous storage of the long document to be stored is achieved, facilitating efficient and flexible retrieval of the long document according to different query requests.

[0009] In some embodiments of this application, based on the aforementioned scheme, the token sequence is divided into multiple token fragments according to a preset segmentation rule, including: dividing the long document to be processed into multiple character ranges according to the character segmentation rule in the preset segmentation rule; determining multiple initial token fragments based on the multiple character ranges and the token sequence; and outputting multiple token fragments according to the boundary segmentation rule in the preset segmentation rule and each initial token fragment.

[0010] Through the above embodiments, the long document to be processed is quickly segmented into initial token fragments based on character segmentation rules, and then the initial token fragments are segmented into multiple token fragments through boundary segmentation rules, so as to improve segmentation efficiency while improving segmentation quality.

[0011] In some embodiments of this application, based on the aforementioned scheme, multiple token fragments are output according to the boundary segmentation rules in the preset segmentation rules and each initial token fragment, including: determining the structural organization mark corresponding to each token in the initial token fragment; segmenting the initial token fragment according to the structural organization mark and the structural segmentation conditions in the boundary segmentation rules to obtain multiple token fragments corresponding to the initial token fragment.

[0012] Through the above embodiments, the initial token sequence is segmented according to the structural organization marker and structural segmentation conditions corresponding to each token in the initial token sequence to obtain multiple token fragments.

[0013] In some embodiments of this application, based on the aforementioned scheme, multiple token segments are output according to the boundary segmentation rules in the preset segmentation rules and each initial token segment, including: generating multiple windows to be identified on the initial token segment according to the preset sliding window; calculating the semantic similarity between each window to be identified and its adjacent windows to be identified in the initial token segment to obtain a similarity sequence corresponding to the initial token segment; and segmenting the initial token segment according to the similarity sequence and the similarity segmentation conditions in the boundary segmentation rules to obtain multiple token segments corresponding to the initial token segment.

[0014] Through the above embodiments, after generating multiple windows to be identified on the initial token fragment according to the preset sliding window, the initial token fragment is segmented according to the semantic similarity between each window to be identified and the adjacent windows to be identified and the similarity segmentation conditions in the boundary segmentation rules, so as to obtain multiple token fragments.

[0015] In some embodiments of this application, based on the aforementioned scheme, generating the information unit to be stored for each token fragment under different storage modes includes: configuring a unique identifier corresponding to each token fragment; and generating the information unit to be stored for each token fragment under different storage modes based on the unique identifier of each token fragment and the preprocessing rules corresponding to each storage mode.

[0016] Through the above embodiments, each token fragment, generated based on the unique identifier of each token fragment, is a unit of information to be stored in different storage modes. This allows the units of information to be stored in different storage modes to be associated, queried, and maintained in consistency through the unique identifier, thereby achieving unified management across storage modes.

[0017] In some embodiments of this application, based on the aforementioned scheme, different storage modes include vector storage mode, key-value pair storage mode, and relational graph storage mode.

[0018] In some embodiments of this application, based on the aforementioned scheme, the preprocessing rules corresponding to the vector storage mode are used to obtain the information vector features of the token fragment, and the information vector features and the unique identifier of the token fragment are used as the information unit to be stored corresponding to the token fragment in the vector storage mode; the preprocessing rules corresponding to the key-value pair storage mode are used to extract the key-value pair fields in the token fragment, and the key-value pair fields and the unique identifier of the token fragment are used as the information unit to be stored corresponding to the token fragment in the key-value pair storage mode; the preprocessing rules corresponding to the relational graph storage mode are used to determine the information association features between the token fragment and other token fragments according to the information vector features and key-value pair fields of each token fragment, and the information association features and the unique identifier of the token fragment are used as the information unit to be stored corresponding to the token fragment in the relational graph storage mode.

[0019] The above embodiments are used to construct information units to be stored under different storage modes for token fragments.

[0020] According to one aspect of the embodiments of this application, a data storage device for long documents is provided, comprising: an identification module configured to acquire a token sequence corresponding to the long document to be processed; a segmentation module configured to segment the token sequence into multiple token fragments according to a preset segmentation rule; a generation module configured to generate information units to be stored for each token fragment in different storage modes; and a storage module configured to store the information units to be stored for each token fragment in different storage modes in preset databases corresponding to different storage modes, so as to store the long document to be processed.

[0021] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, on which computer-readable instructions are stored, which, when executed by a computer's processor, cause the computer to perform the data storage method for long documents as described in the above embodiments.

[0022] According to one aspect of the embodiments of this application, an electronic device is provided, including: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements the data storage method for long documents as described in the above embodiments. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings: Figure 1 This is a flowchart illustrating an exemplary embodiment of a data storage method for long documents; Figure 2 yes Figure 1 The flowchart of step S120 in the illustrated embodiment is shown in an example embodiment; Figure 3 This is a block diagram illustrating a data storage device applied to a long document, as shown in an exemplary embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device shown in an exemplary embodiment of this application. Detailed Implementation

[0024] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0025] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0026] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0027] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0028] It should be noted that "multiple" in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0029] The technical solution of this application proposes a data storage method for long documents, as detailed in the embodiments below. Figure 1 As shown. The method includes at least steps S110 to S140, which are described in detail below: In step S110, the token sequence corresponding to the long document to be processed is obtained.

[0030] Among them, the long documents to be processed are the long documents that need to be stored in the database.

[0031] The token sequence can be obtained by inputting the long document to be processed into a preset word segmentation encoding model, and then performing word segmentation and encoding through the word segmenter built into the word segmentation encoding model to obtain the token sequence of the long document to be processed.

[0032] The pre-defined word segmentation encoding model includes models based on the transformer encoder structure, such as BERT (Bidirectional Encoder Representations from Transformers), RoBERTa (Robustly Optimized BERT Pretraining Approach), or ALBERT (A Lite BERT).

[0033] In addition, the preset word segmentation encoding model, based on the token sequence output by the long document to be processed, can be referred to as the following expression: ; Where T is the token sequence. Let be the i-th token in the long document to be processed, and m be the total number of tokens in the long document to be processed.

[0034] In step S120, the token sequence is divided into multiple token fragments according to a preset segmentation rule.

[0035] In the embodiments of this application, after obtaining the token sequence, the token sequence can be divided into multiple token segments according to a preset segmentation rule. Each token segment consists of a continuous segment of tokens in the token sequence, and the multiple token segments do not overlap and cover all sequences in the token sequence.

[0036] The method of dividing a token sequence into multiple token fragments according to a preset segmentation rule can be flexibly configured. In some embodiments of this application, the token sequence can be divided into multiple token fragments according to a preset number of tokens in the preset segmentation rule.

[0037] In some embodiments of this application, multiple token fragments can be output according to the boundary segmentation rules in the preset segmentation rules and each token in the token sequence.

[0038] In one example, the boundary segmentation rules can include structural segmentation conditions. Based on this, in the process of outputting multiple token fragments, the structural organization mark corresponding to each token in the token sequence can be determined first, and then the token sequence can be segmented according to the structural organization mark and the structural segmentation conditions in the preset segmentation rules to obtain multiple token fragments.

[0039] It should be noted that the token's structural organization markers are used to indicate the location and type of the text corresponding to that token within a long document, such as a heading, body paragraph, table, or image.

[0040] In the above process, under the condition of structural segmentation, we can start from the first token in the token sequence and traverse the structural organization marker of each token sequentially. The structural organization marker of the currently traversed token is compared with that of the previously traversed token. If the structural organization marker of the currently traversed token is the same as that of the previously traversed token, then the currently traversed token is added to the token fragment currently being constructed. If the structural organization marker of the currently traversed token is different from that of the previously traversed token, then the currently constructed token fragment is completed, and the construction of a new token fragment begins, while the currently traversed token is added as the first token in the new token fragment. Thus, the token sequence is segmented using the structural segmentation condition, resulting in multiple token fragments.

[0041] Secondly, in the above process, if multiple consecutive tokens have the same structural organization marker, causing the token fragment to be too long, the structural segmentation condition can be further segmented according to the preset token threshold to avoid the token fragment exceeding the context window limit of the language processing model.

[0042] Furthermore, in the above process, for the title in a long document, the structural segmentation condition can also be adjusted after the structural organization marker of the currently traversed token differs from that of the previously traversed token. If the structural organization marker of the previously traversed token is a title, then the currently traversed token is included in the token fragment currently being built, meaning the token fragment is not terminated. If the structural organization marker of the previously traversed token is not a title, then the currently built token fragment is completed, and the construction of a new token fragment begins. Simultaneously, the currently traversed token is included as the first token in the new token fragment, thus avoiding splitting the title in the long document into a separate token fragment. This reduces the total number of token fragments, thereby reducing the complexity and resource overhead when storing, indexing, or retrieving long documents.

[0043] In another example, the boundary segmentation rule can include similarity segmentation conditions. Based on this, in the process of outputting multiple token fragments, multiple windows to be identified can be generated on the token sequence according to a preset sliding window. Then, the semantic similarity between each window to be identified and its adjacent windows to be identified is calculated to obtain a similarity sequence. Finally, the token sequence is segmented according to the similarity sequence and the similarity segmentation conditions in the boundary segmentation rule to obtain multiple token fragments.

[0044] It should be noted that the preset sliding window length can be flexibly set according to the developer's needs. For example, 8 tokens, 16 tokens, or 32 tokens can be used as the preset sliding window length. There are no restrictions here.

[0045] In the above process, under the similarity segmentation condition, the semantic vector corresponding to each window to be identified can be calculated first. Then, based on the semantic vector of each window to be identified, the cosine similarity between each window to be identified and its neighboring windows can be calculated to obtain the semantic similarity between each window to be identified and its neighboring windows. Afterwards, the semantic similarities between each window to be identified and its neighboring windows are arranged to obtain a similarity sequence. Finally, the position of the lowest similarity value in the similarity sequence is used as the segmentation point to segment the token sequence into multiple token fragments. Thus, the token sequence is segmented using the similarity segmentation condition, resulting in multiple token fragments.

[0046] Secondly, in the above process, the semantic vector can be calculated based on the text corresponding to the window to be identified in the long document to be processed, or it can be calculated based on the token corresponding to the window to be identified.

[0047] In addition, in the above process, the similarity trough value is the similarity of the similarity sequence that is lower than the similarity of the two adjacent similarities and is lower than the preset similarity threshold. This avoids misjudging normal semantic fluctuations as segmentation points, reduces redundant segmentation segments, and improves segmentation quality.

[0048] In step S130, information units to be stored for each token fragment under different storage modes are generated.

[0049] It should be noted that storage modalities represent the data form of token fragments in the database, such as vector storage modalities, key-value pair storage modalities, relational graph storage modalities, index storage modalities, etc.

[0050] In the embodiments of this application, after the token sequence is segmented into multiple token fragments, information units to be stored for each token fragment under different storage modes can be generated to construct a heterogeneous representation of each token fragment, thereby supporting the parallel storage and multimodal retrieval of subsequent heterogeneous databases.

[0051] In some embodiments of this application, the method for generating the information unit to be stored for each token fragment under different storage modes can be achieved by using the preprocessing rules corresponding to each storage mode.

[0052] For example, in the vector storage mode, the information unit to be stored for each token fragment can be generated in the vector storage mode by using the preprocessing rules corresponding to the vector storage mode.

[0053] For example, the preprocessing rule corresponding to this vector storage mode can first obtain the information vector features of the token fragment, and then use the information vector features as the information unit to be stored corresponding to the token fragment in the vector storage mode.

[0054] In order to obtain the information vector features of token fragments, in some examples, we can first obtain the token vector matrix corresponding to the long document to be processed, then extract the token vector sequence corresponding to the token fragments from the token vector matrix, and then calculate the information vector features based on the token vector sequence.

[0055] This calculation method includes, but is not limited to, mean pooling, weighted pooling, and extreme value pooling.

[0056] In addition, the token vector matrix can be obtained by forward reasoning from a preset encoding embedding model by inputting the long document to be processed.

[0057] The preset encoding embedding model can be a model based on the transformer encoder structure, or a vector model based on the static word encoding structure, such as Word2Vec (Word to Vector), GloVe (Global Vectors for Word Representation), or FastText model.

[0058] Correspondingly, the preset encoding embedding model, based on the token vector matrix output by the long document to be processed, can be expressed as follows: ; Where U is the token vector matrix. d is a vector of individual tokens, m is the total number of tokens in the long document to be processed, and d is the vector dimension.

[0059] In the key-value pair storage mode, the information unit to be stored for each token fragment can be generated by the preprocessing rules corresponding to the key-value pair storage mode.

[0060] For example, the preprocessing rule corresponding to the key-value pair storage mode can first extract the key-value pair fields in the token fragment, and then use the key-value pair fields as the information unit to be stored corresponding to the token fragment in the key-value pair storage mode.

[0061] In order to extract key-value pairs from a token fragment, some examples suggest that the corresponding text fragment in the long document to be processed can be obtained first, and then the key-value pairs can be extracted from the text fragment.

[0062] This extraction method includes, but is not limited to, rule-based extraction methods, such as regular expressions and parsers, as well as learning model-based extraction methods, such as natural language processing models and neural network models.

[0063] In addition, the text fragment corresponding to the token fragment in the long document to be processed can be determined by the mapping relationship between the token and the text content in the long document to be processed. This mapping relationship can be established and recorded when the word segmentation encoding model performs word segmentation encoding on the long document to be processed.

[0064] In the relational graph storage mode, the information unit to be stored for each token fragment can be generated by the preprocessing rules corresponding to the relational graph storage mode.

[0065] For example, the preprocessing rules corresponding to the relational graph storage mode can first determine the information association features between the token fragment and other token fragments based on the information vector features and key-value pair fields of each token fragment, and then use the information association features as the information unit to be stored corresponding to the token fragment in the relational graph storage mode.

[0066] In order to obtain the information association features of token fragments, in some examples, the similarity and difference relationships between token fragments and other token fragments can be determined based on the information vector features and key-value pair fields of each token fragment, and then the information association features can be extracted from the similarity and difference relationships.

[0067] To determine the similarity between token fragments and other token fragments, the k-nearest neighbor search method can be used to identify k similar token fragments corresponding to the current token fragment from other token fragments based on the information vector features of each token fragment. These k similar token fragments are the k token fragments that are closest to the current token fragment among the other token fragments. Then, similarity edges are established between the current token fragment and the k similar token fragments to serve as the similarity relationship of the current token fragment.

[0068] The value 'k' is a preset quantity that can be flexibly set according to the developer's needs; no limit is imposed here.

[0069] Furthermore, in the process of determining the k similar token fragments corresponding to the current token fragment from other token fragments based on the information vector features of each token fragment using the k-nearest neighbor search method, the target token fragment under the text block where the current token fragment is located can be obtained first from multiple token fragments, and then the k-nearest neighbor search can be performed based on the information vector features of the target token fragment. Thus, by pre-filtering the text block, the search scope is further focused, thereby improving the relevance and accuracy of similarity retrieval.

[0070] This text block refers to the text block determined after the long document has been pre-divided and categorized according to document types such as chapters, paragraphs, or topics. The text block where the token fragment is located is the text block in the long document where the text corresponding to the token fragment is located.

[0071] In establishing similarity edges between the current token fragment and k similar token fragments, we can first identify target similar token fragments from the k similar token fragments whose similarity to the current token fragment reaches a preset similarity threshold, and then establish similarity edges between the current token fragment and the target similar token fragment. Thus, through further filtering using the preset similarity threshold, the reliability and accuracy of the information association features subsequently determined by similarity relationships are improved.

[0072] To determine the difference relationship between token fragments and other token fragments, we can first determine the key-value pair fields to be compared contained in the key-value pair fields of the current token fragment, and then determine the target token fragment from multiple token fragments based on the key-value pair fields of each token fragment. The field key of the target token fragment in the key-value pair field is the same as the field key of the key-value pair field to be compared. After that, a difference edge is established between the target token fragment and the current token fragment to serve as the difference relationship of the current token fragment.

[0073] The key-value pairs to be compared are key-value pairs belonging to a preset field type within the current token fragment. Preset field types include, but are not limited to, numeric fields, enumerated fields, time fields, and unit fields.

[0074] In some embodiments of this application, the method for generating the information unit to be stored for each token fragment under different storage modes can also be to first configure a unique identifier corresponding to each token fragment, and then, based on the unique identifier of each token fragment, generate the information unit to be stored for each token fragment under different storage modes through the preprocessing rules corresponding to each storage mode. This allows the information units to be stored under different storage modes to be associated, queried, and maintained in consistency through the unique identifier, thereby achieving unified management across storage modes.

[0075] In some embodiments of this application, the unique identifier can be configured according to the order of the token fragments in the long document to be processed, or the hash value corresponding to the token fragment can be calculated and then the hash value can be used as its unique identifier.

[0076] For example, given that each token fragment has its own unique identifier, the preprocessing rules of the vector storage mode can first obtain the information vector features of the token fragment, and then use the information vector features and the unique identifier of the token fragment as the information unit to be stored in the vector storage mode. In other words, the information vector features and the unique identifier are encapsulated and used as the information unit to be stored in the vector storage mode.

[0077] The preprocessing rules corresponding to this key-value pair storage mode can first extract the key-value pair fields from the token fragment, and then use the key-value pair fields and the unique identifier of the token fragment as the information unit to be stored in the key-value pair storage mode. In other words, the key-value pair fields and the unique identifier are encapsulated and used as the information unit to be stored in the key-value pair storage mode.

[0078] The preprocessing rules corresponding to this relational graph storage mode can first determine the information association features between token fragments and other token fragments based on the information vector features and key-value pair fields of each token fragment. Then, the information association features and the unique identifier of the token fragment are used as the information unit to be stored in the relational graph storage mode. In other words, the information association features and the unique identifier are encapsulated and used as the information unit to be stored in the relational graph storage mode.

[0079] In step S140, the information units to be stored for each token fragment under different storage modes are stored in the preset databases corresponding to the different storage modes.

[0080] In the embodiments of this application, after generating the information units to be stored for each token fragment under different storage modes, the information units to be stored for each token fragment under different modes can be stored in the preset databases corresponding to the different modes for storing the long document to be processed. Thus, each token fragment is obtained based on token segmentation, which avoids the problem of token truncation due to segmentation boundaries, thereby ensuring that the subsequent language processing model can correctly parse the document content.

[0081] Meanwhile, by generating information units to be stored for each token fragment under different storage modes and storing them in the preset database of the corresponding storage mode, heterogeneous storage of long documents to be stored is achieved, which facilitates efficient and flexible retrieval of the long documents according to different query requests in the future.

[0082] See Figure 2 , Figure 2 Is Figure 1 The flowchart of step S120 in the illustrated embodiment is shown in an exemplary embodiment. Figure 2 As shown, the process of dividing a token sequence into multiple token fragments according to a preset segmentation rule may include steps S210 to S230, which are described in detail below: In step S210, the long document to be processed is divided into multiple character ranges according to the character segmentation rules in the preset segmentation rules.

[0083] In the embodiments of this application, in the process of dividing a token sequence into multiple token fragments according to a preset segmentation rule, the long document to be processed can first be divided into multiple character ranges according to the character segmentation rule in the preset segmentation rule.

[0084] In one example, a long document to be processed can be split into multiple character ranges according to the preset number of characters in the character splitting rules.

[0085] In another example, the process can begin by traversing the first character of the long document to be processed. When the number of traversed characters reaches the preset number of characters in the character segmentation rule, the statement boundary point corresponding to the current traversed character sequence is determined. This statement boundary point is the closest statement boundary (e.g., period, question mark, exclamation mark, newline character, etc.) in the long document to be processed. The long document to be processed is then segmented using this statement boundary point to obtain a character interval. New character intervals are then constructed by continuing to traverse the long document from the first character after the current character interval, and the above process is repeated. This continues until every character in the long document to be processed is segmented into a character interval. Thus, the long document to be processed is segmented using the character segmentation rule, resulting in multiple character intervals.

[0086] Secondly, in the above process, when determining the statement boundary point corresponding to the currently traversed character sequence, if the number of characters in the currently traversed character sequence exceeds the preset maximum character threshold after adjustment at that statement boundary point, then the nearest token boundary point in the token sequence is determined, so as to segment the long document to be processed through the token boundary point. On the one hand, this allows character intervals to be segmented at sentence boundaries, maintaining semantic coherence. On the other hand, it avoids the problem of excessively long character intervals due to excessive pursuit of sentence boundaries, ensuring that the length of each character interval is suitable for the context window constraint of the language processing model.

[0087] Furthermore, in the above process, when determining the statement boundary point corresponding to the currently traversed character sequence, one can search forward or backward. Based on this, if the number of characters in the currently traversed character sequence is lower than a preset minimum character threshold after adjustment at the statement boundary point, the next statement boundary point for the currently traversed character sequence is determined, until it is determined that the number of characters in the currently traversed character sequence is not lower than the preset minimum character threshold after adjustment at the statement boundary point. This avoids generating a large number of short fragments due to segmentation, reducing fragmentation in storage and retrieval, while ensuring that each fragment has sufficient semantic information, which is beneficial for subsequent processing.

[0088] In step S220, multiple initial token fragments are determined based on multiple character ranges and token sequences.

[0089] In the embodiments of this application, after segmenting into multiple character ranges, multiple initial token fragments can be determined based on the multiple character ranges and the token sequence.

[0090] For example, the mapping relationship between the text content in each character interval and the tokens in the token sequence can be determined. That is, the start character and end character of each character interval are determined, and then the tokens corresponding to the start character and end character are used as the token boundary points for each initial token fragment. Finally, the token sequence is divided according to the token boundary points to obtain multiple initial token fragments.

[0091] In step S230, multiple token fragments are output according to the boundary segmentation rules in the preset segmentation rules and each initial token fragment.

[0092] In the embodiments of this application, the long document to be processed is divided into multiple initial token fragments based on the token sequence by the character segmentation rule in the preset segmentation rules. Then, multiple token fragments are output according to the boundary segmentation rule in the preset segmentation rules and each initial token fragment. Thus, the long document to be processed is quickly segmented into initial token fragments by the character segmentation rule, and then the initial token fragments are segmented into multiple token fragments by the boundary segmentation rule, so as to improve the segmentation efficiency while improving the segmentation quality.

[0093] The process of outputting multiple token fragments based on the boundary segmentation rules in the preset segmentation rules and each initial token fragment. In some examples, for each initial token fragment, the structural organization mark corresponding to each token in the initial token fragment can be determined first, and then the initial token fragment can be segmented according to the structural organization mark and the structural segmentation conditions in the boundary segmentation rules to obtain multiple token fragments corresponding to the initial token fragment.

[0094] The specific implementation process can be referred to the description of the structural segmentation conditions in step S120 above, and will not be repeated here.

[0095] In other examples, for each initial token fragment, multiple windows to be identified can be generated on the initial token fragment according to a preset sliding window. Then, the semantic similarity between each window to be identified and its adjacent windows in the initial token fragment is calculated to obtain the similarity sequence corresponding to the initial token fragment. Finally, the initial token fragment is segmented according to the similarity sequence and the similarity segmentation conditions in the boundary segmentation rules to obtain multiple token fragments under the initial token fragment.

[0096] The specific implementation process can be referred to the description of similarity segmentation conditions in step S120 above, and will not be repeated here.

[0097] In other examples, for each initial token fragment, the initial token fragment can be segmented first using the structural segmentation conditions in the boundary segmentation rules to obtain multiple intermediate token fragments corresponding to the initial token fragment. Then, the multiple intermediate token fragments can be segmented a second time using the similarity segmentation conditions in the boundary segmentation rules to obtain multiple token fragments corresponding to the initial token fragment, thereby further improving the segmentation quality.

[0098] Specifically, we can first determine the structural organization marker corresponding to each token in the initial token fragment, and then segment the initial token fragment according to the structural organization marker and the structural segmentation conditions in the boundary segmentation rules to obtain multiple intermediate token fragments. Then, according to the preset sliding window, we can generate multiple windows to be identified on each intermediate token fragment, and then calculate the semantic similarity between each window to be identified and its adjacent windows to be identified to obtain a similarity sequence. Finally, according to the similarity sequence and the similarity segmentation conditions in the boundary segmentation rules, we can segment each intermediate token fragment to obtain multiple token fragments.

[0099] The following describes an apparatus embodiment of this application, which can be used to execute the data storage method for long documents described in the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the above embodiments of the data storage method for long documents described in this application.

[0100] Figure 3 A block diagram of a data storage device 100 for long documents according to an embodiment of this application is shown.

[0101] Reference Figure 3 As shown, a data storage device 100 for long documents according to an embodiment of this application includes: an identification module 110 configured to acquire a token sequence corresponding to the long document to be processed; a segmentation module 120 configured to segment the token sequence into multiple token fragments according to a preset segmentation rule; a generation module 130 configured to generate information units to be stored for each token fragment in different storage modes; and a storage module 140 configured to store the information units to be stored for each token fragment in different storage modes in preset databases corresponding to different storage modes, so as to store the long document to be processed.

[0102] In some embodiments of this application, based on the foregoing scheme, the segmentation module 120 is further configured to: segment the long document to be processed into multiple character intervals according to the character segmentation rules in the preset segmentation rules; determine multiple initial token fragments based on the multiple character intervals and the token sequence; and output multiple token fragments according to the boundary segmentation rules in the preset segmentation rules and each initial token fragment.

[0103] In some embodiments of this application, based on the foregoing scheme, the segmentation module 120 is further configured to: determine the structural organization mark corresponding to each token in the initial token segment; and segment the initial token segment according to the structural organization mark and the structural segmentation conditions in the boundary segmentation rules to obtain multiple token segments corresponding to the initial token segment.

[0104] In some embodiments of this application, based on the aforementioned scheme, the segmentation module 120 is further configured to: generate multiple windows to be identified on the initial token fragment according to a preset sliding window; calculate the semantic similarity between each window to be identified and its adjacent windows to be identified in the initial token fragment to obtain a similarity sequence corresponding to the initial token fragment; and segment the initial token fragment according to the similarity sequence and the similarity segmentation conditions in the boundary segmentation rules to obtain multiple token fragments corresponding to the initial token fragment.

[0105] In some embodiments of this application, based on the foregoing scheme, the generation module 130 is further configured to: configure a unique identifier corresponding to each token fragment; and generate a storage information unit for each token fragment in different storage modes based on the unique identifier of each token fragment and the preprocessing rules corresponding to different storage modes.

[0106] It should be noted that the data storage device 100 for long documents provided in the above embodiments and the data storage method for long documents provided in the above embodiments belong to the same concept. The specific way in which each module and unit performs operations has been described in detail in the method embodiments, and will not be repeated here.

[0107] Embodiments of this application also provide an electronic device, including a processor and a memory, wherein the memory stores computer-readable instructions that, when executed by the processor, implement the data storage method for long documents as described above.

[0108] Figure 4 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown.

[0109] It should be noted that, Figure 4The computer system 200 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0110] like Figure 4 As shown, the computer system 200 includes a Central Processing Unit (CPU) 201, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 202 or programs loaded from storage portion 208 into Random Access Memory (RAM) 203, such as performing the methods described in the above embodiments. The RAM 203 also stores various programs and data required for system operation. The CPU 201, ROM 202, and RAM 203 are interconnected via a bus 204. An Input / Output (I / O) interface 205 is also connected to the bus 204.

[0111] The following components are connected to I / O interface 205: an input section 206 including a keyboard, mouse, etc.; an output section 207 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 208 including a hard disk, etc.; and a communication section 209 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 209 performs communication processing via a network such as the Internet. A drive 210 is also connected to I / O interface 205 as needed. Removable media 211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 210 as needed so that computer programs read from them can be installed into storage section 208 as needed.

[0112] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 209, and / or installed from removable medium 211. When the computer program is executed by central processing unit (CPU) 201, it performs various functions defined in the system of this application.

[0113] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0114] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0115] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0116] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.

[0117] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0118] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the methods according to the embodiments of this application.

[0119] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0120] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data storage method for long documents, characterized in that, The method includes: Obtain the token sequence corresponding to the long document to be processed; The token sequence is divided into multiple token fragments according to a preset segmentation rule; Generate the information unit to be stored for each of the token fragments under different storage modes; Each token fragment is stored in a preset database corresponding to a different storage mode, which stores the information unit to be stored in each storage mode, so as to store the long document to be processed.

2. The method according to claim 1, characterized in that, The step of dividing the token sequence into multiple token fragments according to a preset segmentation rule includes: The long document to be processed is divided into multiple character ranges according to the character segmentation rules in the preset segmentation rules; Multiple initial token fragments are determined based on the multiple character ranges and the token sequence; Multiple token fragments are output based on the boundary segmentation rules in the preset segmentation rules and each initial token fragment.

3. The method according to claim 2, characterized in that, The step of outputting multiple token fragments according to the boundary segmentation rules in the preset segmentation rules and each initial token fragment includes: Determine the structural organization marker corresponding to each token in the initial token fragment; The initial token fragment is segmented according to the structural organization mark and the structural segmentation conditions in the boundary segmentation rules to obtain multiple token fragments corresponding to the initial token fragment.

4. The method according to claim 2, characterized in that, The step of outputting multiple token fragments according to the boundary segmentation rules in the preset segmentation rules and each initial token fragment includes: Multiple windows to be identified are generated on the initial token fragment according to a preset sliding window; Calculate the semantic similarity between each window to be identified and its adjacent windows to be identified in the initial token fragment to obtain the similarity sequence corresponding to the initial token fragment; The initial token fragment is segmented according to the similarity sequence and the similarity segmentation conditions in the boundary segmentation rules to obtain multiple token fragments corresponding to the initial token fragment.

5. The method according to claim 1, characterized in that, The step of generating the information unit to be stored for each token fragment in different storage modes includes: Configure a unique identifier for each of the aforementioned token fragments; Based on the unique identifier of each token fragment, the information units to be stored for each token fragment in different storage modes are generated according to the preprocessing rules corresponding to the different storage modes.

6. The method according to claim 5, characterized in that, The different storage modes include vector storage mode, key-value pair storage mode, and graph storage mode.

7. The method according to claim 6, characterized in that, The preprocessing rules corresponding to the vector storage mode are used to obtain the information vector features of the token fragment, and the information vector features and the unique identifier of the token fragment are used as the information unit to be stored corresponding to the token fragment in the vector storage mode. The preprocessing rules corresponding to the key-value pair storage mode are used to extract the key-value pair fields in the token fragment, and the key-value pair fields and the unique identifier of the token fragment are used as the information unit to be stored corresponding to the token fragment under the key-value pair storage mode. The preprocessing rules corresponding to the relational graph storage mode are used to determine the information association features between the token fragment and other token fragments based on the information vector features and key-value pair fields of each token fragment, and use the information association features and the unique identifier of the token fragment as the information unit to be stored corresponding to the token fragment in the relational graph storage mode.

8. A data storage device for long documents, characterized in that, include: The recognition module is configured to obtain the token sequence corresponding to the long document to be processed; The segmentation module is configured to segment the token sequence into multiple token fragments according to a preset segmentation rule; The generation module is configured to generate information units to be stored for each of the token fragments under different storage modes; The storage module is configured to store the information units to be stored for each token fragment in different storage modes in a preset database corresponding to the different storage modes, so as to store the long document to be processed.

9. A computer-readable storage medium, characterized in that, It stores computer-readable instructions that, when executed by a computer's processor, cause the computer to perform the data storage method for long documents as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the electronic device to implement the data storage method for long documents as described in any one of claims 1 to 7.

Images