Data slicing method, system and equipment and storage medium

Abstract

The invention relates to the field of computers, and discloses a data slicing method, system and device and a storage medium. The method comprises the following steps: receiving a knowledge operation instruction uploaded by a user; creating knowledge entries through the selected knowledge parser according to the uploaded file, and generating corresponding knowledge identifiers for the knowledge entries; receiving a knowledge slicing instruction sent by a user; according to the specified knowledge identifier and the type of the uploaded file, slicing the content of the uploaded file through the selected knowledge slicing strategy to obtain a plurality of knowledge fragments; and respectively inserting the plurality of knowledge fragments into different types of databases to finish classification. In the embodiment of the invention, intelligent segmentation can be carried out on a large batch of data, and management of the large batch of data is completed, so that the AI large language model and the reference data of multi-modal pre-training are provided, and the training efficiency and effect of the AI large model are improved.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a data slicing method, system, device and storage medium. Background Art

[0002] As the complexity of deep learning models increases, especially with the widespread use of large language models (LLMs) and multimodal pre-trained models, the amount of data required to train these models is also growing exponentially. However, existing data management and processing methods face many challenges when faced with massive amounts of data: as the construction of integrated budget management platforms in provinces and cities across the country gradually improves, the scope of business coverage becomes wider and wider, the number of users gradually increases, the demand for business consulting services increases, and more and more knowledge documents on relevant industry regulations are needed.

[0003] Traditional knowledge bases rely on manual classification and updating, which is time-consuming and labor-intensive and prone to outdated or inaccurate information. General large-scale language models may not fully understand the terminology and rules of specific industries, resulting in the generated content being not professional or relevant enough. Therefore, a data slicing method is needed to solve the above problems and improve the efficiency and effectiveness of AI large model training. Summary of the invention

[0004] The main purpose of this invention is to solve the technical problem of how to intelligently segment large batches of data to provide benchmark data for pre-training of large AI models.

[0005] A first aspect of the present invention provides a data slicing method, the data slicing method comprising: Receiving a knowledge operation instruction uploaded by a user, wherein the knowledge operation instruction carries an uploaded file; According to the uploaded file, a knowledge entry is created by a selected knowledge parser, and a corresponding knowledge identifier is generated for the knowledge entry; Receiving a knowledge slicing instruction sent by a user, wherein the knowledge slicing instruction carries a specified knowledge identifier; According to the specified knowledge identifier and the type of the uploaded file, the content of the uploaded file is sliced ​​by a selected knowledge slicing strategy to obtain multiple knowledge fragments; The multiple knowledge fragments are respectively inserted into different types of databases to complete the classification.

[0006] Optionally, in a first implementation of the first aspect of the present invention, creating a knowledge entry according to the uploaded file by using a selected knowledge parser, and generating a corresponding knowledge identifier for the knowledge entry includes: According to the uploaded file and the preset knowledge base, a knowledge entry is created through the selected knowledge parser; Inserting the knowledge item into the database according to the preset knowledge tag and tag mapping relationship to associate the relevant knowledge item in the database; A corresponding knowledge identifier is generated for the knowledge item.

[0007] Optionally, in a second implementation of the first aspect of the present invention, slicing the content of the uploaded file according to the specified knowledge identifier and the type of the uploaded file through a selected knowledge slicing strategy to obtain multiple knowledge fragments includes: Performing feature extraction on the content of the uploaded file; By using the selected knowledge slicing strategy, a corresponding automatic slicer is obtained, wherein the automatic slicer adopts an optimization algorithm of a user-predefined objective function; According to the result of feature extraction, the specified knowledge identifier and the type of the uploaded file, the content of the uploaded file is sliced ​​by the automatic slicer to obtain multiple knowledge fragments; The multiple knowledge fragments are formatted and sent to the client.

[0008] Optionally, in a third implementation of the first aspect of the present invention, slicing the content of the uploaded file by the automatic slicer according to the result of feature extraction, the specified knowledge identifier, and the type of the uploaded file to obtain multiple knowledge fragments includes: If the automatic slicer is a regular slicer, obtaining a matching rule defined based on a regular expression engine; According to the result of feature extraction, the specified knowledge identifier, the type of the uploaded file, and the matching rule, the content of the uploaded file is sliced ​​step by step in a coarse-to-fine manner through the regular slicer to obtain multiple knowledge fragments.

[0009] Optionally, in a fourth implementation of the first aspect of the present invention, the extracting features from the content of the uploaded file includes: Parsing the uploaded file to obtain binary stream data; By comparing the rules of a preset regular expression matcher with the binary stream data, it is determined whether the content of the uploaded file conforms to the expected format and specifications; If it meets the requirements, feature extraction is performed on the content of the uploaded file.

[0010] Optionally, in a fifth implementation of the first aspect of the present invention, after slicing the content of the uploaded file according to the specified knowledge identifier and the type of the uploaded file by a selected knowledge slicing strategy to obtain multiple knowledge fragments, the method further includes: Generate a parent-child structure document with a hierarchical relationship according to the multiple knowledge fragments; According to the content of the uploaded file and the parent-child structure document, based on the pre-trained language model, parent-child layer knowledge items are constructed and keyword tags are dynamically adapted and added to facilitate search or classification; Identify and delete meaningless or redundant knowledge fragments in the knowledge base.

[0011] Optionally, in a sixth implementation of the first aspect of the present invention, the inserting the multiple knowledge fragments into different types of databases to complete classification includes: sending the plurality of knowledge fragments to a client for secondary adjustment; Insert the adjusted knowledge fragments into different types of databases to complete the classification.

[0012] A second aspect of the present invention provides a data slicing device, comprising: a memory and at least one processor, wherein instructions are stored in the memory, and the memory and the at least one processor are interconnected via lines; the at least one processor calls the instructions in the memory so that the data slicing device executes the above-mentioned data slicing method.

[0013] A third aspect of the present invention provides a computer-readable storage medium, in which instructions are stored, which, when executed on a computer, enable the computer to execute the above-mentioned data slicing method.

[0014] In an embodiment of the present invention, a knowledge operation instruction uploaded by a user is received, wherein the knowledge operation instruction carries an uploaded file; based on the uploaded file, a knowledge entry is created through a selected knowledge parser, and a corresponding knowledge identifier is generated for the knowledge entry; a knowledge slicing instruction sent by a user is received, wherein the knowledge slicing instruction carries a specified knowledge identifier; based on the specified knowledge identifier and the type of the uploaded file, the content of the uploaded file is sliced ​​through a selected knowledge slicing strategy to obtain a plurality of knowledge fragments; and the plurality of knowledge fragments are respectively inserted into different types of databases to complete classification. In the present invention, first, a knowledge operation instruction is received, and a knowledge entry is created and a corresponding knowledge identifier is generated through a selected knowledge parser; then, a knowledge slicing instruction sent by a user is received, and data slicing is performed through a selected knowledge slicing strategy based on the specified knowledge identifier, so that large quantities of data can be intelligently segmented and large quantities of data management can be completed, so as to provide benchmark data for large AI language models and multimodal pre-training, and improve the efficiency and effectiveness of AI large model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A schematic diagram of an embodiment of a data slicing method in an embodiment of the present invention; Figure 2 This is a schematic diagram of the data slicing strategy service interaction process in an embodiment of the present invention; Figure 3 A schematic diagram of an embodiment of a data slicing system in an embodiment of the present invention; Figure 4 Schematic diagram of an embodiment of a data slicing device in an embodiment of the present invention. DETAILED DESCRIPTION

[0016] Embodiments of the present invention provide a data slicing method, system, device and storage medium.

[0017] The embodiments disclosed in the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments disclosed in the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments disclosed in the present invention are only for exemplary purposes and are not intended to limit the scope of protection disclosed in the present invention.

[0018] In the description of the embodiments disclosed in the present invention, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0019] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , an embodiment of the data slicing method in the embodiment of the present invention includes: S100, receiving knowledge operation instructions uploaded by the user.

[0020] In this embodiment, AI brains with relevant built-in definitions are classified according to different privatized knowledge bases, and the privatized knowledge base fields are subdivided using a multi-level model to upload knowledge to AI brains in specific fields. Select three methods to identify and process document data streams (general, question and answer, manual), and through knowledge parsing selection (OCR-general), data cleaning and other parsing strategies, identify the types of basic knowledge and other basic values, and store them in the database. Specifically, when a user logs in to the business system, the user can send knowledge operation instructions according to the authorized different knowledge base AI brains. The browser will send the binary file stream uploaded by the user and the business parameters required to generate the interface to the server and click Next.

[0021] S200, creating a knowledge entry based on the uploaded file through the selected knowledge parser, and generating a corresponding knowledge identifier for the knowledge entry.

[0022] In this embodiment, the uploaded file can be different types of files, such as doc, Mackdown, docx, PDF, etc. Optionally, it can also be uploaded in the form of question and answer or manually entered. The uploaded file is preprocessed by identifying the file stream, and the knowledge name and knowledge label are marked for knowledge. Different knowledge parsers, such as general document parser, text parser, and OCR recognition parser are selected to generate professional knowledge identifiers for preliminary classification and save in the database. Specifically, the basic knowledge in the uploaded file is extracted, and basic parameters such as knowledge tags and knowledge labels are pre-filled for page preview. Through the selected knowledge parser, the browser passes the business parameters required for parsing the basic parameter generation interface to the server to create knowledge entries. Specifically, through the knowledge wildcard module, users are supported to use natural language, keyword combinations or other forms of input to query basic parameters such as knowledge tags or knowledge labels. Even if the input is not completely accurate, the related knowledge entries can be found. If the related knowledge entry cannot be found, the knowledge entry is created, and the corresponding knowledge ID is generated for the knowledge entry for interaction. The user can modify and edit and update the metadata. Among them, the knowledge universal matching module is a key component used for flexible matching and retrieval of knowledge entries in AI intelligent operation and maintenance platforms or similar systems. After receiving the uploaded file, it first uses data stream parsing technology to perform knowledge classification, knowledge marking, knowledge labeling, knowledge label content analysis and other operations on the file content to extract the core vocabulary and grammatical structure.

[0023] S300, receiving the knowledge slicing instruction sent by the user.

[0024] In this embodiment, a knowledge slicing instruction sent by a user is received, and the knowledge slicing instruction carries a specified knowledge id, so that the content of the uploaded file is sliced ​​according to the specified knowledge id.

[0025] S400, according to the specified knowledge identifier and the type of the uploaded file, the content of the uploaded file is sliced ​​through the selected knowledge slicing strategy to obtain multiple knowledge fragments.

[0026] In this embodiment, different slicing pages are jumped according to different knowledge types, and general type files enter the slicing selection page, select different knowledge slicing strategies, and according to the type of uploaded files, parameters are transmitted to the server through the interface type, and the parameters include the specified knowledge identifier and the type of uploaded files. Then, the overall slicing is performed by assembling the specified knowledge ID. For example, there is an online education platform where users can upload course materials such as handouts, videos, test questions, etc. These materials need to be organized and classified according to different knowledge points or chapters. When a user uploads a PDF document containing multiple chapters, the system automatically identifies each chapter based on the document content and creates a "knowledge slice" for each chapter. The front end packages this information (for example, chapter title, page range, etc.) together with the file itself and passes it to the server through the API interface. In this process, the file type (PDF), slicing strategy (by chapter), and specific chapter information are transmitted as parameters. Then, the server generates a unique knowledge ID for each chapter and returns the processing results to the front end, including the knowledge ID, status (success / failure) and other relevant information of each chapter. After receiving the response from the server, the front end checks the status of each chapter. If any chapter fails to be processed successfully, the user is prompted to make corrections; otherwise, proceed to the next step. Once all chapters are successfully processed and obtain their respective knowledge IDs, the front end will assemble these IDs into an overall structure, indicating that the original file has been correctly decomposed into multiple knowledge points.

[0027] S500, inserting multiple knowledge fragments into different types of databases to complete classification.

[0028] In this embodiment, the knowledge fragments are inserted into different types of databases to complete the classification through the knowledge fragment management and scheduling module. The knowledge fragment management and scheduling module is the core component in the AI ​​intelligent operation and maintenance platform responsible for efficient management and scheduling of sliced ​​knowledge data. It ensures that the storage, retrieval, update and allocation process of knowledge fragments is fast and accurate, and can flexibly adjust resource allocation strategies according to business needs. This module combines the design principles of the dynamic vector library (elasticsearch) and the dynamic vector model (bge-large) to maintain a lightweight metadata library, recording the location information and attributes of each knowledge fragment (such as version number, creation time, etc.) for quick positioning and management.

[0029] The knowledge fragment management scheduling module mainly provides a simple way for users to enable or disable multiple knowledge fragments at one time and control whether they can be used for training or other processing. For example, freezing some content during model maintenance, or temporarily hiding certain sensitive information, the status management mechanism of the background service usually involves changes in status fields (such as status), and the corresponding permission check logic to ensure the security and consistency of operations. The results of data slicing are seamlessly passed to the knowledge fragment management scheduling module through the API interface. The knowledge fragment management scheduling module is responsible for subsequent storage, classification and retrieval work. It can dynamically select knowledge to be stored; such as: inserting type A text data into the mongodb database, and inserting type B vector data into the ElasticSearch database; completing efficient dynamic classification.

[0030] In an optional implementation of the first aspect of the present invention, creating a knowledge entry according to the uploaded file by using a selected knowledge parser, and generating a corresponding knowledge identifier for the knowledge entry includes: According to the uploaded file and the preset knowledge base, the knowledge entry is created through the selected knowledge parser; according to the preset knowledge tag and tag mapping relationship, the knowledge entry is inserted into the database to associate the relevant knowledge entries in the database; and the corresponding knowledge identifier is generated for the knowledge entry.

[0031] In this embodiment, by selecting different knowledge parsers, the browser passes the business parameters required by the parsing parameter generation interface to the server, inserts the result into a piece of data into the persistent database according to the preset knowledge tag and tag mapping relationship, and returns the knowledge id for interaction, and the user can modify and edit and update the metadata. On the server side, the received request data will be further processed and compared with the existing knowledge base to determine the most matching answer or create a new knowledge entry. At this time, the system will create a knowledge entry based on the pre-set knowledge tags (such as subject categories, keywords, etc.) and tag mapping relationships (such as associating other similar questions or answers). For example, if the question is about inheritance of Python classes, the system may create "Python" and "inheritance" tag fields for this record and associate it with all related knowledge entries. Once the data is successfully inserted into the database, the system will assign a unique knowledge id to the newly created knowledge entry, which is then returned to the client so that the user can reference or link to this specific knowledge point in the future.

[0032] In an optional implementation of the first aspect of the present invention, according to the specified knowledge identifier and the type of the uploaded file, the content of the uploaded file is sliced ​​by a selected knowledge slicing strategy to obtain multiple knowledge fragments including: Perform feature extraction on the content of the uploaded file; obtain the corresponding automatic slicer through the selected knowledge slicing strategy, and the automatic slicer adopts the optimization algorithm of the user-defined objective function; slice the content of the uploaded file through the automatic slicer according to the result of feature extraction, the specified knowledge identifier and the type of uploaded file to obtain multiple knowledge fragments; convert the multiple knowledge fragments into different formats and send them to the client.

[0033] In this embodiment, the automatic slicer is selected for slicing by the selected knowledge slicing strategy, and the specific steps of the automatic slicer interface are as follows: (1) Based on the results of feature extraction, use optimization algorithms to slice the data reasonably. This may include but is not limited to cluster analysis, decision trees, random forests, neural networks, and other methods.

[0034] (2) The algorithm can determine the optimal slicing strategy based on the user-defined objective function (such as maximizing the amount of information, minimizing redundancy, etc.) and knowledge labels.

[0035] (3) The sliced ​​data subsets are presented to users in a format that is easy to understand and use, providing basic data for subsequent module previews and user feedback for further adjustment and improvement of the slicing logic.

[0036] In an optional implementation of the first aspect of the present invention, according to the result of feature extraction, the specified knowledge identifier and the type of the uploaded file, the content of the uploaded file is sliced ​​by an automatic slicer to obtain multiple knowledge fragments including: If the automatic slicer is a regular slicer, the matching rules defined based on the regular expression engine are obtained; according to the result of feature extraction, the specified knowledge identifier, the type of uploaded file, and the matching rules, the content of the uploaded file is gradually sliced ​​in a coarse-to-fine manner through the regular slicer to obtain multiple knowledge fragments.

[0037] In this embodiment, if Figure 2 As shown, by selecting the dynamic regular slice strategy, using the dynamic segment to add text to match each segment title header for text segmentation and data cleaning, using dynamic wildcards to introduce domain-specific knowledge graphs, combined with custom rules, ensure that the sliced ​​data subset can accurately reflect the characteristics and requirements of a specific industry. For example, in the budget management scenario, priority is given to parts related to regulatory updates and policy changes. Specifically, if a regular slicer is selected, the specific steps of the regular slicer interface are as follows: (1) The regular expression engine allows users to define complex matching rules. These rules can be used to locate custom patterns in the text and perform cutting operations accordingly, such as adding text in front of the fragment, allowing users to append a specified text content to each data fragment that is cut out. This is particularly useful for creating information blocks with contextual prompts, such as adding metadata such as source descriptions or timestamps to each document. Wildcards. In addition to using regular expressions directly, the slicer also introduces the concept of wildcards, making rule definitions more flexible. For example, * can represent any number of arbitrary characters; ? represents a single arbitrary character. With the help of wildcards, even when faced with text with unfixed formats, you can easily customize an effective cutting plan.

[0038] (2) The data after cutting is gradually refined in a coarse-to-fine manner. First, a large-scale preliminary segmentation result is quickly generated, and then further refined to the final required level of detail, providing basic data for subsequent modules with high speed and accuracy.

[0039] In an optional implementation of the first aspect of the present invention, extracting features from the content of the uploaded file includes: Parse the uploaded file to obtain binary stream data; compare the binary stream data with the rules of the preset regular expression matcher to determine whether the content of the uploaded file conforms to the expected format and specifications; if so, perform feature extraction on the content of the uploaded file.

[0040] In this embodiment, according to the assembled knowledge id information, the basic data and business data to be sliced ​​are pushed to the target system interface. After the target system interface receives the data reporting information, the transmission type is determined through the regular matcher to clarify the rules of the regular expression matcher. The binary stream data received by the back end needs to be correctly parsed, and the rules of the preset regular expression matcher are compared with the binary stream data to determine whether the content of the uploaded file conforms to the expected format and specifications. If it conforms to the format and specifications, the data is returned to the preview module for the next editing, and the knowledge id and status information fed back by the system are returned; if it does not conform to the specifications, the source system error message is returned to prompt the user to modify the dynamic slicing strategy, and relevant logs are recorded at the same time; after the above operations are completed, the user can still select the advanced intelligent calculation strategy, and transmit different parameters to the server for calculation according to different check boxes; after the overall interface returns the successful processing, the user clicks the next step to complete the information processing and persist the updated information to the database.

[0041] In an optional implementation of the first aspect of the present invention, according to the specified knowledge identifier and the type of the uploaded file, the content of the uploaded file is sliced ​​by the selected knowledge slicing strategy to obtain multiple knowledge fragments, and further includes: Generate parent-child structure documents with hierarchical relationships based on multiple knowledge fragments; construct parent-child layer knowledge entries based on the content of the uploaded file and the parent-child structure document and dynamically adapt and add keyword tags based on the pre-trained language model to facilitate search or classification; identify and delete meaningless or redundant knowledge fragment content in the knowledge base.

[0042] In this embodiment, by converting the pre-sliced ​​unstructured or semi-structured data in the knowledge base into parent-child structured documents with further hierarchical relationships, based on the pre-trained language model, the text content is understood and classified into appropriate categories, and the parent-child layer knowledge items are dynamically adapted to add appropriate keyword tags to facilitate subsequent search and classification management. For the automatically generated keywords, visualization tools are provided for experts to review and correct to ensure the accuracy and applicability of the tags. By automatically clearing blank fragments, meaningless or redundant content fragments in the knowledge base are identified and removed to ensure the neatness and effectiveness of the data.

[0043] like Figure 2 As shown in the figure, it provides advanced processing of slice strategy data, including: (1) Automatically generate parent-child structured documents, convert the unstructured or semi-structured data in the basic knowledge into parent-child structured documents with hierarchical relationships, use NLP technology to perform word segmentation and syntactic analysis on the input text, and identify structural elements such as titles, paragraphs, and lists.

[0044] (2) Automatically generate similar questions and answers, and mine similar question-answer pairs from the existing knowledge base to help users quickly find answers to related questions. Use word embedding or sentence embedding methods to convert questions and answers into vector form to facilitate similarity calculation. Use clustering algorithms such as K-means and DBSCAN to identify a group of similar questions and select the most representative question-answer pairs for display.

[0045] (3) Automatically generate keyword tags and add appropriate keyword tags to each knowledge item to facilitate subsequent search and classification management. Use algorithms such as TF-IDF and TextRank to extract important words from the text as candidate keywords.

[0046] (4) Automatically remove blank segments, identify and remove meaningless or redundant content segments in the knowledge base to ensure the cleanliness and validity of the data. Check whether there are continuous blank characters, punctuation marks or other invalid marks in the text and delete them. Use metrics such as Levenshtein Distance or cosine similarity to find and merge highly similar repeated segments to avoid information redundancy.

[0047] In an optional implementation of the first aspect of the present invention, inserting the plurality of knowledge fragments into different types of databases to complete classification includes: Send multiple knowledge fragments to the client for secondary adjustment; insert the adjusted knowledge fragments into different types of databases to complete classification.

[0048] In this embodiment, the knowledge fragments obtained after slicing are uploaded to the knowledge segmentation preview module for preview preprocessing. Specifically, the preprocessed knowledge fragments are displayed to the current page module for manual secondary adjustment, and the final slice knowledge id is returned. According to the above final processed knowledge, the user dynamically chooses to insert different types of databases to complete efficient classification. Among them, the page is previewed through memory management, and an efficient memory management and caching mechanism is adopted to reduce unnecessary I / O operations, ensuring that data loading and transmission during the slicing process are as fast as possible and occupy as few resources. For data sets that have completed preliminary slicing, only partial operations are performed on the newly added or changed parts to avoid duplication of work and save time and resources.

[0049] Users can divide long documents into multiple logically coherent segments according to actual needs. Each segment is edited as an independent unit, providing an intuitive drag-and-drop interface that allows users to freely adjust the boundaries between segments, ensuring that the content organization meets expectations and allowing content insertion and reorganization. The information of each segment is stored in a tree or linked list structure, which facilitates efficient traversal and retrieval. Each node contains attributes such as segment ID, content, creation time, and last modification time. Necessary metadata (such as creation time, author, number of revisions, etc.) is attached to each segment to facilitate later statistical analysis and permission control.

[0050] See also Figure 3 A second aspect of the present invention provides a data slicing system, the data slicing system comprising: The operation instruction receiving module 10 is used to receive the knowledge operation instruction uploaded by the user, and the knowledge operation instruction carries the uploaded file; The knowledge entry creation module 20 is used to create a knowledge entry according to the uploaded file through the selected knowledge parser, and generate a corresponding knowledge identifier for the knowledge entry; The slicing instruction receiving module 30 is used to receive the knowledge slicing instruction sent by the user, and the knowledge slicing instruction carries a specified knowledge identifier; The slicing strategy selection module 40 is used to slice the content of the uploaded file according to the specified knowledge identifier and the type of the uploaded file through the selected knowledge slicing strategy to obtain multiple knowledge fragments; The knowledge fragment classification module 50 is used to insert multiple knowledge fragments into different types of databases to complete classification.

[0051] In an optional implementation of the second aspect of the present invention, the knowledge entry creation module 20 is also used to create knowledge entries based on the uploaded file and the preset knowledge base through the selected knowledge parser; insert the knowledge entries into the database according to the preset knowledge tags and label mapping relationships to associate related knowledge entries in the database; and generate corresponding knowledge identifiers for the knowledge entries.

[0052] In an optional implementation of the second aspect of the present invention, the slicing strategy selection module 40 is also used to perform feature extraction on the content of the uploaded file; through the selected knowledge slicing strategy, a corresponding automatic slicer is obtained, and the automatic slicer adopts an optimization algorithm of a user-predefined objective function; according to the result of feature extraction, the specified knowledge identifier and the type of uploaded file, the content of the uploaded file is sliced ​​by the automatic slicer to obtain multiple knowledge fragments; the multiple knowledge fragments are format-converted and sent to the client.

[0053] In an optional implementation of the second aspect of the present invention, the slicing strategy selection module 40 is also used to obtain matching rules defined based on a regular expression engine if the automatic slicer is a regular slicer; based on the result of feature extraction, the specified knowledge identifier, the type of uploaded file, and the matching rules, the regular slicer is used to gradually slice the content of the uploaded file in a coarse-to-fine manner to obtain multiple knowledge fragments.

[0054] In an optional implementation of the second aspect of the present invention, the slicing strategy selection module 40 is also used to parse the uploaded file to obtain binary stream data; compare the binary stream data with the rules of a preset regular expression matcher to determine whether the content of the uploaded file conforms to the expected format and specifications; if so, perform feature extraction on the content of the uploaded file.

[0055] In an optional implementation of the second aspect of the present invention, the data slicing system further includes: The slicing advanced processing module is used to generate parent-child structure documents with hierarchical relationships based on multiple knowledge fragments; based on the content of the uploaded file and the parent-child structure document, it builds parent-child layer knowledge entries and dynamically adapts and adds keyword tags to facilitate search or classification based on the pre-trained language model; it identifies and deletes meaningless or redundant knowledge fragment content in the knowledge base.

[0056] In an optional implementation of the second aspect of the present invention, the knowledge fragment classification module 50 is also used to send multiple knowledge fragments to the client for secondary adjustment; and insert the adjusted knowledge fragments into different types of databases to complete the classification.

[0057] Figure 4: is a structural schematic diagram of a data slicing device provided by an embodiment of the present invention. The data slicing device 500 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 510 (for example, one or more processors) and a memory 520, and one or more storage media 530 (for example, one or more mass storage devices) storing application programs 533 or data 532. Among them, the memory 520 and the storage medium 530 can be short-term storage or permanent storage. The program stored in the storage medium 530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the data slicing device 500. Furthermore, the processor 510 can be configured to communicate with the storage medium 530 and execute a series of instruction operations in the storage medium 530 on the data slicing device 500.

[0058] The data slicing device 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input and output interfaces 560, and / or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, Free BSD, etc. Those skilled in the art will appreciate that Figure 4 The data slicing device structure shown does not constitute a limitation on the data slicing device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0059] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium, and when the instructions are executed on a computer, the computer executes the steps of the data slicing method.

[0060] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0061] In addition, although each operation is described in a specific order, this should be understood as requiring such operation to be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations should be performed to obtain desired results. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination mode.

[0062] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A data slicing method, characterized in that: The data slicing method comprises: Receiving a knowledge operation instruction uploaded by a user, wherein the knowledge operation instruction carries an uploaded file; According to the uploaded file, a knowledge entry is created by a selected knowledge parser, and a corresponding knowledge identifier is generated for the knowledge entry; Receiving a knowledge slicing instruction sent by a user, wherein the knowledge slicing instruction carries a specified knowledge identifier; According to the specified knowledge identifier and the type of the uploaded file, the content of the uploaded file is sliced ​​by a selected knowledge slicing strategy to obtain multiple knowledge fragments; The multiple knowledge fragments are respectively inserted into different types of databases to complete the classification.

2. The data slicing method according to claim 1, characterized in that: The step of creating a knowledge entry according to the uploaded file by using a selected knowledge parser, and generating a corresponding knowledge identifier for the knowledge entry comprises: According to the uploaded file and the preset knowledge base, a knowledge entry is created through the selected knowledge parser; Inserting the knowledge item into the database according to the preset knowledge tag and tag mapping relationship to associate the relevant knowledge item in the database; A corresponding knowledge identifier is generated for the knowledge item.

3. The data slicing method according to claim 1, characterized in that: The method of slicing the content of the uploaded file according to the specified knowledge identifier and the type of the uploaded file by using the selected knowledge slicing strategy to obtain multiple knowledge fragments includes: Performing feature extraction on the content of the uploaded file; By using the selected knowledge slicing strategy, a corresponding automatic slicer is obtained, wherein the automatic slicer adopts an optimization algorithm of a user-predefined objective function; According to the result of feature extraction, the specified knowledge identifier and the type of the uploaded file, the content of the uploaded file is sliced ​​by the automatic slicer to obtain multiple knowledge fragments; The multiple knowledge fragments are formatted and sent to the client.

4. The data slicing method according to claim 3, characterized in that: The step of slicing the content of the uploaded file by the automatic slicer according to the result of feature extraction, the specified knowledge identifier and the type of the uploaded file to obtain a plurality of knowledge fragments includes: If the automatic slicer is a regular slicer, obtaining a matching rule defined based on a regular expression engine; According to the result of feature extraction, the specified knowledge identifier, the type of the uploaded file, and the matching rule, the content of the uploaded file is sliced ​​step by step in a coarse-to-fine manner through the regular slicer to obtain multiple knowledge fragments.

5. The data slicing method according to claim 3, characterized in that: The feature extraction of the content of the uploaded file includes: Parsing the uploaded file to obtain binary stream data; By comparing the rules of a preset regular expression matcher with the binary stream data, it is determined whether the content of the uploaded file conforms to the expected format and specifications; If it meets the requirements, feature extraction is performed on the content of the uploaded file.

6. The data slicing method according to claim 1, characterized in that: After slicing the content of the uploaded file according to the specified knowledge identifier and the type of the uploaded file and obtaining a plurality of knowledge fragments through the selected knowledge slicing strategy, the method further includes: Generate a parent-child structure document with a hierarchical relationship according to the multiple knowledge fragments; According to the content of the uploaded file and the parent-child structure document, based on the pre-trained language model, parent-child layer knowledge items are constructed and keyword tags are dynamically adapted and added to facilitate search or classification; Identify and delete meaningless or redundant knowledge fragments in the knowledge base.

7. The data slicing method according to claim 1, characterized in that: The step of inserting the plurality of knowledge fragments into different types of databases to complete the classification comprises: sending the plurality of knowledge fragments to a client for secondary adjustment; Insert the adjusted knowledge fragments into different types of databases to complete the classification.

8. A data slicing system, characterized in that: The data slicing system comprises: An operation instruction receiving module, used to receive a knowledge operation instruction uploaded by a user, wherein the knowledge operation instruction carries an uploaded file; A knowledge entry creation module, used to create a knowledge entry according to the uploaded file through a selected knowledge parser, and generate a corresponding knowledge identifier for the knowledge entry; A slicing instruction receiving module, used to receive a knowledge slicing instruction sent by a user, wherein the knowledge slicing instruction carries a specified knowledge identifier; A slicing strategy selection module, used to slice the content of the uploaded file according to the specified knowledge identifier and the type of the uploaded file, and obtain multiple knowledge fragments through the selected knowledge slicing strategy; The knowledge fragment classification module is used to insert the multiple knowledge fragments into different types of databases to complete classification.

9. A data slicing device, characterized in that: The data slicing device comprises: a memory and at least one processor, the memory stores instructions, and the memory and the at least one processor are interconnected via a line; The at least one processor calls the instruction in the memory so that the data slicing device executes the data slicing method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data slicing method according to any one of claims 1 to 7 is implemented.

Images