Teaching material book data processing method and device, electronic equipment and storage medium

Abstract

The invention provides a textbook data processing method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring an original textbook data file; extracting a data type of the original textbook data file, and analyzing the original textbook data file according to the data type to obtain structured content data; inputting the structured content data and the corresponding data type into a preset multi-data type hybrid model for processing, and obtaining processed result data and the data type corresponding to the structured content data; wherein the multi-modal data set is used for generating a multi-modal sample set for training the corresponding wisdom education large model according to the training demand of the wisdom education large model. According to the method, the efficiency and accuracy of constructing the multi-modal data set can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, specifically to methods, apparatuses, electronic devices, and storage media for processing textbook and book data. Background Technology

[0002] Smart education is at the forefront of modern educational technology. With the rapid development of large-scale modeling technology, intelligent teaching assistance, smart tutoring, and homework grading are gradually becoming a reality. However, to maximize the effectiveness of large-scale models in smart education, it is necessary to process a large amount of high-quality, specialized textbook data. This data is not only an important support for training large-scale models but also a key resource for model fine-tuning and optimization. Especially in specialized fields such as science and engineering, medicine, and law, the content of textbooks involves multiple data formats, and processing this complex data presents many challenges.

[0003] The challenges of processing textbook and book data largely stem from the diversity of data formats and the complexity of data sources. Textbook and book data in the field of smart education is highly complex, encompassing various data types such as text, formulas and tables, images, and code. Currently, the application of large-scale models in smart education is mostly focused on general content levels. When dealing with more complex textbook and book data, existing general models have significant limitations in understanding and processing the data. Therefore, the efficiency and accuracy of constructing multimodal datasets are currently relatively low. Summary of the Invention

[0004] In view of this, this application provides a method, apparatus, electronic device, and storage medium for processing textbook data, which can improve the efficiency and accuracy of constructing multimodal datasets.

[0005] To solve the above-mentioned technical problems, the technical solution of this application is implemented as follows: In one embodiment, a method for processing textbook data is provided, the method comprising: Obtain the original textbook data file; Extract the data type from the original textbook data file, and parse the original textbook data file according to the data type to obtain structured content data; The structured content data and its corresponding data type are input into a preset multi-data type hybrid model for processing to obtain the processed result data and data type corresponding to the structured content data. The preset multi-data type hybrid model includes a routing module and preset processing models corresponding to various data types. The routing module routes the original data to the preset processing model corresponding to the input data type. The preset processing model processes the original data routed by the routing module and outputs the corresponding result data and data type. The resulting data is then deduplicated and filtered. The deduplicated and filtered result data is stored in the corresponding format according to the corresponding data type to generate a multimodal dataset; wherein, the multimodal dataset is used to generate a multimodal sample set for training the corresponding smart education big model according to the training requirements of the smart education big model.

[0006] The deduplication and filtering of the result data includes: Identify and remove completely duplicate paragraphs, formulas, or code snippets using string matching methods; By using hash algorithms to extract features from content, similar but not completely different text or code can be identified. Remove parts that are incorrectly identified by OCR or are incomplete, and eliminate invalid or noisy data according to format validation rules.

[0007] The data types mentioned include: text, formulas and tables, images, and code; The preset processing models include: preset processing models for text, preset processing models for formulas and tables, preset processing models for images, and preset processing models for code.

[0008] in, The preset processing model corresponding to the text is a model built based on GPT-3.5; The preset processing model corresponding to the formulas and tables is a model built based on GOT; The preset processing model corresponding to the image is a model built based on BLIP; The preset processing model corresponding to the code is a model built based on CodeBERT.

[0009] The step of storing the deduplication and filtering results in the corresponding format according to the corresponding data type includes: If the data type corresponding to the result data is text, then the corresponding storage format is text; If the data type corresponding to the result data is a formula, then the corresponding storage format is LaTeX or MathML. If the data type corresponding to the result data is tabular data, then the corresponding storage format is CSV or Excel file format; If the data type corresponding to the result data is an image, then the corresponding storage format is the specified image format; If the data type corresponding to the result data is code, then the corresponding format stored is the standardized format of an executable file or code fragment.

[0010] Each piece of data in the multimodal dataset also stores additional metadata; the additional metadata is used to retrieve and manage the data in the multimodal dataset; the additional metadata includes the file source, generation time, processing tool, and data type.

[0011] In another embodiment, a textbook data processing device is provided, the device comprising: The acquisition unit is used to acquire the original textbook data file; The analysis unit is used to extract the data type of the original textbook data file and parse the original textbook data file according to the data type to obtain structured content data. A processing unit is used to input the structured content data and its corresponding data type into a preset multi-data type hybrid model for processing, and to obtain the processed result data and data type corresponding to the structured content data; wherein, the preset multi-data type hybrid model includes a routing module and preset processing models corresponding to various data types; the routing module is used to route the corresponding original data to the preset processing model corresponding to the data type according to the input data type; the preset processing model is used to process the original data routed by the routing module and output the corresponding result data and data type; The deduplication and filtering unit is used to deduplicatize and filter the result data. The generation unit is used to store the deduplicated and filtered result data in the corresponding format according to the corresponding data type to generate a multimodal dataset; wherein, the multimodal dataset is used to generate a multimodal sample set for training the corresponding smart education big model according to the training requirements of the smart education big model.

[0012] in, The deduplication filtering unit is specifically used to deduplicatize the result data and, during the filtering process, identify and remove completely duplicate paragraphs, formulas, or code snippets by using string matching; extract features from the content using a hash algorithm to identify similar but not completely different text or code; remove parts that are incorrectly identified by OCR or are incomplete; and eliminate invalid or noisy data according to format validation rules.

[0013] In another embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a textbook data processing method.

[0014] In another embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements a data processing method based on textbooks and books.

[0015] As can be seen from the above technical solution, the embodiments described above address the various data types in textbook data by using preset processing models corresponding to different data types within a pre-defined multi-data type hybrid model. After deduplication and filtering, a multimodal dataset is generated, making it easier to generate a multimodal sample set for training a corresponding smart education model. This solution improves the efficiency and accuracy of constructing multimodal datasets. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the structure of the preset multi-data type hybrid model in the embodiments of this application; Figure 2 This is a schematic diagram of a textbook data processing flow in an embodiment of this application; Figure 3 This is a schematic diagram of another textbook data processing flow in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of the textbook and book data processing device in the embodiments of this application; Figure 5 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe the order or sequence of objects. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0020] The technical solution of the present invention will be described in detail below with reference to specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0021] Smart education is at the forefront of modern educational technology. With the rapid development of large-scale modeling technology, intelligent teaching assistance, smart tutoring, and homework grading are gradually becoming a reality. However, to maximize the effectiveness of large-scale models in smart education, it is necessary to process a large amount of high-quality, specialized textbook data. This data is not only an important support for training large-scale models but also a key resource for model fine-tuning and optimization. Especially in specialized fields such as science and engineering, medicine, and law, the content of textbooks involves multiple data formats, and processing this complex data presents many challenges.

[0022] The challenges of textbook and book data processing largely stem from the diversity of data formats and the complexity of data sources. Textbook and book data in the field of smart education is highly complex, encompassing multiple data types: Text: A large amount of knowledge in textbooks is presented in text form, including textbook content, exercises, solutions, and annotations. This text data needs to be processed through semantic understanding, sentiment analysis, and translation, especially in scenarios involving student question-and-answer sessions or teacher lesson preparation, where models need to accurately understand and generate specialized content.

[0023] Formulas and Tables: Numerous mathematical formulas in STEM textbooks serve as an important form of knowledge expression. These formulas are typically presented in LaTeX or MathML formats, but in many textbooks they are embedded as images, making extraction and parsing them very complex. A large amount of data in textbooks is presented in tables and charts to illustrate relationships and comparisons between data points. The data information and contextual relationships contained in these charts are crucial for student comprehension.

[0024] Images and Code: Images commonly found in textbooks, such as experimental diagrams, schematic diagrams, and structural diagrams, as well as code, such as code snippets in the field of computer science, also need to be understood and processed by the model. This involves multimodal tasks such as image feature extraction and code semantic understanding.

[0025] Data sources in the field of smart education are extensive and diverse. Textbook data may come from different publishers, different versions, and even different countries. This results in inconsistent data formats, making processing quite difficult. Different textbooks may have significant differences in the order of knowledge points, narrative style, and the way tables and formulas are presented. Furthermore, some data may originate from scanned documents or images, requiring complex OCR (Optical Character Recognition) processing, which further increases the difficulty of data processing.

[0026] Currently, the application of large-scale models in smart education is mostly concentrated on processing general content. For example, large-scale language models such as GPT-3.5 can process text data in textbooks and books, supporting teachers' lesson preparation and students' learning through tasks such as text generation, question answering, and translation. However, when dealing with more complex data, such as mathematical formulas, tables, and charts, existing general-purpose models still have limitations in understanding and generating such data. Especially when combined with multimodal data such as images and code, existing multimodal large-scale models such as GPT-4v, while capable of handling the combination of images and text, are not accurate or detailed enough in processing complex professional educational content. For example, when processing experimental data charts and complex function graphs in complex STEM textbooks, they still face problems of low accuracy and insufficient contextual understanding. In addition, for code parsing, current technologies also lack the ability to efficiently understand and execute cross-language and multi-structure code.

[0027] Each expert network focuses on solving a specific type of data or task. The final solution to the problem is obtained by weighted combination of the outputs of each expert network. However, the processing is usually performed on one type of data. Therefore, the efficiency and accuracy of processing textbook data are relatively low, and consequently, the efficiency and accuracy of constructing multimodal datasets are also relatively low.

[0028] To address the aforementioned technical issues, this application provides a method for processing textbook data. For textbook data containing various data types, a pre-defined processing model corresponding to each data type within a pre-defined multi-data type hybrid model is used to process the corresponding data types. After deduplication and filtering, a multimodal dataset is generated, making it easier to generate a multimodal sample set for training a corresponding smart education model. This solution improves the efficiency and accuracy of constructing multimodal datasets.

[0029] In this embodiment of the application, a preset multi-data type hybrid model needs to be established before processing the textbook data.

[0030] See Figure 1 , Figure 1 This is a schematic diagram of the structure of the preset multi-data type hybrid model in the embodiments of this application. Figure 1 The preset multi-data type hybrid model includes: a routing module and preset processing models corresponding to various data types; The routing module is used to route the corresponding raw data to the preset processing model corresponding to the data type based on the input data type. Preset processing models corresponding to various data types are used to process the raw data routed from the routing module and output the corresponding result data and data types; Data types include: text, formulas and tables, images and code, etc. Therefore, the preset processing models corresponding to these various data types are as follows: preset processing models for text, preset processing models for formulas and tables, preset processing models for images and preset processing models for code.

[0031] If a new data type emerges, a preset processing model for the corresponding data type can be added as disclosed in this application. Therefore, the preset multi-data type hybrid model provided in the embodiments of this application has good flexibility and scalability.

[0032] The preset processing models corresponding to various data types in the embodiments of this application can directly use the better models currently available for that data type. There are no specific restrictions on the specific models used. The following are some examples of what can be used: The pre-defined processing model for the text is based on GPT-3.5 (Generative Pre-trained Transformer 3.5). The preset processing model corresponding to the code is a model built on CodeBERT (CodeBidirectional Encoder Representations from Transformers).

[0033] GPT-3.5 was used for text processing. GPT-3.5 is a widely used large-scale pre-trained language model with powerful natural language processing capabilities, capable of generating high-quality question-and-answer, summarization, and automated content generation. Input: Segmented text data, typically 100-500 characters in length. Output: Processed text features, such as a 1024-dimensional vector, or the results of the text generation task, such as a courseware outline or question-and-answer pairs.

[0034] In practice, it is not limited to GPT-3.5; BERT can also be used to build the model, and other generative pre-trained models can be used depending on the different task requirements.

[0035] For large-scale text data, parallel processing using distributed processing frameworks such as MapReduce can improve processing efficiency.

[0036] The preset processing model corresponding to the formulas and tables is a model built based on GOT (General OCR Theory); GOT is a unified end-to-end OCR solution capable of efficiently handling inputs across various scenarios and document styles: Input: Image data or PDF fragments containing mathematical formulas and tables. Output: Structured LaTeX or MathML formulas, or tabular data in CSV / JSON format.

[0037] Specifically, not limited to GOT, mathematical formulas and tables are often embedded in PDF documents and need to be extracted using optical character recognition (OCR) tools. Open-source or commercial OCR tools such as MathPix and Tesseract can be used.

[0038] The pre-defined processing model for the image is based on BLIP (Bootstrapping Language-Image Pre-training). BLIP is a powerful image-text alignment model, particularly suitable for multimodal tasks such as image content analysis and generation. Input: Image data or image fragments extracted from PDFs. Output: Image feature vectors (512-dimensional or 1280-dimensional), and the multimodal semantic alignment results between the image and text.

[0039] Image data processing, besides BLIP, can also be handled by multimodal models such as CLIP (Contrastive Language–Image Pretraining) and ViT (Vision Transformer) to extract feature vectors from images and align them with text or other content. CLIP models can understand the relationship between images and text and are suitable for constructing multimodal data.

[0040] In addition, traditional computer vision algorithms such as OpenCV (Open Source Computer Vision Library) can be used for image preprocessing, such as cropping, enhancement, and noise reduction.

[0041] The default processing model corresponding to the code is based on the CodeBERT (Code Bidirectional Encoder Representations from Transformers) model.

[0042] CodeBERT, a bidirectional transformer for code, can efficiently handle code parsing and semantic understanding tasks in various programming languages: Input: Code snippets, supporting multiple programming languages ​​(such as Python, C++, Java). Output: Syntax tree, functional summary, and feature vectors of the code logic, used for code generation tasks or intelligent programming tutoring.

[0043] In practical implementation, it is not limited to CodeBERT; it can also be parsed and understood using models specifically designed for code understanding and generation, such as Codex and GPT-3.5. The parsed code will be stored in an executable format, retaining key information such as comments and variable names.

[0044] The data processing procedure for textbooks and books in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0045] See Figure 2 , Figure 2 This is a schematic diagram of a textbook data processing flow according to an embodiment of this application. The specific steps are as follows: Step 201: Obtain the original textbook data file.

[0046] Obtain data from various sources, including PDF files, images, and code snippets, from textbooks and books.

[0047] These original teaching material data files can be obtained via the network or local address devices.

[0048] Step 202: Extract the data type of the original textbook data file, and parse the original textbook data file according to the data type to obtain structured content data.

[0049] This step utilizes parsing tools such as PyMuPDF and PDFMiner to extract data from PDFs of various data types, including text, images, tables, mathematical formulas, and code snippets, to obtain processable data, i.e., structured content data.

[0050] Step 203: Input the structured content data and its corresponding data type into a preset multi-data type hybrid model for processing to obtain the processed result data and data type corresponding to the structured content data.

[0051] Structured content data and data types are input into a preset multi-data type hybrid model. In this model, the routing module categorizes the structured content data according to the data type. Structured content data of type text is routed to the preset processing model corresponding to text, structured content data of type formulas and tables is routed to the preset processing model corresponding to formulas and tables, structured content data of type images is routed to the preset processing model corresponding to images, and structured content data of type code is routed to the preset processing model corresponding to code.

[0052] The pre-defined processing model for the received data can directly process the received data.

[0053] Step 204: Deduplicatize and filter the result data.

[0054] Step 205: Store the deduplicated and filtered result data in the corresponding format according to the corresponding data type to generate a multimodal dataset; wherein, the multimodal dataset is used to generate a multimodal sample set for training the corresponding smart education big model according to the training requirements of the smart education big model.

[0055] Specifically, the deduplication and filtering results are stored in the corresponding formats according to their data types, including: If the data type corresponding to the result is text, then the corresponding format stored is text; If the data type corresponding to the result is a formula, then the corresponding storage format is LaTeX or MathML. If the data type of the result is tabular data, then the corresponding storage format is CSV or Excel file format; If the data type corresponding to the result is an image, then the corresponding storage format is the specified image format; If the data type corresponding to the result is code, then the corresponding format stored is the standardized format of an executable file or code fragment.

[0056] This embodiment addresses the various data types within textbook data by employing pre-defined processing models corresponding to each data type within a multi-data type hybrid model. After deduplication and filtering, a multimodal dataset is generated, making it easier to generate multimodal sample sets for training corresponding smart education models. This approach improves the efficiency and accuracy of constructing multimodal datasets.

[0057] In another example, step 204 involves deduplication and filtering of the resulting data, specifically including: Identify and remove completely duplicate paragraphs, formulas, or code snippets using string matching methods; By using hash algorithms to extract features from content, similar but not completely different text or code can be identified. Remove parts that are incorrectly identified by OCR or are incomplete, and eliminate invalid or noisy data according to format validation rules.

[0058] See Figure 3 , Figure 3 This is a schematic diagram of another textbook data processing flow in an embodiment of this application. The specific steps are as follows: Step 301: Obtain the original textbook data file.

[0059] Obtain data from various sources, including PDF files, images, and code snippets, from textbooks and books.

[0060] These original teaching material data files can be obtained via the network or local address devices.

[0061] Step 302: Extract the data type of the original textbook data file, and parse the original textbook data file according to the data type to obtain structured content data.

[0062] This step utilizes parsing tools such as PyMuPDF and PDFMiner to extract data from PDFs of various data types, including text, images, tables, mathematical formulas, and code snippets, to obtain processable data, i.e., structured content data.

[0063] Step 303: Input the structured content data and its corresponding data type into a preset multi-data type hybrid model for processing to obtain the processed result data and data type corresponding to the structured content data. Simultaneously execute steps 304 and 305.

[0064] Structured content data and data types are input into a preset multi-data type hybrid model. In this model, the routing module categorizes the structured content data according to the data type. Structured content data of type text is routed to the preset processing model corresponding to text, structured content data of type formulas and tables is routed to the preset processing model corresponding to formulas and tables, structured content data of type images is routed to the preset processing model corresponding to images, and structured content data of type code is routed to the preset processing model corresponding to code.

[0065] The pre-defined processing model for the received data can directly process the received data.

[0066] Step 304: Identify and remove completely duplicate paragraphs, formulas, or code snippets using string matching; proceed to step 306.

[0067] Step 305: Extract features from the content using a hash algorithm to identify similar but not completely different text or code.

[0068] Combining absolute matching and approximate matching for deduplication can accurately remove duplicate or similar content from teaching materials.

[0069] Techniques such as MinHash and SimHash can be used to identify nearly identical data, ensuring that the final generated spindle mode dataset has high uniqueness and diversity, thus providing a high-quality data foundation for training large models.

[0070] Step 306: Remove parts that are incorrectly recognized by OCR or are incomplete, and eliminate invalid or noisy data according to the format validation rules.

[0071] Step 307: The deduplicated and filtered result data is stored in the corresponding format according to the corresponding data type to generate a multimodal dataset; wherein, the multimodal dataset is used to generate a multimodal sample set for training the corresponding smart education big model according to the training requirements of the smart education big model.

[0072] This multimodal dataset not only stores the content and data type of each data entry, but also stores additional metadata. This additional metadata is used to retrieve and manage data within the multimodal dataset. The additional metadata includes the file source and generation time.

[0073] The source information of the document can be pre-configured to record the source, such as which textbook or book it is from, or which page of the content is from. The generation time records the time when the data was stored.

[0074] Specifically, the deduplication and filtering results are stored in the corresponding formats according to their data types, including: If the data type corresponding to the result is text, then the corresponding format stored is text; If the data type corresponding to the result is a formula, then the corresponding storage format is LaTeX or MathML. If the data type of the result is tabular data, then the corresponding storage format is CSV or Excel file format; If the data type corresponding to the result is an image, then the corresponding storage format is the specified image format; Specify the image format as PNG or JPEG; If the data type corresponding to the result is code, then the corresponding format stored is the standardized format of an executable file or code snippet, such as Python or JavaScript.

[0075] For code data, in practical implementation, it's possible to distinguish between storing it as source code text or an executable file, and choose the appropriate storage method based on specific needs. If readability and editability of the code need to be preserved, it should be stored in source code text format, ensuring that the code snippets conform to the standardized format of the corresponding programming language, such as Python code following the PEP8 specification; if it needs to be saved as an executable code file, it should be stored directly as an executable code file.

[0076] The multimodal dataset here can contain text, images, formulas and tables, and code, or it can contain only one of these or any combination thereof. The specific data included is related to the data types contained in the textbooks and books used for data processing. The method provided in this application enables the integration of data of different data types into the same multimodal dataset.

[0077] After the multimodal dataset is stored, when a large model related to smart education needs to be trained, a specific data format set, such as TFRecord or LMDB, is generated according to the task requirements to serve as training samples for the large model.

[0078] This embodiment addresses the various data types within textbook data by employing pre-defined processing models corresponding to each data type within a multi-data type hybrid model. After deduplication and filtering, a multimodal dataset is generated, making it easier to generate multimodal sample sets for training corresponding smart education models. This approach improves the efficiency and accuracy of constructing multimodal datasets.

[0079] The smart education model presented here can be applied to scenarios such as intelligent question answering, intelligent tutoring, and intelligent homework grading for students. The multimodal dataset in this application provides efficient data support and promotes the development of intelligent applications in the field of education.

[0080] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0081] Based on the same inventive concept, this application also provides a textbook data processing device. See also Figure 4 , Figure 4 This is a schematic diagram of the textbook data processing device in an embodiment of this application. The device includes: Acquisition unit 401 is used to acquire the original textbook data file; Analysis unit 402 is used to extract the data type of the original textbook data file and parse the original textbook data file according to the data type to obtain structured content data; The processing unit 403 is used to input structured content data and its corresponding data type into a preset multi-data type hybrid model for processing, and to obtain the processed result data and data type corresponding to the structured content data. The preset multi-data type hybrid model includes a routing module and preset processing models corresponding to various data types. The routing module is used to route the corresponding original data to the preset processing model corresponding to the data type according to the input data type. The preset processing model is used to process the original data routed by the routing module and output the corresponding result data and data type. The deduplication and filtering unit 404 is used to deduplicatize and filter the result data; The generation unit 405 is used to store the deduplicated and filtered result data in the corresponding format according to the corresponding data type to generate a multimodal dataset; wherein, the multimodal dataset is used to generate a multimodal sample set for training the corresponding smart education big model according to the training requirements of the smart education big model.

[0082] In another example, The deduplication filtering unit 404 is specifically used to deduplicatize the result data and, during the filtering process, to identify and remove completely duplicate paragraphs, formulas, or code snippets by means of string matching; to extract features from the content by means of hash algorithm to identify similar but not completely different text or code; to remove parts that are incorrectly identified by OCR or are incomplete; and to remove invalid or noisy data according to format validation rules.

[0083] In another example, data types include: text, formulas and tables, images, and code; The preset processing models include: preset processing models for text, preset processing models for formulas and tables, preset processing models for images, and preset processing models for code.

[0084] In another example, The pre-defined processing model for the text is based on GPT-3. The preset processing models corresponding to the formulas and tables are based on models built using GOT. The preset processing model for the image is a model built based on BLIP; The default processing model corresponding to the code is a model built based on CodeBERT.

[0085] In another example, If the data type corresponding to the result is text, then the corresponding format stored is text; If the data type corresponding to the result data is a formula, then the corresponding storage format is LaTeX or MathML. If the data type corresponding to the result data is tabular data, then the corresponding storage format is CSV or Excel file format; If the data type corresponding to the result is an image, then the corresponding storage format is the specified image format; If the data type corresponding to the result is code, then the corresponding format stored is the standardized format of an executable file or code fragment.

[0086] In another example, each piece of data in the multimodal dataset also stores additional metadata; this additional metadata is used to retrieve and manage the data in the multimodal dataset; the additional metadata includes the file source, generation time, processing tool, and data type.

[0087] The units in the above embodiments can be integrated into one unit or deployed separately; they can be merged into one unit or further divided into multiple sub-units.

[0088] In another embodiment, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a textbook data processing method.

[0089] In another embodiment, a computer-readable storage medium is also provided, on which computer instructions are stored, which, when executed by a processor, implement a textbook data processing method.

[0090] Figure 5 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention. Figure 5As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute the following methods: Obtain the original textbook data file; Extract the data type from the original textbook data file, and parse the original textbook data file according to the data type to obtain structured content data. Structured content data and its corresponding data type are input into a preset multi-data type hybrid model for processing to obtain the processed result data and data type of the structured content data. The preset multi-data type hybrid model includes a routing module and preset processing models corresponding to various data types. The routing module is used to route the corresponding raw data to the preset processing model corresponding to the data type based on the input data type. The preset processing model is used to process the raw data routed by the routing module and output the corresponding result data and data type. The results data are deduplicated and filtered. The deduplication and filtering results are stored in the corresponding format according to the data type to generate a multimodal dataset. The multimodal dataset is used to generate a multimodal sample set for training the smart education big model according to the training requirements of the smart education big model.

[0091] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0092] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0093] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0094] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments disclosed in this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those shown in the drawings. For example, two blocks shown connectedly may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0095] Those skilled in the art will understand that the features described in the various embodiments and / or claims disclosed in this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, without departing from the spirit and teachings of this application, the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways, and all such combinations and / or combinations fall within the scope of this application.

[0096] This document uses specific embodiments to illustrate the principles and implementation methods of the present invention. The descriptions of these embodiments are merely illustrative of the method and core concepts of the present invention and are not intended to limit this application. Those skilled in the art can make changes to the specific implementation methods and application scope based on the ideas, spirit, and principles of the present invention. Any modifications, equivalent substitutions, or improvements made should be included within the scope of protection of this application.

Claims

1. A method for processing textbook data, characterized in that, The method includes: Obtain the original textbook data file; Extract the data type of the original textbook data file, and parse the original textbook data file according to the data type to obtain structured content data; The structured content data and its corresponding data type are input into a preset multi-data type hybrid model for processing to obtain the processed result data and data type corresponding to the structured content data. The preset multi-data type hybrid model includes a routing module and preset processing models corresponding to various data types. The routing module routes the original data to the preset processing model corresponding to the input data type. The preset processing model processes the original data routed by the routing module and outputs the corresponding result data and data type. The resulting data is then deduplicated and filtered. The deduplicated and filtered result data is stored in the corresponding format according to the corresponding data type to generate a multimodal dataset; wherein, the multimodal dataset is used to generate a multimodal sample set for training the corresponding smart education big model according to the training requirements of the smart education big model.

2. The method according to claim 1, characterized in that, The deduplication and filtering process for the result data includes: Identify and remove completely duplicate paragraphs, formulas, or code snippets using string matching methods; By using hash algorithms to extract features from content, similar but not completely different text or code can be identified. Remove parts that are incorrectly identified by OCR or are incomplete, and eliminate invalid or noisy data according to format validation rules.

3. The method according to claim 1, characterized in that, The data types include: text, formulas and tables, images, and code; The preset processing models include: preset processing models for text, preset processing models for formulas and tables, preset processing models for images, and preset processing models for code.

4. The method according to claim 3, characterized in that, The preset processing model corresponding to the text is a model built based on GPT-3.5; The preset processing model corresponding to the formulas and tables is a model built based on GOT; The preset processing model corresponding to the image is a model built based on BLIP; The preset processing model corresponding to the code is a model built based on CodeBERT.

5. The method according to claim 3, characterized in that, The process of storing the deduplication and filtering results in the corresponding format according to the corresponding data type includes: If the data type corresponding to the result data is text, then the corresponding storage format is text; If the data type corresponding to the result data is a formula, then the corresponding storage format is LaTeX or MathML. If the data type corresponding to the result data is tabular data, then the corresponding storage format is CSV or Excel file format; If the data type corresponding to the result data is an image, then the corresponding storage format is the specified image format; If the data type corresponding to the result data is code, then the corresponding format stored is the standardized format of an executable file or code fragment.

6. The method according to any one of claims 1-5, characterized in that, Each piece of data in the multimodal dataset also stores additional metadata; wherein, the additional metadata is used to retrieve and manage the data in the multimodal dataset; the additional metadata includes file source, generation time, processing tool and data type.

7. A textbook data processing device, characterized in that, The device includes: The acquisition unit is used to acquire the original textbook data file; The analysis unit is used to extract the data type of the original textbook data file and parse the original textbook data file according to the data type to obtain structured content data; A processing unit is used to input the structured content data and its corresponding data type into a preset multi-data type hybrid model for processing, and to obtain the processed result data and data type corresponding to the structured content data; wherein, the preset multi-data type hybrid model includes a routing module and preset processing models corresponding to various data types; the routing module is used to route the corresponding original data to the preset processing model corresponding to the data type according to the input data type; the preset processing model is used to process the original data routed by the routing module and output the corresponding result data and data type; The deduplication and filtering unit is used to deduplicatize and filter the result data. The generation unit is used to store the deduplicated and filtered result data in the corresponding format according to the corresponding data type to generate a multimodal dataset; wherein, the multimodal dataset is used to generate a multimodal sample set for training the corresponding smart education big model according to the training requirements of the smart education big model.

8. The apparatus according to claim 7, characterized in that, The deduplication filtering unit is specifically used to deduplicatize the result data and, during the filtering process, identify and remove completely duplicate paragraphs, formulas, or code snippets by using string matching; extract features from the content using a hash algorithm to identify similar but not completely different text or code; remove parts that are incorrectly identified by OCR or are incomplete; and eliminate invalid or noisy data according to format validation rules.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the method described in any one of claims 1-6.

Images