Scientific data set acquisition method and device, storage medium and equipment

Abstract

The invention relates to a scientific data set acquisition method and device, a storage medium and equipment, and the method comprises the steps: obtaining data set information and link information of a scientific data set, generating a data file, and training a loaded large language model through the data file, when the big language model receives the question information containing the identification information of the scientific data set, the reply information containing the link of the scientific data set is generated, and the acquisition efficiency of the scientific data is improved.

Description

Technical Field

[0001] This invention relates to the field of scientific data acquisition, and in particular to a method, apparatus, storage medium, and device for acquiring scientific datasets. Background Technology

[0002] Large Language Models (LLMs) are large neural network models in the field of artificial intelligence, specifically designed to understand and generate natural language text. LLMs can use learned knowledge to generate new text. Based on this ability, LLMs can understand natural language well and provide corresponding responses.

[0003] However, since scientific datasets are often scattered across multiple platforms, when researchers use large language models to search for scientific datasets, the large language models can only indicate the platforms where the scientific data may exist. Researchers need to search each platform one by one to obtain the scientific data, which is difficult and inefficient. Summary of the Invention

[0004] This application provides a method, apparatus, storage medium, and device for acquiring scientific datasets, which can improve the efficiency of acquiring scientific datasets.

[0005] In a first aspect, embodiments of this application provide a method for acquiring scientific datasets, including:

[0006] Obtain dataset information and link information for at least one scientific dataset, and generate a data file;

[0007] Loading a large language model;

[0008] The data file is embedded into the large language model, so that when the trained large language model receives a question containing the identification information of the scientific dataset, it generates a response containing a link to the scientific dataset.

[0009] Secondly, embodiments of this application provide a scientific dataset acquisition device, comprising:

[0010] A data file generation module is used to obtain dataset information and link information of at least one scientific dataset, and generate a data file based on the dataset information and the link information;

[0011] The model loading module is used to load large language models;

[0012] An embedding module is used to embed the data file into the large language model, so that when the large language model receives a question containing the identification information of the scientific dataset, it generates a response containing a link to the scientific dataset.

[0013] Thirdly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the scientific dataset acquisition method as described in any of the preceding claims.

[0014] Fourthly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable by the processor;

[0015] When the processor executes the computer program, it implements the steps of the scientific dataset acquisition method as described in any of the above.

[0016] In this embodiment of the application, by acquiring the dataset information and link information of the scientific dataset and generating a data file, the loaded large language model is trained using the data file. This enables the large language model to generate response information including the link of the scientific dataset when it receives a question containing the identification information of the scientific dataset, thereby improving the efficiency of acquiring scientific data.

[0017] To better understand and implement this invention, the following detailed description is provided in conjunction with the accompanying drawings. Attached Figure Description

[0018] Figure 1 This is a flowchart of a scientific dataset acquisition method in one embodiment of the present invention;

[0019] Figure 2 This is a flowchart of step S120 in one embodiment of the present invention;

[0020] Figure 3 This is a flowchart of step S130 in one embodiment of the present invention;

[0021] Figure 4 This is a schematic diagram of the display interface in one embodiment of the present invention;

[0022] Figure 5 This is a flowchart of a scientific dataset acquisition method according to another embodiment of the present invention;

[0023] Figure 6 shows the response information obtained after posing a question to the existing deepseek model in one embodiment of the present invention;

[0024] Figure 7 This is the response information obtained after adjusting the deepseek model using the method of this application in one embodiment of the present invention;

[0025] Figure 8 This is a schematic diagram of the structure of a scientific dataset acquisition device in one embodiment of the present invention;

[0026] Figure 9 This is a schematic diagram of the structure of a computer device according to one embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0028] It should be understood that the described embodiments are merely some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of the embodiments of this application.

[0029] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the embodiments of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0030] In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims. In the description of this application, it should be understood that the terms "first," "second," "third," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0031] Furthermore, in the description of this application, unless otherwise stated, "several" refers to two or more. "And / or" describes the correspondence between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0032] Please see Figure 1 This application provides a method for acquiring scientific datasets, the method comprising:

[0033] S110: Obtain dataset information and link information for at least one scientific dataset, and generate a data file;

[0034] Dataset information may include the Chinese name, English name, Chinese abbreviation, and English abbreviation of the scientific dataset; scientific datasets may include scientific datasets from national scientific data centers such as the National High Energy Physics Data Center, the National Genome Data Center, and the National Earth Observation Data Center.

[0035] Data files may include identification information for scientific datasets and corresponding links. This identification information and the links may be stored in the data file as a directory.

[0036] Identification information is used to identify the corresponding scientific dataset. Identification information can be the name or abbreviation of the scientific dataset, etc.

[0037] The link information can be a webpage link to a scientific dataset, which allows access to and download of the scientific dataset.

[0038] Data set information and links for each scientific dataset can be obtained through search engines or manual input.

[0039] S120: Loading a large language model;

[0040] Large language models can be existing large language models such as ChatGPT and deepseek.

[0041] When loading a large language model, it can be loaded onto a local device, such as a local computer, so that the large language model can be run and used on the local device. Alternatively, the large language model can be loaded onto a cloud device, such as a server.

[0042] S130: The data file is embedded into the large language model, so that when the large language model receives a question containing the identification information of the scientific dataset, it generates a response containing a link to the scientific dataset.

[0043] Specifically, embedding can be used to train the language model.

[0044] In this embodiment of the application, by acquiring the dataset information and link information of the scientific dataset and generating a data file, the loaded large language model is trained using the data file. This enables the large language model to generate response information including the link of the scientific dataset when it receives a question containing the identification information of the scientific dataset, thereby improving the efficiency of acquiring scientific data.

[0045] like Figure 2 As shown, in one embodiment, loading a large language model includes:

[0046] S121: Obtain the large language model and convert the data format of the large language model into the target data format corresponding to the loading tool;

[0047] Large language models can be downloaded from their official website, resource websites, etc.

[0048] Loading tools are used to load and run large language models. Loading tools can be tools such as LM Studio and Ollama that can be used to load large language models.

[0049] When converting the data format of a big data model, the data format of the big language model can be converted into a data format that the loading tool can support, or the data format of the big language model can be converted into a data format that is easier to load, such as gguf format, thereby improving the loading efficiency of the big language model.

[0050] S121: Load the large language model of the target data format using the loading tool.

[0051] In this embodiment, the loading tool can be LM Studio, a cross-platform desktop application developed by Element Labs that allows users to discover, download, and run large language models (LLMs) on their local devices. LM Studio integrates Hugging Face's model library, providing numerous models such as Llama, Mistral, Phi, Gemma, Falcon, MPT, and StarCoder. It features a simple and intuitive user interface, powerful offline capabilities, allowing users to run models locally offline, ensuring privacy and data security. Furthermore, LM Studio supports macOS, Windows, and Linux systems, exhibiting excellent cross-platform compatibility.

[0052] The loading tool can be pre-installed on the local device. The data format of the large language model can be safetensors or bin format, and the target data format can be gguf format. This is achieved by converting the large language model data format to gguf format. The gguf format uses a compact binary encoding scheme, reducing storage space by about 30%-50% compared to traditional text formats. It also achieves "zero-copy" loading through memory mapping (mmap), allowing direct access to model parameters without copying them to memory, greatly shortening the model loading time. Furthermore, this format supports efficient inference, has low storage costs, low memory consumption, and strong compatibility. It is compatible with mainstream loading tools and local inference tools, meeting the user's needs in different scenarios.

[0053] In this embodiment of the application, by converting the large language model into the target data format corresponding to the loading tool, the loading efficiency of the large language model can be improved when the loading tool is run on the local device to load the large language model.

[0054] like Figure 3 As shown, in one embodiment, embedding the data file into the large language model includes:

[0055] S131: Use the embedding tool to select the corresponding large language model and create a workspace; the workspace includes an upload control and an embedding control;

[0056] Embedding tools are used to embed data files into large language models, enabling the large language models to respond accordingly based on the content of the data files at runtime.

[0057] S132: Upload the data file to the workspace using the upload control, and display the data processing window; the data processing window includes an embedded control;

[0058] S133: The data file is segmented and vectorized using the embedded control and added to the knowledge base of the large language model.

[0059] like Figure 4 As shown, this is a display interface of a local device in one embodiment. The display interface includes a workspace 1, which includes an upload control 101. Data files are uploaded to the workspace 1 by triggering the upload control 101. After uploading the data file, a data processing window is displayed. This window includes the uploaded data file and an embedding control 102. The embedding control 102 is triggered to segment and vectorize the content of the data file, adding it to the knowledge base of the large language model. This allows the large language model to use the content of the data file to generate response information, improving the efficiency of acquiring scientific datasets.

[0060] like Figure 5 As shown, in one embodiment, the method further includes:

[0061] S210: Obtain the fine-tuning dataset; the fine-tuning dataset includes input information and output information;

[0062] Fine-tuning datasets are used to further adjust and optimize large language models. These datasets can be prepared for specific tasks or domain scenarios. Commonly used fine-tuning datasets include those for instruction fine-tuning, dialogue fine-tuning, domain adaptation, text classification, model inference fine-tuning, and knowledge distillation.

[0063] In this embodiment, the fine-tuning dataset is used to adjust the language style of the large language model output, making the output of the large language model more concise.

[0064] The input information can include multiple input keywords. These keywords allow the large language model to identify valid tokens in the question information. When the large language model receives the question information, it can divide the question information into valid and invalid tokens, output corresponding response information for valid tokens, and ignore invalid tokens. This results in more accurate and concise response information, improves the relevance between the response information and the question information, reduces the amount of data processing, and improves the efficiency of response information generation.

[0065] The output information may include multiple ideal response texts to adjust the output text of the large language model, making the model's responses closer to the user's ideal response style. Optionally, the output information may include string count limits to restrict the number of strings in the output response, making the responses more concise and improving the efficiency of acquiring scientific data.

[0066] S220: Based on the fine-tuning dataset and fine-tuning tool, adjust the response information generated by the large language model.

[0067] Fine-tuning tools are used to fine-tune large language models, enabling them to better meet user needs.

[0068] Optionally, the fine-tuning tool can be Unsloth, Text Generation WebUI, SwanLab, or other tools that can be used to fine-tune large language models. These tools can be pre-installed on the user's local device, allowing fine-tuning of the large language model by invoking the local device's fine-tuning tools.

[0069] In this embodiment, the fine-tuning tool can be Unsloth. Unsloth is an open-source tool for fine-tuning large language models. Unsloth can significantly improve fine-tuning speed. By optimizing the internal structure of the model and using techniques such as QLoRA (which combines LoRA and 4-bit quantization), it can reduce memory usage by 50%-80%, meeting the training needs of large models. Unsloth supports a variety of mainstream models and quantization methods, and can adapt to different models and different training requirements.

[0070] Please refer to Figures 6(a) and (b), which show the response information obtained after asking the existing deepseek model in one embodiment, “Where can I find the 1:10,000,000 basic geographic background dataset for Ukraine from 2012 to 2015?”

[0071] Please see Figure 7This is the response information obtained by adjusting the deepseek model using the method of this application in one embodiment.

[0072] As shown in Figure 6, the existing deepseek model provides some possible ways to find datasets to address the above problems. Users need to switch platforms one by one according to the given ways to find the corresponding datasets, which results in low data acquisition efficiency.

[0073] like Figure 7 As shown in the figure, after adjusting the deepseek model using the method of this application, the deepseek model provides a specific link to the dataset to address the above-mentioned issues. Users can directly download or access the relevant scientific data through this link, improving the efficiency of obtaining scientific datasets. At the same time, as can be seen from the figure, after adjustment, the response information generated by the deepseek model is more concise, making it easier for users to quickly obtain the link information of scientific datasets.

[0074] like Figure 8 As shown in the illustration, this application also provides a scientific dataset acquisition device, the device comprising:

[0075] The data file generation module 210 is used to obtain dataset information and link information of at least one scientific dataset, and generate a data file based on the dataset information and the link information;

[0076] Model loading module 220 is used to load large language models;

[0077] The embedding module 230 is used to embed the data file into the large language model, so that when the large language model receives a question containing the identification information of the scientific dataset, it generates a response containing a link to the scientific dataset.

[0078] In one embodiment, the model loading module 220 includes:

[0079] A large language model acquisition unit is used to acquire a large language model and convert the data format of the large language model into a target data format corresponding to the loading tool.

[0080] The loading unit is used to load the large language model of the target data format using the loading tool.

[0081] In one embodiment, the embedded module 230 includes:

[0082] The workspace creation unit is used to select the corresponding large language model and create a workspace using an embedding tool; the workspace includes an upload control.

[0083] A data processing window display unit is used to upload the data file to the workspace via the upload control and display the data processing window; the data processing window includes embedded controls;

[0084] The data file adding unit is used to segment and vectorize the content of the data file through the embedded control and add it to the knowledge base of the large language model.

[0085] In one embodiment, the apparatus further includes:

[0086] The fine-tuning dataset acquisition module is used to acquire the fine-tuning dataset; the fine-tuning dataset includes input information and output information.

[0087] The response information generation module is used to adjust the response information generated by the large language model based on the fine-tuning dataset and fine-tuning tools.

[0088] It should be noted that the scientific dataset acquisition device provided in the above embodiments is only illustrated by the division of the above functional modules when executing the scientific dataset acquisition method. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the scientific dataset acquisition device provided in the above embodiments and the scientific dataset acquisition method in the above embodiments belong to the same concept, and its implementation process is detailed in the method embodiments, which will not be repeated here.

[0089] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the scientific dataset acquisition method as described in any of the preceding claims.

[0090] The embodiments of this application may take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-readable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information may be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to: phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0091] like Figure 9 As shown, this application embodiment also provides a computer device 300, including a memory 310, a processor 320, and a computer program stored in the memory 310 and executable by the processor 320;

[0092] When the processor 320 executes the computer program, it implements the steps of the scientific dataset acquisition method as described in any of the above.

[0093] The memory 310 includes a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0094] The processor 320 is the control unit of the computer device 300. It connects to various components of the computer device 300 via various interfaces and lines. By running or executing programs or modules stored in the memory 310, and by calling data stored in the memory 310, it performs various functions of the computer device 300 and processes data. For example, when the processor 320 executes the computer program stored in the memory 310, it implements all or part of the steps of the scientific dataset acquisition method described in this application embodiment; or it implements all or part of the functions of the scientific dataset acquisition device. The processor 320 can be composed of integrated circuits, such as a single packaged integrated circuit, or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips.

[0095] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0096] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for acquiring scientific datasets, characterized in that, include: Obtain dataset information and link information for at least one scientific dataset, and generate a data file based on the dataset information and the link information; Loading a large language model; The data file is embedded into the large language model, so that when the large language model receives a question containing the identification information of the scientific dataset, it generates a response containing a link to the scientific dataset.

2. The method for acquiring scientific datasets according to claim 1, characterized in that, Loading large language models includes: Obtain a large language model and convert the data format of the large language model into a target data format corresponding to the loading tool; The loading tool is used to load the large language model of the target data format.

3. The method for acquiring scientific datasets according to claim 1, characterized in that, Embedding the data file into the large language model includes: Use an embedding tool to select the corresponding large language model and create a workspace; the workspace includes an upload control. The data file is uploaded to the workspace using the upload control, and a data processing window is displayed; the data processing window includes embedded controls. The embedded control is used to segment and vectorize the content of the data file and add it to the knowledge base of the large language model.

4. The method for acquiring scientific datasets according to claim 1, characterized in that, It also includes the following steps: Obtain the fine-tuning dataset; the fine-tuning dataset includes input information and output information; Based on the fine-tuning dataset and fine-tuning tools, the response information generated by the large language model is adjusted.

5. A scientific dataset acquisition device, characterized in that, include: A data file generation module is used to obtain dataset information and link information of at least one scientific dataset, and generate a data file based on the dataset information and the link information; The model loading module is used to load large language models; An embedding module is used to embed the data file into the large language model, so that when the large language model receives a question containing the identification information of the scientific dataset, it generates a response containing a link to the scientific dataset.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When executed by a processor, the computer program implements the steps of the scientific dataset acquisition method as described in any one of claims 1-4.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable by the processor; When the processor executes the computer program, it implements the steps of the scientific dataset acquisition method as described in any one of claims 1-4.

Images