Data crawling method, training method and device based on computing power of intelligent computing center

Abstract

The invention provides a data crawling method and device and a training method and device based on computing power of an intelligent computing center, and the data crawling method comprises the steps: S1, receiving a target region sent by an interaction end, the target region being a region selected by a user on a target webpage through the interaction end; s2, in response to the target area, a preset crawling strategy is obtained, and the crawling strategy comprises the crawling depth and the data type of the data needing to be crawled; furthermore, calling a data crawling large language model deployed in the intelligent computing center, crawling data of each link in the target area according to the crawling depth and the data type, and combining the crawled data of each link to obtain a first data set; s3, abnormal data in the first data set are cleared away, and the first data set with the abnormal data cleared away is used as a second data set. According to the invention, high crawling efficiency and high crawling accuracy can be realized.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of computing power infrastructure, and in particular, to a data crawling method, training method and device based on the computing power of an intelligent computing center. Background Art

[0002] With the development of artificial intelligence technology and computing power technology, the concept of intelligent computing center has emerged. "Intelligent computing center" refers to the use of large-scale heterogeneous computing power resources, including general computing power and intelligent computing power, mainly for artificial intelligence applications (such as artificial intelligence deep learning model development, model training and model reasoning and other scenarios) to provide the required computing power, data and algorithms. Intelligent computing center covers facilities, hardware, software, and can provide full-stack capabilities from bottom-level computing power to top-level application enablement.

[0003] The size of the data set, computing resources, and model scale constitute the Scalling Law of large language models. Among them, the importance of data is particularly prominent. High-quality, comprehensive and rich data determines the performance of large language models. With the rapid development of the Internet, a large amount of high-value data is carried on the Internet, which is an important source of training corpus for large language models.

[0004] However, existing data crawling methods have low crawling efficiency and low accuracy. Summary of the invention

[0005] The embodiments of the present invention provide a data crawling method, a training method and a device based on the computing power of an intelligent computing center to solve the problems of low crawling efficiency and low accuracy of existing data crawling methods.

[0006] In order to solve the above-mentioned technical problems, the present invention is achieved as follows:

[0007] In a first aspect, an embodiment of the present invention provides a data crawling method based on the computing power of an intelligent computing center, comprising:

[0008] Step S1: receiving a target area sent by an interactive terminal, wherein the target area is an area selected by a user on a target web page through the interactive terminal;

[0009] Step S2: in response to the target area, a preset crawling strategy is obtained, wherein the crawling strategy includes: crawling depth and data type of data to be crawled; and a data crawling language model deployed in an intelligent computing center is called to crawl data of each link in the target area according to the crawling depth and the data type, and the crawled data of each link is combined to obtain a first data set;

[0010] Step S3: Clear the abnormal data in the first data set to complete the first data set with the abnormal data cleared as the second data set.

[0011] Optionally, the data type includes at least one of the following: text, file, audio, video, and picture.

[0012] Optionally, the step S3 then includes:

[0013] Step S4: calling the expert group model deployed in the intelligent computing center to extract data of each modality type from the second data set; the expert group model includes a large language model for extracting data of at least one of the following modal types: text, file, audio, video, picture;

[0014] Step S5: Generate a message according to the data of each modality type, and send the message to the interaction end.

[0015] In a second aspect, an embodiment of the present invention provides a method for training a large language model for data crawling, comprising:

[0016] Step S01: Acquire multiple sets of training data, each set of training data includes: a training website, a training crawling strategy corresponding to the training website, and crawling data results crawled from the training website according to the training crawling strategy; the training crawling strategy includes: crawling depth and data type of data to be crawled;

[0017] Step S02: using the multiple sets of training data to train a large language model to obtain a data crawling large language model;

[0018] Step S03: deploying the data crawling large language model to the intelligent computing center.

[0019] Optionally, the step S02 includes:

[0020] Step S021: taking the first group of training data among the multiple groups of training data as the first training data;

[0021] Step S022: using the training website of the first training data and the training crawling strategy to train a large language model; after each round of training, determining a loss function value between the data crawled from the training website by the large language model after the training round is completed and the crawling data result of the current first training data;

[0022] Step S023: If the loss function value is greater than a preset loss threshold, continue to use the current first training data for the next round of training;

[0023] Step S024: If the loss function value is less than or equal to the loss threshold, terminate the next round of training using the current first training data, use the large language model after the completion of this round of training as the new large language model, and use the next group of training data in the multiple groups of training data as the new first training data, return to step S022 until the loss function value under the last group of training data is less than or equal to the loss threshold, and determine that the large language model under the last group of training data is the data crawling large language model.

[0024] In a third aspect, an embodiment of the present invention provides a data crawling device based on the computing power of an intelligent computing center, comprising:

[0025] A receiving module, configured to receive a target area sent by an interactive terminal, wherein the target area is an area selected by a user on a target web page through the interactive terminal;

[0026] A response module is used to obtain a preset crawling strategy in response to the target area, wherein the crawling strategy includes: crawling depth and data type of data to be crawled; and, calling a data crawling language model deployed in an intelligent computing center, crawling data of each link in the target area according to the crawling depth and the data type, and combining the crawled data of each link to obtain a first data set;

[0027] The clearing module is used to clear the abnormal data in the first data set, so as to complete the first data set clearing the abnormal data as the second data set.

[0028] In a fourth aspect, an embodiment of the present invention provides a training device for a large language model for data crawling, comprising:

[0029] An acquisition module is used to acquire multiple sets of training data, each set of training data includes: a training website, a training crawling strategy corresponding to the training website, and crawling data results crawled from the training website according to the training crawling strategy; the training crawling strategy includes: crawling depth and data type of data to be crawled;

[0030] A training module, used to train a large language model using the multiple sets of training data to obtain a data crawling large language model;

[0031] A deployment module is used to deploy the data crawling large language model to an intelligent computing center.

[0032] In a fifth aspect, an embodiment of the present invention provides an electronic device, comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the data crawling method based on the computing power of an intelligent computing center as described in any one of the first aspects, or implements the steps of the data crawling large language model training method as described in any one of the second aspects.

[0033] In a sixth aspect, an embodiment of the present invention provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the data crawling method based on the computing power of an intelligent computing center as described in any one of the first aspects are implemented, or the steps of the data crawling large language model training method as described in any one of the second aspects are implemented.

[0034] In the seventh aspect, an embodiment of the present invention provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the data crawling method based on the computing power of an intelligent computing center as described in any one of the first aspects, or implement the steps of the data crawling large language model training method as described in any one of the second aspects.

[0035] In an embodiment of the present invention, step S1 is performed: receiving a target area sent by an interactive terminal, where the target area is an area selected by a user on a target web page through the interactive terminal; step S2 is performed: in response to the target area, obtaining a preset crawling strategy, where the crawling strategy includes: crawling depth and data type of data to be crawled; and calling a data crawling large language model deployed in an intelligent computing center, crawling data of each link in the target area according to the crawling depth and data type, combining the crawled data of each link to obtain a first data set; step S3 is performed: removing abnormal data in the first data set to complete the first data set that removes abnormal data Cooperation is the second data set, the crawling strategy includes the crawling depth and the data type of the data to be crawled, and combined with the target area obtained by box selection, the embodiment of the present invention realizes the precise definition of the crawling task; calls the data crawling big language model deployed in the intelligent computing center, crawls the data of each link in the target area according to the crawling depth and data type, and uses the computing power of the intelligent computing center to crawl, which speeds up the crawling speed. The embodiment of the present invention can achieve high crawling efficiency; in addition, based on the precisely defined crawling tasks and the high cognitive ability of the data crawling big language model for crawling tasks, the embodiment of the present invention can achieve high crawling accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0037] Figure 1 A flowchart of a data crawling method based on the computing power of an intelligent computing center according to an embodiment of the present invention;

[0038] Figure 2 This is one of the flow charts of the method for training a large language model by crawling data according to an embodiment of the present invention;

[0039] Figure 3 This is a second flow chart of a method for training a large language model by crawling data according to an embodiment of the present invention;

[0040] Figure 4 This is a principle block diagram of a data crawling device based on the computing power of an intelligent computing center according to an embodiment of the present invention;

[0041] Figure 5 This is a principle block diagram of a training device for a large language model for data crawling according to an embodiment of the present invention;

[0042] Figure 6 The figure is a principle block diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] The terms "first", "second", etc. of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of one type, and the number of objects is not limited, for example, the first object can be one or more. In addition, "or" in the present application represents at least one of the connected objects. For example, "A or B" covers three schemes, namely, Scheme 1: including A but not including B; Scheme 2: including B but not including A; Scheme 3: including both A and B. The character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0045] In the technical solution of the present disclosure, words such as “connect”, “couple” or “connected” are not limited to physical or mechanical connections, but may include electrical connections.

[0046] In addition, the technical features involved in the different embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0047] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for personal information to prevent illegal access to personal information data and maintain personal information security and network security.

[0048] First, the technical terms involved in the present invention are briefly explained below.

[0049] The "computing power" mentioned in the present invention is the ability of computer equipment or computing / data center to process information. It is the ability of computer hardware and software to work together to execute certain computing requirements. It is the computing power to achieve target result output by processing information data. It is a new type of productivity that integrates information computing power, network carrying capacity, and data storage capacity. It mainly provides services to the society through computing power infrastructure.

[0050] The "computing power" (Computational Power, CP) described in the present invention is the ability of a data center server to process data and output results. It is a comprehensive indicator to measure the computing power of a data center, including general computing power, super computing power and intelligent computing power. The commonly used unit of measurement is the number of floating point operations performed per second (FLOPS: Floating Point Operations Per Second, 1EFLOPS = 10^18FLOPS). The larger the value, the stronger the comprehensive computing power. According to calculations, 1EFLOPS is approximately the computing power output of 5 Tianhe-2A or 500,000 mainstream server CPUs or 2 million mainstream notebooks. The calculation formula is: CP = CP 通用 +CP 智能 +CP 超级 .

[0051] The "Network Power" (NP) described in the present invention is a manifestation of the data transmission capability of computing facilities, including comprehensive capabilities such as network architecture, network bandwidth, transmission latency, intelligent management and scheduling, etc. The carrying capacity involves network transmission within and between data centers, and is a comprehensive indicator for measuring network transmission scheduling capabilities. In the embodiment of the present invention, the carrying capacity uses the memory bandwidth.

[0052] The "Storage Power" (SP) described in the present invention is the comprehensive ability of a data center in terms of data storage capacity, performance, safety and reliability, and green and low-carbon. It is a comprehensive indicator for measuring the data storage capacity of a data center, including external storage devices such as storage arrays and server built-in storage devices. The commonly used unit of measurement for storage capacity is exabyte (EB, 1EB = 2^60bytes), and the commonly used unit of measurement for performance is the number of reads and writes per second per unit capacity (IOPS / TB, Input / Output Operations Per Second / TB). The disaster recovery ratio is an important manifestation of safety and reliability.

[0053] The "computing power infrastructure" described in the present invention is a new type of information infrastructure that integrates information computing power, network carrying capacity, and data storage capacity. It can realize centralized computing, storage, transmission and application of information, and presents characteristics such as diversity and ubiquity, intelligence and agility, security and reliability, and green and low-carbon.

[0054] The “computing power” mentioned in the present invention includes general computing power, intelligent computing power and super computing power.

[0055] The “general computing power” mentioned in the present invention refers to the computing power provided by servers based on CPU (Central Processing Unit) chips, which is used to support basic general computing such as cloud computing and edge computing.

[0056] The "intelligent computing power" described in the present invention is a computing platform for large-scale deployment of special chips such as GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit) for various innovative applications of artificial intelligence, such as natural language processing and machine vision.

[0057] The "super computing power" mentioned in the present invention is mainly the computing power provided by high-performance computing clusters such as supercomputers. It utilizes the centralized computing resources of multiple computer systems working in parallel and uses a dedicated operating system to handle extremely complex or data-intensive problems. It is mainly used for calculations in cutting-edge scientific fields, such as planetary simulation, drug molecule design, and gene analysis.

[0058] The "intelligent computing center" described in the present invention refers to a facility that provides the required computing power, data and algorithms for artificial intelligence applications (such as artificial intelligence deep learning model development, model training and model reasoning scenarios) by using large-scale heterogeneous computing power resources, including general computing power (CPU: Central Processing Unit) and intelligent computing power (GPU: Graphics Processing Unit, FPGA: Field Programmable Gate Array, ASIC: Application Specific Integrated Circuit, etc.). The intelligent computing center covers facilities, hardware, and software, and can provide full-stack capabilities from bottom-level computing power to top-level application enablement.

[0059] The "computing resources" mentioned in the present invention refer to the technologies and facilities with information calculation, transmission, storage and application capabilities required for the development of the digital society, including but not limited to computing resources such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), network resources such as switches and routers, storage resources such as storage arrays and distributed storage, security resources such as firewalls and intrusion detection systems, as well as supporting and guarantee resources such as wind, fire, water and electricity.

[0060] The "Large Language Model (LLM)" described in the present invention is an artificial intelligence model based on deep learning, which is designed to understand and generate natural language. These models are usually trained with large amounts of text data to learn the structure, grammar, semantics and context of the language, and can perform a wide range of tasks including text summarization, translation, sentiment analysis, etc.

[0061] The embodiment of the present invention provides a data crawling method based on the computing power of an intelligent computing center, see Figure 1 As shown, Figure 1 The following is a flow chart of a data crawling method based on the computing power of an intelligent computing center according to an embodiment of the present invention. The data crawling method based on the computing power of an intelligent computing center includes:

[0062] Step S1: receiving a target area sent by the interactive terminal, where the target area is an area selected by the user on the target web page through the interactive terminal;

[0063] Step S2: in response to the target area, a preset crawling strategy is obtained, the crawling strategy includes: crawling depth and data type of data to be crawled; and, a data crawling language model deployed in the intelligent computing center is called to crawl the data of each link in the target area according to the crawling depth and data type, and the crawled data of each link is combined to obtain a first data set;

[0064] Step S3: Clear the abnormal data in the first data set, so as to complete the first data set with the abnormal data cleared as the second data set.

[0065] In some embodiments of the present invention, optionally, the user can select the target area on the target web page by at least one of the following methods: operating the mouse to select, operating the finger on the touch screen of the interactive end to select, typing area parameter information on the keyboard, and operating the stylus to select.

[0066] In an embodiment of the present invention, the specific form of the target area can be a data packet containing all the links in the target area; it can be a picture. In step S2, a data crawling language model deployed in the intelligent computing center is called to crawl the data of each link in the target area according to the crawling depth and data type, and the crawled data of each link is combined to obtain the first data set. The steps include: identifying the target area, determining that the website where the target area is located is the target website, visiting the target website, and determining all the links on the target website that are in the target area.

[0067] In some optional embodiments, the interactive end includes a VR (Virtual Reality) device or an AR (Augmented Reality) device, and the user can select the target area through body movements. In some optional embodiments, the interactive device may also include a device capable of brain-computer connection, and the user can select the target area through a brain-computer interface (Brain-Computer Interface, BCI). It should be noted that there are many types of interactive ends in actual applications, and there are many ways of interaction, which cannot be exhaustively listed, and the interactive ends and interactive methods specifically selected by ordinary technicians in this field under the enlightenment of the present invention should all fall within the protection of the present invention.

[0068] In step S2 of the embodiment of the present invention, the crawling strategy can be a strategy preset by the user corresponding to the web page. That is to say, based on the data features of different web pages, the user presets crawling strategies for multiple web pages, and the crawling strategies correspond to the web pages one by one. For example, the correspondence between the crawling strategy and the web page is a crawling strategy-web page relationship mapping table, and the crawling strategies corresponding to all web pages are combined to form a crawling strategy set. The crawling strategy-web page relationship mapping table and the crawling strategy set are both pre-stored in the database; after receiving the target area sent by the interactive end, in response to the target area, the step of obtaining the preset crawling strategy includes: determining the target web page according to the target area; determining the crawling strategy corresponding to the target web page according to the crawling strategy-web page relationship mapping table retrieved from the database; and obtaining the crawling strategy corresponding to the target web page from the crawling strategy set pre-stored in the database.

[0069] In the embodiment of the present invention, crawling depth refers to the level or depth to which the crawler crawls the web page when crawling web page data. Specifically, in a web crawler, links on the web page are obtained and recursively accessed to obtain deeper page data, and the crawling depth represents how many levels of page data are obtained. For example, if the crawling depth is 2 layers, the data that meets the data type requirements of the next layer of pages after each link jumps in the target area will be crawled, and the data that meets the data type requirements of the next layer of pages after each link jumps on the next layer of pages will be crawled. By using a crawling strategy including crawling depth and the data type of the data to be crawled, as well as a target area obtained by box selection, the embodiment of the present invention achieves a precise definition of the crawling task.

[0070] In some embodiments of the present invention, optionally, the training method of the data crawling large language model includes: obtaining multiple sets of training data, each set of training data includes: a training website, a training crawling strategy corresponding to the training website, and crawling data results crawled from the training website according to the training crawling strategy; the training crawling strategy includes: crawling depth and data type of data to be crawled; using multiple sets of training data to train the large language model to obtain the data crawling large language model; deploying the data crawling large language model to the intelligent computing center. It can be obtained that the trained data crawling large language model has a high cognitive ability for crawling tasks. When called, the data crawling large language model can crawl the data of each link in the target area according to the crawling depth and data type, and realize high-efficiency and high-accuracy data crawling.

[0071] The embodiment of the present invention calls the data crawling big language model deployed in the intelligent computing center, crawls the data of each link in the target area according to the crawling depth and data type, and uses the computing power of the intelligent computing center to crawl, which speeds up the crawling speed and improves the crawling efficiency; in addition, based on the precisely defined crawling tasks and the high cognitive ability of the data crawling big language model for crawling tasks, the embodiment of the present invention ensures high crawling accuracy.

[0072] In practical applications, the data directly crawled often includes data with at least one defect of missing, repeated, or disordered, that is, abnormal data. Based on this, in step S3 of the embodiment of the present invention, the abnormal data in the first data set directly crawled is cleared, which can improve the data quality and ensure the subsequent high-accuracy and high-quality application of the crawled data.

[0073] In an embodiment of the present invention, step S1 is performed: receiving a target area sent by an interactive terminal, where the target area is an area selected by a user on a target web page through the interactive terminal; step S2 is performed: in response to the target area, obtaining a preset crawling strategy, where the crawling strategy includes: crawling depth and data type of data to be crawled; and calling a data crawling large language model deployed in an intelligent computing center, crawling data of each link in the target area according to the crawling depth and data type, combining the crawled data of each link to obtain a first data set; step S3 is performed: removing abnormal data in the first data set to complete the first data set that removes abnormal data Cooperation is the second data set, the crawling strategy includes the crawling depth and the data type of the data to be crawled, and combined with the target area obtained by box selection, the embodiment of the present invention realizes the precise definition of the crawling task; calls the data crawling big language model deployed in the intelligent computing center, crawls the data of each link in the target area according to the crawling depth and data type, and uses the computing power of the intelligent computing center to crawl, which speeds up the crawling speed. The embodiment of the present invention can achieve high crawling efficiency; in addition, based on the precisely defined crawling tasks and the high cognitive ability of the data crawling big language model for crawling tasks, the embodiment of the present invention can achieve high crawling accuracy.

[0074] In some embodiments of the present invention, optionally, the data type includes at least one of the following: text, file, audio, video, picture. The embodiments of the present invention can realize the crawling of multimodal (multiple data types) data, have high adaptability, and can be conveniently applied to the crawling task of multimodal data.

[0075] In some embodiments of the present invention, optionally, step S3 then includes:

[0076] Step S4: calling the expert group model deployed in the intelligent computing center to extract data of each modality type from the second data set; the expert group model includes a large language model for extracting data of at least one of the following modal types: text, file, audio, video, and picture;

[0077] Step S5: Generate a message according to the data of each modality type, and send the message to the interaction end.

[0078] The Expert System Model is an artificial intelligence (AI) application that aims to simulate the decision-making process of human experts to solve problems in specific fields. Expert systems usually use knowledge bases and reasoning mechanisms to make inferences and decisions, and can provide advice and solutions in certain professional fields.

[0079] In an embodiment of the present invention, when applying the crawled data, the expert group model deployed in the intelligent computing center is called to extract data of each modal type from the second data set, thereby realizing data classification; further, a message is generated according to the data of each modal type, that is, the data of each modal type is sorted and summarized, and a message is formed with the sorted and summarized data, which can greatly facilitate users to read and analyze the data status.

[0080] It should be noted that the message can be the data sorted and summarized into a preset template, or it can be the pre-trained large language model that extracts features and data change trends from data of various modal types and forms a message with the features and data change trends.

[0081] The embodiment of the present invention provides a method for training a large language model using data crawling. Figure 2 As shown, Figure 2 This is a flow chart of a method for training a large language model using data crawling according to an embodiment of the present invention. The method for training a large language model using data crawling includes:

[0082] Step S01: obtaining multiple sets of training data, each set of training data includes: a training website, a training crawling strategy corresponding to the training website, and crawling data results crawled from the training website according to the training crawling strategy; the training crawling strategy includes: crawling depth and data type of data to be crawled;

[0083] Step S02: using multiple sets of training data to train a large language model to obtain a data crawling large language model;

[0084] Step S03: Deploy the data crawling large language model to the intelligent computing center.

[0085] In the embodiment of the present invention, the training URL represents all the links of the training website corresponding to the training URL.

[0086] In the embodiment of the present invention, the training data (i.e., training task) is defined by a training URL, a training crawling strategy corresponding to the training URL, and a crawling data result crawled from the training URL according to the training crawling strategy, and a large language model is trained using multiple sets of training data, so that a data crawling large language model with high cognitive ability for crawling tasks can be trained. When the data crawling large language model is called to perform a crawling task, the data crawling large language model can crawl the data of each link in the target area according to the crawling depth and data type in the actual crawling strategy, thereby achieving efficient and accurate data crawling.

[0087] In some embodiments of the present invention, optionally, see Figure 3 As shown, Figure 3The second flowchart of the method for training a large language model by crawling data according to an embodiment of the present invention is shown in step S02, including:

[0088] Step S021: taking a first set of training data among the multiple sets of training data as first training data;

[0089] Step S022: using the training URL and training crawling strategy of the first training data to train the large language model; after each round of training, determining the loss function value between the data crawled from the training URL by the large language model after the training round is completed and the crawling data result of the current first training data;

[0090] Step S023: If the loss function value is greater than the preset loss threshold, continue to use the current first training data for the next round of training;

[0091] Step S024: If the loss function value is less than or equal to the loss threshold, terminate the next round of training using the current first training data, use the large language model after the completion of this round of training as the new large language model, and use the next set of training data in the multiple sets of training data as the new first training data, return to step S022 until the loss function value under the last set of training data is less than or equal to the loss threshold, and determine that the large language model under the last set of training data is the data crawling large language model.

[0092] In the embodiment of the present invention, the loss threshold is preset by the user according to actual needs.

[0093] In the embodiment of the present invention, through the loop training between step S022 and step S024, all the training data are used to train the large language model in sequence, and each training data is trained to meet the training loss function requirements (the loss function value is less than or equal to the loss threshold), and the large language model after the last training data is trained and meets the loss function requirements is the data crawling large language model. Executing the above steps is conducive to improving the cognitive ability of the large language model for crawling tasks. When calling the data crawling large language model to perform crawling tasks, it is ensured that the data crawling large language model can crawl the data of each link in the target area according to the crawling depth and data type in the actual crawling strategy, so as to achieve efficient and accurate data crawling.

[0094] The embodiment of the present invention provides a data crawling device based on the computing power of an intelligent computing center, see Figure 4 As shown, Figure 4 The principle block diagram of the data crawling device based on the computing power of the intelligent computing center according to the embodiment of the present invention, the principle 40 of the data crawling device based on the computing power of the intelligent computing center includes:

[0095] A receiving module 41 is used to receive a target area sent by an interactive terminal, where the target area is an area selected by a user on a target web page through the interactive terminal;

[0096] The response module 42 is used to obtain a preset crawling strategy in response to the target area, wherein the crawling strategy includes: crawling depth and data type of data to be crawled; and call the data crawling language model deployed in the intelligent computing center to crawl the data of each link in the target area according to the crawling depth and the data type, and combine the crawled data of each link to obtain a first data set;

[0097] The clearing module 43 is used to clear the abnormal data in the first data set, so as to complete the first data set cleared of the abnormal data as the second data set.

[0098] In some embodiments of the present invention, optionally, the data type includes at least one of the following: text, file, audio, video, and picture.

[0099] In some embodiments of the present invention, optionally, the method further comprises:

[0100] A message module, used for calling the expert group model deployed in the intelligent computing center to extract data of each modality type from the second data set; the expert group model includes a large language model for extracting data of at least one of the following modal types: text, file, audio, video, picture;

[0101] The message module is also used to generate a message according to the data of each modal type and send the message to the interaction end.

[0102] The data crawling device based on the computing power of the intelligent computing center provided in the embodiment of the present application can achieve Figure 1 The various processes implemented by the method embodiment and achieving the same technical effect are not described here to avoid repetition.

[0103] The embodiment of the present invention provides a data crawling large language model training device, see Figure 5 As shown, Figure 5 The principle block diagram of the training device for crawling a large language model with data according to an embodiment of the present invention is shown in FIG. 50. The training device 50 for crawling a large language model with data includes:

[0104] The acquisition module 51 is used to acquire multiple sets of training data, each set of training data includes: a training website, a training crawling strategy corresponding to the training website, and crawling data results crawled from the training website according to the training crawling strategy; the training crawling strategy includes: crawling depth and data type of data to be crawled;

[0105] A training module 52, configured to train a large language model using the plurality of sets of training data to obtain a data crawling large language model;

[0106] The deployment module 53 is used to deploy the data crawling large language model to the intelligent computing center.

[0107] In some embodiments of the present invention, optionally, the training module 52 includes:

[0108] A first training module, configured to use a first group of training data among the multiple groups of training data as first training data;

[0109] A second training module is used to train a large language model using the training URL of the first training data and the training crawling strategy; after each round of training, a loss function value between the data crawled from the training URL by the large language model after the round of training is completed and the crawling data result of the current first training data is determined;

[0110] A third training module, configured to continue to use the current first training data for the next round of training if the loss function value is greater than a preset loss threshold;

[0111] The fourth training module is used to terminate the use of the current first training data for the next round of training if the loss function value is less than or equal to the loss threshold, use the large language model after the completion of this round of training as the new large language model, and use the next group of training data in the multiple groups of training data as the new first training data, return to the second training module until the loss function value under the last group of training data is less than or equal to the loss threshold, and determine that the large language model under the last group of training data is the data crawling large language model.

[0112] The data crawling large language model training device provided in the embodiment of the present application can achieve Figures 2 to 3 The various processes implemented by the method embodiment and achieving the same technical effect are not described here to avoid repetition.

[0113] The embodiment of the present invention provides an electronic device 60, see Figure 6 As shown, Figure 6 This is a principle block diagram of an electronic device 60 according to an embodiment of the present invention, including a processor 61, a memory 62, and a program or instruction stored in the memory 62 and executable on the processor 61. When the program or instruction is executed by the processor, the steps of any one of the data crawling methods based on the computing power of an intelligent computing center of the present invention are implemented, or the steps of any one of the data crawling large language model training methods of the present invention are implemented.

[0114] An embodiment of the present invention provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps in the data crawling method based on the computing power of an intelligent computing center as described above are implemented, or the various processes of an embodiment of the data crawling large language model training method as described above are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0115] The readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc. In some examples, the readable storage medium may be a non-transitory readable storage medium.

[0116] An embodiment of the present invention also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, the steps in any of the above-mentioned data crawling methods based on the computing power of an intelligent computing center are implemented, or the various processes of the embodiments of any of the above-mentioned data crawling large language model training methods are implemented, and the same technical effect can be achieved. To avoid repetition, they will not be repeated here.

[0117] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A data crawling method based on the computing power of an intelligent computing center, characterized in that: include: Step S1: receiving a target area sent by an interactive terminal, wherein the target area is an area selected by a user on a target web page through the interactive terminal; Step S2: in response to the target area, a preset crawling strategy is obtained, wherein the crawling strategy includes: crawling depth and data type of data to be crawled; and a data crawling language model deployed in an intelligent computing center is called to crawl data of each link in the target area according to the crawling depth and the data type, and the crawled data of each link is combined to obtain a first data set; Step S3: Clear the abnormal data in the first data set to complete the first data set with the abnormal data cleared as the second data set.

2. The data crawling method based on the computing power of the intelligent computing center according to claim 1 is characterized in that: The data type includes at least one of the following: text, file, audio, video, and picture.

3. The data crawling method based on the computing power of the intelligent computing center according to claim 1 is characterized in that: The step S3 then comprises: Step S4: calling the expert group model deployed in the intelligent computing center to extract data of each modality type from the second data set; the expert group model includes a large language model for extracting data of at least one of the following modal types: text, file, audio, video, picture; Step S5: Generate a message according to the data of each modality type, and send the message to the interaction end.

4. A method for training a large language model by crawling data, characterized in that: include: Step S01: Acquire multiple sets of training data, each set of training data includes: a training website, a training crawling strategy corresponding to the training website, and crawling data results crawled from the training website according to the training crawling strategy; the training crawling strategy includes: crawling depth and data type of data to be crawled; Step S02: using the multiple sets of training data to train a large language model to obtain a data crawling large language model; Step S03: deploying the data crawling large language model to the intelligent computing center.

5. The method for training a large language model for data crawling according to claim 4, characterized in that: The step S02 comprises: Step S021: taking the first group of training data among the multiple groups of training data as the first training data; Step S022: using the training website of the first training data and the training crawling strategy to train a large language model; after each round of training, determining a loss function value between the data crawled from the training website by the large language model after the training round is completed and the crawling data result of the current first training data; Step S023: If the loss function value is greater than a preset loss threshold, continue to use the current first training data for the next round of training; Step S024: If the loss function value is less than or equal to the loss threshold, terminate the next round of training using the current first training data, use the large language model after the completion of this round of training as the new large language model, and use the next group of training data in the multiple groups of training data as the new first training data, return to step S022 until the loss function value under the last group of training data is less than or equal to the loss threshold, and determine that the large language model under the last group of training data is the data crawling large language model.

6. A data crawling device based on the computing power of an intelligent computing center, characterized in that: include: A receiving module, configured to receive a target area sent by an interactive terminal, wherein the target area is an area selected by a user on a target web page through the interactive terminal; A response module is used to obtain a preset crawling strategy in response to the target area, wherein the crawling strategy includes: crawling depth and data type of data to be crawled; and, calling a data crawling language model deployed in an intelligent computing center, crawling data of each link in the target area according to the crawling depth and the data type, and combining the crawled data of each link to obtain a first data set; The clearing module is used to clear the abnormal data in the first data set, so as to complete the first data set clearing the abnormal data as the second data set.

7. A data crawling large language model training device, characterized in that: include: An acquisition module is used to acquire multiple sets of training data, each set of training data includes: a training website, a training crawling strategy corresponding to the training website, and crawling data results crawled from the training website according to the training crawling strategy; the training crawling strategy includes: crawling depth and data type of data to be crawled; A training module, used to train a large language model using the multiple sets of training data to obtain a data crawling large language model; A deployment module is used to deploy the data crawling large language model to an intelligent computing center.

8. An electronic device, characterized in that: It includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor. When the program or instruction is executed by the processor, it implements the steps of the data crawling method based on the computing power of an intelligent computing center as described in any one of claims 1 to 3, or implements the steps of the data crawling large language model training method as described in any one of claims 4 to 5.

9. A readable storage medium, characterized in that: The readable storage medium stores programs or instructions, which, when executed by the processor, implement the steps of the data crawling method based on the computing power of the intelligent computing center as described in any one of claims 1 to 3, or implement the steps of the data crawling large language model training method as described in any one of claims 4 to 5.

10. A computer program product, characterized in that It includes computer instructions, which, when executed by a processor, implement the steps of the data crawling method based on the computing power of an intelligent computing center as described in any one of claims 1 to 3, or implement the steps of the data crawling large language model training method as described in any one of claims 4 to 5.

Images