Web page data collection method, system, computer device and readable storage medium

Abstract

The present application relates to a web page data collection method, system, computer device and readable storage medium. The web page data collected by the method is used to pre-train a large language model in a target field, and the method comprises: determining a target scene template and a target keyword; filling the target keyword into the target scene template, calling a general large language model, and generating multiple search terms; calling a search service, and generating a first URL list based on the search terms; and collecting web page data based on crawler technology and the first URL list. The use of this method can reduce the cost of hardware resource consumption, labor cost, and improve the efficiency of obtaining pre-training data.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and in particular to a web page data collection method, system, computer device and readable storage medium. Background Art

[0002] With the development of natural language processing technology, large language model technology has emerged. Large language model is an advanced artificial intelligence model based on deep learning and natural language processing technology. Due to its powerful expression, generalization and problem-solving capabilities, it is currently widely used. Among them, large language models are divided into general and special ones.

[0003] However, the text corpus in a specific field is relatively small and cannot meet the pre-training requirements of a large language model, while the web page data publicly available on the Internet is huge. To improve the accuracy of a large language model in a specific field, the web page data in a specific field can be used as training data to pre-train a dedicated large language model, but in the prior art, there is a great waste of resources and costs in obtaining a large amount of web page data in a specific field. Summary of the invention

[0004] Based on this, it is necessary to provide a web page data collection method, system, computer device and readable storage medium that can reduce hardware resource consumption costs, labor costs and improve the efficiency of obtaining pre-training data in response to the above technical problems.

[0005] In a first aspect, the present application provides a web page data collection method, wherein the web page data collected by the method is used to pre-train a large language model in a target domain, the method comprising:

[0006] Determine the target scenario template and target keywords;

[0007] Fill the target keyword into the target scenario template, call the universal large language model, and generate multiple search terms;

[0008] Invoke a search service to generate a first URL list based on the search term;

[0009] The web page data is collected based on the crawler technology and the first URL list.

[0010] In one embodiment, determining the target scene template and the target keyword includes:

[0011] In response to a service call instruction for webpage data collection, obtaining parameters carried by the service call instruction; the parameters at least include target keywords;

[0012] If the parameter further includes a target number, the first scene template corresponding to the target number is used as the target scene template.

[0013] In one embodiment, the responding to the service call instruction for webpage data collection, after obtaining the parameters carried by the service call instruction, further comprises:

[0014] If the parameter does not include the target number, determining a second scene template and at least two scenes to be selected;

[0015] Filling the candidate scene and the target keyword into the second scene template, calling a universal large language model, and generating a target scene;

[0016] The first scene template corresponding to the target scene is used as the target scene template.

[0017] In one embodiment, the calling of the search service and generating a first URL list based on the search term includes:

[0018] If the search words are questions, determining the question similarity between two of the questions;

[0019] If there is a target question similarity greater than or equal to a preset question similarity threshold among the question similarities, deduplication processing is performed on the two target questions corresponding to the target question similarity;

[0020] If the search terms are concepts, determining the conceptual similarities between the concepts;

[0021] If there is a target concept similarity in the concept similarities that is greater than or equal to a preset concept similarity threshold, deduplication processing is performed on the two target concepts corresponding to the target concept similarity;

[0022] If the search term is a noun, then the same target nouns in the noun are deduplicated;

[0023] The search service is called to generate a first URL list corresponding to the search terms after deduplication processing.

[0024] In one embodiment, the webpage data collected based on crawler technology and the first URL list includes:

[0025] Deduplication processing is performed on the first URL to obtain a second URL list;

[0026] Counting the number of occurrences of each domain name in the second URL list;

[0027] The domain name in the second URL list that appears more than the first preset number threshold is used as the target domain name, and the URL list corresponding to the target domain name is the third URL list;

[0028] Web page data is collected based on crawler technology, the second URL list and the third URL list.

[0029] In one embodiment, the webpage data collected based on the crawler technology, the second URL list and the third URL list includes:

[0030] The longest identical path of each URL in the third URL list is used as the starting URL, and a fourth URL list is obtained based on the starting URL; the fourth URL list at least includes the URL extracted from the web page corresponding to the starting URL;

[0031] Combining the fourth URL list and the second URL list to obtain a fifth URL list;

[0032] Deduplication processing is performed on the fifth URL list to obtain a sixth URL list;

[0033] Based on the crawler technology, web page data is crawled from the web pages corresponding to the URLs in the sixth URL list.

[0034] In one embodiment, extracting a fourth URL list based on the start URL includes:

[0035] Performing a URL extraction operation on the target start URL corresponding to each target domain name respectively, the URL extraction operation includes:

[0036] Collecting webpage content corresponding to the target start URL;

[0037] Filtering the content corresponding to the preset module in the webpage content, and extracting a seventh URL list from the remaining part of the webpage content;

[0038] Determine whether the number of acquisitions reaches a second preset number threshold;

[0039] If not, each URL in the seventh URL list is used as the target start URL, and the process returns to the step of collecting the webpage content corresponding to the target start URL;

[0040] If so, the loop ends, and all seventh URL lists extracted are merged into the fourth URL list.

[0041] In one embodiment, after the webpage data is collected based on the crawler technology and the first URL list, the following further comprises:

[0042] Sampling target web page data from the web page data;

[0043] Outputting to display the target webpage data, so that relevant personnel can analyze the proportion of data belonging to the target field in the target webpage data;

[0044] receiving the proportion, and if the proportion reaches a preset proportion threshold, performing data cleaning on the web page data, and using the cleaned web page data to pre-train a large language model for a target domain;

[0045] If the proportion is lower than the preset proportion threshold, the web page data is classified to obtain partial web page data corresponding to the target field; the data classification refers to classifying and screening the web page data through a specially designed and implemented classification algorithm.

[0046] In one embodiment, the method further comprises:

[0047] The cleaned web page data or the cleaned partial web page data is used to pre-train an untrained model to obtain a large language model of the target domain.

[0048] In a second aspect, the present application further provides a web page data collection system, the system comprising:

[0049] A collection interface, wherein a first input box is provided in the collection interface for inputting a target scene template and a target keyword;

[0050] A search phrase output interface, used to display search phrases, wherein the search phrases are generated by filling the target keywords into the target scenario template and calling a general large language model;

[0051] A model training interface, wherein the model training interface is used to display training process data or result data of pre-training a large language model using web page data; wherein the web page data is web page data collected based on crawler technology and the first URL list after calling a search service and generating a first URL list based on the search term.

[0052] In one embodiment, the system further comprises:

[0053] The URL list display interface is used to display the first URL list.

[0054] In one embodiment, the system further comprises:

[0055] The target web page data display interface is used to display the target web page data, and the target web page data is collected from the web page data.

[0056] In one embodiment, the system further comprises:

[0057] A scene template management interface, wherein the scene template management interface is provided with a template maintenance entry, a template addition entry, a template deletion entry, and a template modification entry;

[0058] Among them, the template maintenance entry is used to maintain the number and content corresponding to the scene template, the template addition entry is used to add a new scene template, the template deletion entry is used to delete an existing scene template, and the template modification entry is used to modify an existing scene template.

[0059] In a third aspect, the present application further provides a web page data collection system, the system comprising:

[0060] A collection interface, wherein a first input box is provided in the collection interface for inputting a target scene template and a target keyword;

[0061] A search phrase output interface, used to display search phrases, wherein the search phrases are generated by filling the target keywords into the target scenario template and calling a general large language model;

[0062] The web page data collection monitoring page is used to display the collected web page data and the current data collection status. The web page data is the web page data collected based on the crawler technology and the first URL list after calling the search service and generating the first URL list based on the search term.

[0063] In one embodiment, the system further comprises:

[0064] The URL list display interface is used to display the first URL list.

[0065] In one embodiment, the system further comprises:

[0066] The target web page data display interface is used to display the target web page data, and the target web page data is collected from the web page data.

[0067] In one embodiment, the system further comprises:

[0068] A scene template management interface, wherein the scene template management interface is provided with a template maintenance entry, a template addition entry, a template deletion entry, and a template modification entry;

[0069] Among them, the template maintenance entry is used to maintain the number and content corresponding to the scene template, the template addition entry is used to add a new scene template, the template deletion entry is used to delete an existing scene template, and the template modification entry is used to modify an existing scene template.

[0070] In a fourth aspect, the present application further provides a web page data collection device, comprising:

[0071] A determination module, used to determine a target scene template and target keywords;

[0072] A filling module, used to fill the target keyword into the target scenario template, call a universal large language model, and generate multiple search terms;

[0073] A generating module, configured to call a search service and generate a first URL list based on the search term;

[0074] The collection module is used to collect web page data based on crawler technology and the first URL list.

[0075] In a fifth aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0076] Determine the target scenario template and target keywords;

[0077] Fill the target keyword into the target scenario template, call the universal large language model, and generate multiple search terms;

[0078] Invoke a search service to generate a first URL list based on the search term;

[0079] The web page data is collected based on the crawler technology and the first URL list.

[0080] In a sixth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0081] Determine the target scenario template and target keywords;

[0082] Fill the target keyword into the target scenario template, call the universal large language model, and generate multiple search terms;

[0083] Invoke a search service to generate a first URL list based on the search term;

[0084] The web page data is collected based on the crawler technology and the first URL list.

[0085] In a seventh aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0086] Determine the target scenario template and target keywords;

[0087] Fill the target keyword into the target scenario template, call the universal large language model, and generate multiple search terms;

[0088] Invoke a search service to generate a first URL list based on the search term;

[0089] The web page data is collected based on the crawler technology and the first URL list.

[0090] The web page data collection method, system, computer device and readable storage medium described above, the method generates multiple search phrases corresponding to the target keyword by calling a general large language model, which is equivalent to increasing the scale of the keyword and realizing the expansion of the target keyword in the target field. Under this premise, the search service is subsequently called to generate a first URL list based on the search phrase, and sufficient web page data is collected based on this, so as to ensure the scale requirement of the pre-training data of the large language model in the target field. In addition, since the search phrase is generated by the target keyword and the target scene template, the search phrase has a high correlation with the target keyword, so it can be ensured that the collected web page data has a high correlation with the target field. This makes its data scale much smaller than that of general web page data, so that no more storage resources are required for storage, thereby reducing the cost of hardware resource consumption; and since the web page data has a high correlation with the target field, it is not necessary to spend a lot of time like obtaining the web page data of the target field from general web page data, thereby improving the efficiency of obtaining pre-training data. In addition, since the method does not need to manually limit the data source, that is, specify the URL, as in the case of domain experts, but automatically generates it through the target keyword, that is, the automatic generation of the URL is realized, so the labor cost is also reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0092] Figure 1 A diagram showing an application environment of a web page data collection method in one embodiment;

[0093] Figure 2 A schematic diagram of a flow chart of a web page data collection method in one embodiment;

[0094] Figure 3 is a structural block diagram of a web page data collection device in one embodiment;

[0095] Figure 4 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0096] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0097] The webpage data collection method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 can determine the target scene template and the target keyword, which can be uploaded by the user through the terminal 102, and then fill the target keyword into the target scene template, call the general large language model, and generate multiple search terms; call the search service to generate a first URL list based on the search terms; collect and obtain web page data based on the crawler technology and the first URL list. After obtaining the web page data, the server 104 can send the web page data to the terminal 102 for the user to use. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices, and the Internet of Things devices can be smart TVs, smart car-mounted devices, etc. Portable wearable devices can be smart watches, head-mounted devices, etc. The head mounted device may be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0098] As mentioned in the background technology, the text corpus in a specific field is relatively small and cannot meet the pre-training requirements of a large language model, while the web page data publicly available on the Internet is huge. To improve the accuracy of a large language model in a specific field, the web page data in a specific field can be used as training data to pre-train a dedicated large language model, but in the prior art, there is a great waste of resources and costs in obtaining a large amount of web page data in a specific field.

[0099] For large language models, the quality and quantity of pre-training data are the primary factors affecting the performance of natural language processing algorithm models. In order to build a powerful large language model, it is necessary to collect massive data from diversified data sources to pre-train the large language model.

[0100] According to different sources, pre-training data can be divided into two types: general text data and special text data. General text data covers web pages, books, conversation texts, etc.

[0101] Existing large language models mainly mix various public text data as pre-training corpus, but in the current training corpus composition of large language models, web pages are still the most widely used pre-training data for building large language models.

[0102] The public web pages on the Internet provide a large amount of data resources for pre-training of large language models. These data are large in scale, easy to obtain, and cover a wide range. Most large language models collect a large amount of web page data to enhance their language modeling capabilities.

[0103] In traditional technology, there are two main ways to obtain web page data: one is for users to build web page datasets themselves, and the other is to directly use open source web page datasets. Among them, users building web page datasets themselves means that users manually use crawler technology to perform large-scale parallel data crawling on web pages; open source web page datasets include Common Crawl, C4, RefinedWeb, etc.

[0104] The applicant has found that the above users can be roughly divided into two categories according to their professional level, one is domain experts and the other is non-professionals. For non-professionals, when building a web page data set, they generally directly perform large-scale parallel data crawling on general websites. For domain experts, although they understand and master some professional websites corresponding to the field, they can directly crawl from the professional website through crawler technology when obtaining web page data, but the scale of web page data crawled in this way is small and cannot meet the needs of pre-training large language models in specific fields.

[0105] However, the above web page dataset is suitable for pre-training a general large language model, but not for pre-training a large language model for a specific field, because the web page data corresponding to the specific field in the web page dataset is often less than 1%. For example, the data volume of the web page dataset is 100GB, and the data volume of the web page data belonging to the specific field is 10MB.

[0106] Therefore, on the one hand, the data size of general web page data is large, and it requires more storage resources than storing web page data in specific fields; on the other hand, it is necessary to classify and filter general web page data through specially designed and implemented classification algorithms to obtain web page data in specific fields. This classification and filtering process is quite time-consuming. In addition, since domain experts are required to limit the data source, that is, specify the URL, the labor cost is also relatively high.

[0107] In summary, traditional technologies have the problems of high hardware resource consumption, high labor costs and high time costs. In addition, due to the construction time of open source web page datasets, it is impossible to obtain the latest corpus data on the Internet in this specific field, which affects the accuracy of the model.

[0108] To solve this problem, in an exemplary embodiment, Figure 2 As shown, a web page data collection method is provided, and the web page data collected by the method is used to pre-train a large language model in a target domain, wherein the target domain refers to a certain field, and the field refers to a discipline or industry category with clear boundaries and specific research objects, methods, theories, and application scopes. In this category, people focus on in-depth research on a certain type of problem, technology, phenomenon, or knowledge system to promote theoretical development, technological innovation, and practical application in this field.

[0109] The fields include but are not limited to aerospace, medicine, and biology. The target field can be aerospace, medicine, or biology, etc. The target field is determined according to the needs of the user. For example, if the user needs to pre-train a large language model in the medical field, the target field refers to the medical field.

[0110] The method generates multiple search phrases corresponding to the target keyword by calling a general large language model, which is equivalent to increasing the scale of the keyword and realizing the expansion of the target keyword in the target field. Under this premise, the search service is subsequently called to generate a first URL list based on the search phrase, and sufficient web page data is collected based on this, so as to ensure the scale requirement of the pre-trained data of the large language model in the target field. In addition, since the search phrase is generated by the target keyword and the target scene template, the search phrase has a high correlation with the target keyword, so it can be ensured that the collected web page data has a high correlation with the target field. This makes its data scale much smaller than that of general web page data, so that no more storage resources are required for storage, thereby reducing the cost of hardware resource consumption; and since the web page data has a high correlation with the target field, it is not necessary to spend a lot of time like obtaining the web page data of the target field from the general web page data, thereby improving the efficiency of obtaining pre-trained data, and avoiding the consumption of a large amount of server computing resources, thereby further reducing the cost of hardware resource consumption.

[0111] In addition, it should be noted that if the proportion of data in the target domain in the collected web page data meets the requirements of pre-training, the web page data can be cleaned and the cleaned web page data can be used to pre-train the large language model of the target domain, thereby eliminating the need to classify and screen the web page data, thereby improving the efficiency of obtaining pre-training data.

[0112] Among them, data cleaning refers to removing noise such as HTML (HyperText Markup Language) tags, removing sensitive information such as contact information, etc.

[0113] In addition, since this method does not require domain experts to manually define the data source, that is, specify the URL, but automatically generates it through target keywords, that is, the URL is automatically generated, thus reducing labor costs.

[0114] This method is applied to Figure 1 The server 104 in the example is used as an example to illustrate, including the following steps 202 to 208. Among them:

[0115] Step 202, determining the target scene template and target keywords.

[0116] The scene template refers to the template corresponding to the scene.

[0117] The scenarios include, but are not limited to, question-and-answer competition scenarios, professional examination scenarios, knowledge reasoning scenarios, and preference recommendation scenarios.

[0118] It should be noted that there is a corresponding relationship between the scenario and the above-mentioned target domain, that is, after the general large language model is subsequently called, the general large language model will give the output result in combination with the scenario, so that the output result is more relevant to the target domain. This is because although the large language model has powerful natural language processing capabilities and extensive knowledge reserves, its application method and optimization direction will be different in different scenarios. Among them, the general large language model can be a closed-source commercial large model or a locally deployed open source large model.

[0119] For example, if the scenario is a question-answering competition scenario, when calling a general large language model, after specifying the scenario as a question-answering competition scenario, the output results generated by the general large language model reasoning will be more scientific and cover a wider range of levels. If the specified scenario is a professional exam scenario, knowledge reasoning scenario, preference recommendation scenario, etc., this effect will not be achieved.

[0120] For example, if the target field is "medical field", among the four scenarios, namely, question-answering competition scenario, professional examination scenario, knowledge reasoning scenario and preference recommendation scenario, the knowledge reasoning scenario corresponds to the target field; if the scenario template corresponding to the knowledge reasoning scenario is used to call the general large language model, the output result of the general large language model will have a deeper professional depth and be more concise; if the scenario template corresponding to other scenarios is used to call the general large language model, the output result of the general large language model will not have this effect.

[0121] The template in this embodiment refers to a pre-prompt template, which is input as a part of the general large language model when the general large language model is called.

[0122] Among them, the pre-prompt refers to the text prompt designed before the user interacts with the general large language model to guide the model to output specific content or perform specific tasks. Effective pre-prompt can clearly convey the user's intention and guide the general large language model to generate content or perform operations in the direction expected by the user.

[0123] For example, if the scenario is a quiz contest scenario, the template corresponding to the quiz contest scenario can be: "You are now the organizer of a knowledge quiz contest. You need to test the contestants' mastery of knowledge in a specific field. Please help me generate N questions. The keywords of this specific field are: ". N can be set as needed, but the setting needs to be reasonable to avoid the scale of the collected web page data being too large or too small. The larger N is, the larger the scale is, and the smaller N is, the smaller the scale is.

[0124] For example, if the scenario is a knowledge reasoning scenario, the template corresponding to the knowledge reasoning scenario can be: "Please imitate the function and role of the knowledge graph, starting from the keywords of a specific field, and find N professional terms related to the specific field. These professional terms need to be as detailed and specific as possible. The keywords of this specific field are: "

[0125] It is worth mentioning that the scene template can be set up in multiple languages ​​according to the required language types.

[0126] In one embodiment, the scene template can be dynamically expanded or deleted according to the needs of the user. This can be achieved through the scene template management interface in the web data collection system, and the scene template management interface is provided with a template maintenance entry, a template addition entry, a template deletion entry, and a template modification entry;

[0127] Among them, the template maintenance entry is used to maintain the number and content corresponding to the scene template, the template addition entry is used to add a new scene template, the template deletion entry is used to delete an existing scene template, and the template modification entry is used to modify an existing scene template.

[0128] The target scene template refers to a scene template selected from a number of candidate scene templates, that is, a template corresponding to a scene selected from a number of candidate scene corresponding templates, for example, the target scene template is a template corresponding to a quiz competition scene.

[0129] Among them, the target keywords are the keywords corresponding to the target field.

[0130] Among them, keywords are brief summaries of information that can help people quickly understand the core of the content. In this embodiment, keywords are a bridge between users and the general large language model. Users describe their information needs by entering keywords, and the general large language model generates relevant content based on these keywords and returns relevant results to users.

[0131] It can be understood that the target keyword can improve the relevance of the output results of the general large language model to the target field to a greater extent. For example, if the target field is the biological field, the target keyword can be biology. If the target field is the medical field, the target keyword can be immunology.

[0132] Among them, the target keyword needs to be specified directly by the user. Specifically, after the user enters the target keyword in the first input box of the collection interface of the web data collection system, the web data collection system uses the target keyword as a parameter carried by the instruction to generate a service call instruction for web data collection, and sends the service call instruction to the server.

[0133] It should be noted that there is at least one target keyword, and the specific number is not limited in this embodiment.

[0134] In addition, the user is required to input the target domain in the first input box to specify the domain to which the collected web page data belongs. Of course, the collection interface may also include an option box to provide options corresponding to several domains for the user to select the target domain in the option box.

[0135] It should be noted that the parameters carried by the service call instruction may or may not include a target number, wherein the number is used to uniquely identify the above scene template, and the target number refers to the target scene template.

[0136] For example, the scenario templates include templates corresponding to a quiz contest, templates corresponding to a professional exam, and templates corresponding to knowledge reasoning, where the template corresponding to the quiz contest is numbered 0, the template corresponding to the professional exam is numbered 1, and the template corresponding to knowledge reasoning is numbered 2. If the parameters carried by the service call instruction also include a target number, and the target number is 1, it can be determined that the target scenario template is the template corresponding to the professional exam.

[0137] In one embodiment, if the user specifies the scenario, he or she can actively select the target scenario template, that is, in response to the service call instruction for web page data collection, obtain the parameters carried by the service call instruction; if the parameters also include a target number, the first scenario template corresponding to the target number is used as the target scenario template.

[0138] Among them, the first scenario template includes but is not limited to a template corresponding to a question-and-answer competition scenario, a template corresponding to a professional examination scenario, a template corresponding to a knowledge reasoning scenario, and a template corresponding to a preference recommendation scenario.

[0139] Exemplarily, the service call instruction may include a target keyword and a target number, that is, when the user calls the service through the client, he gives: ("target keyword", target number), thereby determining the target keyword and target number. For example, given ("immunology", 3), where the number of the template corresponding to the preferred recommendation scenario is 3, the target keyword is "immunology", and the target scenario template is the template corresponding to the preferred recommendation scenario.

[0140] In another embodiment, if the user is not clear about the scenario, the target scenario template can be automatically selected through a universal large language model. That is, in response to the service call instruction for web page data collection, the parameters carried by the service call instruction are obtained; if the parameters do not include the target number, a second scenario template and at least two to-be-selected scenarios are determined; the to-be-selected scenarios and target keywords are filled into the second scenario template, the universal large language model is called, and the target scenario is generated; the first scenario template corresponding to the target scenario is used as the target scenario template.

[0141] Among them, the second scene template is a scene template different from the first scene template. For example, the second scene template can be "Please select a scene that is most relevant to these keywords from the candidate scenes based on the keywords and candidate scenes I give. The keywords I give are: and the candidate scenes I give are:".

[0142] Among them, the scenarios to be selected may be question-answering competition scenarios, professional examination scenarios, knowledge reasoning scenarios, preference recommendation scenarios, and the like.

[0143] Among them, taking at least two candidate scenarios as "knowledge reasoning scenario" and "question and answer competition scenario" and the second scenario template as "Please select a scenario that is most relevant to the keywords and candidate scenarios I give, the keywords I give are:, the candidate scenarios I give are:" as an example, after filling the candidate scenarios and target keywords into the second scenario template, the result is "Please select a scenario that is most relevant to the keywords and candidate scenarios I give, the keywords I give are: immunology, the candidate scenarios I give are: knowledge reasoning scenario, question and answer competition scenario".

[0144] This embodiment automatically selects a target scene template for the user by calling a universal large language model, so that the user does not need to determine the target scene template corresponding to the target field by himself, thereby enabling the user to obtain web page data more conveniently; or the target scene template can be automatically and accurately provided without the user clarifying the target scene template corresponding to the target field.

[0145] Step 204, fill the target keyword into the target scenario template, call the general large language model, and generate multiple search terms.

[0146] The search terms include but are not limited to concepts, nouns, and questions. Since the output results of the general large language model in this embodiment are used for subsequent searches, they are called search terms.

[0147] Exemplarily, the target scenario template is used as the pre-prompt of the target keyword to obtain a complete input sentence, and the input sentence is passed as a function parameter to the general large language model, so that the general large language model generates multiple search terms after reasoning. At this time, the multiple search terms can be sent to the client, and the search term output interface of the web page data collection system displays the multiple search terms generated after calling the general large language model.

[0148] It should be noted that the purpose of generating multiple search terms through target keywords is to obtain more concepts, nouns or questions related to the target field based on the target keyword, so as to obtain more web page data in the target field based on the concepts, nouns or questions in the target field, thereby increasing the source range and scale of collectible web page data.

[0149] Taking the target scenario template as the template corresponding to the quiz competition scenario and the target keyword as "biology" as an example, after filling the target keyword into the target scenario template, a complete model input is: "You are now the organizer of a knowledge quiz competition and need to test the contestants' mastery of knowledge in a specific field. Please help me generate N questions. The keyword for this specific field is: biology."

[0150] Step 206: call the search service to generate a first URL (Uniform Resource Locator) list based on the search term.

[0151] Among them, search services are provided by search service providers such as Google and Baidu. These search service providers often have mature search and recommendation technologies and can filter out websites with a high relevance to the search terms from existing Internet website resources, thereby providing websites in the target field as the source of web page data collection.

[0152] Exemplarily, when calling a search service, a corresponding authentication key can be obtained, and the authentication key and the deduplicated search terms can be used as input to the search service API (Application Programming Interface) one by one to obtain a URL list corresponding to each search term. After merging the URL lists corresponding to all search terms, a first URL list is obtained.

[0153] The de-duplication process refers to retaining only one search term for at least two identical search terms.

[0154] In one embodiment, due to the limitations of the corpus in the training phase of the existing general large language model, the search terms generated by the general large language model are sometimes highly similar or identical. In order to avoid wasting server resources, it is necessary to perform deduplication processing on the search terms, and only retain one of two or more highly similar or identical search terms. That is, the search terms are first deduplicated, and then the search service is called to generate a first URL list corresponding to the deduplicated search terms.

[0155] It should be noted that when the search terms are concepts or questions, since they have a large number of characters, the similarities can be compared two by two, while when the search terms are nouns, since they have a small number of characters, they can be compared two by two for similarity.

[0156] That is, if the search term is a question, the question similarity between each pair of questions is determined; if there is a target question similarity greater than or equal to the preset question similarity threshold among the question similarities, the two target questions corresponding to the target question similarities are deduplicated; if the search term is a concept, the concept similarity between each pair of concepts is determined; if there is a target concept similarity greater than or equal to the preset concept similarity threshold among the concept similarities, the two target concepts corresponding to the target concept similarity are deduplicated; if the search term is a noun, the same target nouns in the nouns are deduplicated.

[0157] Among them, question similarity and concept similarity can be calculated by algorithms such as Simhash.

[0158] Among them, whether the nouns are the same can be compared according to the code values ​​corresponding to the noun encoding standards such as GBK, GB18030, etc.

[0159] Step 208: Collect and obtain web page data based on crawler technology and the first URL list.

[0160] Exemplarily, web page data is crawled from web pages corresponding to the URLs in the first URL list based on crawler technology. It is worth mentioning that when using crawler technology to collect web page data, it is necessary to pay attention to comply with the crawler protocol.

[0161] In an exemplary embodiment, the above step 208 also includes: sampling target web page data from the web page data; outputting and displaying the target web page data for relevant personnel to analyze the proportion of data belonging to the target field in the target web page data; receiving the proportion, and if the proportion reaches a preset proportion threshold, cleaning the web page data, and using the cleaned web page data to pre-train the large language model of the target field.

[0162] It can be understood that this embodiment is aimed at determining whether web page data needs to be classified and screened before it can be used as pre-training data. Therefore, if the proportion is lower than the preset proportion threshold, the web page data is classified to obtain partial web page data corresponding to the target field. Among them, data classification refers to classifying and screening the web page data through a specially designed and implemented classification algorithm to obtain partial web page data corresponding to the target field.

[0163] The preset percentage threshold is not limited in this embodiment.

[0164] Among them, the specially designed and implemented classification algorithm can be implemented based on a keyword matching algorithm or an algorithm based on a neural network.

[0165] Exemplarily, the target web page data may be displayed through a target web page data display interface in a web page data collection system.

[0166] Exemplarily, pre-training specifically refers to, after determining whether the web page data needs to be classified and screened, performing data cleaning on the web page data or part of the web page data obtained after classification and screening, and using the cleaned web page data or part of the cleaned web page data to pre-train the untrained model, thereby obtaining a large language model of the target domain.

[0167] Among them, the untrained model is generally a neural network model based on the Transformer structure. The neural network model based on the Transformer structure can be an untrained original model or a trained open source model.

[0168] Exemplarily, the first URL list may be displayed via a URL list display interface of the webpage data collection system, and a user may browse all URLs in the first URL list on the URL list display interface.

[0169] Exemplarily, during the process of collecting web page data, the web page data may be sent to the client, and the web page data collection monitoring page of the web page data collection system may display the collected web page data and the current data collection status for the user to browse.

[0170] The data collection status includes but is not limited to the URL of the currently collected web page, data collection progress, and data volume.

[0171] Exemplarily, after displaying the web page data, you can jump to the model training interface of the web page data collection system, which is used to display the training process data or result data of pre-training a large language model using the web page data.

[0172] It is worth mentioning that in the conventional technology, crawler tools need to be customized for web pages with different structures, while the crawler tools in the embodiments of the present application can directly use general tools, thereby reducing the labor cost during writing. Among them, the crawler tool is the implementation of the crawler technology.

[0173] In one embodiment, considering that there are certain similarities between search terms, even if searching is performed using different search terms, some duplicate URLs may be obtained in the end.

[0174] In order to avoid collecting duplicate pages through the same URL, in this embodiment, the first URL is deduplicated to obtain the second URL list. That is, after all search terms are searched, all URLs in the first URL list returned by the search engine are deduplicated, and then the web page data is crawled.

[0175] In an exemplary embodiment, in order to increase the proportion of web page data in the target domain in all collected web page data, thereby improving the model accuracy of the pre-trained large language model of the target domain, this embodiment does not directly use crawler technology to crawl web page data through the URLs in the second URL list, but performs deep crawling of web page data on the target domain name.

[0176] First, the number of occurrences of each domain name in the second URL list is counted; the domain name whose number of occurrences in the second URL list is greater than the first preset number threshold is taken as the target domain name, and the URL list corresponding to the target domain name is the third URL list; web page data is collected based on crawler technology, the second URL list and the third URL list.

[0177] In this embodiment, deep crawling is performed based on the third URL list to obtain more web page data with high relevance to the target field.

[0178] It is understandable that, since the target domain name in the third URL list appears more often than other domain names, the web page data under the target domain name is more likely to be more relevant to the target domain. Therefore, collecting more web page data through the URL corresponding to the target domain name may be more suitable for the target domain.

[0179] Among them, the domain name, also known as the network domain, is the name of a computer or a group of computers on the Internet composed of a string of names separated by dots. It is used to locate the computer during data transmission (sometimes also refers to the geographical location).

[0180] For example, for the URL: "https: / / zh.wikipedia.org / wiki / Category:Transportation" and the URL: "https: / / zh.wikipedia.org / wiki / Central America", these two URLs have a common domain name "zh.wikipedia.org".

[0181] The first preset number threshold is not limited in this embodiment. It can be understood that the larger the first preset number threshold is, the fewer the target domain names in the second URL list are.

[0182] For example, the second URL list contains the URL: "https: / / zh.wikipedia.org / wiki / Category:Traffic" and the URL: "https: / / zh.wikipedia.org / wiki / Central America", whose common domain name is "zh.wikipedia.org", and the number of occurrences of this domain name is 2. If the first preset number threshold is 2 or greater than 2, the domain name "zh.wikipedia.org" is not the target domain name; if the first preset number threshold is less than 2, such as 1, the domain name "zh.wikipedia.org" is used as the target domain name. Among them, the URLs in the third URL list are "https: / / zh.wikipedia.org / wiki / Category:Traffic" and "https: / / zh.wikipedia.org / wiki / Central America".

[0183] In one embodiment, the URLs in the webpage content corresponding to the URLs in the third URL list can be directly extracted, and the URLs in the next-level pages corresponding to the URLs in the third URL list can be extracted. For example, if the URLs in the third URL list include a second-level page corresponding to a domain name (such as "zh.wikipedia.org / wiki"), the URLs in the second-level page can be extracted, and the URLs in the third-level page can be extracted.

[0184] In another embodiment, the longest identical path of each URL in the third URL list is used as the starting URL, and a fourth URL list is extracted based on the starting URL; the fourth URL list at least includes the URL extracted from the web page corresponding to the starting URL.

[0185] For example, the third URL list contains the URL: "https: / / zh.wikipedia.org / wiki / Category:Transportation" and the URL: "https: / / zh.wikipedia.org / wiki / Central America", where the same paths include "zh.wikipedia.org" and "zh.wikipedia.org / wiki", and "zh.wikipedia.org / wiki" is longer than "zh.wikipedia.org", so the longest same path is "zh.wikipedia.org / wiki".

[0186] Wherein, obtaining the fourth URL list based on the starting URL extraction includes: performing a URL extraction operation on the target starting URL corresponding to each target domain name respectively, and the URL extraction operation includes steps 1 to 5. Wherein:

[0187] Step 1, collect the webpage content corresponding to the target start URL;

[0188] Step 2, filtering the content corresponding to the preset module in the webpage content, and extracting the seventh URL list from the remaining part of the webpage content;

[0189] Step 3, determining whether the number of acquisitions reaches a second preset number threshold;

[0190] Step 4: If not, use each URL in the seventh URL list as the target start URL and return to step 1;

[0191] Step 5: If yes, end the loop and merge all the seventh URL lists extracted into the fourth URL list.

[0192] The preset modules refer to non-core modules such as header and footer on the web page. The web page data corresponding to the URL in the web page content of the non-core module does not contain web page data in the target field. By filtering the content corresponding to the preset modules, the collected web page data can be prevented from being polluted.

[0193] The second preset number threshold is an empirical value, and preferably setting the second preset number threshold can both increase the proportion of web page data in the target field in all web page data and avoid data explosion. For example, the second preset number threshold is 2.

[0194] In one embodiment, web page data is obtained by crawling from web pages corresponding to each URL in the second URL list and the fourth URL list based on crawler technology.

[0195] In another embodiment, since the URLs in the second URL list may be partially identical to the URLs in the fourth URL list, in order to avoid repeated collection of web page data based on the same URL, the collection efficiency of web page data is improved.

[0196] In this embodiment, the fourth URL list and the second URL list are merged to obtain the fifth URL list; the fifth URL list is deduplicated to obtain the sixth URL list; and web page data is crawled from the web pages corresponding to the URLs in the sixth URL list based on crawler technology.

[0197] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0198] Based on the same inventive concept, the embodiment of the present application also provides a web page data collection system. The implementation scheme for solving the problem provided by the system is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more web page data collection system embodiments provided below can refer to the limitations of the web page data collection method above, and will not be repeated here.

[0199] It should be emphasized that, in this embodiment, the web page data collection system is generally used in enterprises that do large language model training.

[0200] The system can be applied to a client device, and the system includes:

[0201] A collection interface, wherein a first input box is provided in the collection interface for inputting a target scene template and a target keyword;

[0202] A search phrase output interface, used to display search phrases, wherein the search phrases are generated by filling the target keywords into the target scenario template and calling a general large language model;

[0203] The web page data collection monitoring page is used to display the collected web page data and the current data collection status. The web page data is the web page data collected based on the crawler technology and the first URL list after calling the search service and generating the first URL list based on the search term.

[0204] In one embodiment, the system further comprises:

[0205] The URL list display interface is used to display the first URL list.

[0206] In one embodiment, the system further comprises:

[0207] The target web page data display interface is used to display the target web page data, and the target web page data is collected from the web page data.

[0208] In one embodiment, the system further comprises:

[0209] A scene template management interface, wherein the scene template management interface is provided with a template maintenance entry, a template addition entry, a template deletion entry, and a template modification entry;

[0210] Among them, the template maintenance entry is used to maintain the number and content corresponding to the scene template, the template addition entry is used to add a new scene template, the template deletion entry is used to delete an existing scene template, and the template modification entry is used to modify an existing scene template.

[0211] Based on the same inventive concept, the embodiment of the present application also provides a web page data collection system. The implementation scheme for solving the problem provided by the system is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more web page data collection system embodiments provided below can refer to the limitations of the web page data collection method above, and will not be repeated here.

[0212] It should be emphasized that, in this embodiment, the web page data collection system is generally used in enterprises that generate training data.

[0213] The system can be applied to a client device, and the system includes:

[0214] A collection interface, wherein a first input box is provided in the collection interface for inputting a target scene template and a target keyword;

[0215] A search phrase output interface, used to display search phrases, wherein the search phrases are generated by filling the target keywords into the target scenario template and calling a general large language model;

[0216] The web page data collection monitoring page is used to display the collected web page data and the current data collection status. The web page data is the web page data collected based on the crawler technology and the first URL list after calling the search service and generating the first URL list based on the search term.

[0217] In one embodiment, the system further comprises:

[0218] The URL list display interface is used to display the first URL list.

[0219] In one embodiment, the system further comprises:

[0220] The target web page data display interface is used to display the target web page data, and the target web page data is collected from the web page data.

[0221] In one embodiment, the system further comprises:

[0222] A scene template management interface, wherein the scene template management interface is provided with a template maintenance entry, a template addition entry, a template deletion entry, and a template modification entry;

[0223] Among them, the template maintenance entry is used to maintain the number and content corresponding to the scene template, the template addition entry is used to add a new scene template, the template deletion entry is used to delete an existing scene template, and the template modification entry is used to modify an existing scene template.

[0224] Based on the same inventive concept, the embodiment of the present application also provides a web page data collection device for implementing the web page data collection method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more web page data collection device embodiments provided below can refer to the limitations of the web page data collection method above, and will not be repeated here.

[0225] In an exemplary embodiment, Figure 3 As shown, a web page data collection device is provided, including: a determination module 302, a filling module 304, a generation module 306 and a collection module 308, wherein:

[0226] A determination module 302 is used to determine a target scene template and a target keyword;

[0227] A filling module 304 is used to fill the target keyword into the target scenario template, call a common large language model, and generate multiple search terms;

[0228] A generating module 306, configured to call a search service and generate a first URL list based on the search term;

[0229] The collection module 308 is used to collect web page data based on crawler technology and the first URL list.

[0230] In one embodiment, the determining module 302 includes:

[0231] The acquisition unit is used to respond to the service call instruction for web page data collection and acquire the parameters carried by the service call instruction; the parameters include at least target keywords; if the parameters also include a target number, the first scene template corresponding to the target number is used as the target scene template.

[0232] In one embodiment, the determining module 302 further includes:

[0233] a determining unit, configured to determine a second scene template and at least two scenes to be selected if the parameter does not include a target number;

[0234] The first generating unit is used to fill the selected scene and the target keyword into the second scene template, call the general large language model, generate the target scene; and use the first scene template corresponding to the target scene as the target scene template.

[0235] In one embodiment, the generating module 306 includes:

[0236] A first deduplication unit is used for, if the search words are questions, determining the question similarities between the questions; if there is a target question similarity greater than or equal to a preset question similarity threshold among the question similarities, deduplication processing is performed on the two target questions corresponding to the target question similarities; if the search words are concepts, determining the concept similarities between the concepts; if there is a target concept similarity greater than or equal to a preset concept similarity threshold among the concept similarities, deduplication processing is performed on the two target concepts corresponding to the target concept similarities; if the search words are nouns, deduplication processing is performed on the same target nouns among the nouns;

[0237] The second generating unit is used to call the search service to generate a first URL list corresponding to the search terms after deduplication processing.

[0238] In one embodiment, the acquisition module 308 includes:

[0239] A second deduplication unit, configured to perform deduplication processing on the first URL to obtain a second URL list;

[0240] a statistical unit, configured to count the number of occurrences of each domain name in the second URL list; taking a domain name in the second URL list whose number of occurrences is greater than a first preset number threshold as a target domain name, and the URL list corresponding to the target domain name is a third URL list;

[0241] The collection unit is used to collect web page data based on crawler technology, the second URL list and the third URL list.

[0242] In one embodiment, the acquisition unit includes:

[0243] an extraction subunit, configured to use the longest identical path of each URL in the third URL list as a starting URL, and extract a fourth URL list based on the starting URL; the fourth URL list at least includes the URL extracted from the web page corresponding to the starting URL;

[0244] a merging subunit, configured to merge the fourth URL list and the second URL list to obtain a fifth URL list;

[0245] a deduplication subunit, configured to perform deduplication processing on the fifth URL list to obtain a sixth URL list;

[0246] The crawling subunit is used to crawl and obtain web page data from the web pages corresponding to each URL in the sixth URL list based on the crawler technology.

[0247] In one embodiment, the extraction subunit is specifically used for:

[0248] Performing a URL extraction operation on the target start URL corresponding to each target domain name respectively, the URL extraction operation includes:

[0249] Collecting webpage content corresponding to the target start URL;

[0250] Filtering the content corresponding to the preset module in the webpage content, and extracting a seventh URL list from the remaining part of the webpage content;

[0251] Determine whether the number of acquisitions reaches a second preset number threshold;

[0252] If not, each URL in the seventh URL list is used as the target start URL, and the process returns to the step of collecting the webpage content corresponding to the target start URL;

[0253] If so, the loop ends, and all seventh URL lists extracted are merged into the fourth URL list.

[0254] In one embodiment, the webpage data collection device further includes:

[0255] A sampling module, used for sampling the target web page data from the web page data;

[0256] An output module, used for outputting and displaying the target webpage data, so that corresponding personnel can analyze the proportion of data belonging to the target field in the target webpage data;

[0257] A receiving module is used to receive the proportion. If the proportion reaches a preset proportion threshold, the web page data is cleaned and the cleaned web page data is used to pre-train a large language model of a target domain. If the proportion is lower than the preset proportion threshold, the web page data is classified to obtain partial web page data corresponding to the target domain. The data classification refers to classifying and screening the web page data through a specially designed and implemented classification algorithm.

[0258] In one embodiment, the webpage data collection device further includes:

[0259] The pre-training module is used to use the cleaned web page data or the cleaned part of the web page data to pre-train the untrained model to obtain a large language model of the target domain.

[0260] Each module in the webpage data collection device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0261] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 4 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store web page data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a web page data collection method is implemented.

[0262] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0263] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0264] Determine the target scenario template and target keywords;

[0265] Fill the target keyword into the target scenario template, call the universal large language model, and generate multiple search terms;

[0266] Invoke a search service to generate a first URL list based on the search term;

[0267] The web page data is collected based on the crawler technology and the first URL list.

[0268] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0269] In response to a service call instruction for webpage data collection, obtaining parameters carried by the service call instruction; the parameters at least include target keywords;

[0270] If the parameter further includes a target number, the first scene template corresponding to the target number is used as the target scene template.

[0271] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0272] If the parameter does not include the target number, determining a second scene template and at least two scenes to be selected;

[0273] Filling the candidate scene and the target keyword into the second scene template, calling a universal large language model, and generating a target scene;

[0274] The first scene template corresponding to the target scene is used as the target scene template.

[0275] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0276] If the search words are questions, determining the question similarity between two of the questions;

[0277] If there is a target question similarity greater than or equal to a preset question similarity threshold among the question similarities, deduplication processing is performed on the two target questions corresponding to the target question similarity;

[0278] If the search terms are concepts, determining the conceptual similarities between the concepts;

[0279] If there is a target concept similarity in the concept similarities that is greater than or equal to a preset concept similarity threshold, deduplication processing is performed on the two target concepts corresponding to the target concept similarity;

[0280] If the search term is a noun, then the same target nouns in the noun are deduplicated;

[0281] The search service is called to generate a first URL list corresponding to the search terms after deduplication processing.

[0282] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0283] Deduplication processing is performed on the first URL to obtain a second URL list;

[0284] Counting the number of occurrences of each domain name in the second URL list;

[0285] The domain name in the second URL list that appears more than the first preset number threshold is used as the target domain name, and the URL list corresponding to the target domain name is the third URL list;

[0286] Web page data is collected based on crawler technology, the second URL list and the third URL list.

[0287] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0288] The longest identical path of each URL in the third URL list is used as the starting URL, and a fourth URL list is obtained based on the starting URL; the fourth URL list at least includes the URL extracted from the web page corresponding to the starting URL;

[0289] Combining the fourth URL list and the second URL list to obtain a fifth URL list;

[0290] Deduplication processing is performed on the fifth URL list to obtain a sixth URL list;

[0291] The web page data is crawled from the web pages corresponding to the URLs in the sixth URL list based on the crawler technology.

[0292] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0293] Performing a URL extraction operation on the target start URL corresponding to each target domain name respectively, the URL extraction operation includes:

[0294] Collecting webpage content corresponding to the target start URL;

[0295] Filtering the content corresponding to the preset module in the webpage content, and extracting a seventh URL list from the remaining part of the webpage content;

[0296] Determine whether the number of acquisitions reaches a second preset number threshold;

[0297] If not, each URL in the seventh URL list is used as the target start URL, and the process returns to the step of collecting the webpage content corresponding to the target start URL;

[0298] If so, the loop ends, and all seventh URL lists extracted are merged into the fourth URL list.

[0299] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0300] Sampling target web page data from the web page data;

[0301] Outputting to display the target webpage data, so that relevant personnel can analyze the proportion of data belonging to the target field in the target webpage data;

[0302] receiving the proportion, and if the proportion reaches a preset proportion threshold, performing data cleaning on the web page data, and using the cleaned web page data to pre-train a large language model for a target domain;

[0303] If the proportion is lower than the preset proportion threshold, the web page data is classified to obtain partial web page data corresponding to the target field; the data classification refers to classifying and screening the web page data through a specially designed and implemented classification algorithm.

[0304] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0305] The cleaned web page data or the cleaned partial web page data is used to pre-train an untrained model to obtain a large language model of the target domain.

[0306] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0307] Determine the target scenario template and target keywords;

[0308] Fill the target keyword into the target scenario template, call the universal large language model, and generate multiple search terms;

[0309] Invoke a search service to generate a first URL list based on the search term;

[0310] The web page data is collected based on the crawler technology and the first URL list.

[0311] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0312] In response to a service call instruction for webpage data collection, obtaining parameters carried by the service call instruction; the parameters at least include target keywords;

[0313] If the parameter further includes a target number, the first scene template corresponding to the target number is used as the target scene template.

[0314] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0315] If the parameter does not include the target number, determining a second scene template and at least two scenes to be selected;

[0316] Filling the candidate scene and the target keyword into the second scene template, calling a universal large language model, and generating a target scene;

[0317] The first scene template corresponding to the target scene is used as the target scene template.

[0318] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0319] If the search words are questions, determining the question similarity between two of the questions;

[0320] If there is a target question similarity greater than or equal to a preset question similarity threshold among the question similarities, deduplication processing is performed on the two target questions corresponding to the target question similarity;

[0321] If the search terms are concepts, determining the conceptual similarities between the concepts;

[0322] If there is a target concept similarity in the concept similarities that is greater than or equal to a preset concept similarity threshold, deduplication processing is performed on the two target concepts corresponding to the target concept similarity;

[0323] If the search term is a noun, then the same target nouns in the noun are deduplicated;

[0324] The search service is called to generate a first URL list corresponding to the search terms after deduplication processing.

[0325] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0326] Deduplication processing is performed on the first URL to obtain a second URL list;

[0327] Counting the number of occurrences of each domain name in the second URL list;

[0328] The domain name in the second URL list that appears more than the first preset number threshold is used as the target domain name, and the URL list corresponding to the target domain name is the third URL list;

[0329] Web page data is collected based on crawler technology, the second URL list and the third URL list.

[0330] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0331] The longest identical path of each URL in the third URL list is used as the starting URL, and a fourth URL list is obtained based on the starting URL; the fourth URL list at least includes the URL extracted from the web page corresponding to the starting URL;

[0332] Combining the fourth URL list and the second URL list to obtain a fifth URL list;

[0333] Deduplication processing is performed on the fifth URL list to obtain a sixth URL list;

[0334] Based on the crawler technology, web page data is crawled from the web pages corresponding to the URLs in the sixth URL list.

[0335] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0336] Performing a URL extraction operation on the target start URL corresponding to each target domain name respectively, the URL extraction operation includes:

[0337] Collecting webpage content corresponding to the target start URL;

[0338] Filtering the content corresponding to the preset module in the webpage content, and extracting a seventh URL list from the remaining part of the webpage content;

[0339] Determine whether the number of acquisitions reaches a second preset number threshold;

[0340] If not, each URL in the seventh URL list is used as the target start URL, and the process returns to the step of collecting the webpage content corresponding to the target start URL;

[0341] If so, the loop ends, and all seventh URL lists extracted are merged into the fourth URL list.

[0342] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0343] Sampling target web page data from the web page data;

[0344] Outputting to display the target webpage data, so that relevant personnel can analyze the proportion of data belonging to the target field in the target webpage data;

[0345] receiving the proportion, and if the proportion reaches a preset proportion threshold, performing data cleaning on the web page data, and using the cleaned web page data to pre-train a large language model for a target domain;

[0346] If the proportion is lower than the preset proportion threshold, the web page data is classified to obtain partial web page data corresponding to the target field; the data classification refers to classifying and screening the web page data through a specially designed and implemented classification algorithm.

[0347] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0348] The cleaned web page data or the cleaned partial web page data is used to pre-train an untrained model to obtain a large language model of the target domain.

[0349] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0350] Determine the target scenario template and target keywords;

[0351] Fill the target keyword into the target scenario template, call the universal large language model, and generate multiple search terms;

[0352] Invoke a search service to generate a first URL list based on the search term;

[0353] The web page data is collected based on the crawler technology and the first URL list.

[0354] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0355] In response to a service call instruction for webpage data collection, obtaining parameters carried by the service call instruction; the parameters at least include target keywords;

[0356] If the parameter further includes a target number, the first scene template corresponding to the target number is used as the target scene template.

[0357] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0358] If the parameter does not include the target number, determining a second scene template and at least two scenes to be selected;

[0359] Filling the candidate scene and the target keyword into the second scene template, calling a universal large language model, and generating a target scene;

[0360] The first scene template corresponding to the target scene is used as the target scene template.

[0361] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0362] If the search words are questions, determining the question similarity between two of the questions;

[0363] If there is a target question similarity greater than or equal to a preset question similarity threshold among the question similarities, deduplication processing is performed on the two target questions corresponding to the target question similarity;

[0364] If the search terms are concepts, determining the conceptual similarities between the concepts;

[0365] If there is a target concept similarity in the concept similarities that is greater than or equal to a preset concept similarity threshold, deduplication processing is performed on the two target concepts corresponding to the target concept similarity;

[0366] If the search term is a noun, then the same target nouns in the noun are deduplicated;

[0367] The search service is called to generate a first URL list corresponding to the search terms after deduplication processing.

[0368] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0369] Deduplication processing is performed on the first URL to obtain a second URL list;

[0370] Counting the number of occurrences of each domain name in the second URL list;

[0371] The domain name in the second URL list that appears more than the first preset number threshold is used as the target domain name, and the URL list corresponding to the target domain name is the third URL list;

[0372] Web page data is collected based on crawler technology, the second URL list and the third URL list.

[0373] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0374] The longest identical path of each URL in the third URL list is used as the starting URL, and a fourth URL list is obtained based on the starting URL; the fourth URL list at least includes the URL extracted from the web page corresponding to the starting URL;

[0375] Combining the fourth URL list and the second URL list to obtain a fifth URL list;

[0376] Deduplication processing is performed on the fifth URL list to obtain a sixth URL list;

[0377] The web page data is crawled from the web pages corresponding to the URLs in the sixth URL list based on the crawler technology.

[0378] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0379] Performing a URL extraction operation on the target start URL corresponding to each target domain name respectively, the URL extraction operation includes:

[0380] Collecting webpage content corresponding to the target start URL;

[0381] Filtering the content corresponding to the preset module in the webpage content, and extracting a seventh URL list from the remaining part of the webpage content;

[0382] Determine whether the number of acquisitions reaches a second preset number threshold;

[0383] If not, each URL in the seventh URL list is used as the target start URL, and the process returns to the step of collecting the webpage content corresponding to the target start URL;

[0384] If so, the loop ends, and all seventh URL lists extracted are merged into the fourth URL list.

[0385] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0386] Sampling target web page data from the web page data;

[0387] Outputting to display the target webpage data, so that relevant personnel can analyze the proportion of data belonging to the target field in the target webpage data;

[0388] receiving the proportion, and if the proportion reaches a preset proportion threshold, performing data cleaning on the web page data, and using the cleaned web page data to pre-train a large language model for a target domain;

[0389] If the proportion is lower than the preset proportion threshold, the web page data is classified to obtain partial web page data corresponding to the target field; the data classification refers to classifying and screening the web page data through a specially designed and implemented classification algorithm.

[0390] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0391] The cleaned web page data or the cleaned partial web page data is used to pre-train an untrained model to obtain a large language model of the target domain.

[0392] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0393] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0394] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A web page data collection method, characterized in that: The web page data collected by the method is used to pre-train a large language model in the target domain, and the method includes: Determine a target scene template and a target keyword; the target scene template refers to a template corresponding to a scene selected from templates corresponding to a plurality of candidate scenes; Fill the target keyword into the target scenario template, call the universal large language model, and generate multiple search terms; Invoke a search service to generate a first URL list based on the search term; Based on the first URL list, a domain name whose appearance times is greater than a first preset number threshold is obtained, and the domain name is used as a target domain name; The longest identical path of the URL corresponding to the target domain name is used as the starting URL; Performing a URL extraction operation on the target start URL corresponding to each target domain name respectively, the URL extraction operation includes: Collecting webpage content corresponding to the target start URL; Filtering the content corresponding to the preset module in the webpage content, and extracting a seventh URL list from the remaining part of the webpage content; Determine whether the number of acquisitions reaches a second preset number threshold; If not, each URL in the seventh URL list is used as the target start URL, and the process returns to the step of collecting the webpage content corresponding to the target start URL; If yes, the loop ends, and all seventh URL lists extracted are merged into the fourth URL list; Acquire web page data based on crawler technology and the fourth URL list; Wherein, the determining of the target scene template and the target keyword comprises: In response to a service call instruction for webpage data collection, obtaining parameters carried by the service call instruction; the parameters at least include target keywords; If the parameter also includes a target number, the first scene template corresponding to the target number is used as the target scene template; If the parameter does not include the target number, determining a second scene template and at least two scenes to be selected; Filling the candidate scene and the target keyword into the second scene template, calling a universal large language model, and generating a target scene; The first scene template corresponding to the target scene is used as the target scene template.

2. The method according to claim 1, characterized in that The calling of the search service and generating a first URL list based on the search term comprises: If the search words are questions, determining the question similarity between two of the questions; If there is a target question similarity greater than or equal to a preset question similarity threshold among the question similarities, deduplication processing is performed on the two target questions corresponding to the target question similarity; If the search terms are concepts, determining the conceptual similarities between the concepts; If there is a target concept similarity in the concept similarities that is greater than or equal to a preset concept similarity threshold, deduplication processing is performed on the two target concepts corresponding to the target concept similarity; If the search term is a noun, then the same target nouns in the noun are deduplicated; The search service is called to generate a first URL list corresponding to the search terms after deduplication processing.

3. The method according to claim 1, characterized in that The webpage data collected based on the crawler technology and the fourth URL list includes: The fourth URL list and the second URL list are combined to obtain a fifth URL list; the second URL list is a second URL list obtained by performing a deduplication process on the first URL list; Deduplication processing is performed on the fifth URL list to obtain a sixth URL list; Based on the crawler technology, web page data is crawled from the web pages corresponding to the URLs in the sixth URL list.

4. The method according to any one of claims 1 to 3, characterized in that: After the webpage data is collected based on the crawler technology and the fourth URL list, the following further includes: Sampling target web page data from the web page data; Outputting to display the target webpage data, so that relevant personnel can analyze the proportion of data belonging to the target field in the target webpage data; receiving the proportion, and if the proportion reaches a preset proportion threshold, performing data cleaning on the web page data, and using the cleaned web page data to pre-train a large language model for a target domain; If the proportion is lower than the preset proportion threshold, the web page data is classified to obtain partial web page data corresponding to the target field; the data classification refers to classifying and screening the web page data through a specially designed and implemented classification algorithm.

5. The method according to claim 4, characterized in that The method further comprises: The cleaned web page data or the cleaned partial web page data is used to pre-train an untrained model to obtain a large language model of the target domain.

6. A web page data collection system, characterized in that: The system comprises: A collection interface, wherein a first input box is provided in the collection interface for inputting a target scene template and a target keyword; A search phrase output interface, used to display search phrases, wherein the search phrases are generated by filling the target keywords into the target scenario template and calling a general large language model; A model training interface, wherein the model training interface is used to display training process data or result data of pre-training a large language model using web page data; wherein the web page data is generated after a first URL list is generated based on the search term by calling a search service; Based on the first URL list, a domain name whose appearance times is greater than a first preset number threshold is obtained, and the domain name is used as a target domain name; The longest identical path of the URL corresponding to the target domain name is used as the starting URL; Performing a URL extraction operation on the target start URL corresponding to each target domain name respectively, the URL extraction operation includes: Collecting webpage content corresponding to the target start URL; Filtering the content corresponding to the preset module in the webpage content, and extracting a seventh URL list from the remaining part of the webpage content; Determine whether the number of acquisitions reaches a second preset number threshold; If not, each URL in the seventh URL list is used as the target start URL, and the process returns to the step of collecting the webpage content corresponding to the target start URL; If yes, the loop ends, and all seventh URL lists extracted are merged into the fourth URL list; The web page data is collected based on the crawler technology and the fourth URL list.

7. The system according to claim 6, characterized in that The system further comprises: A scene template management interface, wherein the scene template management interface is provided with a template maintenance entry, a template addition entry, a template deletion entry, and a template modification entry; Among them, the template maintenance entry is used to maintain the number and content corresponding to the scene template, the template addition entry is used to add a new scene template, the template deletion entry is used to delete an existing scene template, and the template modification entry is used to modify an existing scene template.

8. A web page data collection system, characterized in that: The system comprises: A collection interface, wherein a first input box is provided in the collection interface for inputting a target scene template and a target keyword; A search phrase output interface, used to display search phrases, wherein the search phrases are generated by filling the target keywords into the target scenario template and calling a general large language model; A web page data collection monitoring page is used to display the collected web page data and the current data collection status, wherein the web page data is generated after the first URL list is generated based on the search term by calling the search service; Based on the first URL list, a domain name whose appearance times is greater than a first preset number threshold is obtained, and the domain name is used as a target domain name; The longest identical path of the URL corresponding to the target domain name is used as the starting URL; Performing a URL extraction operation on the target start URL corresponding to each target domain name respectively, the URL extraction operation includes: Collecting webpage content corresponding to the target start URL; Filtering the content corresponding to the preset module in the webpage content, and extracting a seventh URL list from the remaining part of the webpage content; Determine whether the number of acquisitions reaches a second preset number threshold; If not, each URL in the seventh URL list is used as the target start URL, and the process returns to the step of collecting the webpage content corresponding to the target start URL; If yes, the loop ends, and all seventh URL lists extracted are merged into the fourth URL list; The web page data is collected based on the crawler technology and the fourth URL list.

9. The system according to claim 8, characterized in that The system further comprises: A scene template management interface, wherein the scene template management interface is provided with a template maintenance entry, a template addition entry, a template deletion entry, and a template modification entry; Among them, the template maintenance entry is used to maintain the number and content corresponding to the scene template, the template addition entry is used to add a new scene template, the template deletion entry is used to delete an existing scene template, and the template modification entry is used to modify an existing scene template.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images