Webpage article content extraction and target model training method and device
By fine-tuning the pre-trained large language model, the target model is built to extract web article content, solving the problem of low efficiency in web page extraction on different sites, and achieving efficient and accurate content extraction.
Patent Information
- Application Number
- CN202410955105.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2024-11-26
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing technology requires the development of code separately when extracting web article contents on different sites, resulting in high labor and time costs and low efficiency.
The pre-trained large language model is used for fine-tuning, and the target model is built to extract the content of web pages, which are suitable for web pages on different sites, saving manpower and time and improving efficiency.
It realizes efficient extraction of article content on web pages on different sites, improves the accuracy of the extraction results, and has wide applicability.
Smart Images

Figure CN119025734A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to methods and devices for extracting content from webpage articles and training target models in the fields of deep learning, large models, and natural language processing. Background Art
[0002] Currently, when it is necessary to extract content from articles on web pages of different sites, it is usually necessary to develop a set of corresponding codes for each site, which consumes a lot of manpower and time costs and is inefficient. Summary of the invention
[0003] The present invention provides a method and device for extracting web page article content and training a target model.
[0004] A method for extracting content from a webpage article, comprising:
[0005] Obtaining a web page to be processed, wherein the web page to be processed includes web page data in the form of an article;
[0006] The web page to be processed is used as the input of the target model to obtain the web page article content extraction result output by the target model, and the target model is obtained by fine-tuning the pre-trained large language model.
[0007] A target model training method, comprising:
[0008] Get the pre-trained large language model;
[0009] A first training sample set is constructed, and the large language model is fine-tuned using the first training sample set to obtain a target model, wherein the target model is used to generate a web page article content extraction result corresponding to an input web page to be processed, wherein the web page to be processed includes web page data in the form of articles.
[0010] A webpage article content extraction device comprises: a webpage acquisition module and a content extraction module;
[0011] The webpage acquisition module is used to acquire a webpage to be processed, wherein the webpage to be processed includes webpage data in the form of an article;
[0012] The content extraction module is used to use the web page to be processed as the input of the target model to obtain the web page article content extraction result output by the target model, and the target model is obtained by fine-tuning the pre-trained large language model.
[0013] A target model training device, comprising: a model acquisition module and a model training module;
[0014] The model acquisition module is used to acquire a pre-trained large language model;
[0015] The model training module is used to construct a first training sample set, and use the first training sample set to fine-tune the large language model to obtain a target model. The target model is used to generate a web page article content extraction result corresponding to the input web page to be processed, and the web page to be processed includes web page data in the form of articles.
[0016] An electronic device, comprising:
[0017] at least one processor; and
[0018] a memory communicatively connected to the at least one processor; wherein,
[0019] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method described above.
[0020] A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method as described above.
[0021] A computer program product comprises a computer program / instruction, wherein the computer program / instruction implements the above method when executed by a processor.
[0022] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.
[0024] Figure 1 This is a flow chart of an embodiment of the method for extracting content from a webpage article according to the present disclosure;
[0025] Figure 2 A flowchart of an embodiment of the target model training method disclosed in the present invention;
[0026] Figure 3 It is a schematic diagram of the composition structure of an embodiment 300 of the webpage article content extraction device of the present disclosure;
[0027] Figure 4 Schematic diagram of the structure of the target model training device embodiment 400 of the present disclosure;
[0028] Figure 5 A schematic block diagram of an electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0029] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0030] In addition, it should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.
[0031] Figure 1 Flow chart of an embodiment of the method for extracting content of webpage articles described in the present disclosure. Figure 1 As shown, the following specific implementation methods are included.
[0032] In step 101, a web page to be processed is obtained, wherein the web page to be processed includes web page data in the form of an article.
[0033] In step 102, the web page to be processed is used as the input of the target model to obtain the web page article content extraction result output by the target model. The target model is obtained by fine-tuning the pre-trained large language model (LLM).
[0034] It can be seen that by adopting the scheme described in the above method embodiment, the pre-trained target model can be used to generate the web page article content extraction result corresponding to the web page to be processed, thereby saving manpower and time costs and improving processing efficiency. Moreover, the target model is a large language model obtained by pre-training + fine-tuning. The powerful reasoning ability of the large language model can be used to improve the accuracy of the obtained web page article content extraction result. In addition, the scheme is applicable to web pages from different sites, that is, it has wide applicability.
[0035] The web page to be processed may include web page data in the form of articles, that is, the web page to be processed may include at least one article. There is no limitation on how to obtain the web page to be processed, for example, the web page to be processed may be a web page captured from a predetermined site.
[0036] Furthermore, the web page to be processed can be used as the input of the target model, thereby obtaining the web page article content extraction result output by the target model.
[0037] In some embodiments of the present disclosure, the web page article content extraction results output by the target model may include: the web page article content extraction results in a lightweight markup language (markdown) format output by the target model. In addition, the web page article content extraction results may include: article title (i.e., main title), paragraph title, and paragraph content.
[0038] That is to say, with the help of the target model, various valid information of the article in the web page to be processed can be extracted, so that the user can have a comprehensive and accurate understanding of the content of the article. In addition, with the help of the markdown format, the extracted article titles, paragraph titles and paragraph contents can be organized into hierarchical related content. For example, different tags can be used to distinguish article titles, different paragraph titles and different paragraph contents, etc., so as to facilitate users to view and perform other uses.
[0039] The target model may be pre-trained, such as obtained by fine-tuning a pre-trained large language model. The following is a detailed description of the training method of the target model.
[0040] Figure 2 Flow chart of an embodiment of the target model training method disclosed in the present invention. Figure 2 As shown, the following specific implementation methods are included.
[0041] In step 201, a pre-trained large language model is obtained.
[0042] In step 202, a first training sample set is constructed, and the large language model is fine-tuned using the first training sample set to obtain a target model, which is used to generate a web page article content extraction result corresponding to the input web page to be processed, and the web page to be processed includes web page data in the form of articles.
[0043] By adopting the scheme described in the above method embodiment, a target model can be trained and used to generate a web page article content extraction result corresponding to the web page to be processed, thereby saving manpower and time costs and improving processing efficiency. Moreover, the target model is a large language model obtained by pre-training + fine-tuning. The powerful reasoning ability of the large language model can be used to improve the accuracy of the web page article content extraction result. In addition, the scheme is applicable to web pages from different sites, that is, it has wide applicability.
[0044] A large language model refers to a deep learning model trained using a large amount of text data, which can generate natural language text or understand the meaning of natural language text, etc. It can be used to process various natural language tasks, such as text classification, question answering, dialogue, etc., and is an important path to artificial intelligence. The specific model of the large language model in the solution described in this disclosure is not limited.
[0045] The pre-trained large language model has a certain degree of versatility because it has been trained with a large amount of text data. On this basis, it can be fine-tuned to make the large language model adapt to specific tasks, that is, to improve its model performance on specific tasks.
[0046] Accordingly, in the solution described in the present disclosure, a first training sample set can be constructed, and the first training sample set can be used to fine-tune the pre-trained large language model to obtain the desired target model.
[0047] In some embodiments of the present disclosure, web pages can be captured from a predetermined site, and the captured web pages may include web page data in the form of articles. Thereafter, the following processing can be performed on any captured web page: the web page is used as a training web page, and the web page article content extraction result corresponding to the training web page is obtained, and the training web page and the corresponding web page article content extraction result are used to form a training sample, which is added to the first training sample set.
[0048] The predetermined site is usually an application (APP) platform authorized by some partners. In addition, the captured web page can be a whitelist web page authorized by the partner, etc.
[0049] For any captured web page, it can be used as a training web page, and the web page article content extraction result corresponding to the training web page can be obtained. Accordingly, the training web page and the corresponding web page article content extraction result can be used to form a training sample, and the training sample can be added to the first training sample set. In this way, multiple training samples can be constructed. When the number of training samples reaches the requirement, the first training sample set can be used to fine-tune the pre-trained large language model, so that the large language model learns the ability to generate web page article content extraction results.
[0050] In some embodiments of the present disclosure, for any training web page, after obtaining the web page article content extraction result corresponding to the training web page, the web page article content extraction result corresponding to the training web page can also be subject to a quality review. In response to determining that the review is passed, the training web page and the corresponding web page article content extraction result can be used to form a training sample. In response to determining that the review is not passed, the training web page can be discarded.
[0051] There is also no restriction on how to conduct quality review. For example, it can be determined whether the number of characters included in the web article content extraction result is less than a predetermined threshold. The specific value of the predetermined threshold can be determined according to actual needs, that is, it can be determined whether the content in the web article content extraction result is too little. If it is too little, it will be detrimental to the learning of the large language model, so it can be considered that the review has failed. For another example, it can be determined whether the web article content extraction result includes sensitive content. If so, it can also be considered that the review has failed. For another example, the web article content extraction result and the corresponding training web page can also be displayed to the user. If an extraction error instruction is returned after the user performs manual inspection, it can also be considered that the review has failed.
[0052] By conducting quality audits, the quality of generated training samples can be improved, thereby improving the training effect of the model.
[0053] In some embodiments of the present disclosure, for any training web page, the article title, paragraph title and paragraph content can be extracted from the training web page, and the extracted content can be organized into markdown format as the web page article content extraction result corresponding to the training web page.
[0054] It can extract various valid information from articles in the training web pages, and use the markdown format to organize the extracted article titles, paragraph titles, and paragraph contents into hierarchical related content, which is convenient for large language models to learn.
[0055] In addition, in some embodiments of the present disclosure, for any training web page, regular expressions may be used to extract article titles, paragraph titles, and paragraph contents from the training web page.
[0056] Regular expressions can be used to define specific extraction rules. Accordingly, with the help of the extraction rules, the required article titles, paragraph titles and paragraph contents can be extracted conveniently and efficiently.
[0057] After fine-tuning the pre-trained large language model using the first training sample set to obtain the target model, the target model can be applied to the actual web page article content extraction work, such as Figure 1 As described in, the web page to be processed can be used as the input of the target model to obtain the web page article content extraction result output by the target model.
[0058] In addition, in some embodiments of the present disclosure, a matching score between the web page to be processed and the corresponding web page article content extraction result can also be obtained. In response to determining that the matching score is greater than a predetermined threshold, the web page to be processed and the corresponding web page article content extraction result can be used to form a training sample and added to a second training sample set. The second training sample set is used to optimize the training of the target model.
[0059] There is no restriction on how to obtain the matching score between the web page to be processed and the corresponding web article content extraction result. For example, the matching score given by humans after comparing the web page to be processed and the corresponding web article content extraction result can be obtained. For another example, the web page to be processed and the corresponding web article content extraction result can be used as the input of a pre-trained scoring model to obtain the matching score output by the scoring model. The higher the matching score, the higher the accuracy of the extracted web article content extraction result.
[0060] Afterwards, the obtained matching score can be compared with a predetermined threshold. The specific value of the predetermined threshold can be determined according to actual needs. If the obtained matching score is greater than the predetermined threshold, then it can be considered that the accuracy of the web page article content extraction result corresponding to the web page to be processed is relatively high. Accordingly, the web page to be processed and the corresponding web page article content extraction result can be used to form a training sample and added to the second training sample set.
[0061] Subsequently, the second training sample set can be used to optimize the target model for training, thereby achieving continuous optimization of the target model in the actual application process, thereby further improving the performance of the target model and further improving the accuracy of the subsequent web article content extraction results.
[0062] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the described order of actions, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present disclosure. In addition, for parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.
[0063] The above is an introduction to the method embodiment. The following is a further explanation of the scheme disclosed in the present invention through an apparatus embodiment.
[0064] Figure 3 FIG. 3 is a schematic diagram of the composition structure of the webpage article content extraction device embodiment 300 described in the present disclosure. Figure 3 As shown, it includes: a web page acquisition module 301 and a content extraction module 302.
[0065] The web page acquisition module 301 is used to acquire the web page to be processed, and the web page to be processed includes web page data in the form of articles.
[0066] The content extraction module 302 is used to take the web page to be processed as the input of the target model to obtain the web page article content extraction result output by the target model. The target model is obtained by fine-tuning the pre-trained large language model.
[0067] By adopting the scheme described in the above-mentioned device embodiment, the pre-trained target model can be used to generate the web page article content extraction result corresponding to the web page to be processed, thereby saving manpower and time costs and improving processing efficiency. Moreover, the target model is a large language model obtained by pre-training + fine-tuning. The powerful reasoning ability of the large language model can be used to improve the accuracy of the obtained web page article content extraction result. In addition, the scheme is applicable to web pages from different sites, that is, it has wide applicability.
[0068] In some embodiments of the present disclosure, the web page article content extraction results output by the target model may include: the web page article content extraction results in markdown format output by the target model. In addition, the web page article content extraction results may include: article title, paragraph title and paragraph content.
[0069] Figure 4 FIG. 4 is a schematic diagram of the structure of the target model training device embodiment 400 described in the present disclosure. Figure 4 As shown, it includes: a model acquisition module 401 and a model training module 402.
[0070] The model acquisition module 401 is used to acquire a pre-trained large language model.
[0071] The model training module 402 is used to construct a first training sample set, and use the first training sample set to fine-tune the large language model to obtain a target model. The target model is used to generate a web page article content extraction result corresponding to the input web page to be processed, and the web page to be processed includes web page data in the form of articles.
[0072] By adopting the scheme described in the above-mentioned device embodiment, a target model can be trained, and then the target model can be used to generate a web page article content extraction result corresponding to the web page to be processed, thereby saving manpower and time costs and improving processing efficiency. Moreover, the target model is a large language model obtained by pre-training + fine-tuning. The powerful reasoning ability of the large language model can be used to improve the accuracy of the obtained web page article content extraction results. In addition, the scheme is applicable to web pages from different sites, that is, it has wide applicability.
[0073] The pre-trained large language model has a certain degree of versatility because it has been trained with a large amount of text data. On this basis, it can be fine-tuned to make the large language model adapt to specific tasks, that is, to improve its model performance on specific tasks.
[0074] Accordingly, the model training module 402 may construct a first training sample set, and may use the first training sample set to fine-tune the pre-trained large language model, thereby obtaining a desired target model.
[0075] In some embodiments of the present disclosure, the model training module 402 can crawl web pages from a predetermined site, and the crawled web pages may include web page data in the form of articles. Thereafter, the following processing can be performed on any crawled web page: the web page is used as a training web page, and the web page article content extraction result corresponding to the training web page is obtained, and the training web page and the corresponding web page article content extraction result are used to form a training sample, which is added to the first training sample set.
[0076] In some embodiments of the present disclosure, the model training module 402 can, for any training web page, after obtaining the web page article content extraction result corresponding to the training web page, also perform a quality review of the web page article content extraction result corresponding to the training web page. In response to determining that the review is passed, the training web page and the corresponding web page article content extraction result can be used to form a training sample. In response to determining that the review is not passed, the training web page can be discarded.
[0077] In addition, in some embodiments of the present disclosure, for any training web page, the model training module 402 can extract the article title, paragraph title and paragraph content from the training web page, and can organize the extracted content into markdown format as the web page article content extraction result corresponding to the training web page.
[0078] Specifically, in some embodiments of the present disclosure, for any training web page, the model training module 402 can use regular expressions to extract article titles, paragraph titles, and paragraph contents from the training web page.
[0079] According to the above method, the model training module 402 can construct multiple training samples. When the number of training samples reaches the requirement, the first training sample set can be used to fine-tune the pre-trained large language model to obtain the required target model, and the target model can be applied to the actual web page article content extraction work, that is, the web page to be processed is used as the input of the target model to obtain the web page article content extraction result output by the target model.
[0080] In addition, in some embodiments of the present disclosure, the model training module 402 can also obtain a matching score between the web page to be processed and the corresponding web page article content extraction result. In response to determining that the matching score is greater than a predetermined threshold, the web page to be processed and the corresponding web page article content extraction result can be used to form a training sample and added to a second training sample set. The second training sample set is used to optimize the training of the target model.
[0081] For example, a matching score given manually by comparing the web page to be processed and the corresponding web article content extraction result can be obtained. For another example, the web page to be processed and the corresponding web article content extraction result can be used as the input of the pre-trained scoring model to obtain the matching score output by the scoring model. The higher the matching score, the higher the accuracy of the extracted web article content extraction result. Afterwards, the obtained matching score can be compared with a predetermined threshold. If the obtained matching score is greater than the predetermined threshold, then it can be considered that the accuracy of the web article content extraction result corresponding to the web page to be processed is high. Accordingly, the web page to be processed and the corresponding web article content extraction result can be used to form a training sample and added to the second training sample set. Subsequently, the second training sample set can be used to optimize the target model for training, thereby achieving continuous optimization of the target model in the actual application process.
[0082] Figure 3 and Figure 4 The specific working process of the device embodiment shown can refer to the relevant description in the aforementioned method embodiment, which will not be repeated here.
[0083] The scheme disclosed in the present invention can be applied to the field of artificial intelligence, especially to the fields of deep learning, large models and natural language processing. Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It has both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.
[0084] In addition, the articles and the like in the embodiments described in this disclosure are not intended for a specific user and do not reflect the personal information of a specific user. In the technical solution of this disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good customs.
[0085] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
[0086] Figure 5A schematic block diagram of an electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0087] like Figure 5 As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 to a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.
[0088] A number of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0089] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI, Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP, Digital Signal Processing), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as the methods described in the present disclosure. For example, in some embodiments, the methods described in the present disclosure may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the methods described in the present disclosure may be executed. Alternatively, in other embodiments, the computing unit 501 may be configured to execute the method described in the present disclosure in any other appropriate manner (for example, by means of firmware).
[0090] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), system on chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0091] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0092] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM, Electronically Programmable Read-Only Memory), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM, Compact Disc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0093] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0094] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by digital data communication (e.g., a communication network) in any form or medium. Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0095] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0096] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0097] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A method for extracting content from a webpage article, comprising: Obtaining a web page to be processed, wherein the web page to be processed includes web page data in the form of an article; The web page to be processed is used as the input of the target model to obtain the web page article content extraction result output by the target model, and the target model is obtained by fine-tuning the pre-trained large language model.
2. The method according to claim 1, wherein: The obtaining of the webpage article content extraction result output by the target model comprises: obtaining the webpage article content extraction result in a lightweight markup language format output by the target model; The web page article content extraction result includes: article title, paragraph title and paragraph content.
3. A target model training method, comprising: Get the pre-trained large language model; A first training sample set is constructed, and the large language model is fine-tuned using the first training sample set to obtain a target model, wherein the target model is used to generate a web page article content extraction result corresponding to an input web page to be processed, wherein the web page to be processed includes web page data in the form of articles.
4. The method according to claim 3, wherein: The constructing of the first training sample set comprises: crawling web pages from a predetermined site, wherein the crawled web pages include web page data in the form of articles; For any captured web page, the following processing is performed respectively: the web page is used as a training web page, and the web page article content extraction result corresponding to the training web page is obtained, and the training web page and the corresponding web page article content extraction result are used to form a training sample, which is added to the first training sample set.
5. The method according to claim 4, wherein: The method of using the training web page and the corresponding web page article content extraction result to form a training sample includes: Performing a quality review on the webpage article content extraction results corresponding to the training webpage; In response to determining that the review is passed, the training web page and the corresponding web page article content extraction result are used to form the training sample.
6. The method according to claim 4, wherein: The step of obtaining the webpage article content extraction result corresponding to the training webpage includes: Extracting article titles, paragraph titles, and paragraph contents from the training web pages; The extracted content is organized into a lightweight markup language format as the web page article content extraction result.
7. The method according to claim 6, wherein: The extracting of article titles, paragraph titles and paragraph contents from the training webpage includes: The article title, the paragraph title and the paragraph content are extracted from the training web page using regular expressions.
8. The method according to any one of claims 4 to 7, further comprising: Obtaining a matching score between the web page to be processed and the corresponding web page article content extraction result; In response to determining that the matching score is greater than a predetermined threshold, a training sample is formed using the to-be-processed web page and the corresponding web page article content extraction result, and added to a second training sample set, and the second training sample set is used to optimize the training of the target model.
9. A device for extracting content from a webpage article, comprising: Web page acquisition module and content extraction module; The webpage acquisition module is used to acquire a webpage to be processed, wherein the webpage to be processed includes webpage data in the form of an article; The content extraction module is used to use the web page to be processed as the input of the target model to obtain the web page article content extraction result output by the target model, and the target model is obtained by fine-tuning the pre-trained large language model.
10. The device according to claim 9, wherein: The content extraction module obtains the webpage article content extraction result in the lightweight markup language format output by the target model; The web page article content extraction result includes: article title, paragraph title and paragraph content.
11. A target model training device, comprising: Model acquisition module and model training module; The model acquisition module is used to acquire a pre-trained large language model; The model training module is used to construct a first training sample set, and use the first training sample set to fine-tune the large language model to obtain a target model. The target model is used to generate a web page article content extraction result corresponding to the input web page to be processed, and the web page to be processed includes web page data in the form of articles.
12. The device according to claim 11, wherein The model training module crawls web pages from a predetermined site, and the crawled web pages include web page data in the form of articles. For any crawled web page, the following processing is performed: the web page is used as a training web page, and the web page article content extraction result corresponding to the training web page is obtained, and the training web page and the corresponding web page article content extraction result are used to form a training sample, which is added to the first training sample set.
13. The device according to claim 12, wherein: The model training module is further used to perform a quality review on the web page article content extraction results corresponding to the training web page, and in response to determining that the review is passed, use the training web page and the corresponding web page article content extraction results to form the training sample.
14. The device according to claim 12, wherein: The model training module extracts article titles, paragraph titles and paragraph contents from the training web page, and organizes the extracted contents into a lightweight markup language format as the web page article content extraction result.
15. The device according to claim 14, wherein: The model training module uses regular expressions to extract the article title, the paragraph title and the paragraph content from the training web page.
16. The device according to any one of claims 12 to 15, wherein: The model training module is further used to obtain a matching score between the web page to be processed and the corresponding web page article content extraction result. In response to determining that the matching score is greater than a predetermined threshold, a training sample is formed using the web page to be processed and the corresponding web page article content extraction result, and added to a second training sample set. The second training sample set is used to optimize the training of the target model.
17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.
18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1 to 8.
19. A computer program product, comprising a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 8 is implemented.
Citation Information
Patent Citations
Webpage data collection method and device, terminal and storage medium
CN117349519A
Medical event extraction method based on generative pre-training language model
CN117493504A
Powerpoint content generation method and device, electronic equipment and storage medium
CN117951290A
