Information acquisition method and device and electronic equipment

Abstract

The embodiment of the invention provides an information acquisition method and device and electronic equipment, and relates to the technical field of computers. The information acquisition method comprises the following steps: acquiring a webpage address of a first webpage; collecting first original webpage information corresponding to the webpage address of the first webpage; according to an information extraction rule, first webpage information is extracted from the first original webpage information, the information extraction rule is data generated through a large language model according to content description information and second original webpage information of a second webpage, and the content description information indicates webpage information needing to be obtained from the webpage. According to the method and the device, the adaptability to variable webpage structures can be improved, and the webpage information extraction efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an information acquisition method, device and electronic equipment. Background Art

[0002] With the development of Internet technology, various information web pages are increasing day by day, and web page content is also growing massively. Therefore, how to effectively extract the required web page information from the massive content has become a problem that needs to be solved urgently.

[0003] At present, web crawlers are often used to extract web page information. A web crawler is a program or script that automatically grabs web page information according to certain rules. Among them, web crawlers rely on fixed element selectors such as Cascading Style Sheets (CSS) selectors and XPath selectors to filter several fixed page elements in a web page, thereby obtaining the web page information corresponding to the page elements and realizing the extraction of web page information. XPath is an Extensible Markup Language (XML) path language (XMLPathLanguage).

[0004] However, with the development of web page construction technology, web page structure changes frequently. The current web page extraction method relies on fixed element selectors, which is difficult to adapt to the changing web page structure, resulting in low efficiency in web page information extraction. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide an information acquisition method, device and electronic device, which to a certain extent solves the problem of low efficiency in current web page information extraction. The specific technical solution is as follows:

[0006] In a first aspect of the present invention, a method for obtaining information is provided, the method comprising:

[0007] Obtaining a webpage address of a first webpage;

[0008] Collecting first original web page information corresponding to the web page address of the first web page;

[0009] The first web page information is extracted from the first original web page information according to an information extraction rule, wherein the information extraction rule is data generated by a large language model based on content description information and second original web page information of a second web page, wherein the content description information indicates web page information to be obtained from a web page, and the second web page is a web page having the same web page structure as the first web page.

[0010] In a second aspect of the present invention, there is also provided an information acquisition device, the device comprising:

[0011] An acquisition module, used to acquire the webpage address of the first webpage;

[0012] A collection module, used for collecting first original web page information corresponding to the web page address of the first web page;

[0013] An extraction module is used to extract the first web page information from the first original web page information according to an information extraction rule, wherein the information extraction rule is data generated by a large language model based on content description information and second original web page information of a second web page, wherein the content description information indicates the web page information required to be obtained from the web page, and the second web page is a web page having the same web page structure as the first web page.

[0014] In a third aspect of the present invention, there is also provided an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0015] Memory, used to store computer programs;

[0016] The processor is used to implement any method step described in the first aspect when executing the program stored in the memory.

[0017] In a fourth aspect of the implementation of the present invention, a computer-readable storage medium is further provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer executes any of the information acquisition methods described in the first aspect above.

[0018] In a fifth aspect of the present invention, a computer program product is provided, including a computer program / instruction, wherein when the computer program / instruction is executed by a processor, any of the methods described in the first aspect is implemented.

[0019] The information acquisition method, device and electronic device provided in the embodiment of the present application, after acquiring the webpage address of the first webpage, collects the first original webpage information corresponding to the webpage address of the first webpage, and then extracts the first webpage information from the first original webpage information according to the information extraction rule, so as to realize the extraction of the required webpage information. Among them, the information extraction rule of the first webpage is generated by a large language model according to the content description information and the second original webpage information of the second webpage having the same webpage structure as the first webpage. In this technical solution, since the large language model can be used to generate an information extraction rule for extracting the webpage information indicated by the content description information for a class of webpages with the same webpage structure, the information extraction rule is used to realize the extraction of the webpage information indicated by the content description information in a class of webpages with the same webpage structure. Therefore, compared with the related art, the webpage information extraction method of the present application gets rid of the webpage extraction rule limited by the fixed element selector, but uses more webpage structure-specific information extraction rules for webpages with different webpage structures, thereby improving the adaptability to the variable webpage structure and improving the efficiency of webpage information extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0021] Figure 1 A flowchart of an information acquisition method provided in an embodiment of the present application;

[0022] Figure 2 A flowchart of a method for generating information extraction rules provided in an embodiment of the present application;

[0023] Figure 3 A block diagram of a web crawler automation system provided by another embodiment of the present application;

[0024] Figure 4 A flowchart of an information acquisition method provided in another embodiment of the present application;

[0025] Figure 5 A block diagram of an information acquisition device provided in an embodiment of the present application;

[0026] Figure 6 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0028] With the development of Internet technology, various information web pages are increasing day by day, and the content of web pages is also growing massively. In many scenarios, web page information needs to be extracted to support specific functions. Among them, the extraction scenarios of web page information include but are not limited to: data analysis scenarios, automated task execution scenarios, public opinion monitoring scenarios, scientific research and academic research scenarios, information aggregation scenarios, etc.

[0029] In the data analysis scenario, data from various web pages can be extracted to support the company's market research, industry analysis, and competitive product intelligence collection. Among them, competitive product intelligence collection can help companies detect the dynamics of competitive companies in real time and fully understand market changes. In the automated task execution scenario, information extraction tasks on specific web pages can be performed. For example, submission forms and download files on web pages can be extracted to improve the efficiency of file aggregation. In the public opinion monitoring scenario, news web pages, social web pages, and other web pages can be extracted regularly to understand the public's response to specific events, thereby supporting corporate crisis management. In the scientific research and academic research scenario, paper web pages, forum web pages, etc. can be extracted to support academic researchers to quickly obtain scientific research intelligence in related fields. In the information aggregation scenario, information from different web pages can be extracted to support multi-website information integration, so as to provide users with more comprehensive data. Therefore, how to effectively extract the required web page information from massive content has become a problem that needs to be solved urgently.

[0030] At present, web crawlers are often used to extract web page information. A web crawler is a program or script that automatically grabs web page information according to certain rules. Among them, web crawlers rely on fixed element selectors such as CSS selectors and XPath selectors to filter several fixed page elements in a web page, thereby obtaining the web page information corresponding to the page elements and realizing the extraction of web page information.

[0031] However, with the development of web page construction technology, web page structure changes frequently. The current web page extraction method relies on fixed element selectors, which is difficult to adapt to the changing web page results, resulting in low efficiency of web page information extraction.

[0032] Please refer to Figure 1 , which shows a flow chart of an information acquisition method provided by an embodiment of the present application. The information acquisition method can be applied to electronic devices. Optionally, the electronic device can include a personal computer, a tablet computer, a mobile phone, etc. Figure 1 As shown, the information acquisition method includes:

[0033] Step 101: Obtain the webpage address of a first webpage.

[0034] Optionally, the electronic device may obtain the webpage address of the first webpage to be extracted input by the user. The webpage address is also called a webpage link.

[0035] In one example, if the user wants to extract web page information of the first web page, the user can manually input the web page address of the first web page on the electronic device, so that the electronic device obtains the web page address of the first web page. In another example, the electronic device can record the web page addresses of web page information that have been extracted historically, and provide web page identifiers of web page information that have been extracted historically for the user to select. The user can select the web page identifier of the first web page on the electronic device, so that the electronic device obtains the web page identifier of the first web page, and then obtains the web page address corresponding to the web page identifier.

[0036] Step 102: Collect first original web page information corresponding to the web page address of the first web page.

[0037] Wherein, when the webpage is a static page, the original webpage information of the webpage is the page source code information. Wherein, when the webpage is a dynamic page, the original webpage information of the webpage (i.e., the dynamic webpage information) includes the page source code information and the page dynamic data. Wherein, the page dynamic data refers to the page data obtained in response to the trigger operation on the operable page elements on the webpage.

[0038] In an optional implementation, when the first webpage is a static page, the process of the electronic device collecting the first original webpage information corresponding to the webpage address of the first webpage may include:

[0039] The electronic device can initiate a first webpage request to the page server according to the webpage address of the first webpage, obtain the page source code information of the first webpage, and obtain the first original webpage information. The first webpage request is used to request the page server to return the page source code information of the first page to the electronic device.

[0040] For example, the electronic device may run a browser and initiate a first web page request through the browser, where the first web page request includes the web page address of the first web page. Upon receiving the first web page request, the page server sends the page source code information of the first page stored therein to the electronic device, so that the electronic device obtains the first original web page information of the first page. The first original web page information includes the page source code information of the first web page.

[0041] In another optional implementation, when the first web page is a dynamic page, the process of the electronic device collecting the first original web page information corresponding to the web page address of the first web page may include:

[0042] The electronic device can load a web page according to the web page address of the first web page, obtain dynamic web page information of the first web page, and obtain first original web page information, wherein the first original web page information may include page source code information and page dynamic data of the first page.

[0043] As an example, the process in which the electronic device loads a web page according to the web page address of the first web page, obtains dynamic web page information of the first web page, and obtains the first original web page information may include the following steps 1021 to 1022.

[0044] In step 1021, a headless browser is called to load a web page according to the web page address of the first web page.

[0045] Optionally, the electronic device may initiate a first web page request according to the web page address of the first web page, and obtain the page source code information of the first web page. Afterwards, the electronic device calls a headless browser to load the first web page according to the page source code information. For example, the headless browser may be a Selenium browser in headless mode. Alternatively, the headless browser may also be a PhantomJS browser.

[0046] In step 1022, on the first page loaded by the headless browser, a trigger event is executed according to the behavior trigger information to request a web page resource, dynamic web page information of the first web page is obtained, and first original web page information is obtained.

[0047] Among them, the behavior trigger information is used to simulate the user's operation on the operable page elements on the web page. Optionally, the behavior trigger information may be information pre-edited in the electronic device, so that after the electronic device loads the first page on the headless browser, it reads the behavior trigger information stored in itself. Alternatively, the behavior trigger information may also be information stored in a third-party device. After the electronic device loads the first page on the headless browser, it may request the behavior trigger information from the third-party device to obtain the behavior trigger information. Optionally, the behavior trigger information may be used to simulate triggering the user to click, slide, long press or double-click on the operable page elements.

[0048] The electronic device can simulate triggering an operation on an operable page element on the first page through the behavior trigger information on the page loaded by the headless browser, so as to trigger a trigger event on the operable page element to request a webpage resource, obtain page dynamic data corresponding to the operable page element, and obtain dynamic webpage information of the first webpage. The dynamic page information includes page source code information and page dynamic data of the first page.

[0049] In one example, there is a secondary drop-down menu control (i.e., an operational page element) on the first page. The primary menu in the secondary drop-down menu control is used to click to trigger the display of the secondary menu. The behavior trigger information can be used to simulate the user's click operation on the primary menu on the first page. On the first page, the electronic device can trigger the click operation on the primary menu through the behavior trigger information to trigger the click event of the primary menu to request the resource of the secondary menu, and obtain page dynamic data: secondary menu data.

[0050] In another example, a video playback control (i.e., an operable page element) is provided on the first page. The video playback control is used to slide and trigger the display of a target video. The behavior trigger information can be used to simulate a user's sliding operation on the video playback control on the first page. On the first page, the electronic device can trigger a sliding operation on the video playback control through the behavior trigger information to trigger a resource request for the target video for the sliding event of the video playback control, and obtain page dynamic data: target video data.

[0051] In the embodiment of the present application, by simulating the user's operation on the operable page elements on the page, the headless browser can be used to obtain the page dynamic data of the dynamic page, which effectively ensures the integrity of the original web page information of the dynamic page. In addition, using the headless browser to obtain the original web page information can also effectively circumvent the anti-extraction strategy of the web page and improve the success rate of obtaining the original web page information of the web page.

[0052] Step 103: extract the first web page information from the first original web page information according to the information extraction rule. The information extraction rule is data generated by the large language model according to the content description information and the second original web page information of the second web page. The content description information indicates the web page information required to be obtained from the web page.

[0053] The second web page is a web page having the same web page structure as the first web page. For example, the second web page can be the same web page as the first web page. The content description information indicates the web page information required to be obtained from the web page. Furthermore, the web page information required to be obtained from the second web page indicated by the content description information is the same as the web page information required to be obtained from the first web page. Accordingly, the first web page information is the web page information indicated by the content description information in the first web page.

[0054] Optionally, the content description information may include extraction keywords, extraction content types, etc. The extraction keywords refer to keywords of the web page information to be acquired. The extraction content types may include text types, table types, picture types, video types, audio types, etc.

[0055] For example, suppose that the operating conditions of company x in year z need to be extracted from a web page. Keywords extracted from the content description information may be: company x, year z, profit, financial report, etc., and the extracted content types may be: table type and text type. Accordingly, the first web page may be a web page of financial business statistics. The first web page information is the profit information and financial report information of company x in year z contained in the first web page.

[0056] In an embodiment of the present application, the information extraction rule is used to extract the web page information indicated by the content description information from the original web page information of the page. Optionally, the information extraction rule may include a tag filtering expression and an information filtering expression of the tag content. The tag filtering expression may be used to filter tags having the web page information indicated by the content description information from the original web page information. The information filtering expression may be used to filter the web page information indicated by the content description information from the tag content of the tag.

[0057] For example, suppose that the operating conditions of company x in year z need to be extracted from a web page. Keywords extracted from the content description information may be: company x, year z, profit, financial report, etc. And suppose that the operating conditions of company x in year z can be extracted from the title tag, and the text tag under the title tag whose title content has the extracted keyword. In this way, the information extraction rule may include regular expression 1, regular expression 2, regular expression 3 and regular expression 4. Among them, regular expression 1 is used to filter title tags from the original web page information. Regular expression 2 is used to filter target title tags whose title content has any of the extracted keywords of company x, year z, profit, and financial report. Regular expression 3 is used to filter text tags under the target title tag from the original web page information. Regular expression 4 is used to filter text tag content that has the four extracted keywords of company x, year z, profit, and financial report from the text tag, that is, the web page information indicated by the content description information.

[0058] Optionally, the information extraction rule may be data periodically generated through a large language model based on content description information and original web page information of a web page having the same web page structure as the first web page. The update period of the information extraction rule may be determined based on the update period of the large language model, or the update period of the information extraction rule may also be custom data. For example, the electronic device may call a scheduled task to execute once every 5 days to generate an information extraction rule based on content description information and the second original web page information through a large language model.

[0059] In this way, since the large language model can be further optimized during long-term use, it can further learn the semantic information that users need more. Therefore, the electronic device periodically generates information extraction rules based on the content description information and the original web page information with the same web page structure as the first web page through the large language model, so that the newly generated information extraction rules can be more adaptable to the changing website structure. In addition, the method of regularly updating the information extraction rules using the large language model reduces human participation and reduces the manual maintenance cost of the information extraction rules.

[0060] In some embodiments of the present application, as shown above, the information extraction rule is data generated by a large language model based on the content description information and the second original web page information of the second web page. The information acquisition method performed by the electronic device includes a rule generation phase and a rule use phase. In the rule generation phase, the electronic device can perform a rule generation process to obtain an information extraction rule. In the rule use phase, the electronic device can use the information extraction rule obtained in the rule generation phase to perform the aforementioned steps 101 to 103 to extract the first web page information in the first web page.

[0061] Based on this, before the aforementioned step 101, the electronic device may also perform rule generation processing. Figure 2 As shown, the rule generation process includes:

[0062] Step 201: Obtain content description information and the webpage address of a second webpage.

[0063] Optionally, the electronic device can obtain the content description information input by the user and the webpage address of the second webpage. For example, the user can manually input the webpage address of the second page to be extracted and the content description information of the webpage information to be obtained into the electronic device, so that the electronic device can obtain the content description information and the webpage address of the second webpage.

[0064] Step 202: collect the second original web page information corresponding to the web page address of the second web page.

[0065] The explanation and implementation of this step can refer to the explanation and implementation of the aforementioned step 102, and will not be elaborated in detail in the embodiment of the present application.

[0066] Step 203: Input the first indication information, the content description information and the second original web page information into the large language model to obtain the second web page information.

[0067] The first indication information is used to instruct the large language model to perform the task of extracting the web page information indicated by the content description information. When performing the extraction task, the large language model is used to perform semantic understanding of the content description information and extract the web page information indicated by the content description information from the second original web page information.

[0068] Optionally, the electronic device may input the first indication information, the content description information, and the second original web page information into the large language model to obtain the second web page information output by the large language model. The second web page information is the web page information indicated by the content description information extracted from the second original web page information by the large language model. For example, the large language model may be a Generative Pre-trained Transformer (GPT) model, a ChatGLM2-6B model, etc.

[0069] In an optional case, the large language model can be trained with a plurality of first training sample data. The first training sample data includes: content description information, third original web page information of a third web page, and a first label. The first label is web page information indicated by the content description information in the artificially marked third original web page information.

[0070] The convergence condition of the large language model may be: the output accuracy of the large language model is greater than the first accuracy threshold. The output accuracy refers to the quotient of the number of times the model outputs accurate results and the total number of outputs. When the similarity between the output result of the large language model based on the content description information and the third original web page information and the content of the corresponding first tag is greater than the first similarity threshold, the output result is determined to be an accurate result.

[0071] In this way, since the second web page information is the web page information indicated by the content description information in the second web page extracted by using the large language model, and the large language model has a certain output accuracy, the extraction accuracy of the second web page information is guaranteed to a certain extent, and the accuracy of the information extraction rule subsequently determined by using the second web page information is guaranteed.

[0072] Further optionally, the electronic device may further perform content verification on the output result of the large language model according to the content description information and the second original web page information to further ensure the extraction accuracy of the second web page information. Based on this, the process of the electronic device inputting the first indication information, the content description information and the second original web page information into the large language model to obtain the second web page information may include the following steps 2031 to 2034.

[0073] In step 2031, the first indication information, the content description information and the second original web page information are input into a large language model to generate candidate web page information.

[0074] The candidate web page information is the web page information indicated by the content description information extracted from the second original web page information after the large language model performs semantic understanding on the content description information.

[0075] In step 2032, the candidate web page information is checked for accuracy of extracted content.

[0076] Optionally, the extracted content accuracy check can be used to check whether the candidate web page information is the web page information indicated by the content description information in the second web page, and to check whether the candidate web page information includes all the web page information indicated by the content description information in the second web page.

[0077] In an optional implementation, the electronic device may obtain target web page information indicated by content description information in the second web page. Determine content similarity between the candidate web page information and the target web page information. If the content similarity is greater than a second similarity threshold, determine that the verification is passed. If the content similarity is less than or equal to the second similarity threshold, determine that the verification is not passed.

[0078] The target web page information of the second web page may be web page information indicated by the content description information in the second web page extracted manually. Alternatively, the target web page information of the second web page may also be web page information indicated by the content description information in the second web page obtained by the electronic device through other data sources. For example, the electronic device may call a preset application programming interface (Application Programming Interface, API) to obtain the web page information indicated by the content description information in the second web page.

[0079] In another optional implementation, the electronic device may also display the candidate web page information to receive a manual verification result of the accuracy of the extracted content of the candidate web page information. The electronic device determines that the verification is passed when the result is accurately input. If the result is inaccurately input, the electronic device determines that the verification is not passed.

[0080] For example, the electronic device may display candidate web page information, an accurate result control, and an inaccurate result control. If the verification personnel determines that the candidate web page information is accurate, they may click on the accurate result control so that the electronic device receives the accurate result input and determines that the verification has passed. If the verification personnel determines that the candidate web page information is inaccurate, they may click on the inaccurate result control so that the electronic device receives the inaccurate result input and determines that the verification has failed.

[0081] In the embodiment of the present application, the former verification method of candidate web page information can guarantee the verification efficiency of candidate web page information to a certain extent compared with the latter manual verification method. However, the latter verification method, which uses manual verification of the content accuracy of candidate web page information, can improve the verification accuracy of candidate web page information to a certain extent.

[0082] In step 2033, when the verification passes, the candidate web page information is determined to be the second web page information.

[0083] If the verification is passed, it indicates that the content of the candidate web page information is highly accurate. The electronic device may determine that the candidate web page information is the second web page information.

[0084] In step 2034, if the verification fails, the process returns to the step of inputting the first indication information, the content description information and the second original web page information into the large language model to generate candidate web page information.

[0085] In an optional case, if the verification fails, it indicates that the content accuracy of the candidate web page information is low. The electronic device can modify the target parameters of the large language model and return to execute step 2031 until the accuracy verification of the captured content of the candidate web page information obtained by the large language model passes.

[0086] Among them, the target parameter is used to set the output distribution of the large language model, which affects the output result of the large language model. Optionally, the target parameter can be a temperature parameter, a Max New Tokens parameter, etc. The temperature parameter is an important parameter for regulating the randomness and diversity when generating a large language model. The Max New Tokens parameter is used to regulate the maximum token length of the large language model output.

[0087] In another optional case, when the verification fails, the electronic device may update the first indication information and then return to execute step 2031 until the accuracy verification of the captured content of the candidate web page information obtained through the large language model passes.

[0088] The updated first indication information is used to instruct the large language model to use a different information extraction strategy than the previous one to perform the task of extracting the web page information indicated by the content description information. When performing the extraction task, the large language model is used to: after semantically understanding the content description information, use a different information extraction strategy than the previous one to extract the web page information indicated by the content description information from the second original web page information.

[0089] Further optionally, the restriction condition for the large language model to select the information extraction strategy can be pre-configured. For example, the updated first indication information can include the restriction condition for the large language model to select the information extraction strategy by the user. In this way, the updated first indication information is used to instruct the large language model to use an information extraction strategy different from the previous one within the restriction condition to perform the task of extracting the webpage information indicated by the content description information.

[0090] For example, the first indication information is to extract the web page information indicated by the content description information from the second web page. The updated first indication information may be to extract the web page information indicated by the content description information from the second web page, and when the second web page fails to extract information, the web page information indicated by the description information in the web page corresponding to the hyperlink in the second web page may be added to increase the extraction scope, and when the web page corresponding to the hyperlink in the second web page fails to extract information, a prompt indicating that the extraction cannot be performed is returned.

[0091] In another optional case, when the verification fails, the electronic device can modify the target parameters of the large language model, and after updating the first indication information, return to execute step 2031 until the accuracy verification of the captured content of the candidate web page information obtained by the large language model passes.

[0092] In this way, after obtaining the candidate web page information using the large language model, the accuracy of the extracted content of the candidate web page information is increased, so that when the content accuracy of the candidate web page information output by the large language model is poor, the candidate web page information is re-determined by the large language model until the candidate web page information with higher content accuracy is obtained, and the second web page information is determined. The accuracy of the extracted content can be effectively improved by checking the accuracy of the extracted content, thereby effectively ensuring the accuracy of the information extraction rules determined subsequently using the second web page information.

[0093] Step 204: input the second indication information, the second webpage information and the second original webpage information into the large language model to obtain information extraction rules.

[0094] The second instruction information is used to instruct the large language model to perform the task of generating information extraction rules. When performing the generation task, the large language model is also used to generate information extraction rules that can extract the second web page information from the second original web page information.

[0095] Optionally, the electronic device may input the second indication information, the second webpage information and the second original webpage information into a large language model to obtain an information extraction rule. The information extraction rule is used to extract the second webpage information from the second original webpage.

[0096] In an optional case, the large language model can be trained with a plurality of second training sample data. The second training sample data includes: the fourth web page information indicated by the content description information in the third original web page information, and the third original web page information of the third web page.

[0097] Among them, the convergence condition of the large language model can be: the output accuracy of the large language model is greater than the second accuracy threshold. The output accuracy refers to the quotient of the number of times the model outputs accurate results and the total number of outputs. When the content similarity between the fifth web page information and the fourth web page information is greater than the third similarity threshold, the output result is determined to be an accurate result. The fifth web page information is the web page information extracted from the third original web page information according to the output result of the large language model.

[0098] In this way, since the information extraction rule is data generated by the large language model according to the second web page information and the second original web page information, and the large language model has a certain output accuracy, the generation accuracy of the information extraction rule is effectively guaranteed to a certain extent.

[0099] Further optionally, the electronic device may further verify the output result of the large language model based on the second web page information and the second original web page information to further ensure the accuracy of the generation of the information extraction rule. Based on this, the process of the electronic device inputting the second indication information, the second web page information and the second original web page information into the large language model to obtain the information extraction rule may include the following steps 2041 to 2045.

[0100] In step 2041, the second indication information, the second web page information and the second original web page information are input into the large language model to generate candidate extraction rules.

[0101] The candidate extraction rule is an information extraction rule generated by the large language model and capable of extracting the second web page information from the second original web page information.

[0102] In step 2042, third web page information is extracted from the second original web page information according to the candidate extraction rule.

[0103] Optionally, the electronic device may extract the third web page information defined by the candidate extraction rule from the second original web page information.

[0104] In step 2043, the accuracy of the extracted content of the third web page information is verified.

[0105] Optionally, the extracted content accuracy check may be used to check whether the third web page information is the web page information indicated by the content description information in the second web page.

[0106] In an optional implementation, the electronic device may obtain target web page information indicated by content description information in the second web page. Determine content similarity between the candidate web page information and the target web page information. If the content similarity is greater than a second similarity threshold, determine that the verification is passed. If the content similarity is less than or equal to the second similarity threshold, determine that the verification is not passed.

[0107] The target web page information of the second web page may be web page information indicated by content description information in the second web page extracted manually. Alternatively, the target web page information of the second web page may also be web page information indicated by content description information in the second web page obtained by the electronic device through other data sources. Alternatively, the target web page information may also be the second web page information.

[0108] In another optional implementation, the electronic device may also display the candidate web page information to receive the manual verification result of the accuracy of the extracted content of the third web page information. The electronic device determines that the verification is passed when the result is accurately input. If the result is inaccurately input, the electronic device determines that the verification is not passed.

[0109] In step 2044, if the verification passes, the candidate extraction rule is determined to be an information extraction rule.

[0110] If the verification is passed, it indicates that the candidate extraction rule has a high accuracy. The electronic device may determine that the candidate extraction rule is an information extraction rule.

[0111] In step 2045, if the verification fails, the process returns to the step of inputting the second indication information, the second web page information and the second original web page information into the large language model to generate candidate extraction rules.

[0112] In an optional case, if the verification fails, it indicates that the candidate extraction rule has low accuracy. The electronic device can modify the target parameters of the large language model and then return to execute step 2041 until the accuracy of the extracted content of the third web page information extracted by the candidate extraction rule passes the verification.

[0113] In another optional case, when the verification fails, the electronic device may update the second indication information and then return to execute step 2041 until the accuracy verification of the extracted content of the third web page information extracted by the candidate extraction rule passes.

[0114] The updated second indication information is used to instruct the large language model to use a rule generation strategy different from the previous one to perform the task of generating information extraction rules. When performing the generation task, the large language model is used to: use a rule generation strategy different from the previous one to generate an information extraction rule that can extract the second web page information from the second original web page information.

[0115] Further optionally, the restriction condition for the large language model to select the rule generation strategy may be pre-configured. For example, the updated second instruction information may include the restriction condition for the large language model to select the rule generation strategy. In this way, the updated second instruction information is used to instruct the large language model to use a rule generation strategy different from the previous one to perform the task of generating information extraction rules within the restriction condition.

[0116] For example, the second indication information is to generate an information extraction rule for extracting the second web page information from the second original web page information. The updated second indication information can be to generate an information extraction rule for extracting the second web page information from the second original web page information. However, the information extraction rule needs to be in an information extraction language different from the aforementioned one, such as the XPath language.

[0117] In another optional case, when the verification fails, the electronic device can modify the target parameters of the large language model, and after updating the second indication information, return to execute step 2041 until the accuracy verification of the extracted content of the third web page information extracted by the candidate extraction rule passes.

[0118] In this way, after obtaining the candidate extraction rules by using the large language model, the third web page information is extracted from the second original web page information according to the candidate extraction rules, and the accuracy of the extracted content of the third web page information is verified. When the content accuracy of the third web page information is poor, indicating that the accuracy of the candidate extraction rules is poor, the candidate extraction rules are re-determined through the large language model until the content accuracy of the third web page information extracted by the candidate extraction rules is high, and the information extraction rules are determined.

[0119] By extracting the third web page information from the second original web page information according to the candidate extraction rules output by the large language model, and then verifying the accuracy of the extracted content of the third web page information, the accuracy of the final information extraction rules can be effectively guaranteed.

[0120] It should be noted that, in some embodiments, the information extraction rule may also be information output by directly inputting the third indication information, the content description information, and the second original web page information of the second web page into the large language model. The third indication information is used to instruct the large language model to perform the rule generation task. When performing the rule generation task, the large language model is used to perform semantic understanding of the content description information, extract the web page information indicated by the content description information from the second original web page information, and generate an information extraction rule to extract the second web page information from the second original web page information.

[0121] In some embodiments of the present application, the electronic device may also obtain the web page information indicated by the content description information from multiple different data sources to obtain the sixth web page information. Furthermore, the electronic device may perform information integration processing on the sixth web page information and the first web page information extracted from the first web page, and output the integrated web page information to provide more comprehensive, detailed, and accurate required web page information. Among them, the information integration processing includes but is not limited to deduplication processing, invalid information clearing processing, data standardization processing, etc.

[0122] In an embodiment of the present application, after acquiring the web address of the first web page to be extracted, the information acquisition method can collect the first original web page information corresponding to the web address of the first web page, and then extract the first web page information from the first original web page information according to the information extraction rule, so as to extract the required web page information. Among them, the information extraction rule is data generated by a large language model according to the content description information and the second original web page information of the second web page. And because the information extraction rule is data generated by the large language model for the second web page for extracting the web page information indicated by the content description information. Therefore, compared with the related art, the web page information extraction method gets rid of the web page extraction rules limited by the fixed element selector, but uses information extraction rules that are more targeted to the web page, improves the adaptability to the changing web page structure, and improves the efficiency of web page information extraction.

[0123] For ease of understanding, the information acquisition method provided by the embodiment of the present application is further described with the following example. For example, the second web page and the first web page are the same web page. The electronic device includes a web crawler automation system, and the web crawler automation system can execute the information acquisition method provided by the embodiment of the present application. Figure 3 and Figure 4 As shown, the web crawler automation system 30 includes: a data collection module 301, a user behavior simulation module 302, an adaptive content analysis module 303, an automatic generation and extraction rule module 304, a multi-source data fusion module 305, an intelligent strategy adjustment module 306, a feedback and debugging module 307 and a long-term learning and optimization module 308.

[0124] The web crawler automation system can obtain the content description information input by the user and the web page address of the second web page. The content description information includes extracted keywords and extracted content types.

[0125] The web crawler automation system 30 calls the data acquisition module 301 and the simulated user behavior module 302 to obtain the second original web page information of the second web page. The second original web page information includes the page source code information and page dynamic data of the second web page. Among them, the data acquisition module 301 obtains the page source code information of the second web page according to the web page address of the second web page. The simulated user behavior module 302 obtains the page dynamic data of the second web page by simulating user behavior. Specifically, the simulated user behavior module 302 can call a headless browser to load a web page according to the web page address of the second web page, and on the second page loaded by the headless browser, execute a trigger event according to the behavior trigger information to request a web page resource, and obtain the page dynamic data of the second web page.

[0126] The web crawler automation system 30 calls the adaptive content parsing module 303 to extract candidate web page information. Specifically, the adaptive content parsing module 303 inputs the first indication information, the content description information and the second original web page information into the large language model to obtain candidate web page information. The large language model can perform semantic understanding of the content description information under the instruction of the first indication information, and then extract the web page information indicated by the content description information from the second original web page information.

[0127] The web crawler automation system 30 calls the feedback and debugging module 307 to determine whether the candidate web page information has passed the accuracy check of the extracted content. Specifically: the feedback and debugging module 307 can perform an accuracy check on the extracted content of the candidate web page information, so that if the check passes, indicating that the extraction effect of the candidate web page information is good, the candidate web page information is determined to be the second web page information; if the check fails, indicating that the extraction effect of the candidate web page information is poor, it returns to call the adaptive content parsing module 303 to return to the execution of inputting the first indication information, content description information and the second original web page information into the large language model to obtain the candidate web page information, until the accuracy check of the extracted content of the candidate web page information passes. In this way, extracting the second web page information from the second web page through the adaptive content parsing module 303 can ensure the extraction accuracy of the second web page information to a certain extent.

[0128] The web crawler automation system 30 calls the automatic generation of extraction rules module 304 to extract candidate extraction rules. Specifically, the automatic generation of extraction rules module 304 can input the second indication information, the second web page information and the second original web page information into the large language model to generate candidate extraction rules.

[0129] The web crawler automation system 30 calls the feedback and debugging module 307 to determine whether the candidate extraction rule meets the requirements. Specifically, the feedback and debugging module 307 can be called to extract the third web page information from the second original web page information according to the candidate extraction rule, and the third web page information is checked for the accuracy of the extracted content. When the check is passed, indicating that the extraction effect of the candidate extraction rule is good, the candidate extraction rule is determined to be the information extraction rule. When the check fails, indicating that the extraction effect of the candidate extraction rule is poor, the automatic generation of extraction rule module 304 is returned to return to execute the second indication information, the second web page information, and the second original web page information input into the large language model to generate the candidate extraction rule, until the accuracy of the extraction content of the third web page information is checked, so as to determine that the data extracted according to the candidate extraction rule is accurate. In this way, by automatically generating the extraction rule module 304 and the feedback and debugging module 307, the information extraction rule can be automatically parsed and generated according to the second web page information reflecting the user's needs, without the need for manual maintenance of the information extraction rule, thereby improving the generation efficiency of the information extraction rule.

[0130] In the case of a second web page to be extracted again, the web crawler automation system 30 can extract the web page information indicated by the content description information from the second web page according to the information extraction rule extracted by the automatic generation of extraction rule module 304 .

[0131] Furthermore, the web crawler automation system 30 can also call the multi-source data fusion module 305 to output integrated web page information. Specifically, the multi-source data fusion module 305 can obtain web page information indicated by content description information from multiple different data sources to obtain the sixth web page information. Then, the sixth web page information and the first web page information extracted from the second web page are integrated and processed, and the integrated web page information is output to provide more comprehensive, detailed and accurate required web page information.

[0132] In addition, the web crawler automation system 30 can also call the intelligent policy adjustment module 306 to regularly update the information extraction rules. Specifically, the intelligent policy adjustment module 306 can call the timed task, and periodically generate information extraction rules based on the content description information and the second original web page information of the second web page through the large language model. The intelligent policy adjustment module 306 can periodically generate information extraction rules based on the content description information and the second original web page information through the large language model, so that the newly generated information extraction rules can be more adaptable to the changing website structure. In addition, the method of regularly updating the information extraction rules using the large language model reduces human participation, reduces the manual maintenance cost of the information extraction rules, and reduces the frequent call cost of the large language model. The long-term learning and optimization module 308 can be used to learn and optimize strategies in the process of long-term extraction of web page information, and improve the stability and efficiency of the web crawler automation system 30. For example, the long-term learning and optimization module 308 can call the timed intelligent policy adjustment module 306.

[0133] In summary, the information acquisition method provided by the embodiment of the present application, after obtaining the webpage address of the first webpage, collects the first original webpage information corresponding to the webpage address of the first webpage, and then extracts the first webpage information from the first original webpage information according to the information extraction rule, so as to realize the extraction of the required webpage information. Among them, the information extraction rule of the first webpage is generated by a large language model according to the content description information and the second original webpage information of the second webpage having the same webpage structure as the first webpage. In this technical solution, since the large language model can be used to generate an information extraction rule for extracting the webpage information indicated by the content description information for a class of webpages with the same webpage structure, the information extraction rule is used to realize the extraction of the webpage information indicated by the content description information in a class of webpages with the same webpage structure. Therefore, compared with the related art, the webpage information extraction method of the present application gets rid of the webpage extraction rule limited by the fixed element selector, but uses more webpage structure-specific information extraction rules for webpages with different webpage structures, thereby improving the adaptability to the variable webpage structure and improving the efficiency of webpage information extraction.

[0134] Please refer to Figure 5 , which shows a block diagram of an information acquisition device provided by an embodiment of the present application. Figure 5 As shown, the information acquisition device 500 includes: an acquisition module 501 , a collection module 502 and an extraction module 503 .

[0135] The acquisition module 501 is used to acquire the webpage address of the first webpage;

[0136] A collection module 502 is used to collect first original web page information corresponding to the web page address of the first web page;

[0137] The extraction module 503 is used to extract the first web page information from the first original web page information according to the information extraction rule. The information extraction rule is data generated by a large language model based on the content description information and the second original web page information of the second web page. The content description information indicates the web page information to be obtained from the web page. The second web page is a web page having the same web page structure as the first web page.

[0138] Optionally, the acquisition module 501 is further used to acquire content description information and a webpage address of a second webpage;

[0139] The collection module 502 is further used to collect the second original web page information corresponding to the web page address of the second web page;

[0140] The information acquisition device 500 further includes: a model calling module, which is used to input the first indication information, the content description information, and the second original web page information into the large language model to obtain the second web page information, wherein the first indication information is used to instruct the large language model to perform a task of extracting the web page information indicated by the content description information, and the large language model is used to perform semantic understanding of the content description information and then extract the web page information indicated by the content description information from the second original web page information when performing the extraction task;

[0141] The model calling module is also used to input the second indication information, the second web page information and the second original web page information into the large language model to obtain information extraction rules. The second indication information is used to instruct the large language model to perform the task of generating information extraction rules. When performing the generation task, the large language model is used to generate information extraction rules to extract the second web page information from the second original web page information.

[0142] Optionally, the model calling module is also used to: input the first indication information, content description information and second original web page information into the large language model to generate candidate web page information; verify the accuracy of the extracted content of the candidate web page information; if the verification passes, determine that the candidate web page information is the second web page information; if the verification fails, return to execute the step of inputting the first indication information, content description information and second original web page information into the large language model to generate candidate web page information.

[0143] Optionally, the model calling module is also used to: input the second indication information, the second web page information and the second original web page information into the large language model to generate candidate extraction rules; extract the third web page information from the second original web page information according to the candidate extraction rules; verify the accuracy of the extracted content of the third web page information; if the verification passes, determine the candidate extraction rule as the information extraction rule; if the verification fails, return to execute the step of inputting the second indication information, the second web page information and the second original web page information into the large language model to generate candidate extraction rules.

[0144] Optionally, the acquisition module 502 is also used to call a headless browser to load a web page according to the web address of the first web page; on the first page loaded by the headless browser, a trigger event is executed according to the behavior trigger information to request web page resources, dynamic web page information of the first web page is obtained, and the first original web page information is obtained. The behavior trigger information is used to simulate the user's operation on the operable page elements on the web page.

[0145] Optionally, the information extraction rule is data periodically generated through a large language model based on content description information and original web page information of web pages having the same web page structure as the first web page.

[0146] In summary, the information acquisition device provided in the embodiment of the present application, after acquiring the webpage address of the first webpage, collects the first original webpage information corresponding to the webpage address of the first webpage, and then extracts the first webpage information from the first original webpage information according to the information extraction rule, so as to realize the extraction of the required webpage information. Among them, the information extraction rule of the first webpage is generated by a large language model according to the content description information and the second original webpage information of the second webpage having the same webpage structure as the first webpage. In this technical solution, since the large language model can be used to generate an information extraction rule for extracting the webpage information indicated by the content description information for a class of webpages with the same webpage structure, the information extraction rule is used to realize the extraction of the webpage information indicated by the content description information in a class of webpages with the same webpage structure. Therefore, compared with the related art, the webpage information extraction method of the present application gets rid of the webpage extraction rule limited by the fixed element selector, but uses more webpage structure-specific information extraction rules for webpages with different webpage structures, thereby improving the adaptability to the variable webpage structure and improving the efficiency of webpage information extraction.

[0147] The present application also provides an electronic device, which may be the above-mentioned terminal or server. Figure 6 As shown, it includes a processor 601, a communication interface 602, a memory 603 and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604.

[0148] Memory 603, used for storing computer programs;

[0149] The processor 601 is used to implement any information acquisition method provided in the embodiments of the present application when executing the program stored in the memory 603.

[0150] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0151] The communication interface 604 is used for communication between the above electronic device and other devices.

[0152] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0153] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0154] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor of a terminal, the steps of the information acquisition method described in any of the above embodiments are executed.

[0155] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the steps of the information acquisition method described in any one of the above embodiments.

[0156] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0157] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "and / or" means and and or. For example, "A and / or B" means: A, B and A and B three situations. The term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements that are not clearly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements limited by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements.

[0158] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0159] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for obtaining information, characterized in that: The method comprises: Obtaining a webpage address of a first webpage; Collecting first original web page information corresponding to the web page address of the first web page; The first web page information is extracted from the first original web page information according to an information extraction rule, wherein the information extraction rule is data generated by a large language model based on content description information and second original web page information of a second web page, wherein the content description information indicates web page information to be obtained from a web page, and the second web page is a web page having the same web page structure as the first web page.

2. The method according to claim 1, characterized in that Before obtaining the webpage address of the first webpage, the method further includes: Obtaining the content description information and the webpage address of the second webpage; Collecting the second original web page information corresponding to the web page address of the second web page; Inputting the first indication information, the content description information and the second original web page information into the large language model to obtain the second web page information, wherein the first indication information is used to instruct the large language model to perform a task of extracting the web page information indicated by the content description information, and the large language model is used to perform semantic understanding of the content description information and then extract the web page information indicated by the content description information from the second original web page information when performing the extraction task; The second indication information, the second web page information and the second original web page information are input into the large language model to obtain the information extraction rule, wherein the second indication information is used to instruct the large language model to perform a task of generating information extraction rules, and the large language model is used to generate information extraction rules to extract the second web page information from the second original web page information when performing the generation task.

3. The method according to claim 2, characterized in that The step of inputting the first indication information, the content description information and the second original webpage information into the large language model to obtain the second webpage information includes: Inputting the first indication information, the content description information and the second original web page information into the large language model to generate candidate web page information; Verifying the accuracy of the extracted content of the candidate web page information; If the verification passes, determining the candidate web page information as the second web page information; If the verification fails, the process returns to the step of inputting the first indication information, the content description information and the second original web page information into the large language model to generate candidate web page information.

4. The method according to claim 2 or 3, characterized in that: The step of inputting the second indication information, the second webpage information and the second original webpage information into the large language model to obtain the information extraction rule includes: Inputting the second indication information, the second webpage information and the second original webpage information into the large language model to generate candidate extraction rules; extracting third web page information from the second original web page information according to the candidate extraction rule; Performing an accuracy check on the extracted content of the third web page information; If the verification passes, determining the candidate extraction rule as the information extraction rule; If the verification fails, the process returns to the step of inputting the second indication information, the second web page information and the second original web page information into the large language model to generate candidate extraction rules.

5. The method according to claim 1, characterized in that The collecting of the first original web page information corresponding to the web page address of the first web page includes: Calling a headless browser to load a web page according to the web page address of the first web page; On the first page loaded by the headless browser, a trigger event is executed according to the behavior trigger information to request web page resources, dynamic web page information of the first web page is obtained, and the first original web page information is obtained. The behavior trigger information is used to simulate the user's operation on the operable page elements on the web page.

6. The method according to claim 1, characterized in that The information extraction rule is data generated periodically through a large language model based on the content description information and original web page information of web pages having the same web page structure as the first web page.

7. An information acquisition device, characterized in that: The device comprises: An acquisition module, used to acquire the webpage address of the first webpage; A collection module, used for collecting first original web page information corresponding to the web page address of the first web page; An extraction module is used to extract the first web page information from the first original web page information according to an information extraction rule, wherein the information extraction rule is data generated by a large language model based on content description information and second original web page information of a second web page, wherein the content description information indicates the web page information required to be obtained from the web page, and the second web page is a web page having the same web page structure as the first web page.

8. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing any method described in claims 1 to 6 when executing a program stored in a memory.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images